From patchwork Fri Oct 15 14:39:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101772 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE15CA0C45; Fri, 15 Oct 2021 16:40:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B848541255; Fri, 15 Oct 2021 16:40:20 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 1138E4124D for ; Fri, 15 Oct 2021 16:40:18 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816105" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816105" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542780" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:16 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Ciara Power , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:44 +0000 Message-Id: <20211015143957.842499-2-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 01/14] drivers/crypto: introduce IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Fan Zhang This patch introduces the new framework to share common code between the SW crypto PMDs that depend on the intel-ipsec-mb library. This change helps to reduce future effort on the code maintenance and feature updates. The PMDs that will be added to this framework in subsequent patches are: - AESNI MB - AESNI GCM - CHACHA20_POLY1305 - KASUMI - SNOW3G - ZUC The use of these PMDs will not change, they will still be supported for x86, and will use the same EAL args as before. The minimum required version for the intel-ipsec-mb library is now v1.0. Signed-off-by: Fan Zhang Signed-off-by: Ciara Power Acked-by: Ray Kinsella --- v4: - Added chacha20_poly1305 to commit log list. - Fixed remove function to also release each queue pair. - Fix copyright year for new files. - Fix return values. - Fix alphabetical order of PMD list in meson build file. - Rename files and functions to remove rte_ prefix and pmd references. - Added basic release note for framework. v3: - Updated intel-ipsec-mb macros. - Added use of auto init function for IMB_MGR. - Added detail to commit log. v2: - Added qp NULL check in get stats function. - Added maintainers file entry. - Replaced strlcpy with rte_strlcpy. --- MAINTAINERS | 4 + doc/guides/rel_notes/release_21_11.rst | 5 + drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 293 +++++++++++++++++++++ drivers/crypto/ipsec_mb/ipsec_mb_private.c | 173 ++++++++++++ drivers/crypto/ipsec_mb/ipsec_mb_private.h | 274 +++++++++++++++++++ drivers/crypto/ipsec_mb/meson.build | 27 ++ drivers/crypto/ipsec_mb/version.map | 3 + drivers/crypto/meson.build | 1 + 8 files changed, 780 insertions(+) create mode 100644 drivers/crypto/ipsec_mb/ipsec_mb_ops.c create mode 100644 drivers/crypto/ipsec_mb/ipsec_mb_private.c create mode 100644 drivers/crypto/ipsec_mb/ipsec_mb_private.h create mode 100644 drivers/crypto/ipsec_mb/meson.build create mode 100644 drivers/crypto/ipsec_mb/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 278e5b3226..c038ceae6d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1067,6 +1067,10 @@ F: drivers/common/qat/ F: doc/guides/cryptodevs/qat.rst F: doc/guides/cryptodevs/features/qat.ini +IPsec MB +M: Fan Zhang +F: drivers/crypto/ipsec_mb/ + KASUMI M: Pablo de Lara F: drivers/crypto/kasumi/ diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 54718ff367..fc2c85b074 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -94,6 +94,11 @@ New Features * Added Intel ixgbe support on Windows. +* **Added framework for consolidation of IPSec_MB dependent SW Crypto PMDs.** + + The IPSec_MB framework was added to share common code between Intel + SW Crypto PMDs that depend on the intel-ipsec-mb library. + * **Updated Marvell cnxk ethdev driver.** * Added rte_flow support for dual VLAN insert and strip actions. diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c new file mode 100644 index 0000000000..409c7556bb --- /dev/null +++ b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c @@ -0,0 +1,293 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include + +#include +#include + +#include "ipsec_mb_private.h" + +/** Configure device */ +int +ipsec_mb_config(__rte_unused struct rte_cryptodev *dev, + __rte_unused struct rte_cryptodev_config *config) +{ + return 0; +} + +/** Start device */ +int +ipsec_mb_start(__rte_unused struct rte_cryptodev *dev) +{ + return 0; +} + +/** Stop device */ +void +ipsec_mb_stop(__rte_unused struct rte_cryptodev *dev) +{ +} + +/** Close device */ +int +ipsec_mb_close(__rte_unused struct rte_cryptodev *dev) +{ + return 0; +} + +/** Get device statistics */ +void +ipsec_mb_stats_get(struct rte_cryptodev *dev, + struct rte_cryptodev_stats *stats) +{ + int qp_id; + + for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { + struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; + if (qp == NULL) { + IPSEC_MB_LOG(DEBUG, "Uninitialised qp %d", qp_id); + continue; + } + + stats->enqueued_count += qp->stats.enqueued_count; + stats->dequeued_count += qp->stats.dequeued_count; + + stats->enqueue_err_count += qp->stats.enqueue_err_count; + stats->dequeue_err_count += qp->stats.dequeue_err_count; + } +} + +/** Reset device statistics */ +void +ipsec_mb_stats_reset(struct rte_cryptodev *dev) +{ + int qp_id; + + for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { + struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; + + memset(&qp->stats, 0, sizeof(qp->stats)); + } +} + +/** Get device info */ +void +ipsec_mb_info_get(struct rte_cryptodev *dev, + struct rte_cryptodev_info *dev_info) +{ + struct ipsec_mb_dev_private *internals = dev->data->dev_private; + struct ipsec_mb_internals *pmd_info = + &ipsec_mb_pmds[internals->pmd_type]; + + if (dev_info != NULL) { + dev_info->driver_id = dev->driver_id; + dev_info->feature_flags = dev->feature_flags; + dev_info->capabilities = pmd_info->caps; + dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; + /* No limit of number of sessions */ + dev_info->sym.max_nb_sessions = 0; + } +} + +/** Release queue pair */ +int +ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) +{ + struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; + struct rte_ring *r = NULL; + + if (qp != NULL) { + r = rte_ring_lookup(qp->name); + if (r) + rte_ring_free(r); + rte_free(qp); + dev->data->queue_pairs[qp_id] = NULL; + } + return 0; +} + +/** Set a unique name for the queue pair */ +int +ipsec_mb_qp_set_unique_name(struct rte_cryptodev *dev, + struct ipsec_mb_qp *qp) +{ + uint32_t n = + snprintf(qp->name, sizeof(qp->name), "ipsec_mb_pmd_%u_qp_%u", + dev->data->dev_id, qp->id); + + if (n >= sizeof(qp->name)) + return -1; + + return 0; +} + +/** Create a ring to place processed operations on */ +static struct rte_ring +*ipsec_mb_qp_create_processed_ops_ring( + struct ipsec_mb_qp *qp, unsigned int ring_size, int socket_id) +{ + struct rte_ring *r; + char ring_name[RTE_CRYPTODEV_NAME_MAX_LEN]; + + unsigned int n = rte_strlcpy(ring_name, qp->name, sizeof(ring_name)); + + if (n >= sizeof(ring_name)) + return NULL; + + r = rte_ring_lookup(ring_name); + if (r) { + if (rte_ring_get_size(r) >= ring_size) { + IPSEC_MB_LOG( + INFO, "Reusing existing ring %s for processed ops", + ring_name); + return r; + } + IPSEC_MB_LOG( + ERR, "Unable to reuse existing ring %s for processed ops", + ring_name); + return NULL; + } + + return rte_ring_create(ring_name, ring_size, socket_id, + RING_F_SP_ENQ | RING_F_SC_DEQ); +} + +/** Setup a queue pair */ +int +ipsec_mb_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, + const struct rte_cryptodev_qp_conf *qp_conf, + int socket_id) +{ + struct ipsec_mb_qp *qp = NULL; + struct ipsec_mb_dev_private *internals = dev->data->dev_private; + struct ipsec_mb_internals *pmd_data = + &ipsec_mb_pmds[internals->pmd_type]; + uint32_t qp_size; + int ret; + + /* Free memory prior to re-allocation if needed. */ + if (dev->data->queue_pairs[qp_id] != NULL) + ipsec_mb_qp_release(dev, qp_id); + + qp_size = sizeof(*qp) + pmd_data->qp_priv_size; + /* Allocate the queue pair data structure. */ + qp = rte_zmalloc_socket("IPSEC PMD Queue Pair", qp_size, + RTE_CACHE_LINE_SIZE, socket_id); + if (qp == NULL) + return -ENOMEM; + + qp->id = qp_id; + dev->data->queue_pairs[qp_id] = qp; + if (ipsec_mb_qp_set_unique_name(dev, qp)) { + ret = -EINVAL; + goto qp_setup_cleanup; + } + + qp->pmd_type = internals->pmd_type; + qp->sess_mp = qp_conf->mp_session; + qp->sess_mp_priv = qp_conf->mp_session_private; + + qp->ingress_queue = ipsec_mb_qp_create_processed_ops_ring(qp, + qp_conf->nb_descriptors, socket_id); + if (qp->ingress_queue == NULL) { + ret = -EINVAL; + goto qp_setup_cleanup; + } + + qp->mb_mgr = alloc_init_mb_mgr(); + if (!qp->mb_mgr) { + ret = -ENOMEM; + goto qp_setup_cleanup; + } + + memset(&qp->stats, 0, sizeof(qp->stats)); + + if (pmd_data->queue_pair_configure) { + ret = pmd_data->queue_pair_configure(qp); + if (ret < 0) + goto qp_setup_cleanup; + } + + return 0; + +qp_setup_cleanup: + if (qp->mb_mgr) + free_mb_mgr(qp->mb_mgr); + if (qp) + rte_free(qp); + return ret; +} + +/** Return the size of the specific pmd session structure */ +unsigned +ipsec_mb_sym_session_get_size(struct rte_cryptodev *dev) +{ + struct ipsec_mb_dev_private *internals = dev->data->dev_private; + struct ipsec_mb_internals *pmd_data = + &ipsec_mb_pmds[internals->pmd_type]; + + return pmd_data->session_priv_size; +} + +/** Configure pmd specific multi-buffer session from a crypto xform chain */ +int +ipsec_mb_sym_session_configure( + struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform, + struct rte_cryptodev_sym_session *sess, struct rte_mempool *mempool) +{ + void *sess_private_data; + struct ipsec_mb_dev_private *internals = dev->data->dev_private; + struct ipsec_mb_internals *pmd_data = + &ipsec_mb_pmds[internals->pmd_type]; + IMB_MGR *mb_mgr = alloc_init_mb_mgr(); + int ret = 0; + + if (!mb_mgr) + return -ENOMEM; + + if (unlikely(sess == NULL)) { + IPSEC_MB_LOG(ERR, "invalid session struct"); + free_mb_mgr(mb_mgr); + return -EINVAL; + } + + if (rte_mempool_get(mempool, &sess_private_data)) { + IPSEC_MB_LOG(ERR, "Couldn't get object from session mempool"); + free_mb_mgr(mb_mgr); + return -ENOMEM; + } + + ret = (*pmd_data->session_configure)(mb_mgr, sess_private_data, xform); + if (ret != 0) { + IPSEC_MB_LOG(ERR, "failed configure session parameters"); + + /* Return session to mempool */ + rte_mempool_put(mempool, sess_private_data); + free_mb_mgr(mb_mgr); + return ret; + } + + set_sym_session_private_data(sess, dev->driver_id, sess_private_data); + + return 0; +} + +/** Clear the session memory */ +void +ipsec_mb_sym_session_clear(struct rte_cryptodev *dev, + struct rte_cryptodev_sym_session *sess) +{ + uint8_t index = dev->driver_id; + void *sess_priv = get_sym_session_private_data(sess, index); + + /* Zero out the whole structure */ + if (sess_priv) { + memset(sess_priv, 0, ipsec_mb_sym_session_get_size(dev)); + struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); + + set_sym_session_private_data(sess, index, NULL); + rte_mempool_put(sess_mp, sess_priv); + } +} diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.c b/drivers/crypto/ipsec_mb/ipsec_mb_private.c new file mode 100644 index 0000000000..d142d7622b --- /dev/null +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.c @@ -0,0 +1,173 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include +#include +#include + +#include "ipsec_mb_private.h" + +RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); + +struct ipsec_mb_internals ipsec_mb_pmds[IPSEC_MB_N_PMD_TYPES]; +int ipsec_mb_logtype_driver; +enum ipsec_mb_vector_mode vector_mode; + +/** + * Generic burst enqueue, place crypto operations on ingress queue for + * processing. + * + * @param __qp Queue Pair to process + * @param ops Crypto operations for processing + * @param nb_ops Number of crypto operations for processing + * + * @return + * - Number of crypto operations enqueued + */ +static uint16_t +ipsec_mb_enqueue_burst(void *__qp, struct rte_crypto_op **ops, + uint16_t nb_ops) +{ + struct ipsec_mb_qp *qp = __qp; + + unsigned int nb_enqueued; + + nb_enqueued = rte_ring_enqueue_burst(qp->ingress_queue, + (void **)ops, nb_ops, NULL); + + qp->stats.enqueued_count += nb_enqueued; + qp->stats.enqueue_err_count += nb_ops - nb_enqueued; + + return nb_enqueued; +} + +int +ipsec_mb_create(struct rte_vdev_device *vdev, + enum ipsec_mb_pmd_types pmd_type) +{ + struct rte_cryptodev *dev; + struct ipsec_mb_dev_private *internals; + struct ipsec_mb_internals *pmd_data = &ipsec_mb_pmds[pmd_type]; + struct rte_cryptodev_pmd_init_params init_params = {}; + const char *name, *args; + int retval; + + if (vector_mode == IPSEC_MB_NOT_SUPPORTED) { + /* Check CPU for supported vector instruction set */ + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) + vector_mode = IPSEC_MB_AVX512; + else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) + vector_mode = IPSEC_MB_AVX2; + else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) + vector_mode = IPSEC_MB_AVX; + else + vector_mode = IPSEC_MB_SSE; + } + + init_params.private_data_size = sizeof(struct ipsec_mb_dev_private) + + pmd_data->internals_priv_size; + init_params.max_nb_queue_pairs = + RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS; + init_params.socket_id = rte_socket_id(); + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + + args = rte_vdev_device_args(vdev); + + retval = rte_cryptodev_pmd_parse_input_args(&init_params, args); + if (retval) { + IPSEC_MB_LOG( + ERR, "Failed to parse initialisation arguments[%s]", args); + return -EINVAL; + } + + dev = rte_cryptodev_pmd_create(name, &vdev->device, &init_params); + if (dev == NULL) { + IPSEC_MB_LOG(ERR, "driver %s: create failed", + init_params.name); + return -ENODEV; + } + + /* Set vector instructions mode supported */ + internals = dev->data->dev_private; + internals->pmd_type = pmd_type; + internals->max_nb_queue_pairs = init_params.max_nb_queue_pairs; + + dev->driver_id = ipsec_mb_get_driver_id(pmd_type); + if (dev->driver_id == UINT8_MAX) { + IPSEC_MB_LOG(ERR, "driver %s: create failed", + init_params.name); + return -ENODEV; + } + dev->dev_ops = ipsec_mb_pmds[pmd_type].ops; + dev->enqueue_burst = ipsec_mb_enqueue_burst; + dev->dequeue_burst = ipsec_mb_pmds[pmd_type].dequeue_burst; + + if (pmd_data->dev_config) { + retval = (*pmd_data->dev_config)(dev); + if (retval < 0) { + IPSEC_MB_LOG(ERR, + "Failed to configure device %s", name); + rte_cryptodev_pmd_destroy(dev); + return retval; + } + } + + dev->feature_flags = pmd_data->feature_flags; + + switch (vector_mode) { + case IPSEC_MB_AVX512: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; + break; + case IPSEC_MB_AVX2: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + break; + case IPSEC_MB_AVX: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + break; + case IPSEC_MB_SSE: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + break; + default: + break; + } + + IPSEC_MB_LOG(INFO, "IPSec Multi-buffer library version used: %s\n", + imb_get_version_str()); + + return 0; +} + +int +ipsec_mb_remove(struct rte_vdev_device *vdev) +{ + struct rte_cryptodev *cryptodev; + const char *name; + int qp_id; + + name = rte_vdev_device_name(vdev); + if (name == NULL) + return -EINVAL; + + cryptodev = rte_cryptodev_pmd_get_named_dev(name); + if (cryptodev == NULL) + return -ENODEV; + + if (RTE_PER_LCORE(mb_mgr)) { + free_mb_mgr(RTE_PER_LCORE(mb_mgr)); + RTE_PER_LCORE(mb_mgr) = NULL; + } + + if (cryptodev->security_ctx) { + rte_free(cryptodev->security_ctx); + cryptodev->security_ctx = NULL; + } + + for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++) + ipsec_mb_qp_release(cryptodev, qp_id); + + return rte_cryptodev_pmd_destroy(cryptodev); +} diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h new file mode 100644 index 0000000000..a6e05a07db --- /dev/null +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -0,0 +1,274 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef _IPSEC_MB_PRIVATE_H_ +#define _IPSEC_MB_PRIVATE_H_ + +#include +#include +#include + +#if defined(RTE_LIB_SECURITY) +#define IPSEC_MB_DOCSIS_SEC_ENABLED 1 +#include +#include +#endif + +/* Maximum length for digest */ +#define DIGEST_LENGTH_MAX 64 + +enum ipsec_mb_vector_mode { + IPSEC_MB_NOT_SUPPORTED = 0, + IPSEC_MB_SSE, + IPSEC_MB_AVX, + IPSEC_MB_AVX2, + IPSEC_MB_AVX512 +}; + +extern enum ipsec_mb_vector_mode vector_mode; + +/** IMB_MGR instances, one per thread */ +extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); + +/** PMD LOGTYPE DRIVER, common to all PMDs */ +extern int ipsec_mb_logtype_driver; +#define IPSEC_MB_LOG(level, fmt, ...) \ + rte_log(RTE_LOG_##level, ipsec_mb_logtype_driver, \ + "%s() line %u: " fmt "\n", __func__, __LINE__, ##__VA_ARGS__) + +/** All supported device types */ +enum ipsec_mb_pmd_types { + IPSEC_MB_N_PMD_TYPES +}; + +/** Crypto operations */ +enum ipsec_mb_operation { + IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN = 0, + IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT, + IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT, + IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY, + IPSEC_MB_OP_ENCRYPT_ONLY, + IPSEC_MB_OP_DECRYPT_ONLY, + IPSEC_MB_OP_HASH_GEN_ONLY, + IPSEC_MB_OP_HASH_VERIFY_ONLY, + IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT, + IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT, + IPSEC_MB_OP_NOT_SUPPORTED +}; + +/** Helper function. Gets driver ID based on PMD type */ +static __rte_always_inline uint8_t +ipsec_mb_get_driver_id(__rte_unused enum ipsec_mb_pmd_types pmd_type) +{ + return UINT8_MAX; +} + +/** Common private data structure for each PMD */ +struct ipsec_mb_dev_private { + enum ipsec_mb_pmd_types pmd_type; + /**< PMD type */ + uint32_t max_nb_queue_pairs; + /**< Max number of queue pairs supported by device */ + __extension__ uint8_t priv[0]; +}; + +/** IPSEC Multi buffer queue pair common queue pair data for all PMDs */ +struct ipsec_mb_qp { + uint16_t id; + /**< Queue Pair Identifier */ + char name[RTE_CRYPTODEV_NAME_MAX_LEN]; + struct rte_ring *ingress_queue; + /**< Ring for placing operations ready for processing */ + struct rte_mempool *sess_mp; + /**< Session Mempool */ + struct rte_mempool *sess_mp_priv; + /**< Session Private Data Mempool */ + struct rte_cryptodev_stats stats; + /**< Queue pair statistics */ + enum ipsec_mb_pmd_types pmd_type; + /**< pmd type */ + uint8_t digest_idx; + /**< Index of the next + * slot to be used in temp_digests, + * to store the digest for a given operation + */ + IMB_MGR *mb_mgr; + /* Multi buffer manager */ + __extension__ uint8_t additional_data[0]; + /**< Storing PMD specific additional data */ +}; + +static __rte_always_inline void * +ipsec_mb_get_qp_private_data(struct ipsec_mb_qp *qp) +{ + return (void *)qp->additional_data; +} + +/** Helper function. Allocates job manager */ +static __rte_always_inline IMB_MGR * +alloc_init_mb_mgr(void) +{ + IMB_MGR *mb_mgr = alloc_mb_mgr(0); + + if (unlikely(mb_mgr == NULL)) { + IPSEC_MB_LOG(ERR, "Failed to allocate IMB_MGR data\n"); + return NULL; + } + + init_mb_mgr_auto(mb_mgr, NULL); + + return mb_mgr; +} + +/** Helper function. Gets per thread job manager */ +static __rte_always_inline IMB_MGR * +get_per_thread_mb_mgr(void) +{ + if (unlikely(RTE_PER_LCORE(mb_mgr) == NULL)) + RTE_PER_LCORE(mb_mgr) = alloc_init_mb_mgr(); + + return RTE_PER_LCORE(mb_mgr); +} + +/** Device creation function */ +int +ipsec_mb_create(struct rte_vdev_device *vdev, + enum ipsec_mb_pmd_types pmd_type); + +/** Device remove function */ +int +ipsec_mb_remove(struct rte_vdev_device *vdev); + +/** Configure queue pair PMD type specific data */ +typedef int (*ipsec_mb_queue_pair_configure_t)(struct ipsec_mb_qp *qp); + +/** Configure session PMD type specific data */ +typedef int (*ipsec_mb_session_configure_t)(IMB_MGR *mbr_mgr, + void *session_private, + const struct rte_crypto_sym_xform *xform); + +/** Configure internals PMD type specific data */ +typedef int (*ipsec_mb_dev_configure_t)(struct rte_cryptodev *dev); + +/** Per PMD type operation and data */ +struct ipsec_mb_internals { + uint8_t is_configured; + dequeue_pkt_burst_t dequeue_burst; + ipsec_mb_dev_configure_t dev_config; + ipsec_mb_queue_pair_configure_t queue_pair_configure; + ipsec_mb_session_configure_t session_configure; + const struct rte_cryptodev_capabilities *caps; + struct rte_cryptodev_ops *ops; + struct rte_security_ops *security_ops; + uint64_t feature_flags; + uint32_t session_priv_size; + uint32_t qp_priv_size; + uint32_t internals_priv_size; +}; + +/** Global PMD type specific data */ +extern struct ipsec_mb_internals ipsec_mb_pmds[IPSEC_MB_N_PMD_TYPES]; + +int +ipsec_mb_config(struct rte_cryptodev *dev, + struct rte_cryptodev_config *config); + +int +ipsec_mb_start(struct rte_cryptodev *dev); + +void +ipsec_mb_stop(struct rte_cryptodev *dev); + +int +ipsec_mb_close(struct rte_cryptodev *dev); + +void +ipsec_mb_stats_get(struct rte_cryptodev *dev, + struct rte_cryptodev_stats *stats); + +void +ipsec_mb_stats_reset(struct rte_cryptodev *dev); + +void +ipsec_mb_info_get(struct rte_cryptodev *dev, + struct rte_cryptodev_info *dev_info); + +int +ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id); + +int +ipsec_mb_qp_set_unique_name(struct rte_cryptodev *dev, struct ipsec_mb_qp *qp); + +int +ipsec_mb_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, + const struct rte_cryptodev_qp_conf *qp_conf, + int socket_id); + +/** Returns the size of the aesni multi-buffer session structure */ +unsigned +ipsec_mb_sym_session_get_size(struct rte_cryptodev *dev); + +/** Configure an aesni multi-buffer session from a crypto xform chain */ +int ipsec_mb_sym_session_configure( + struct rte_cryptodev *dev, + struct rte_crypto_sym_xform *xform, + struct rte_cryptodev_sym_session *sess, + struct rte_mempool *mempool); + +/** Clear the memory of session so it does not leave key material behind */ +void +ipsec_mb_sym_session_clear(struct rte_cryptodev *dev, + struct rte_cryptodev_sym_session *sess); + +/** Get session from op. If sessionless create a session */ +static __rte_always_inline void * +ipsec_mb_get_session_private(struct ipsec_mb_qp *qp, struct rte_crypto_op *op) +{ + void *sess = NULL; + uint32_t driver_id = ipsec_mb_get_driver_id(qp->pmd_type); + struct rte_crypto_sym_op *sym_op = op->sym; + uint8_t sess_type = op->sess_type; + void *_sess; + void *_sess_private_data = NULL; + struct ipsec_mb_internals *pmd_data = &ipsec_mb_pmds[qp->pmd_type]; + + switch (sess_type) { + case RTE_CRYPTO_OP_WITH_SESSION: + if (likely(sym_op->session != NULL)) + sess = get_sym_session_private_data(sym_op->session, + driver_id); + break; + case RTE_CRYPTO_OP_SESSIONLESS: + if (!qp->sess_mp || + rte_mempool_get(qp->sess_mp, (void **)&_sess)) + return NULL; + + if (!qp->sess_mp_priv || + rte_mempool_get(qp->sess_mp_priv, + (void **)&_sess_private_data)) + return NULL; + + sess = _sess_private_data; + if (unlikely(pmd_data->session_configure(qp->mb_mgr, + sess, sym_op->xform) != 0)) { + rte_mempool_put(qp->sess_mp, _sess); + rte_mempool_put(qp->sess_mp_priv, _sess_private_data); + sess = NULL; + } + + sym_op->session = (struct rte_cryptodev_sym_session *)_sess; + set_sym_session_private_data(sym_op->session, driver_id, + _sess_private_data); + break; + default: + IPSEC_MB_LOG(ERR, "Unrecognized session type %u", sess_type); + } + + if (unlikely(sess == NULL)) + op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; + + return sess; +} + +#endif /* _IPSEC_MB_PRIVATE_H_ */ diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build new file mode 100644 index 0000000000..76676fe990 --- /dev/null +++ b/drivers/crypto/ipsec_mb/meson.build @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2021 Intel Corporation + +IMB_required_ver = '1.0.0' +lib = cc.find_library('IPSec_MB', required: false) +if not lib.found() + build = false + reason = 'missing dependency, "libIPSec_MB"' +else + ext_deps += lib + + # version comes with quotes, so we split based on " and take the middle + imb_ver = cc.get_define('IMB_VERSION_STR', + prefix : '#include').split('"')[1] + + if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) + reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( + IMB_required_ver, imb_ver) + build = false + endif + +endif + +sources = files('ipsec_mb_private.c', + 'ipsec_mb_ops.c', + ) +deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/ipsec_mb/version.map b/drivers/crypto/ipsec_mb/version.map new file mode 100644 index 0000000000..c2e0723b4c --- /dev/null +++ b/drivers/crypto/ipsec_mb/version.map @@ -0,0 +1,3 @@ +DPDK_22 { + local: *; +}; diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index ea239f4c56..4d93dcd216 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -15,6 +15,7 @@ drivers = [ 'cnxk', 'dpaa_sec', 'dpaa2_sec', + 'ipsec_mb', 'kasumi', 'mlx5', 'mvsam', From patchwork Fri Oct 15 14:39:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101773 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 18233A0C45; Fri, 15 Oct 2021 16:40:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0CE064126B; Fri, 15 Oct 2021 16:40:23 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 03FFE4125E for ; Fri, 15 Oct 2021 16:40:20 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816113" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816113" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542788" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:18 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Ciara Power , Anatoly Burakov Date: Fri, 15 Oct 2021 14:39:45 +0000 Message-Id: <20211015143957.842499-3-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 02/14] crypto/ipsec_mb: add multiprocess support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The ipsec_mb SW PMD now has multiprocess support. The queue-pair IMB_MGR is stored in a memzone instead of being allocated externally by the Intel IPSec MB library, when v1.1 is used. If v1.0 is used, multi process is not supported, and allocation is done as before. The secondary process needs to reconfigure the queue-pair to allow for IMB_MGR function pointers be updated. Intel IPsec MB library version 1.1 is required for this support. Signed-off-by: Ciara Power Acked-by: Ray Kinsella --- v4: - Remove call to init mb_mgr auto when setting up queue pair IMB_MGR for a secondary process, this is not needed. - Renamed function to remove pmd reference. - Combined release note with the IPSec_MB framework note. --- doc/guides/rel_notes/release_21_11.rst | 3 + drivers/crypto/ipsec_mb/ipsec_mb_ops.c | 109 ++++++++++++++++++--- drivers/crypto/ipsec_mb/ipsec_mb_private.h | 5 + 3 files changed, 101 insertions(+), 16 deletions(-) diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index fc2c85b074..5771cbbf68 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -99,6 +99,9 @@ New Features The IPSec_MB framework was added to share common code between Intel SW Crypto PMDs that depend on the intel-ipsec-mb library. + * Multiprocess support was added for the consolidated PMDs, + which requires v1.1 of the intel-ipsec-mb library. + * **Updated Marvell cnxk ethdev driver.** * Added rte_flow support for dual VLAN insert and strip actions. diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c index 409c7556bb..189262c4ad 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_ops.c +++ b/drivers/crypto/ipsec_mb/ipsec_mb_ops.c @@ -9,6 +9,8 @@ #include "ipsec_mb_private.h" +#define IMB_MP_REQ_VER_STR "1.1.0" + /** Configure device */ int ipsec_mb_config(__rte_unused struct rte_cryptodev *dev, @@ -98,10 +100,20 @@ ipsec_mb_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; struct rte_ring *r = NULL; - if (qp != NULL) { + if (qp != NULL && rte_eal_process_type() == RTE_PROC_PRIMARY) { r = rte_ring_lookup(qp->name); if (r) rte_ring_free(r); + +#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM + if (qp->mb_mgr) + free_mb_mgr(qp->mb_mgr); +#else + if (qp->mb_mgr_mz) { + rte_memzone_free(qp->mb_mgr_mz); + qp->mb_mgr = NULL; + } +#endif rte_free(qp); dev->data->queue_pairs[qp_id] = NULL; } @@ -154,6 +166,41 @@ static struct rte_ring RING_F_SP_ENQ | RING_F_SC_DEQ); } +#if IMB_VERSION(1, 1, 0) <= IMB_VERSION_NUM +static IMB_MGR * +ipsec_mb_alloc_mgr_from_memzone(const struct rte_memzone **mb_mgr_mz, + const char *mb_mgr_mz_name) +{ + IMB_MGR *mb_mgr; + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + *mb_mgr_mz = rte_memzone_lookup(mb_mgr_mz_name); + if (*mb_mgr_mz == NULL) { + *mb_mgr_mz = rte_memzone_reserve(mb_mgr_mz_name, + imb_get_mb_mgr_size(), + rte_socket_id(), 0); + } + if (*mb_mgr_mz == NULL) { + IPSEC_MB_LOG(DEBUG, "Error allocating memzone for %s", + mb_mgr_mz_name); + return NULL; + } + mb_mgr = imb_set_pointers_mb_mgr((*mb_mgr_mz)->addr, 0, 1); + init_mb_mgr_auto(mb_mgr, NULL); + } else { + *mb_mgr_mz = rte_memzone_lookup(mb_mgr_mz_name); + if (*mb_mgr_mz == NULL) { + IPSEC_MB_LOG(ERR, + "Secondary can't find %s mz, did primary create it?", + mb_mgr_mz_name); + return NULL; + } + mb_mgr = imb_set_pointers_mb_mgr((*mb_mgr_mz)->addr, 0, 0); + } + return mb_mgr; +} +#endif + /** Setup a queue pair */ int ipsec_mb_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, @@ -167,16 +214,44 @@ ipsec_mb_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, uint32_t qp_size; int ret; - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - ipsec_mb_qp_release(dev, qp_id); + if (rte_eal_process_type() == RTE_PROC_SECONDARY) { +#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM + IPSEC_MB_LOG(ERR, "The intel-ipsec-mb version (%s) does not support multiprocess," + "the minimum version required for this feature is %s.", + IMB_VERSION_STR, IMB_MP_REQ_VER_STR); + return -EINVAL; +#endif + if (dev->data->queue_pairs[qp_id] != NULL) + qp = dev->data->queue_pairs[qp_id]; + } else { + /* Free memory prior to re-allocation if needed. */ + if (dev->data->queue_pairs[qp_id] != NULL) + ipsec_mb_qp_release(dev, qp_id); + + qp_size = sizeof(*qp) + pmd_data->qp_priv_size; + /* Allocate the queue pair data structure. */ + qp = rte_zmalloc_socket("IPSEC PMD Queue Pair", qp_size, + RTE_CACHE_LINE_SIZE, socket_id); + if (qp == NULL) + return -ENOMEM; + } + +#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM + qp->mb_mgr = alloc_init_mb_mgr(); +#else + char mz_name[IPSEC_MB_MAX_MZ_NAME]; + snprintf(mz_name, sizeof(mz_name), "IMB_MGR_DEV_%d_QP_%d", + dev->data->dev_id, qp_id); + qp->mb_mgr = ipsec_mb_alloc_mgr_from_memzone(&(qp->mb_mgr_mz), + mz_name); +#endif + if (qp->mb_mgr == NULL) { + ret = -ENOMEM; + goto qp_setup_cleanup; + } - qp_size = sizeof(*qp) + pmd_data->qp_priv_size; - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("IPSEC PMD Queue Pair", qp_size, - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return -ENOMEM; + if (rte_eal_process_type() == RTE_PROC_SECONDARY) + return 0; qp->id = qp_id; dev->data->queue_pairs[qp_id] = qp; @@ -196,12 +271,6 @@ ipsec_mb_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, goto qp_setup_cleanup; } - qp->mb_mgr = alloc_init_mb_mgr(); - if (!qp->mb_mgr) { - ret = -ENOMEM; - goto qp_setup_cleanup; - } - memset(&qp->stats, 0, sizeof(qp->stats)); if (pmd_data->queue_pair_configure) { @@ -213,8 +282,15 @@ ipsec_mb_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, return 0; qp_setup_cleanup: +#if IMB_VERSION(1, 1, 0) > IMB_VERSION_NUM if (qp->mb_mgr) free_mb_mgr(qp->mb_mgr); +#else + if (rte_eal_process_type() == RTE_PROC_SECONDARY) + return ret; + if (qp->mb_mgr_mz) + rte_memzone_free(qp->mb_mgr_mz); +#endif if (qp) rte_free(qp); return ret; @@ -271,6 +347,7 @@ ipsec_mb_sym_session_configure( set_sym_session_private_data(sess, dev->driver_id, sess_private_data); + free_mb_mgr(mb_mgr); return 0; } diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index a6e05a07db..541f759e66 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -18,6 +18,9 @@ /* Maximum length for digest */ #define DIGEST_LENGTH_MAX 64 +/* Maximum length for memzone name */ +#define IPSEC_MB_MAX_MZ_NAME 32 + enum ipsec_mb_vector_mode { IPSEC_MB_NOT_SUPPORTED = 0, IPSEC_MB_SSE, @@ -95,6 +98,8 @@ struct ipsec_mb_qp { */ IMB_MGR *mb_mgr; /* Multi buffer manager */ + const struct rte_memzone *mb_mgr_mz; + /* Shared memzone for storing mb_mgr */ __extension__ uint8_t additional_data[0]; /**< Storing PMD specific additional data */ }; From patchwork Fri Oct 15 14:39:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101774 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 67B75A0C45; Fri, 15 Oct 2021 16:40:38 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 13094411CB; Fri, 15 Oct 2021 16:40:27 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 64454411CB for ; Fri, 15 Oct 2021 16:40:24 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816124" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816124" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542803" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:21 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, =?utf-8?q?Ciara=C2=A0Power?= , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:46 +0000 Message-Id: <20211015143957.842499-4-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 03/14] drivers/crypto: move aesni-mb PMD to IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch removes the crypto/aesni_mb folder and gathers all aesni-mb PMD implementation specific details into a single file, pmd_aesni_mb.c in crypto/ipsec_mb. Now that intel-ipsec-mb v1.0 is the minimum supported version, old macros can be replaced with the newer macros supported by this version. Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - Fixed some struct and function naming. - Added release note. - Fixed max AAD size for ChaCha20_poly1305. - Fixed incorrect error log. - Split into private header file. - Fixed some checkpatch errors. v3: - Updated intel-ipsec-mb library macros. - Fixed some formatting. v2: Updated maintainers file. --- MAINTAINERS | 10 +- doc/guides/cryptodevs/aesni_mb.rst | 4 +- doc/guides/rel_notes/release_21_11.rst | 4 + .../crypto/aesni_mb/aesni_mb_pmd_private.h | 337 ----- drivers/crypto/aesni_mb/meson.build | 25 - drivers/crypto/aesni_mb/version.map | 3 - drivers/crypto/ipsec_mb/ipsec_mb_private.c | 7 +- drivers/crypto/ipsec_mb/ipsec_mb_private.h | 143 +- drivers/crypto/ipsec_mb/meson.build | 1 + .../pmd_aesni_mb.c} | 1214 +++++++---------- .../pmd_aesni_mb_priv.h} | 665 ++++----- drivers/crypto/meson.build | 1 - 12 files changed, 916 insertions(+), 1498 deletions(-) delete mode 100644 drivers/crypto/aesni_mb/aesni_mb_pmd_private.h delete mode 100644 drivers/crypto/aesni_mb/meson.build delete mode 100644 drivers/crypto/aesni_mb/version.map rename drivers/crypto/{aesni_mb/rte_aesni_mb_pmd.c => ipsec_mb/pmd_aesni_mb.c} (60%) rename drivers/crypto/{aesni_mb/rte_aesni_mb_pmd_ops.c => ipsec_mb/pmd_aesni_mb_priv.h} (55%) diff --git a/MAINTAINERS b/MAINTAINERS index c038ceae6d..9aeb0d8e03 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1051,13 +1051,6 @@ F: drivers/crypto/aesni_gcm/ F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/features/aesni_gcm.ini -Intel AES-NI Multi-Buffer -M: Declan Doherty -M: Pablo de Lara -F: drivers/crypto/aesni_mb/ -F: doc/guides/cryptodevs/aesni_mb.rst -F: doc/guides/cryptodevs/features/aesni_mb.ini - Intel QuickAssist M: John Griffin M: Fiona Trahe @@ -1069,7 +1062,10 @@ F: doc/guides/cryptodevs/features/qat.ini IPsec MB M: Fan Zhang +M: Pablo de Lara F: drivers/crypto/ipsec_mb/ +F: doc/guides/cryptodevs/aesni_mb.rst +F: doc/guides/cryptodevs/features/aesni_mb.ini KASUMI M: Pablo de Lara diff --git a/doc/guides/cryptodevs/aesni_mb.rst b/doc/guides/cryptodevs/aesni_mb.rst index a466d0ab48..3551a0dbd7 100644 --- a/doc/guides/cryptodevs/aesni_mb.rst +++ b/doc/guides/cryptodevs/aesni_mb.rst @@ -130,7 +130,9 @@ and the Multi-Buffer library version supported by them: 18.02 0.48 18.05 - 19.02 0.49 - 0.52 19.05 - 19.08 0.52 - 19.11+ 0.52 - 1.0* + 19.11 - 20.08 0.52 - 0.55 + 20.11 - 21.08 0.53 - 1.0* + 21.11+ 1.0* ============== ============================ \* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 5771cbbf68..7628c32052 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -101,6 +101,10 @@ New Features * Multiprocess support was added for the consolidated PMDs, which requires v1.1 of the intel-ipsec-mb library. + * The following PMDs were moved into a single source folder, + however their usage and EAL options remain unchanged. + + * AESNI_MB PMD. * **Updated Marvell cnxk ethdev driver.** diff --git a/drivers/crypto/aesni_mb/aesni_mb_pmd_private.h b/drivers/crypto/aesni_mb/aesni_mb_pmd_private.h deleted file mode 100644 index 11e7bf5d18..0000000000 --- a/drivers/crypto/aesni_mb/aesni_mb_pmd_private.h +++ /dev/null @@ -1,337 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2015-2016 Intel Corporation - */ - -#ifndef _AESNI_MB_PMD_PRIVATE_H_ -#define _AESNI_MB_PMD_PRIVATE_H_ - -#include - -#if defined(RTE_LIB_SECURITY) && (IMB_VERSION_NUM) >= IMB_VERSION(0, 54, 0) -#define AESNI_MB_DOCSIS_SEC_ENABLED 1 -#include -#include -#endif - -enum aesni_mb_vector_mode { - RTE_AESNI_MB_NOT_SUPPORTED = 0, - RTE_AESNI_MB_SSE, - RTE_AESNI_MB_AVX, - RTE_AESNI_MB_AVX2, - RTE_AESNI_MB_AVX512 -}; - -#define CRYPTODEV_NAME_AESNI_MB_PMD crypto_aesni_mb -/**< AES-NI Multi buffer PMD device name */ - -/** AESNI_MB PMD LOGTYPE DRIVER */ -extern int aesni_mb_logtype_driver; - -#define AESNI_MB_LOG(level, fmt, ...) \ - rte_log(RTE_LOG_ ## level, aesni_mb_logtype_driver, \ - "%s() line %u: " fmt "\n", __func__, __LINE__, \ - ## __VA_ARGS__) - - -#define HMAC_IPAD_VALUE (0x36) -#define HMAC_OPAD_VALUE (0x5C) - -/* Maximum length for digest */ -#define DIGEST_LENGTH_MAX 64 -static const unsigned auth_blocksize[] = { - [NULL_HASH] = 0, - [MD5] = 64, - [SHA1] = 64, - [SHA_224] = 64, - [SHA_256] = 64, - [SHA_384] = 128, - [SHA_512] = 128, - [AES_XCBC] = 16, - [AES_CCM] = 16, - [AES_CMAC] = 16, - [AES_GMAC] = 16, - [PLAIN_SHA1] = 64, - [PLAIN_SHA_224] = 64, - [PLAIN_SHA_256] = 64, - [PLAIN_SHA_384] = 128, - [PLAIN_SHA_512] = 128, -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM - [IMB_AUTH_ZUC_EIA3_BITLEN] = 16, - [IMB_AUTH_SNOW3G_UIA2_BITLEN] = 16, - [IMB_AUTH_KASUMI_UIA1] = 16 -#endif -}; - -/** - * Get the blocksize in bytes for a specified authentication algorithm - * - * @Note: this function will not return a valid value for a non-valid - * authentication algorithm - */ -static inline unsigned -get_auth_algo_blocksize(JOB_HASH_ALG algo) -{ - return auth_blocksize[algo]; -} - -static const unsigned auth_truncated_digest_byte_lengths[] = { - [MD5] = 12, - [SHA1] = 12, - [SHA_224] = 14, - [SHA_256] = 16, - [SHA_384] = 24, - [SHA_512] = 32, - [AES_XCBC] = 12, - [AES_CMAC] = 12, - [AES_CCM] = 8, - [NULL_HASH] = 0, - [AES_GMAC] = 12, - [PLAIN_SHA1] = 20, - [PLAIN_SHA_224] = 28, - [PLAIN_SHA_256] = 32, - [PLAIN_SHA_384] = 48, - [PLAIN_SHA_512] = 64, -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM - [IMB_AUTH_ZUC_EIA3_BITLEN] = 4, - [IMB_AUTH_SNOW3G_UIA2_BITLEN] = 4, - [IMB_AUTH_KASUMI_UIA1] = 4 -#endif -}; - -/** - * Get the IPsec specified truncated length in bytes of the HMAC digest for a - * specified authentication algorithm - * - * @Note: this function will not return a valid value for a non-valid - * authentication algorithm - */ -static inline unsigned -get_truncated_digest_byte_length(JOB_HASH_ALG algo) -{ - return auth_truncated_digest_byte_lengths[algo]; -} - -static const unsigned auth_digest_byte_lengths[] = { - [MD5] = 16, - [SHA1] = 20, - [SHA_224] = 28, - [SHA_256] = 32, - [SHA_384] = 48, - [SHA_512] = 64, - [AES_XCBC] = 16, - [AES_CMAC] = 16, - [AES_CCM] = 16, - [AES_GMAC] = 16, - [NULL_HASH] = 0, - [PLAIN_SHA1] = 20, - [PLAIN_SHA_224] = 28, - [PLAIN_SHA_256] = 32, - [PLAIN_SHA_384] = 48, - [PLAIN_SHA_512] = 64, -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM - [IMB_AUTH_ZUC_EIA3_BITLEN] = 4, - [IMB_AUTH_SNOW3G_UIA2_BITLEN] = 4, - [IMB_AUTH_KASUMI_UIA1] = 4 -#endif - /**< Vector mode dependent pointer table of the multi-buffer APIs */ - -}; - -/** - * Get the full digest size in bytes for a specified authentication algorithm - * (if available in the Multi-buffer library) - * - * @Note: this function will not return a valid value for a non-valid - * authentication algorithm - */ -static inline unsigned -get_digest_byte_length(JOB_HASH_ALG algo) -{ - return auth_digest_byte_lengths[algo]; -} - -enum aesni_mb_operation { - AESNI_MB_OP_HASH_CIPHER, - AESNI_MB_OP_CIPHER_HASH, - AESNI_MB_OP_HASH_ONLY, - AESNI_MB_OP_CIPHER_ONLY, - AESNI_MB_OP_AEAD_HASH_CIPHER, - AESNI_MB_OP_AEAD_CIPHER_HASH, - AESNI_MB_OP_NOT_SUPPORTED -}; - -/** private data structure for each virtual AESNI device */ -struct aesni_mb_private { - enum aesni_mb_vector_mode vector_mode; - /**< CPU vector instruction set mode */ - unsigned max_nb_queue_pairs; - /**< Max number of queue pairs supported by device */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ -}; - -/** AESNI Multi buffer queue pair */ -struct aesni_mb_qp { - uint16_t id; - /**< Queue Pair Identifier */ - char name[RTE_CRYPTODEV_NAME_MAX_LEN]; - /**< Unique Queue Pair Name */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ - struct rte_ring *ingress_queue; - /**< Ring for placing operations ready for processing */ - struct rte_mempool *sess_mp; - /**< Session Mempool */ - struct rte_mempool *sess_mp_priv; - /**< Session Private Data Mempool */ - struct rte_cryptodev_stats stats; - /**< Queue pair statistics */ - uint8_t digest_idx; - /**< Index of the next slot to be used in temp_digests, - * to store the digest for a given operation - */ - uint8_t temp_digests[MAX_JOBS][DIGEST_LENGTH_MAX]; - /**< Buffers used to store the digest generated - * by the driver when verifying a digest provided - * by the user (using authentication verify operation) - */ -} __rte_cache_aligned; - -/** AES-NI multi-buffer private session structure */ -struct aesni_mb_session { - JOB_CHAIN_ORDER chain_order; - struct { - uint16_t length; - uint16_t offset; - } iv; - struct { - uint16_t length; - uint16_t offset; - } auth_iv; - /**< IV parameters */ - - /** Cipher Parameters */const struct aesni_mb_op_fns *op_fns; - /**< Vector mode dependent pointer table of the multi-buffer APIs */ - - struct { - /** Cipher direction - encrypt / decrypt */ - JOB_CIPHER_DIRECTION direction; - /** Cipher mode - CBC / Counter */ - JOB_CIPHER_MODE mode; - - uint64_t key_length_in_bytes; - - union { - struct { - uint32_t encode[60] __rte_aligned(16); - /**< encode key */ - uint32_t decode[60] __rte_aligned(16); - /**< decode key */ - } expanded_aes_keys; - /**< Expanded AES keys - Allocating space to - * contain the maximum expanded key size which - * is 240 bytes for 256 bit AES, calculate by: - * ((key size (bytes)) * - * ((number of rounds) + 1)) - */ - struct { - const void *ks_ptr[3]; - uint64_t key[3][16]; - } exp_3des_keys; - /**< Expanded 3DES keys */ - - struct gcm_key_data gcm_key; - /**< Expanded GCM key */ - uint8_t zuc_cipher_key[16]; - /**< ZUC cipher key */ -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM - snow3g_key_schedule_t pKeySched_snow3g_cipher; - /**< SNOW3G scheduled cipher key */ - kasumi_key_sched_t pKeySched_kasumi_cipher; - /**< KASUMI scheduled cipher key */ -#endif - }; - } cipher; - - /** Authentication Parameters */ - struct { - JOB_HASH_ALG algo; /**< Authentication Algorithm */ - enum rte_crypto_auth_operation operation; - /**< auth operation generate or verify */ - union { - struct { - uint8_t inner[128] __rte_aligned(16); - /**< inner pad */ - uint8_t outer[128] __rte_aligned(16); - /**< outer pad */ - } pads; - /**< HMAC Authentication pads - - * allocating space for the maximum pad - * size supported which is 128 bytes for - * SHA512 - */ - - struct { - uint32_t k1_expanded[44] __rte_aligned(16); - /**< k1 (expanded key). */ - uint8_t k2[16] __rte_aligned(16); - /**< k2. */ - uint8_t k3[16] __rte_aligned(16); - /**< k3. */ - } xcbc; - - struct { - uint32_t expkey[60] __rte_aligned(16); - /**< k1 (expanded key). */ - uint32_t skey1[4] __rte_aligned(16); - /**< k2. */ - uint32_t skey2[4] __rte_aligned(16); - /**< k3. */ - } cmac; - /**< Expanded XCBC authentication keys */ - uint8_t zuc_auth_key[16]; - /**< ZUC authentication key */ -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM - snow3g_key_schedule_t pKeySched_snow3g_auth; - /**< SNOW3G scheduled authentication key */ - kasumi_key_sched_t pKeySched_kasumi_auth; - /**< KASUMI scheduled authentication key */ -#endif - }; - /** Generated digest size by the Multi-buffer library */ - uint16_t gen_digest_len; - /** Requested digest size from Cryptodev */ - uint16_t req_digest_len; - - } auth; - struct { - /** AAD data length */ - uint16_t aad_len; - } aead; -} __rte_cache_aligned; - -extern int -aesni_mb_set_session_parameters(const MB_MGR *mb_mgr, - struct aesni_mb_session *sess, - const struct rte_crypto_sym_xform *xform); - -#ifdef AESNI_MB_DOCSIS_SEC_ENABLED -extern int -aesni_mb_set_docsis_sec_session_parameters( - __rte_unused struct rte_cryptodev *dev, - struct rte_security_session_conf *conf, - void *sess); -#endif - -/** device specific operations function pointer structures */ -extern struct rte_cryptodev_ops *rte_aesni_mb_pmd_ops; -#ifdef AESNI_MB_DOCSIS_SEC_ENABLED -extern struct rte_security_ops *rte_aesni_mb_pmd_sec_ops; -#endif - -extern uint32_t -aesni_mb_cpu_crypto_process_bulk(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs sofs, - struct rte_crypto_sym_vec *vec); - -#endif /* _AESNI_MB_PMD_PRIVATE_H_ */ diff --git a/drivers/crypto/aesni_mb/meson.build b/drivers/crypto/aesni_mb/meson.build deleted file mode 100644 index ed6b9f53e4..0000000000 --- a/drivers/crypto/aesni_mb/meson.build +++ /dev/null @@ -1,25 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018 Intel Corporation - -IMB_required_ver = '0.52.0' -lib = cc.find_library('IPSec_MB', required: false) -if not lib.found() - build = false - reason = 'missing dependency, "libIPSec_MB"' -else - ext_deps += lib - - # version comes with quotes, so we split based on " and take the middle - imb_ver = cc.get_define('IMB_VERSION_STR', - prefix : '#include').split('"')[1] - - if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) - reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( - IMB_required_ver, imb_ver) - build = false - endif - -endif - -sources = files('rte_aesni_mb_pmd.c', 'rte_aesni_mb_pmd_ops.c') -deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/aesni_mb/version.map b/drivers/crypto/aesni_mb/version.map deleted file mode 100644 index c2e0723b4c..0000000000 --- a/drivers/crypto/aesni_mb/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_22 { - local: *; -}; diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.c b/drivers/crypto/ipsec_mb/ipsec_mb_private.c index d142d7622b..d896fa20b4 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.c +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.c @@ -105,6 +105,7 @@ ipsec_mb_create(struct rte_vdev_device *vdev, dev->dev_ops = ipsec_mb_pmds[pmd_type].ops; dev->enqueue_burst = ipsec_mb_enqueue_burst; dev->dequeue_burst = ipsec_mb_pmds[pmd_type].dequeue_burst; + dev->feature_flags = pmd_data->feature_flags; if (pmd_data->dev_config) { retval = (*pmd_data->dev_config)(dev); @@ -116,8 +117,6 @@ ipsec_mb_create(struct rte_vdev_device *vdev, } } - dev->feature_flags = pmd_data->feature_flags; - switch (vector_mode) { case IPSEC_MB_AVX512: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; @@ -165,6 +164,10 @@ ipsec_mb_remove(struct rte_vdev_device *vdev) rte_free(cryptodev->security_ctx); cryptodev->security_ctx = NULL; } +#ifdef AESNI_MB_DOCSIS_SEC_ENABLED + rte_free(cryptodev->security_ctx); + cryptodev->security_ctx = NULL; +#endif for (qp_id = 0; qp_id < cryptodev->data->nb_queue_pairs; qp_id++) ipsec_mb_qp_release(cryptodev, qp_id); diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index 541f759e66..1b7bf459ea 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -34,6 +34,9 @@ extern enum ipsec_mb_vector_mode vector_mode; /** IMB_MGR instances, one per thread */ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); +#define CRYPTODEV_NAME_AESNI_MB_PMD crypto_aesni_mb +/**< IPSEC Multi buffer aesni_mb PMD device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -42,6 +45,7 @@ extern int ipsec_mb_logtype_driver; /** All supported device types */ enum ipsec_mb_pmd_types { + IPSEC_MB_PMD_TYPE_AESNI_MB = 0, IPSEC_MB_N_PMD_TYPES }; @@ -60,10 +64,18 @@ enum ipsec_mb_operation { IPSEC_MB_OP_NOT_SUPPORTED }; +extern uint8_t pmd_driver_id_aesni_mb; + /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t -ipsec_mb_get_driver_id(__rte_unused enum ipsec_mb_pmd_types pmd_type) +ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) { + switch (pmd_type) { + case IPSEC_MB_PMD_TYPE_AESNI_MB: + return pmd_driver_id_aesni_mb; + default: + break; + } return UINT8_MAX; } @@ -136,6 +148,135 @@ get_per_thread_mb_mgr(void) return RTE_PER_LCORE(mb_mgr); } +/** Helper function. Gets mode and chained xforms from the xform */ +static __rte_always_inline int +ipsec_mb_parse_xform(const struct rte_crypto_sym_xform *xform, + enum ipsec_mb_operation *mode, + const struct rte_crypto_sym_xform **auth_xform, + const struct rte_crypto_sym_xform **cipher_xform, + const struct rte_crypto_sym_xform **aead_xform) +{ + const struct rte_crypto_sym_xform *next = xform->next; + + if (xform == NULL) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { + if (next == NULL) { + if (xform->cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) { + *mode = IPSEC_MB_OP_ENCRYPT_ONLY; + *cipher_xform = xform; + *auth_xform = NULL; + return 0; + } + *mode = IPSEC_MB_OP_DECRYPT_ONLY; + *cipher_xform = xform; + *auth_xform = NULL; + return 0; + } + + if (next->type != RTE_CRYPTO_SYM_XFORM_AUTH) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + if (xform->cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) { + if (next->auth.op != RTE_CRYPTO_AUTH_OP_GENERATE) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + *mode = IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN; + *cipher_xform = xform; + *auth_xform = xform->next; + return 0; + } + if (next->auth.op != RTE_CRYPTO_AUTH_OP_VERIFY) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + *mode = IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY; + *cipher_xform = xform; + *auth_xform = xform->next; + return 0; + } + + if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { + if (next == NULL) { + if (xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) { + *mode = IPSEC_MB_OP_HASH_GEN_ONLY; + *auth_xform = xform; + *cipher_xform = NULL; + return 0; + } + *mode = IPSEC_MB_OP_HASH_VERIFY_ONLY; + *auth_xform = xform; + *cipher_xform = NULL; + return 0; + } + + if (next->type != RTE_CRYPTO_SYM_XFORM_CIPHER) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + if (xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) { + if (next->cipher.op != RTE_CRYPTO_CIPHER_OP_ENCRYPT) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + *mode = IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT; + *auth_xform = xform; + *cipher_xform = xform->next; + return 0; + } + if (next->cipher.op != RTE_CRYPTO_CIPHER_OP_DECRYPT) { + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; + } + + *mode = IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT; + *auth_xform = xform; + *cipher_xform = xform->next; + return 0; + } + + if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { + if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) { + /* + * CCM requires to hash first and cipher later + * when encrypting + */ + if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM) { + *mode = IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT; + *aead_xform = xform; + return 0; + } else { + *mode = + IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT; + *aead_xform = xform; + return 0; + } + } else { + if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM) { + *mode = IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT; + *aead_xform = xform; + return 0; + } + *mode = IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT; + *aead_xform = xform; + return 0; + } + } + + *mode = IPSEC_MB_OP_NOT_SUPPORTED; + return -ENOTSUP; +} + /** Device creation function */ int ipsec_mb_create(struct rte_vdev_device *vdev, diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index 76676fe990..14e654ba09 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -23,5 +23,6 @@ endif sources = files('ipsec_mb_private.c', 'ipsec_mb_ops.c', + 'pmd_aesni_mb.c' ) deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c b/drivers/crypto/ipsec_mb/pmd_aesni_mb.c similarity index 60% rename from drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c rename to drivers/crypto/ipsec_mb/pmd_aesni_mb.c index 60963a8208..c9c4906722 100644 --- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c +++ b/drivers/crypto/ipsec_mb/pmd_aesni_mb.c @@ -1,39 +1,14 @@ /* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2015-2017 Intel Corporation + * Copyright(c) 2015-2021 Intel Corporation */ -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "aesni_mb_pmd_private.h" - -#define AES_CCM_DIGEST_MIN_LEN 4 -#define AES_CCM_DIGEST_MAX_LEN 16 -#define HMAC_MAX_BLOCK_SIZE 128 -static uint8_t cryptodev_driver_id; - -/* - * Needed to support CPU-CRYPTO API (rte_cryptodev_sym_cpu_crypto_process), - * as we still use JOB based API even for synchronous processing. - */ -static RTE_DEFINE_PER_LCORE(MB_MGR *, sync_mb_mgr); - -typedef void (*hash_one_block_t)(const void *data, void *digest); -typedef void (*aes_keyexp_t)(const void *key, void *enc_exp_keys, void *dec_exp_keys); +#include "pmd_aesni_mb_priv.h" /** * Calculate the authentication pre-computes * - * @param one_block_hash Function pointer to calculate digest on ipad/opad + * @param one_block_hash Function pointer + * to calculate digest on ipad/opad * @param ipad Inner pad output byte array * @param opad Outer pad output byte array * @param hkey Authentication key @@ -46,7 +21,7 @@ calculate_auth_precomputes(hash_one_block_t one_block_hash, const uint8_t *hkey, uint16_t hkey_len, uint16_t blocksize) { - unsigned i, length; + uint32_t i, length; uint8_t ipad_buf[blocksize] __rte_aligned(16); uint8_t opad_buf[blocksize] __rte_aligned(16); @@ -72,74 +47,18 @@ calculate_auth_precomputes(hash_one_block_t one_block_hash, memset(opad_buf, 0, blocksize); } -/** Get xform chain order */ -static enum aesni_mb_operation -aesni_mb_get_chain_order(const struct rte_crypto_sym_xform *xform) -{ - if (xform == NULL) - return AESNI_MB_OP_NOT_SUPPORTED; - - if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { - if (xform->next == NULL) - return AESNI_MB_OP_CIPHER_ONLY; - if (xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) - return AESNI_MB_OP_CIPHER_HASH; - } - - if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - if (xform->next == NULL) - return AESNI_MB_OP_HASH_ONLY; - if (xform->next->type == RTE_CRYPTO_SYM_XFORM_CIPHER) - return AESNI_MB_OP_HASH_CIPHER; - } -#if IMB_VERSION_NUM > IMB_VERSION(0, 52, 0) - if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { - if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) { - /* - * CCM requires to hash first and cipher later - * when encrypting - */ - if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM) - return AESNI_MB_OP_AEAD_HASH_CIPHER; - else - return AESNI_MB_OP_AEAD_CIPHER_HASH; - } else { - if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM) - return AESNI_MB_OP_AEAD_CIPHER_HASH; - else - return AESNI_MB_OP_AEAD_HASH_CIPHER; - } - } -#else - if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { - if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM || - xform->aead.algo == RTE_CRYPTO_AEAD_AES_GCM) { - if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) - return AESNI_MB_OP_AEAD_CIPHER_HASH; - else - return AESNI_MB_OP_AEAD_HASH_CIPHER; - } - } -#endif - - return AESNI_MB_OP_NOT_SUPPORTED; -} - static inline int -is_aead_algo(JOB_HASH_ALG hash_alg, JOB_CIPHER_MODE cipher_mode) +is_aead_algo(IMB_HASH_ALG hash_alg, IMB_CIPHER_MODE cipher_mode) { -#if IMB_VERSION(0, 54, 3) <= IMB_VERSION_NUM - return (hash_alg == IMB_AUTH_CHACHA20_POLY1305 || hash_alg == AES_CCM || - (hash_alg == AES_GMAC && cipher_mode == GCM)); -#else - return ((hash_alg == AES_GMAC && cipher_mode == GCM) || - hash_alg == AES_CCM); -#endif + return (hash_alg == IMB_AUTH_CHACHA20_POLY1305 || + hash_alg == IMB_AUTH_AES_CCM || + (hash_alg == IMB_AUTH_AES_GMAC && + cipher_mode == IMB_CIPHER_GCM)); } /** Set session authentication parameters */ static int -aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, +aesni_mb_set_session_auth_parameters(const IMB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { @@ -149,12 +68,12 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, uint32_t auth_precompute = 1; if (xform == NULL) { - sess->auth.algo = NULL_HASH; + sess->auth.algo = IMB_AUTH_NULL; return 0; } if (xform->type != RTE_CRYPTO_SYM_XFORM_AUTH) { - AESNI_MB_LOG(ERR, "Crypto xform struct not of type auth"); + IPSEC_MB_LOG(ERR, "Crypto xform struct not of type auth"); return -1; } @@ -170,12 +89,12 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, /* Set Authentication Parameters */ if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_XCBC_MAC) { - sess->auth.algo = AES_XCBC; + sess->auth.algo = IMB_AUTH_AES_XCBC; uint16_t xcbc_mac_digest_len = - get_truncated_digest_byte_length(AES_XCBC); + get_truncated_digest_byte_length(IMB_AUTH_AES_XCBC); if (sess->auth.req_digest_len != xcbc_mac_digest_len) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } sess->auth.gen_digest_len = sess->auth.req_digest_len; @@ -189,12 +108,13 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_CMAC) { uint32_t dust[4*15]; - sess->auth.algo = AES_CMAC; + sess->auth.algo = IMB_AUTH_AES_CMAC; - uint16_t cmac_digest_len = get_digest_byte_length(AES_CMAC); + uint16_t cmac_digest_len = + get_digest_byte_length(IMB_AUTH_AES_CMAC); if (sess->auth.req_digest_len > cmac_digest_len) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } /* @@ -219,14 +139,15 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { if (xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) { - sess->cipher.direction = ENCRYPT; - sess->chain_order = CIPHER_HASH; + sess->cipher.direction = IMB_DIR_ENCRYPT; + sess->chain_order = IMB_ORDER_CIPHER_HASH; } else - sess->cipher.direction = DECRYPT; + sess->cipher.direction = IMB_DIR_DECRYPT; - sess->auth.algo = AES_GMAC; - if (sess->auth.req_digest_len > get_digest_byte_length(AES_GMAC)) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + sess->auth.algo = IMB_AUTH_AES_GMAC; + if (sess->auth.req_digest_len > + get_digest_byte_length(IMB_AUTH_AES_GMAC)) { + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } sess->auth.gen_digest_len = sess->auth.req_digest_len; @@ -234,36 +155,36 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, sess->iv.offset = xform->auth.iv.offset; switch (xform->auth.key.length) { - case AES_128_BYTES: + case IMB_KEY_128_BYTES: IMB_AES128_GCM_PRE(mb_mgr, xform->auth.key.data, &sess->cipher.gcm_key); - sess->cipher.key_length_in_bytes = AES_128_BYTES; + sess->cipher.key_length_in_bytes = IMB_KEY_128_BYTES; break; - case AES_192_BYTES: + case IMB_KEY_192_BYTES: IMB_AES192_GCM_PRE(mb_mgr, xform->auth.key.data, &sess->cipher.gcm_key); - sess->cipher.key_length_in_bytes = AES_192_BYTES; + sess->cipher.key_length_in_bytes = IMB_KEY_192_BYTES; break; - case AES_256_BYTES: + case IMB_KEY_256_BYTES: IMB_AES256_GCM_PRE(mb_mgr, xform->auth.key.data, &sess->cipher.gcm_key); - sess->cipher.key_length_in_bytes = AES_256_BYTES; + sess->cipher.key_length_in_bytes = IMB_KEY_256_BYTES; break; default: - RTE_LOG(ERR, PMD, "failed to parse test type\n"); + IPSEC_MB_LOG(ERR, "Invalid authentication key length\n"); return -EINVAL; } return 0; } -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM if (xform->auth.algo == RTE_CRYPTO_AUTH_ZUC_EIA3) { sess->auth.algo = IMB_AUTH_ZUC_EIA3_BITLEN; uint16_t zuc_eia3_digest_len = - get_truncated_digest_byte_length(IMB_AUTH_ZUC_EIA3_BITLEN); + get_truncated_digest_byte_length( + IMB_AUTH_ZUC_EIA3_BITLEN); if (sess->auth.req_digest_len != zuc_eia3_digest_len) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } sess->auth.gen_digest_len = sess->auth.req_digest_len; @@ -273,9 +194,10 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, } else if (xform->auth.algo == RTE_CRYPTO_AUTH_SNOW3G_UIA2) { sess->auth.algo = IMB_AUTH_SNOW3G_UIA2_BITLEN; uint16_t snow3g_uia2_digest_len = - get_truncated_digest_byte_length(IMB_AUTH_SNOW3G_UIA2_BITLEN); + get_truncated_digest_byte_length( + IMB_AUTH_SNOW3G_UIA2_BITLEN); if (sess->auth.req_digest_len != snow3g_uia2_digest_len) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } sess->auth.gen_digest_len = sess->auth.req_digest_len; @@ -288,7 +210,7 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, uint16_t kasumi_f9_digest_len = get_truncated_digest_byte_length(IMB_AUTH_KASUMI_UIA1); if (sess->auth.req_digest_len != kasumi_f9_digest_len) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } sess->auth.gen_digest_len = sess->auth.req_digest_len; @@ -297,17 +219,17 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, &sess->auth.pKeySched_kasumi_auth); return 0; } -#endif switch (xform->auth.algo) { case RTE_CRYPTO_AUTH_MD5_HMAC: - sess->auth.algo = MD5; + sess->auth.algo = IMB_AUTH_MD5; hash_oneblock_fn = mb_mgr->md5_one_block; break; case RTE_CRYPTO_AUTH_SHA1_HMAC: - sess->auth.algo = SHA1; + sess->auth.algo = IMB_AUTH_HMAC_SHA_1; hash_oneblock_fn = mb_mgr->sha1_one_block; - if (xform->auth.key.length > get_auth_algo_blocksize(SHA1)) { + if (xform->auth.key.length > get_auth_algo_blocksize( + IMB_AUTH_HMAC_SHA_1)) { IMB_SHA1(mb_mgr, xform->auth.key.data, xform->auth.key.length, @@ -316,13 +238,14 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, } break; case RTE_CRYPTO_AUTH_SHA1: - sess->auth.algo = PLAIN_SHA1; + sess->auth.algo = IMB_AUTH_SHA_1; auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA224_HMAC: - sess->auth.algo = SHA_224; + sess->auth.algo = IMB_AUTH_HMAC_SHA_224; hash_oneblock_fn = mb_mgr->sha224_one_block; - if (xform->auth.key.length > get_auth_algo_blocksize(SHA_224)) { + if (xform->auth.key.length > get_auth_algo_blocksize( + IMB_AUTH_HMAC_SHA_224)) { IMB_SHA224(mb_mgr, xform->auth.key.data, xform->auth.key.length, @@ -331,13 +254,14 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, } break; case RTE_CRYPTO_AUTH_SHA224: - sess->auth.algo = PLAIN_SHA_224; + sess->auth.algo = IMB_AUTH_SHA_224; auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA256_HMAC: - sess->auth.algo = SHA_256; + sess->auth.algo = IMB_AUTH_HMAC_SHA_256; hash_oneblock_fn = mb_mgr->sha256_one_block; - if (xform->auth.key.length > get_auth_algo_blocksize(SHA_256)) { + if (xform->auth.key.length > get_auth_algo_blocksize( + IMB_AUTH_HMAC_SHA_256)) { IMB_SHA256(mb_mgr, xform->auth.key.data, xform->auth.key.length, @@ -346,13 +270,14 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, } break; case RTE_CRYPTO_AUTH_SHA256: - sess->auth.algo = PLAIN_SHA_256; + sess->auth.algo = IMB_AUTH_SHA_256; auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA384_HMAC: - sess->auth.algo = SHA_384; + sess->auth.algo = IMB_AUTH_HMAC_SHA_384; hash_oneblock_fn = mb_mgr->sha384_one_block; - if (xform->auth.key.length > get_auth_algo_blocksize(SHA_384)) { + if (xform->auth.key.length > get_auth_algo_blocksize( + IMB_AUTH_HMAC_SHA_384)) { IMB_SHA384(mb_mgr, xform->auth.key.data, xform->auth.key.length, @@ -361,13 +286,14 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, } break; case RTE_CRYPTO_AUTH_SHA384: - sess->auth.algo = PLAIN_SHA_384; + sess->auth.algo = IMB_AUTH_SHA_384; auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA512_HMAC: - sess->auth.algo = SHA_512; + sess->auth.algo = IMB_AUTH_HMAC_SHA_512; hash_oneblock_fn = mb_mgr->sha512_one_block; - if (xform->auth.key.length > get_auth_algo_blocksize(SHA_512)) { + if (xform->auth.key.length > get_auth_algo_blocksize( + IMB_AUTH_HMAC_SHA_512)) { IMB_SHA512(mb_mgr, xform->auth.key.data, xform->auth.key.length, @@ -376,11 +302,12 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, } break; case RTE_CRYPTO_AUTH_SHA512: - sess->auth.algo = PLAIN_SHA_512; + sess->auth.algo = IMB_AUTH_SHA_512; auth_precompute = 0; break; default: - AESNI_MB_LOG(ERR, "Unsupported authentication algorithm selection"); + IPSEC_MB_LOG(ERR, + "Unsupported authentication algorithm selection"); return -ENOTSUP; } uint16_t trunc_digest_size = @@ -390,7 +317,7 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, if (sess->auth.req_digest_len > full_digest_size || sess->auth.req_digest_len == 0) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } @@ -424,73 +351,68 @@ aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, /** Set session cipher parameters */ static int -aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, +aesni_mb_set_session_cipher_parameters(const IMB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { uint8_t is_aes = 0; uint8_t is_3DES = 0; uint8_t is_docsis = 0; -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM uint8_t is_zuc = 0; uint8_t is_snow3g = 0; uint8_t is_kasumi = 0; -#endif if (xform == NULL) { - sess->cipher.mode = NULL_CIPHER; + sess->cipher.mode = IMB_CIPHER_NULL; return 0; } if (xform->type != RTE_CRYPTO_SYM_XFORM_CIPHER) { - AESNI_MB_LOG(ERR, "Crypto xform struct not of type cipher"); + IPSEC_MB_LOG(ERR, "Crypto xform struct not of type cipher"); return -EINVAL; } /* Select cipher direction */ switch (xform->cipher.op) { case RTE_CRYPTO_CIPHER_OP_ENCRYPT: - sess->cipher.direction = ENCRYPT; + sess->cipher.direction = IMB_DIR_ENCRYPT; break; case RTE_CRYPTO_CIPHER_OP_DECRYPT: - sess->cipher.direction = DECRYPT; + sess->cipher.direction = IMB_DIR_DECRYPT; break; default: - AESNI_MB_LOG(ERR, "Invalid cipher operation parameter"); + IPSEC_MB_LOG(ERR, "Invalid cipher operation parameter"); return -EINVAL; } /* Select cipher mode */ switch (xform->cipher.algo) { case RTE_CRYPTO_CIPHER_AES_CBC: - sess->cipher.mode = CBC; + sess->cipher.mode = IMB_CIPHER_CBC; is_aes = 1; break; case RTE_CRYPTO_CIPHER_AES_CTR: - sess->cipher.mode = CNTR; + sess->cipher.mode = IMB_CIPHER_CNTR; is_aes = 1; break; case RTE_CRYPTO_CIPHER_AES_DOCSISBPI: - sess->cipher.mode = DOCSIS_SEC_BPI; + sess->cipher.mode = IMB_CIPHER_DOCSIS_SEC_BPI; is_docsis = 1; break; case RTE_CRYPTO_CIPHER_DES_CBC: - sess->cipher.mode = DES; + sess->cipher.mode = IMB_CIPHER_DES; break; case RTE_CRYPTO_CIPHER_DES_DOCSISBPI: - sess->cipher.mode = DOCSIS_DES; + sess->cipher.mode = IMB_CIPHER_DOCSIS_DES; break; case RTE_CRYPTO_CIPHER_3DES_CBC: - sess->cipher.mode = DES3; + sess->cipher.mode = IMB_CIPHER_DES3; is_3DES = 1; break; -#if IMB_VERSION(0, 53, 0) <= IMB_VERSION_NUM case RTE_CRYPTO_CIPHER_AES_ECB: - sess->cipher.mode = ECB; + sess->cipher.mode = IMB_CIPHER_ECB; is_aes = 1; break; -#endif -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM case RTE_CRYPTO_CIPHER_ZUC_EEA3: sess->cipher.mode = IMB_CIPHER_ZUC_EEA3; is_zuc = 1; @@ -503,9 +425,8 @@ aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, sess->cipher.mode = IMB_CIPHER_KASUMI_UEA1_BITLEN; is_kasumi = 1; break; -#endif default: - AESNI_MB_LOG(ERR, "Unsupported cipher mode parameter"); + IPSEC_MB_LOG(ERR, "Unsupported cipher mode parameter"); return -ENOTSUP; } @@ -516,46 +437,44 @@ aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, /* Check key length and choose key expansion function for AES */ if (is_aes) { switch (xform->cipher.key.length) { - case AES_128_BYTES: - sess->cipher.key_length_in_bytes = AES_128_BYTES; + case IMB_KEY_128_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_128_BYTES; IMB_AES_KEYEXP_128(mb_mgr, xform->cipher.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; - case AES_192_BYTES: - sess->cipher.key_length_in_bytes = AES_192_BYTES; + case IMB_KEY_192_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_192_BYTES; IMB_AES_KEYEXP_192(mb_mgr, xform->cipher.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; - case AES_256_BYTES: - sess->cipher.key_length_in_bytes = AES_256_BYTES; + case IMB_KEY_256_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_256_BYTES; IMB_AES_KEYEXP_256(mb_mgr, xform->cipher.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; default: - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } } else if (is_docsis) { switch (xform->cipher.key.length) { - case AES_128_BYTES: - sess->cipher.key_length_in_bytes = AES_128_BYTES; + case IMB_KEY_128_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_128_BYTES; IMB_AES_KEYEXP_128(mb_mgr, xform->cipher.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM - case AES_256_BYTES: - sess->cipher.key_length_in_bytes = AES_256_BYTES; + case IMB_KEY_256_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_256_BYTES; IMB_AES_KEYEXP_256(mb_mgr, xform->cipher.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; -#endif default: - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } } else if (is_3DES) { @@ -597,15 +516,14 @@ aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, sess->cipher.exp_3des_keys.ks_ptr[2] = keys[0]; break; default: - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } sess->cipher.key_length_in_bytes = 24; -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM } else if (is_zuc) { if (xform->cipher.key.length != 16) { - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } sess->cipher.key_length_in_bytes = 16; @@ -613,7 +531,7 @@ aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, 16); } else if (is_snow3g) { if (xform->cipher.key.length != 16) { - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } sess->cipher.key_length_in_bytes = 16; @@ -621,16 +539,15 @@ aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, &sess->cipher.pKeySched_snow3g_cipher); } else if (is_kasumi) { if (xform->cipher.key.length != 16) { - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } sess->cipher.key_length_in_bytes = 16; IMB_KASUMI_INIT_F8_KEY_SCHED(mb_mgr, xform->cipher.key.data, &sess->cipher.pKeySched_kasumi_cipher); -#endif } else { if (xform->cipher.key.length != 8) { - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } sess->cipher.key_length_in_bytes = 8; @@ -647,21 +564,21 @@ aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, } static int -aesni_mb_set_session_aead_parameters(const MB_MGR *mb_mgr, +aesni_mb_set_session_aead_parameters(const IMB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { switch (xform->aead.op) { case RTE_CRYPTO_AEAD_OP_ENCRYPT: - sess->cipher.direction = ENCRYPT; + sess->cipher.direction = IMB_DIR_ENCRYPT; sess->auth.operation = RTE_CRYPTO_AUTH_OP_GENERATE; break; case RTE_CRYPTO_AEAD_OP_DECRYPT: - sess->cipher.direction = DECRYPT; + sess->cipher.direction = IMB_DIR_DECRYPT; sess->auth.operation = RTE_CRYPTO_AUTH_OP_VERIFY; break; default: - AESNI_MB_LOG(ERR, "Invalid aead operation parameter"); + IPSEC_MB_LOG(ERR, "Invalid aead operation parameter"); return -EINVAL; } @@ -675,151 +592,151 @@ aesni_mb_set_session_aead_parameters(const MB_MGR *mb_mgr, switch (xform->aead.algo) { case RTE_CRYPTO_AEAD_AES_CCM: - sess->cipher.mode = CCM; - sess->auth.algo = AES_CCM; + sess->cipher.mode = IMB_CIPHER_CCM; + sess->auth.algo = IMB_AUTH_AES_CCM; /* Check key length and choose key expansion function for AES */ switch (xform->aead.key.length) { - case AES_128_BYTES: - sess->cipher.key_length_in_bytes = AES_128_BYTES; + case IMB_KEY_128_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_128_BYTES; IMB_AES_KEYEXP_128(mb_mgr, xform->aead.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; - case AES_256_BYTES: - sess->cipher.key_length_in_bytes = AES_256_BYTES; + case IMB_KEY_256_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_256_BYTES; IMB_AES_KEYEXP_256(mb_mgr, xform->aead.key.data, sess->cipher.expanded_aes_keys.encode, sess->cipher.expanded_aes_keys.decode); break; default: - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } /* CCM digests must be between 4 and 16 and an even number */ if (sess->auth.req_digest_len < AES_CCM_DIGEST_MIN_LEN || - sess->auth.req_digest_len > AES_CCM_DIGEST_MAX_LEN || - (sess->auth.req_digest_len & 1) == 1) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + sess->auth.req_digest_len > AES_CCM_DIGEST_MAX_LEN || + (sess->auth.req_digest_len & 1) == 1) { + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } break; case RTE_CRYPTO_AEAD_AES_GCM: - sess->cipher.mode = GCM; - sess->auth.algo = AES_GMAC; + sess->cipher.mode = IMB_CIPHER_GCM; + sess->auth.algo = IMB_AUTH_AES_GMAC; switch (xform->aead.key.length) { - case AES_128_BYTES: - sess->cipher.key_length_in_bytes = AES_128_BYTES; + case IMB_KEY_128_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_128_BYTES; IMB_AES128_GCM_PRE(mb_mgr, xform->aead.key.data, &sess->cipher.gcm_key); break; - case AES_192_BYTES: - sess->cipher.key_length_in_bytes = AES_192_BYTES; + case IMB_KEY_192_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_192_BYTES; IMB_AES192_GCM_PRE(mb_mgr, xform->aead.key.data, &sess->cipher.gcm_key); break; - case AES_256_BYTES: - sess->cipher.key_length_in_bytes = AES_256_BYTES; + case IMB_KEY_256_BYTES: + sess->cipher.key_length_in_bytes = IMB_KEY_256_BYTES; IMB_AES256_GCM_PRE(mb_mgr, xform->aead.key.data, &sess->cipher.gcm_key); break; default: - AESNI_MB_LOG(ERR, "Invalid cipher key length"); + IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } /* GCM digest size must be between 1 and 16 */ if (sess->auth.req_digest_len == 0 || sess->auth.req_digest_len > 16) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } break; -#if IMB_VERSION(0, 54, 3) <= IMB_VERSION_NUM case RTE_CRYPTO_AEAD_CHACHA20_POLY1305: sess->cipher.mode = IMB_CIPHER_CHACHA20_POLY1305; sess->auth.algo = IMB_AUTH_CHACHA20_POLY1305; if (xform->aead.key.length != 32) { - AESNI_MB_LOG(ERR, "Invalid key length"); + IPSEC_MB_LOG(ERR, "Invalid key length"); return -EINVAL; } sess->cipher.key_length_in_bytes = 32; memcpy(sess->cipher.expanded_aes_keys.encode, xform->aead.key.data, 32); if (sess->auth.req_digest_len != 16) { - AESNI_MB_LOG(ERR, "Invalid digest size\n"); + IPSEC_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } break; -#endif default: - AESNI_MB_LOG(ERR, "Unsupported aead mode parameter"); + IPSEC_MB_LOG(ERR, "Unsupported aead mode parameter"); return -ENOTSUP; } return 0; } -/** Parse crypto xform chain and set private session parameters */ -int -aesni_mb_set_session_parameters(const MB_MGR *mb_mgr, - struct aesni_mb_session *sess, +/** Configure a aesni multi-buffer session from a crypto xform chain */ +static int +aesni_mb_session_configure(IMB_MGR *mb_mgr, + void *priv_sess, const struct rte_crypto_sym_xform *xform) { const struct rte_crypto_sym_xform *auth_xform = NULL; const struct rte_crypto_sym_xform *cipher_xform = NULL; const struct rte_crypto_sym_xform *aead_xform = NULL; + enum ipsec_mb_operation mode; + struct aesni_mb_session *sess = (struct aesni_mb_session *) priv_sess; int ret; + ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, &aead_xform); + if (ret) + return ret; + /* Select Crypto operation - hash then cipher / cipher then hash */ - switch (aesni_mb_get_chain_order(xform)) { - case AESNI_MB_OP_HASH_CIPHER: - sess->chain_order = HASH_CIPHER; - auth_xform = xform; - cipher_xform = xform->next; + switch (mode) { + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + sess->chain_order = IMB_ORDER_HASH_CIPHER; break; - case AESNI_MB_OP_CIPHER_HASH: - sess->chain_order = CIPHER_HASH; - auth_xform = xform->next; - cipher_xform = xform; + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + case IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY: + sess->chain_order = IMB_ORDER_CIPHER_HASH; break; - case AESNI_MB_OP_HASH_ONLY: - sess->chain_order = HASH_CIPHER; - auth_xform = xform; - cipher_xform = NULL; + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + case IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT: + sess->chain_order = IMB_ORDER_HASH_CIPHER; break; - case AESNI_MB_OP_CIPHER_ONLY: - /* - * Multi buffer library operates only at two modes, - * CIPHER_HASH and HASH_CIPHER. When doing ciphering only, - * chain order depends on cipher operation: encryption is always - * the first operation and decryption the last one. - */ - if (xform->cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) - sess->chain_order = CIPHER_HASH; - else - sess->chain_order = HASH_CIPHER; - auth_xform = NULL; - cipher_xform = xform; + /* + * Multi buffer library operates only at two modes, + * IMB_ORDER_CIPHER_HASH and IMB_ORDER_HASH_CIPHER. + * When doing ciphering only, chain order depends + * on cipher operation: encryption is always + * the first operation and decryption the last one. + */ + case IPSEC_MB_OP_ENCRYPT_ONLY: + sess->chain_order = IMB_ORDER_CIPHER_HASH; break; - case AESNI_MB_OP_AEAD_CIPHER_HASH: - sess->chain_order = CIPHER_HASH; + case IPSEC_MB_OP_DECRYPT_ONLY: + sess->chain_order = IMB_ORDER_HASH_CIPHER; + break; + case IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT: + sess->chain_order = IMB_ORDER_CIPHER_HASH; sess->aead.aad_len = xform->aead.aad_length; - aead_xform = xform; break; - case AESNI_MB_OP_AEAD_HASH_CIPHER: - sess->chain_order = HASH_CIPHER; + case IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT: + sess->chain_order = IMB_ORDER_HASH_CIPHER; sess->aead.aad_len = xform->aead.aad_length; - aead_xform = xform; break; - case AESNI_MB_OP_NOT_SUPPORTED: + case IPSEC_MB_OP_NOT_SUPPORTED: default: - AESNI_MB_LOG(ERR, "Unsupported operation chain order parameter"); + IPSEC_MB_LOG(ERR, + "Unsupported operation chain order parameter"); return -ENOTSUP; } @@ -829,14 +746,15 @@ aesni_mb_set_session_parameters(const MB_MGR *mb_mgr, ret = aesni_mb_set_session_auth_parameters(mb_mgr, sess, auth_xform); if (ret != 0) { - AESNI_MB_LOG(ERR, "Invalid/unsupported authentication parameters"); + IPSEC_MB_LOG(ERR, + "Invalid/unsupported authentication parameters"); return ret; } ret = aesni_mb_set_session_cipher_parameters(mb_mgr, sess, cipher_xform); if (ret != 0) { - AESNI_MB_LOG(ERR, "Invalid/unsupported cipher parameters"); + IPSEC_MB_LOG(ERR, "Invalid/unsupported cipher parameters"); return ret; } @@ -844,7 +762,8 @@ aesni_mb_set_session_parameters(const MB_MGR *mb_mgr, ret = aesni_mb_set_session_aead_parameters(mb_mgr, sess, aead_xform); if (ret != 0) { - AESNI_MB_LOG(ERR, "Invalid/unsupported aead parameters"); + IPSEC_MB_LOG(ERR, + "Invalid/unsupported aead parameters"); return ret; } } @@ -868,9 +787,9 @@ check_docsis_sec_session(struct rte_security_session_conf *conf) crypto_sym->cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT && crypto_sym->cipher.algo == RTE_CRYPTO_CIPHER_AES_DOCSISBPI && - (crypto_sym->cipher.key.length == IMB_KEY_AES_128_BYTES || - crypto_sym->cipher.key.length == IMB_KEY_AES_256_BYTES) && - crypto_sym->cipher.iv.length == AES_BLOCK_SIZE && + (crypto_sym->cipher.key.length == IMB_KEY_128_BYTES || + crypto_sym->cipher.key.length == IMB_KEY_256_BYTES) && + crypto_sym->cipher.iv.length == IMB_AES_BLOCK_SIZE && crypto_sym->next == NULL) { return 0; } @@ -882,9 +801,9 @@ check_docsis_sec_session(struct rte_security_session_conf *conf) crypto_sym->cipher.op == RTE_CRYPTO_CIPHER_OP_DECRYPT && crypto_sym->cipher.algo == RTE_CRYPTO_CIPHER_AES_DOCSISBPI && - (crypto_sym->cipher.key.length == IMB_KEY_AES_128_BYTES || - crypto_sym->cipher.key.length == IMB_KEY_AES_256_BYTES) && - crypto_sym->cipher.iv.length == AES_BLOCK_SIZE && + (crypto_sym->cipher.key.length == IMB_KEY_128_BYTES || + crypto_sym->cipher.key.length == IMB_KEY_256_BYTES) && + crypto_sym->cipher.iv.length == IMB_AES_BLOCK_SIZE && crypto_sym->next == NULL) { return 0; } @@ -899,7 +818,7 @@ aesni_mb_set_docsis_sec_session_auth_parameters(struct aesni_mb_session *sess, struct rte_security_docsis_xform *xform) { if (xform == NULL) { - AESNI_MB_LOG(ERR, "Invalid DOCSIS xform"); + IPSEC_MB_LOG(ERR, "Invalid DOCSIS xform"); return -EINVAL; } @@ -911,7 +830,7 @@ aesni_mb_set_docsis_sec_session_auth_parameters(struct aesni_mb_session *sess, sess->auth.algo = IMB_AUTH_DOCSIS_CRC32; sess->auth.operation = RTE_CRYPTO_AUTH_OP_GENERATE; } else { - AESNI_MB_LOG(ERR, "Unsupported DOCSIS direction"); + IPSEC_MB_LOG(ERR, "Unsupported DOCSIS direction"); return -ENOTSUP; } @@ -925,135 +844,67 @@ aesni_mb_set_docsis_sec_session_auth_parameters(struct aesni_mb_session *sess, * Parse DOCSIS security session configuration and set private session * parameters */ -int +static int aesni_mb_set_docsis_sec_session_parameters( __rte_unused struct rte_cryptodev *dev, struct rte_security_session_conf *conf, void *sess) { + IMB_MGR *mb_mgr = alloc_init_mb_mgr(); struct rte_security_docsis_xform *docsis_xform; struct rte_crypto_sym_xform *cipher_xform; - struct aesni_mb_session *aesni_sess = sess; - struct aesni_mb_private *internals = dev->data->dev_private; - int ret; + struct aesni_mb_session *ipsec_sess = sess; + int ret = 0; + + if (!mb_mgr) + return -ENOMEM; ret = check_docsis_sec_session(conf); if (ret) { - AESNI_MB_LOG(ERR, "Unsupported DOCSIS security configuration"); - return ret; + IPSEC_MB_LOG(ERR, "Unsupported DOCSIS security configuration"); + goto error_exit; } switch (conf->docsis.direction) { case RTE_SECURITY_DOCSIS_UPLINK: - aesni_sess->chain_order = IMB_ORDER_CIPHER_HASH; + ipsec_sess->chain_order = IMB_ORDER_CIPHER_HASH; docsis_xform = &conf->docsis; cipher_xform = conf->crypto_xform; break; case RTE_SECURITY_DOCSIS_DOWNLINK: - aesni_sess->chain_order = IMB_ORDER_HASH_CIPHER; + ipsec_sess->chain_order = IMB_ORDER_HASH_CIPHER; cipher_xform = conf->crypto_xform; docsis_xform = &conf->docsis; break; default: - return -EINVAL; + IPSEC_MB_LOG(ERR, "Unsupported DOCSIS security configuration"); + ret = -EINVAL; + goto error_exit; } /* Default IV length = 0 */ - aesni_sess->iv.length = 0; + ipsec_sess->iv.length = 0; - ret = aesni_mb_set_docsis_sec_session_auth_parameters(aesni_sess, + ret = aesni_mb_set_docsis_sec_session_auth_parameters(ipsec_sess, docsis_xform); if (ret != 0) { - AESNI_MB_LOG(ERR, "Invalid/unsupported DOCSIS parameters"); - return -EINVAL; + IPSEC_MB_LOG(ERR, "Invalid/unsupported DOCSIS parameters"); + goto error_exit; } - ret = aesni_mb_set_session_cipher_parameters(internals->mb_mgr, - aesni_sess, cipher_xform); + ret = aesni_mb_set_session_cipher_parameters(mb_mgr, + ipsec_sess, cipher_xform); if (ret != 0) { - AESNI_MB_LOG(ERR, "Invalid/unsupported cipher parameters"); - return -EINVAL; + IPSEC_MB_LOG(ERR, "Invalid/unsupported cipher parameters"); + goto error_exit; } - return 0; -} -#endif - -/** - * burst enqueue, place crypto operations on ingress queue for processing. - * - * @param __qp Queue Pair to process - * @param ops Crypto operations for processing - * @param nb_ops Number of crypto operations for processing - * - * @return - * - Number of crypto operations enqueued - */ -static uint16_t -aesni_mb_pmd_enqueue_burst(void *__qp, struct rte_crypto_op **ops, - uint16_t nb_ops) -{ - struct aesni_mb_qp *qp = __qp; - - unsigned int nb_enqueued; - - nb_enqueued = rte_ring_enqueue_burst(qp->ingress_queue, - (void **)ops, nb_ops, NULL); - - qp->stats.enqueued_count += nb_enqueued; - - return nb_enqueued; +error_exit: + free_mb_mgr(mb_mgr); + return ret; } - -/** Get multi buffer session */ -static inline struct aesni_mb_session * -get_session(struct aesni_mb_qp *qp, struct rte_crypto_op *op) -{ - struct aesni_mb_session *sess = NULL; - - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { - if (likely(op->sym->session != NULL)) - sess = (struct aesni_mb_session *) - get_sym_session_private_data( - op->sym->session, - cryptodev_driver_id); -#ifdef AESNI_MB_DOCSIS_SEC_ENABLED - } else if (op->sess_type == RTE_CRYPTO_OP_SECURITY_SESSION) { - if (likely(op->sym->sec_session != NULL)) - sess = (struct aesni_mb_session *) - get_sec_session_private_data( - op->sym->sec_session); #endif - } else { - void *_sess = rte_cryptodev_sym_session_create(qp->sess_mp); - void *_sess_private_data = NULL; - - if (_sess == NULL) - return NULL; - - if (rte_mempool_get(qp->sess_mp_priv, - (void **)&_sess_private_data)) - return NULL; - - sess = (struct aesni_mb_session *)_sess_private_data; - - if (unlikely(aesni_mb_set_session_parameters(qp->mb_mgr, - sess, op->sym->xform) != 0)) { - rte_mempool_put(qp->sess_mp, _sess); - rte_mempool_put(qp->sess_mp_priv, _sess_private_data); - sess = NULL; - } - op->sym->session = (struct rte_cryptodev_sym_session *)_sess; - set_sym_session_private_data(op->sym->session, - cryptodev_driver_id, _sess_private_data); - } - - if (unlikely(sess == NULL)) - op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - - return sess; -} static inline uint64_t auth_start_offset(struct rte_crypto_op *op, struct aesni_mb_session *session, @@ -1065,7 +916,7 @@ auth_start_offset(struct rte_crypto_op *op, struct aesni_mb_session *session, uint32_t cipher_end, auth_end; /* Only cipher then hash needs special calculation. */ - if (!oop || session->chain_order != CIPHER_HASH) + if (!oop || session->chain_order != IMB_ORDER_CIPHER_HASH) return op->sym->auth.data.offset; m_src = op->sym->m_src; @@ -1106,7 +957,7 @@ auth_start_offset(struct rte_crypto_op *op, struct aesni_mb_session *session, } static inline void -set_cpu_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_session *session, +set_cpu_mb_job_params(IMB_JOB *job, struct aesni_mb_session *session, union rte_crypto_sym_ofs sofs, void *buf, uint32_t len, struct rte_crypto_va_iova_ptr *iv, struct rte_crypto_va_iova_ptr *aad, void *digest, void *udata) @@ -1118,82 +969,71 @@ set_cpu_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_session *session, job->cipher_direction = session->cipher.direction; job->cipher_mode = session->cipher.mode; - job->aes_key_len_in_bytes = session->cipher.key_length_in_bytes; + job->key_len_in_bytes = session->cipher.key_length_in_bytes; /* Set authentication parameters */ job->hash_alg = session->auth.algo; job->iv = iv->va; switch (job->hash_alg) { - case AES_XCBC: + case IMB_AUTH_AES_XCBC: job->u.XCBC._k1_expanded = session->auth.xcbc.k1_expanded; job->u.XCBC._k2 = session->auth.xcbc.k2; job->u.XCBC._k3 = session->auth.xcbc.k3; - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; break; - case AES_CCM: + case IMB_AUTH_AES_CCM: job->u.CCM.aad = (uint8_t *)aad->va + 18; job->u.CCM.aad_len_in_bytes = session->aead.aad_len; - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; job->iv++; break; - case AES_CMAC: + case IMB_AUTH_AES_CMAC: job->u.CMAC._key_expanded = session->auth.cmac.expkey; job->u.CMAC._skey1 = session->auth.cmac.skey1; job->u.CMAC._skey2 = session->auth.cmac.skey2; - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; break; - case AES_GMAC: - if (session->cipher.mode == GCM) { + case IMB_AUTH_AES_GMAC: + if (session->cipher.mode == IMB_CIPHER_GCM) { job->u.GCM.aad = aad->va; job->u.GCM.aad_len_in_bytes = session->aead.aad_len; } else { /* For GMAC */ job->u.GCM.aad = buf; job->u.GCM.aad_len_in_bytes = len; - job->cipher_mode = GCM; + job->cipher_mode = IMB_CIPHER_GCM; } - job->aes_enc_key_expanded = &session->cipher.gcm_key; - job->aes_dec_key_expanded = &session->cipher.gcm_key; + job->enc_keys = &session->cipher.gcm_key; + job->dec_keys = &session->cipher.gcm_key; break; -#if IMB_VERSION(0, 54, 3) <= IMB_VERSION_NUM case IMB_AUTH_CHACHA20_POLY1305: job->u.CHACHA20_POLY1305.aad = aad->va; - job->u.CHACHA20_POLY1305.aad_len_in_bytes = session->aead.aad_len; - job->aes_enc_key_expanded = session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = session->cipher.expanded_aes_keys.encode; + job->u.CHACHA20_POLY1305.aad_len_in_bytes = + session->aead.aad_len; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.encode; break; -#endif default: job->u.HMAC._hashed_auth_key_xor_ipad = session->auth.pads.inner; job->u.HMAC._hashed_auth_key_xor_opad = session->auth.pads.outer; - if (job->cipher_mode == DES3) { - job->aes_enc_key_expanded = - session->cipher.exp_3des_keys.ks_ptr; - job->aes_dec_key_expanded = - session->cipher.exp_3des_keys.ks_ptr; + if (job->cipher_mode == IMB_CIPHER_DES3) { + job->enc_keys = session->cipher.exp_3des_keys.ks_ptr; + job->dec_keys = session->cipher.exp_3des_keys.ks_ptr; } else { - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; } } @@ -1214,7 +1054,8 @@ set_cpu_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_session *session, job->dst = (uint8_t *)buf + sofs.ofs.cipher.head; job->cipher_start_src_offset_in_bytes = sofs.ofs.cipher.head; job->hash_start_src_offset_in_bytes = sofs.ofs.auth.head; - if (job->hash_alg == AES_GMAC && session->cipher.mode != GCM) { + if (job->hash_alg == IMB_AUTH_AES_GMAC && + session->cipher.mode != IMB_CIPHER_GCM) { job->msg_len_to_hash_in_bytes = 0; job->msg_len_to_cipher_in_bytes = 0; } else { @@ -1228,26 +1069,28 @@ set_cpu_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_session *session, } /** - * Process a crypto operation and complete a JOB_AES_HMAC job structure for + * Process a crypto operation and complete a IMB_JOB job structure for * submission to the multi buffer library for processing. * - * @param qp queue pair - * @param job JOB_AES_HMAC structure to fill - * @param m mbuf to process + * @param qp queue pair + * @param job IMB_JOB structure to fill + * @param op crypto op to process + * @param digest_idx ID for digest to use * * @return - * - Completed JOB_AES_HMAC structure pointer on success - * - NULL pointer if completion of JOB_AES_HMAC structure isn't possible + * - 0 on success, the IMB_JOB will be filled + * - -1 if invalid session, IMB_JOB will not be filled */ static inline int -set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, +set_mb_job_params(IMB_JOB *job, struct ipsec_mb_qp *qp, struct rte_crypto_op *op, uint8_t *digest_idx) { struct rte_mbuf *m_src = op->sym->m_src, *m_dst; + struct aesni_mb_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); struct aesni_mb_session *session; uint32_t m_offset, oop; - session = get_session(qp, op); + session = ipsec_mb_get_session_private(qp, op); if (session == NULL) { op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; return -1; @@ -1260,7 +1103,7 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->cipher_direction = session->cipher.direction; job->cipher_mode = session->cipher.mode; - job->aes_key_len_in_bytes = session->cipher.key_length_in_bytes; + job->key_len_in_bytes = session->cipher.key_length_in_bytes; /* Set authentication parameters */ job->hash_alg = session->auth.algo; @@ -1268,38 +1111,32 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, const int aead = is_aead_algo(job->hash_alg, job->cipher_mode); switch (job->hash_alg) { - case AES_XCBC: + case IMB_AUTH_AES_XCBC: job->u.XCBC._k1_expanded = session->auth.xcbc.k1_expanded; job->u.XCBC._k2 = session->auth.xcbc.k2; job->u.XCBC._k3 = session->auth.xcbc.k3; - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; break; - case AES_CCM: + case IMB_AUTH_AES_CCM: job->u.CCM.aad = op->sym->aead.aad.data + 18; job->u.CCM.aad_len_in_bytes = session->aead.aad_len; - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; break; - case AES_CMAC: + case IMB_AUTH_AES_CMAC: job->u.CMAC._key_expanded = session->auth.cmac.expkey; job->u.CMAC._skey1 = session->auth.cmac.skey1; job->u.CMAC._skey2 = session->auth.cmac.skey2; - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; break; - case AES_GMAC: - if (session->cipher.mode == GCM) { + case IMB_AUTH_AES_GMAC: + if (session->cipher.mode == IMB_CIPHER_GCM) { job->u.GCM.aad = op->sym->aead.aad.data; job->u.GCM.aad_len_in_bytes = session->aead.aad_len; } else { @@ -1307,48 +1144,46 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->u.GCM.aad = rte_pktmbuf_mtod_offset(m_src, uint8_t *, op->sym->auth.data.offset); job->u.GCM.aad_len_in_bytes = op->sym->auth.data.length; - job->cipher_mode = GCM; + job->cipher_mode = IMB_CIPHER_GCM; } - job->aes_enc_key_expanded = &session->cipher.gcm_key; - job->aes_dec_key_expanded = &session->cipher.gcm_key; + job->enc_keys = &session->cipher.gcm_key; + job->dec_keys = &session->cipher.gcm_key; break; -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM case IMB_AUTH_ZUC_EIA3_BITLEN: job->u.ZUC_EIA3._key = session->auth.zuc_auth_key; job->u.ZUC_EIA3._iv = rte_crypto_op_ctod_offset(op, uint8_t *, session->auth_iv.offset); break; case IMB_AUTH_SNOW3G_UIA2_BITLEN: - job->u.SNOW3G_UIA2._key = (void *) &session->auth.pKeySched_snow3g_auth; - job->u.SNOW3G_UIA2._iv = rte_crypto_op_ctod_offset(op, uint8_t *, + job->u.SNOW3G_UIA2._key = (void *) + &session->auth.pKeySched_snow3g_auth; + job->u.SNOW3G_UIA2._iv = + rte_crypto_op_ctod_offset(op, uint8_t *, session->auth_iv.offset); break; case IMB_AUTH_KASUMI_UIA1: - job->u.KASUMI_UIA1._key = (void *) &session->auth.pKeySched_kasumi_auth; + job->u.KASUMI_UIA1._key = (void *) + &session->auth.pKeySched_kasumi_auth; break; -#endif -#if IMB_VERSION(0, 54, 3) <= IMB_VERSION_NUM case IMB_AUTH_CHACHA20_POLY1305: job->u.CHACHA20_POLY1305.aad = op->sym->aead.aad.data; - job->u.CHACHA20_POLY1305.aad_len_in_bytes = session->aead.aad_len; - job->aes_enc_key_expanded = session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = session->cipher.expanded_aes_keys.encode; + job->u.CHACHA20_POLY1305.aad_len_in_bytes = + session->aead.aad_len; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.encode; break; -#endif default: - job->u.HMAC._hashed_auth_key_xor_ipad = session->auth.pads.inner; - job->u.HMAC._hashed_auth_key_xor_opad = session->auth.pads.outer; - - if (job->cipher_mode == DES3) { - job->aes_enc_key_expanded = - session->cipher.exp_3des_keys.ks_ptr; - job->aes_dec_key_expanded = - session->cipher.exp_3des_keys.ks_ptr; + job->u.HMAC._hashed_auth_key_xor_ipad = + session->auth.pads.inner; + job->u.HMAC._hashed_auth_key_xor_opad = + session->auth.pads.outer; + + if (job->cipher_mode == IMB_CIPHER_DES3) { + job->enc_keys = session->cipher.exp_3des_keys.ks_ptr; + job->dec_keys = session->cipher.exp_3des_keys.ks_ptr; } else { - job->aes_enc_key_expanded = - session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = - session->cipher.expanded_aes_keys.decode; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; } } @@ -1357,10 +1192,9 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, else m_offset = op->sym->cipher.data.offset; -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM if (job->cipher_mode == IMB_CIPHER_ZUC_EEA3) { - job->aes_enc_key_expanded = session->cipher.zuc_cipher_key; - job->aes_dec_key_expanded = session->cipher.zuc_cipher_key; + job->enc_keys = session->cipher.zuc_cipher_key; + job->dec_keys = session->cipher.zuc_cipher_key; } else if (job->cipher_mode == IMB_CIPHER_SNOW3G_UEA2_BITLEN) { job->enc_keys = &session->cipher.pKeySched_snow3g_cipher; m_offset = 0; @@ -1368,7 +1202,6 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->enc_keys = &session->cipher.pKeySched_kasumi_cipher; m_offset = 0; } -#endif if (!op->sym->m_dst) { /* in-place operation */ @@ -1385,19 +1218,21 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, } /* Set digest output location */ - if (job->hash_alg != NULL_HASH && + if (job->hash_alg != IMB_AUTH_NULL && session->auth.operation == RTE_CRYPTO_AUTH_OP_VERIFY) { - job->auth_tag_output = qp->temp_digests[*digest_idx]; - *digest_idx = (*digest_idx + 1) % MAX_JOBS; + job->auth_tag_output = qp_data->temp_digests[*digest_idx]; + *digest_idx = (*digest_idx + 1) % IMB_MAX_JOBS; } else { if (aead) job->auth_tag_output = op->sym->aead.digest.data; else job->auth_tag_output = op->sym->auth.digest.data; - if (session->auth.req_digest_len != session->auth.gen_digest_len) { - job->auth_tag_output = qp->temp_digests[*digest_idx]; - *digest_idx = (*digest_idx + 1) % MAX_JOBS; + if (session->auth.req_digest_len != + session->auth.gen_digest_len) { + job->auth_tag_output = + qp_data->temp_digests[*digest_idx]; + *digest_idx = (*digest_idx + 1) % IMB_MAX_JOBS; } } /* @@ -1416,7 +1251,7 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->dst = rte_pktmbuf_mtod_offset(m_dst, uint8_t *, m_offset); switch (job->hash_alg) { - case AES_CCM: + case IMB_AUTH_AES_CCM: job->cipher_start_src_offset_in_bytes = op->sym->aead.data.offset; job->msg_len_to_cipher_in_bytes = op->sym->aead.data.length; @@ -1427,8 +1262,8 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, session->iv.offset + 1); break; - case AES_GMAC: - if (session->cipher.mode == GCM) { + case IMB_AUTH_AES_GMAC: + if (session->cipher.mode == IMB_CIPHER_GCM) { job->cipher_start_src_offset_in_bytes = op->sym->aead.data.offset; job->hash_start_src_offset_in_bytes = @@ -1450,10 +1285,11 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, session->iv.offset); break; -#if IMB_VERSION(0, 54, 3) <= IMB_VERSION_NUM case IMB_AUTH_CHACHA20_POLY1305: - job->cipher_start_src_offset_in_bytes = op->sym->aead.data.offset; - job->hash_start_src_offset_in_bytes = op->sym->aead.data.offset; + job->cipher_start_src_offset_in_bytes = + op->sym->aead.data.offset; + job->hash_start_src_offset_in_bytes = + op->sym->aead.data.offset; job->msg_len_to_cipher_in_bytes = op->sym->aead.data.length; job->msg_len_to_hash_in_bytes = @@ -1462,7 +1298,6 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->iv = rte_crypto_op_ctod_offset(op, uint8_t *, session->iv.offset); break; -#endif default: /* For SNOW3G, length and offsets are already in bits */ job->cipher_start_src_offset_in_bytes = @@ -1477,12 +1312,10 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, session->iv.offset); } -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM if (job->cipher_mode == IMB_CIPHER_ZUC_EEA3) job->msg_len_to_cipher_in_bytes >>= 3; else if (job->hash_alg == IMB_AUTH_KASUMI_UIA1) job->msg_len_to_hash_in_bytes >>= 3; -#endif /* Set user data to be crypto operation data struct */ job->user_data = op; @@ -1493,23 +1326,29 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, #ifdef AESNI_MB_DOCSIS_SEC_ENABLED /** * Process a crypto operation containing a security op and complete a - * JOB_AES_HMAC job structure for submission to the multi buffer library for + * IMB_JOB job structure for submission to the multi buffer library for * processing. */ static inline int -set_sec_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, - struct rte_crypto_op *op, uint8_t *digest_idx) +set_sec_mb_job_params(IMB_JOB *job, struct ipsec_mb_qp *qp, + struct rte_crypto_op *op, uint8_t *digest_idx) { + struct aesni_mb_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); struct rte_mbuf *m_src, *m_dst; struct rte_crypto_sym_op *sym; - struct aesni_mb_session *session; + struct aesni_mb_session *session = NULL; - session = get_session(qp, op); - if (unlikely(session == NULL)) { + if (unlikely(op->sess_type != RTE_CRYPTO_OP_SECURITY_SESSION)) { op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; return -1; } + session = (struct aesni_mb_session *) + get_sec_session_private_data(op->sym->sec_session); + if (unlikely(session == NULL)) { + op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; + return -1; + } /* Only DOCSIS protocol operations supported now */ if (session->cipher.mode != IMB_CIPHER_DOCSIS_SEC_BPI || session->auth.algo != IMB_AUTH_DOCSIS_CRC32) { @@ -1536,9 +1375,9 @@ set_sec_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->cipher_direction = session->cipher.direction; job->cipher_mode = session->cipher.mode; - job->aes_key_len_in_bytes = session->cipher.key_length_in_bytes; - job->aes_enc_key_expanded = session->cipher.expanded_aes_keys.encode; - job->aes_dec_key_expanded = session->cipher.expanded_aes_keys.decode; + job->key_len_in_bytes = session->cipher.key_length_in_bytes; + job->enc_keys = session->cipher.expanded_aes_keys.encode; + job->dec_keys = session->cipher.expanded_aes_keys.decode; /* Set IV parameters */ job->iv_len_in_bytes = session->iv.length; @@ -1548,8 +1387,8 @@ set_sec_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, job->hash_alg = session->auth.algo; /* Set digest output location */ - job->auth_tag_output = qp->temp_digests[*digest_idx]; - *digest_idx = (*digest_idx + 1) % MAX_JOBS; + job->auth_tag_output = qp_data->temp_digests[*digest_idx]; + *digest_idx = (*digest_idx + 1) % IMB_MAX_JOBS; /* Set digest length */ job->auth_tag_output_len_in_bytes = session->auth.gen_digest_len; @@ -1571,7 +1410,7 @@ set_sec_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, } static inline void -verify_docsis_sec_crc(JOB_AES_HMAC *job, uint8_t *status) +verify_docsis_sec_crc(IMB_JOB *job, uint8_t *status) { uint16_t crc_offset; uint8_t *crc; @@ -1591,7 +1430,7 @@ verify_docsis_sec_crc(JOB_AES_HMAC *job, uint8_t *status) #endif static inline void -verify_digest(JOB_AES_HMAC *job, void *digest, uint16_t len, uint8_t *status) +verify_digest(IMB_JOB *job, void *digest, uint16_t len, uint8_t *status) { /* Verify digest if required */ if (memcmp(job->auth_tag_output, digest, len) != 0) @@ -1599,7 +1438,7 @@ verify_digest(JOB_AES_HMAC *job, void *digest, uint16_t len, uint8_t *status) } static inline void -generate_digest(JOB_AES_HMAC *job, struct rte_crypto_op *op, +generate_digest(IMB_JOB *job, struct rte_crypto_op *op, struct aesni_mb_session *sess) { /* No extra copy needed */ @@ -1617,18 +1456,20 @@ generate_digest(JOB_AES_HMAC *job, struct rte_crypto_op *op, /** * Process a completed job and return rte_mbuf which job processed * - * @param qp Queue Pair to process - * @param job JOB_AES_HMAC job to process + * @param qp Queue Pair to process + * @param job IMB_JOB job to process * * @return * - Returns processed crypto operation. * - Returns NULL on invalid job */ static inline struct rte_crypto_op * -post_process_mb_job(struct aesni_mb_qp *qp, JOB_AES_HMAC *job) +post_process_mb_job(struct ipsec_mb_qp *qp, IMB_JOB *job) { struct rte_crypto_op *op = (struct rte_crypto_op *)job->user_data; struct aesni_mb_session *sess = NULL; + uint32_t driver_id = ipsec_mb_get_driver_id( + IPSEC_MB_PMD_TYPE_AESNI_MB); #ifdef AESNI_MB_DOCSIS_SEC_ENABLED uint8_t is_docsis_sec = 0; @@ -1644,7 +1485,7 @@ post_process_mb_job(struct aesni_mb_qp *qp, JOB_AES_HMAC *job) #endif { sess = get_sym_session_private_data(op->sym->session, - cryptodev_driver_id); + driver_id); } if (unlikely(sess == NULL)) { @@ -1654,14 +1495,15 @@ post_process_mb_job(struct aesni_mb_qp *qp, JOB_AES_HMAC *job) if (likely(op->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED)) { switch (job->status) { - case STS_COMPLETED: + case IMB_STATUS_COMPLETED: op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - if (job->hash_alg == NULL_HASH) + if (job->hash_alg == IMB_AUTH_NULL) break; if (sess->auth.operation == RTE_CRYPTO_AUTH_OP_VERIFY) { - if (is_aead_algo(job->hash_alg, sess->cipher.mode)) + if (is_aead_algo(job->hash_alg, + sess->cipher.mode)) verify_digest(job, op->sym->aead.digest.data, sess->auth.req_digest_len, @@ -1699,30 +1541,34 @@ post_process_mb_job(struct aesni_mb_qp *qp, JOB_AES_HMAC *job) } static inline void -post_process_mb_sync_job(JOB_AES_HMAC *job) +post_process_mb_sync_job(IMB_JOB *job) { uint32_t *st; st = job->user_data; - st[0] = (job->status == STS_COMPLETED) ? 0 : EBADMSG; + st[0] = (job->status == IMB_STATUS_COMPLETED) ? 0 : EBADMSG; } /** - * Process a completed JOB_AES_HMAC job and keep processing jobs until + * Process a completed IMB_JOB job and keep processing jobs until * get_completed_job return NULL * * @param qp Queue Pair to process - * @param job JOB_AES_HMAC job + * @param mb_mgr IMB_MGR to use + * @param job IMB_JOB job + * @param ops crypto ops to fill + * @param nb_ops number of crypto ops * * @return * - Number of processed jobs */ static unsigned -handle_completed_jobs(struct aesni_mb_qp *qp, JOB_AES_HMAC *job, - struct rte_crypto_op **ops, uint16_t nb_ops) +handle_completed_jobs(struct ipsec_mb_qp *qp, IMB_MGR *mb_mgr, + IMB_JOB *job, struct rte_crypto_op **ops, + uint16_t nb_ops) { struct rte_crypto_op *op = NULL; - unsigned processed_jobs = 0; + uint16_t processed_jobs = 0; while (job != NULL) { op = post_process_mb_job(qp, job); @@ -1737,14 +1583,14 @@ handle_completed_jobs(struct aesni_mb_qp *qp, JOB_AES_HMAC *job, if (processed_jobs == nb_ops) break; - job = IMB_GET_COMPLETED_JOB(qp->mb_mgr); + job = IMB_GET_COMPLETED_JOB(mb_mgr); } return processed_jobs; } static inline uint32_t -handle_completed_sync_jobs(JOB_AES_HMAC *job, MB_MGR *mb_mgr) +handle_completed_sync_jobs(IMB_JOB *job, IMB_MGR *mb_mgr) { uint32_t i; @@ -1755,37 +1601,37 @@ handle_completed_sync_jobs(JOB_AES_HMAC *job, MB_MGR *mb_mgr) } static inline uint32_t -flush_mb_sync_mgr(MB_MGR *mb_mgr) +flush_mb_sync_mgr(IMB_MGR *mb_mgr) { - JOB_AES_HMAC *job; + IMB_JOB *job; job = IMB_FLUSH_JOB(mb_mgr); return handle_completed_sync_jobs(job, mb_mgr); } static inline uint16_t -flush_mb_mgr(struct aesni_mb_qp *qp, struct rte_crypto_op **ops, - uint16_t nb_ops) +flush_mb_mgr(struct ipsec_mb_qp *qp, IMB_MGR *mb_mgr, + struct rte_crypto_op **ops, uint16_t nb_ops) { int processed_ops = 0; /* Flush the remaining jobs */ - JOB_AES_HMAC *job = IMB_FLUSH_JOB(qp->mb_mgr); + IMB_JOB *job = IMB_FLUSH_JOB(mb_mgr); if (job) - processed_ops += handle_completed_jobs(qp, job, + processed_ops += handle_completed_jobs(qp, mb_mgr, job, &ops[processed_ops], nb_ops - processed_ops); return processed_ops; } -static inline JOB_AES_HMAC * -set_job_null_op(JOB_AES_HMAC *job, struct rte_crypto_op *op) +static inline IMB_JOB * +set_job_null_op(IMB_JOB *job, struct rte_crypto_op *op) { - job->chain_order = HASH_CIPHER; - job->cipher_mode = NULL_CIPHER; - job->hash_alg = NULL_HASH; - job->cipher_direction = DECRYPT; + job->chain_order = IMB_ORDER_HASH_CIPHER; + job->cipher_mode = IMB_CIPHER_NULL; + job->hash_alg = IMB_AUTH_NULL; + job->cipher_direction = IMB_DIR_DECRYPT; /* Set user data to be crypto operation data struct */ job->user_data = op; @@ -1794,39 +1640,39 @@ set_job_null_op(JOB_AES_HMAC *job, struct rte_crypto_op *op) } static uint16_t -aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, +aesni_mb_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, uint16_t nb_ops) { - struct aesni_mb_qp *qp = queue_pair; - + struct ipsec_mb_qp *qp = queue_pair; + IMB_MGR *mb_mgr = qp->mb_mgr; struct rte_crypto_op *op; - JOB_AES_HMAC *job; - + IMB_JOB *job; int retval, processed_jobs = 0; - if (unlikely(nb_ops == 0)) + if (unlikely(nb_ops == 0 || mb_mgr == NULL)) return 0; uint8_t digest_idx = qp->digest_idx; + do { /* Get next free mb job struct from mb manager */ - job = IMB_GET_NEXT_JOB(qp->mb_mgr); + job = IMB_GET_NEXT_JOB(mb_mgr); if (unlikely(job == NULL)) { /* if no free mb job structs we need to flush mb_mgr */ - processed_jobs += flush_mb_mgr(qp, + processed_jobs += flush_mb_mgr(qp, mb_mgr, &ops[processed_jobs], nb_ops - processed_jobs); if (nb_ops == processed_jobs) break; - job = IMB_GET_NEXT_JOB(qp->mb_mgr); + job = IMB_GET_NEXT_JOB(mb_mgr); } /* * Get next operation to process from ingress queue. - * There is no need to return the job to the MB_MGR - * if there are no more operations to process, since the MB_MGR + * There is no need to return the job to the IMB_MGR + * if there are no more operations to process, since the IMB_MGR * can use that pointer again in next get_next calls. */ retval = rte_ring_dequeue(qp->ingress_queue, (void **)&op); @@ -1839,7 +1685,8 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, &digest_idx); else #endif - retval = set_mb_job_params(job, qp, op, &digest_idx); + retval = set_mb_job_params(job, qp, op, + &digest_idx); if (unlikely(retval != 0)) { qp->stats.dequeue_err_count++; @@ -1848,17 +1695,17 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, /* Submit job to multi-buffer for processing */ #ifdef RTE_LIBRTE_PMD_AESNI_MB_DEBUG - job = IMB_SUBMIT_JOB(qp->mb_mgr); + job = IMB_SUBMIT_JOB(mb_mgr); #else - job = IMB_SUBMIT_JOB_NOCHECK(qp->mb_mgr); + job = IMB_SUBMIT_JOB_NOCHECK(mb_mgr); #endif /* * If submit returns a processed job then handle it, * before submitting subsequent jobs */ if (job) - processed_jobs += handle_completed_jobs(qp, job, - &ops[processed_jobs], + processed_jobs += handle_completed_jobs(qp, mb_mgr, + job, &ops[processed_jobs], nb_ops - processed_jobs); } while (processed_jobs < nb_ops); @@ -1866,44 +1713,16 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, qp->digest_idx = digest_idx; if (processed_jobs < 1) - processed_jobs += flush_mb_mgr(qp, + processed_jobs += flush_mb_mgr(qp, mb_mgr, &ops[processed_jobs], nb_ops - processed_jobs); return processed_jobs; } -static MB_MGR * -alloc_init_mb_mgr(enum aesni_mb_vector_mode vector_mode) -{ - MB_MGR *mb_mgr = alloc_mb_mgr(0); - if (mb_mgr == NULL) - return NULL; - - switch (vector_mode) { - case RTE_AESNI_MB_SSE: - init_mb_mgr_sse(mb_mgr); - break; - case RTE_AESNI_MB_AVX: - init_mb_mgr_avx(mb_mgr); - break; - case RTE_AESNI_MB_AVX2: - init_mb_mgr_avx2(mb_mgr); - break; - case RTE_AESNI_MB_AVX512: - init_mb_mgr_avx512(mb_mgr); - break; - default: - AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n", vector_mode); - free_mb_mgr(mb_mgr); - return NULL; - } - - return mb_mgr; -} static inline void -aesni_mb_fill_error_code(struct rte_crypto_sym_vec *vec, int32_t err) +ipsec_mb_fill_error_code(struct rte_crypto_sym_vec *vec, int32_t err) { uint32_t i; @@ -1916,14 +1735,14 @@ check_crypto_sgl(union rte_crypto_sym_ofs so, const struct rte_crypto_sgl *sgl) { /* no multi-seg support with current AESNI-MB PMD */ if (sgl->num != 1) - return ENOTSUP; + return -ENOTSUP; else if (so.ofs.cipher.head + so.ofs.cipher.tail > sgl->vec[0].len) - return EINVAL; + return -EINVAL; return 0; } -static inline JOB_AES_HMAC * -submit_sync_job(MB_MGR *mb_mgr) +static inline IMB_JOB * +submit_sync_job(IMB_MGR *mb_mgr) { #ifdef RTE_LIBRTE_PMD_AESNI_MB_DEBUG return IMB_SUBMIT_JOB(mb_mgr); @@ -1966,42 +1785,31 @@ verify_sync_dgst(struct rte_crypto_sym_vec *vec, return k; } -uint32_t -aesni_mb_cpu_crypto_process_bulk(struct rte_cryptodev *dev, +static uint32_t +aesni_mb_process_bulk(struct rte_cryptodev *dev, struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs sofs, struct rte_crypto_sym_vec *vec) { int32_t ret; uint32_t i, j, k, len; void *buf; - JOB_AES_HMAC *job; - MB_MGR *mb_mgr; - struct aesni_mb_private *priv; + IMB_JOB *job; + IMB_MGR *mb_mgr; struct aesni_mb_session *s; uint8_t tmp_dgst[vec->num][DIGEST_LENGTH_MAX]; s = get_sym_session_private_data(sess, dev->driver_id); if (s == NULL) { - aesni_mb_fill_error_code(vec, EINVAL); + ipsec_mb_fill_error_code(vec, EINVAL); return 0; } /* get per-thread MB MGR, create one if needed */ - mb_mgr = RTE_PER_LCORE(sync_mb_mgr); - if (mb_mgr == NULL) { - - priv = dev->data->dev_private; - mb_mgr = alloc_init_mb_mgr(priv->vector_mode); - if (mb_mgr == NULL) { - aesni_mb_fill_error_code(vec, ENOMEM); - return 0; - } - RTE_PER_LCORE(sync_mb_mgr) = mb_mgr; - } + mb_mgr = get_per_thread_mb_mgr(); + if (unlikely(mb_mgr == NULL)) + return 0; for (i = 0, j = 0, k = 0; i != vec->num; i++) { - - ret = check_crypto_sgl(sofs, vec->sgl + i); if (ret != 0) { vec->status[i] = ret; @@ -2047,186 +1855,172 @@ aesni_mb_cpu_crypto_process_bulk(struct rte_cryptodev *dev, return k; } -static int cryptodev_aesni_mb_remove(struct rte_vdev_device *vdev); +struct rte_cryptodev_ops aesni_mb_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, -static uint64_t -vec_mode_to_flags(enum aesni_mb_vector_mode mode) -{ - switch (mode) { - case RTE_AESNI_MB_SSE: - return RTE_CRYPTODEV_FF_CPU_SSE; - case RTE_AESNI_MB_AVX: - return RTE_CRYPTODEV_FF_CPU_AVX; - case RTE_AESNI_MB_AVX2: - return RTE_CRYPTODEV_FF_CPU_AVX2; - case RTE_AESNI_MB_AVX512: - return RTE_CRYPTODEV_FF_CPU_AVX512; - default: - AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n", mode); - return 0; - } -} + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, -static int -cryptodev_aesni_mb_create(const char *name, - struct rte_vdev_device *vdev, - struct rte_cryptodev_pmd_init_params *init_params) -{ - struct rte_cryptodev *dev; - struct aesni_mb_private *internals; - enum aesni_mb_vector_mode vector_mode; - MB_MGR *mb_mgr; - - dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); - if (dev == NULL) { - AESNI_MB_LOG(ERR, "failed to create cryptodev vdev"); - return -ENODEV; - } + .dev_infos_get = ipsec_mb_info_get, - /* Check CPU for supported vector instruction set */ - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) - vector_mode = RTE_AESNI_MB_AVX512; - else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) - vector_mode = RTE_AESNI_MB_AVX2; - else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) - vector_mode = RTE_AESNI_MB_AVX; - else - vector_mode = RTE_AESNI_MB_SSE; + .queue_pair_setup = ipsec_mb_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, - dev->driver_id = cryptodev_driver_id; - dev->dev_ops = rte_aesni_mb_pmd_ops; + .sym_cpu_process = aesni_mb_process_bulk, - /* register rx/tx burst functions for data path */ - dev->dequeue_burst = aesni_mb_pmd_dequeue_burst; - dev->enqueue_burst = aesni_mb_pmd_enqueue_burst; - - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | - RTE_CRYPTODEV_FF_SYM_CPU_CRYPTO | - RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | - RTE_CRYPTODEV_FF_SYM_SESSIONLESS; + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; #ifdef AESNI_MB_DOCSIS_SEC_ENABLED - struct rte_security_ctx *security_instance; - security_instance = rte_malloc("aesni_mb_sec", - sizeof(struct rte_security_ctx), - RTE_CACHE_LINE_SIZE); - if (security_instance == NULL) { - AESNI_MB_LOG(ERR, "rte_security_ctx memory alloc failed"); - rte_cryptodev_pmd_destroy(dev); - return -ENOMEM; +/** + * Configure a aesni multi-buffer session from a security session + * configuration + */ +static int +aesni_mb_pmd_sec_sess_create(void *dev, struct rte_security_session_conf *conf, + struct rte_security_session *sess, + struct rte_mempool *mempool) +{ + void *sess_private_data; + struct rte_cryptodev *cdev = (struct rte_cryptodev *)dev; + int ret; + + if (conf->action_type != RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL || + conf->protocol != RTE_SECURITY_PROTOCOL_DOCSIS) { + IPSEC_MB_LOG(ERR, "Invalid security protocol"); + return -EINVAL; } - security_instance->device = (void *)dev; - security_instance->ops = rte_aesni_mb_pmd_sec_ops; - security_instance->sess_cnt = 0; - dev->security_ctx = security_instance; - dev->feature_flags |= RTE_CRYPTODEV_FF_SECURITY; -#endif + if (rte_mempool_get(mempool, &sess_private_data)) { + IPSEC_MB_LOG(ERR, "Couldn't get object from session mempool"); + return -ENOMEM; + } - /* Check CPU for support for AES instruction set */ - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AESNI; - else - AESNI_MB_LOG(WARNING, "AES instructions not supported by CPU"); + ret = aesni_mb_set_docsis_sec_session_parameters(cdev, conf, + sess_private_data); - dev->feature_flags |= vec_mode_to_flags(vector_mode); + if (ret != 0) { + IPSEC_MB_LOG(ERR, "Failed to configure session parameters"); - mb_mgr = alloc_init_mb_mgr(vector_mode); - if (mb_mgr == NULL) { -#ifdef AESNI_MB_DOCSIS_SEC_ENABLED - rte_free(dev->security_ctx); - dev->security_ctx = NULL; -#endif - rte_cryptodev_pmd_destroy(dev); - return -ENOMEM; + /* Return session to mempool */ + rte_mempool_put(mempool, sess_private_data); + return ret; } - /* Set vector instructions mode supported */ - internals = dev->data->dev_private; + set_sec_session_private_data(sess, sess_private_data); - internals->vector_mode = vector_mode; - internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; - internals->mb_mgr = mb_mgr; - - AESNI_MB_LOG(INFO, "IPSec Multi-buffer library version used: %s\n", - imb_get_version_str()); - return 0; + return ret; } +/** Clear the memory of session so it does not leave key material behind */ static int -cryptodev_aesni_mb_probe(struct rte_vdev_device *vdev) +aesni_mb_pmd_sec_sess_destroy(void *dev __rte_unused, + struct rte_security_session *sess) { - struct rte_cryptodev_pmd_init_params init_params = { - "", - sizeof(struct aesni_mb_private), - rte_socket_id(), - RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS - }; - const char *name, *args; - int retval; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; + void *sess_priv = get_sec_session_private_data(sess); - args = rte_vdev_device_args(vdev); + if (sess_priv) { + struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - retval = rte_cryptodev_pmd_parse_input_args(&init_params, args); - if (retval) { - AESNI_MB_LOG(ERR, "Failed to parse initialisation arguments[%s]", - args); - return -EINVAL; + memset(sess_priv, 0, sizeof(struct aesni_mb_session)); + set_sec_session_private_data(sess, NULL); + rte_mempool_put(sess_mp, sess_priv); } - - return cryptodev_aesni_mb_create(name, vdev, &init_params); + return 0; } -static int -cryptodev_aesni_mb_remove(struct rte_vdev_device *vdev) +/** Get security capabilities for aesni multi-buffer */ +static const struct rte_security_capability * +aesni_mb_pmd_sec_capa_get(void *device __rte_unused) { - struct rte_cryptodev *cryptodev; - struct aesni_mb_private *internals; - const char *name; + return aesni_mb_pmd_security_cap; +} - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; +static struct rte_security_ops aesni_mb_pmd_sec_ops = { + .session_create = aesni_mb_pmd_sec_sess_create, + .session_update = NULL, + .session_stats_get = NULL, + .session_destroy = aesni_mb_pmd_sec_sess_destroy, + .set_pkt_metadata = NULL, + .capabilities_get = aesni_mb_pmd_sec_capa_get +}; - cryptodev = rte_cryptodev_pmd_get_named_dev(name); - if (cryptodev == NULL) - return -ENODEV; +struct rte_security_ops *rte_aesni_mb_pmd_sec_ops = &aesni_mb_pmd_sec_ops; - internals = cryptodev->data->dev_private; +static int +aesni_mb_configure_dev(struct rte_cryptodev *dev) +{ + struct rte_security_ctx *security_instance; - free_mb_mgr(internals->mb_mgr); - if (RTE_PER_LCORE(sync_mb_mgr)) { - free_mb_mgr(RTE_PER_LCORE(sync_mb_mgr)); - RTE_PER_LCORE(sync_mb_mgr) = NULL; + security_instance = rte_malloc("aesni_mb_sec", + sizeof(struct rte_security_ctx), + RTE_CACHE_LINE_SIZE); + if (security_instance != NULL) { + security_instance->device = (void *)dev; + security_instance->ops = rte_aesni_mb_pmd_sec_ops; + security_instance->sess_cnt = 0; + dev->security_ctx = security_instance; + + return 0; } -#ifdef AESNI_MB_DOCSIS_SEC_ENABLED - rte_free(cryptodev->security_ctx); - cryptodev->security_ctx = NULL; + return -ENOMEM; +} + #endif - return rte_cryptodev_pmd_destroy(cryptodev); +static int +aesni_mb_probe(struct rte_vdev_device *vdev) +{ + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_AESNI_MB); } static struct rte_vdev_driver cryptodev_aesni_mb_pmd_drv = { - .probe = cryptodev_aesni_mb_probe, - .remove = cryptodev_aesni_mb_remove + .probe = aesni_mb_probe, + .remove = ipsec_mb_remove }; static struct cryptodev_driver aesni_mb_crypto_drv; -RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_AESNI_MB_PMD, cryptodev_aesni_mb_pmd_drv); +RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_AESNI_MB_PMD, + cryptodev_aesni_mb_pmd_drv); RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_AESNI_MB_PMD, cryptodev_aesni_mb_pmd); RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_AESNI_MB_PMD, - "max_nb_queue_pairs= " - "socket_id="); -RTE_PMD_REGISTER_CRYPTO_DRIVER(aesni_mb_crypto_drv, - cryptodev_aesni_mb_pmd_drv.driver, - cryptodev_driver_id); -RTE_LOG_REGISTER_DEFAULT(aesni_mb_logtype_driver, NOTICE); + "max_nb_queue_pairs= socket_id="); +RTE_PMD_REGISTER_CRYPTO_DRIVER( + aesni_mb_crypto_drv, + cryptodev_aesni_mb_pmd_drv.driver, + pmd_driver_id_aesni_mb); + +/* Constructor function to register aesni-mb PMD */ +RTE_INIT(ipsec_mb_register_aesni_mb) +{ + struct ipsec_mb_internals *aesni_mb_data = + &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_AESNI_MB]; + + aesni_mb_data->caps = aesni_mb_capabilities; + aesni_mb_data->dequeue_burst = aesni_mb_dequeue_burst; + aesni_mb_data->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | + RTE_CRYPTODEV_FF_SYM_CPU_CRYPTO | + RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | + RTE_CRYPTODEV_FF_SYM_SESSIONLESS; + + aesni_mb_data->internals_priv_size = 0; + aesni_mb_data->ops = &aesni_mb_pmd_ops; + aesni_mb_data->qp_priv_size = sizeof(struct aesni_mb_qp_data); + aesni_mb_data->queue_pair_configure = NULL; +#ifdef AESNI_MB_DOCSIS_SEC_ENABLED + aesni_mb_data->security_ops = &aesni_mb_pmd_sec_ops; + aesni_mb_data->dev_config = aesni_mb_configure_dev; + aesni_mb_data->feature_flags |= RTE_CRYPTODEV_FF_SECURITY; +#endif + aesni_mb_data->session_configure = aesni_mb_session_configure; + aesni_mb_data->session_priv_size = sizeof(struct aesni_mb_session); +} diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c b/drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h similarity index 55% rename from drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c rename to drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h index 48a8f91868..db7d283ca2 100644 --- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c +++ b/drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h @@ -1,19 +1,28 @@ /* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2015-2017 Intel Corporation + * Copyright(c) 2015-2021 Intel Corporation */ -#include +#ifndef _PMD_AESNI_MB_PRIV_H_ +#define _PMD_AESNI_MB_PRIV_H_ -#include -#include -#include +#include + +#if defined(RTE_LIB_SECURITY) +#define AESNI_MB_DOCSIS_SEC_ENABLED 1 +#include +#include #include -#include +#endif -#include "aesni_mb_pmd_private.h" +#include "ipsec_mb_private.h" +#define AES_CCM_DIGEST_MIN_LEN 4 +#define AES_CCM_DIGEST_MAX_LEN 16 +#define HMAC_MAX_BLOCK_SIZE 128 +#define HMAC_IPAD_VALUE (0x36) +#define HMAC_OPAD_VALUE (0x5C) -static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { +static const struct rte_cryptodev_capabilities aesni_mb_capabilities[] = { { /* MD5 HMAC */ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, {.sym = { @@ -315,13 +324,8 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 16, .key_size = { .min = 16, -#if IMB_VERSION_NUM >= IMB_VERSION(0, 53, 3) .max = 32, .increment = 16 -#else - .max = 16, - .increment = 0 -#endif }, .iv_size = { .min = 16, @@ -400,13 +404,8 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 16, .key_size = { .min = 16, -#if IMB_VERSION(0, 54, 2) <= IMB_VERSION_NUM .max = 32, .increment = 16 -#else - .max = 16, - .increment = 0 -#endif }, .digest_size = { .min = 4, @@ -502,7 +501,6 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { }, } }, } }, -#if IMB_VERSION(0, 53, 0) <= IMB_VERSION_NUM { /* AES ECB */ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, {.sym = { @@ -519,8 +517,6 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { }, } }, } }, -#endif -#if IMB_VERSION(0, 53, 3) <= IMB_VERSION_NUM { /* ZUC (EIA3) */ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, {.sym = { @@ -652,8 +648,6 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { }, } }, } }, -#endif -#if IMB_VERSION(0, 54, 3) <= IMB_VERSION_NUM { /* CHACHA20-POLY1305 */ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, {.sym = { @@ -673,7 +667,7 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { }, .aad_size = { .min = 0, - .max = 240, + .max = 1024, .increment = 1 }, .iv_size = { @@ -684,10 +678,244 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { }, } }, } }, -#endif RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() }; +uint8_t pmd_driver_id_aesni_mb; + +struct aesni_mb_qp_data { + uint8_t temp_digests[IMB_MAX_JOBS][DIGEST_LENGTH_MAX]; + /* *< Buffers used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ +}; + +/* Maximum length for digest */ +#define DIGEST_LENGTH_MAX 64 +static const unsigned int auth_blocksize[] = { + [IMB_AUTH_NULL] = 0, + [IMB_AUTH_MD5] = 64, + [IMB_AUTH_HMAC_SHA_1] = 64, + [IMB_AUTH_HMAC_SHA_224] = 64, + [IMB_AUTH_HMAC_SHA_256] = 64, + [IMB_AUTH_HMAC_SHA_384] = 128, + [IMB_AUTH_HMAC_SHA_512] = 128, + [IMB_AUTH_AES_XCBC] = 16, + [IMB_AUTH_AES_CCM] = 16, + [IMB_AUTH_AES_CMAC] = 16, + [IMB_AUTH_AES_GMAC] = 16, + [IMB_AUTH_SHA_1] = 64, + [IMB_AUTH_SHA_224] = 64, + [IMB_AUTH_SHA_256] = 64, + [IMB_AUTH_SHA_384] = 128, + [IMB_AUTH_SHA_512] = 128, + [IMB_AUTH_ZUC_EIA3_BITLEN] = 16, + [IMB_AUTH_SNOW3G_UIA2_BITLEN] = 16, + [IMB_AUTH_KASUMI_UIA1] = 16 +}; + +/** + * Get the blocksize in bytes for a specified authentication algorithm + * + * @Note: this function will not return a valid value for a non-valid + * authentication algorithm + */ +static inline unsigned int +get_auth_algo_blocksize(IMB_HASH_ALG algo) +{ + return auth_blocksize[algo]; +} + +static const unsigned int auth_truncated_digest_byte_lengths[] = { + [IMB_AUTH_MD5] = 12, + [IMB_AUTH_HMAC_SHA_1] = 12, + [IMB_AUTH_HMAC_SHA_224] = 14, + [IMB_AUTH_HMAC_SHA_256] = 16, + [IMB_AUTH_HMAC_SHA_384] = 24, + [IMB_AUTH_HMAC_SHA_512] = 32, + [IMB_AUTH_AES_XCBC] = 12, + [IMB_AUTH_AES_CMAC] = 12, + [IMB_AUTH_AES_CCM] = 8, + [IMB_AUTH_NULL] = 0, + [IMB_AUTH_AES_GMAC] = 12, + [IMB_AUTH_SHA_1] = 20, + [IMB_AUTH_SHA_224] = 28, + [IMB_AUTH_SHA_256] = 32, + [IMB_AUTH_SHA_384] = 48, + [IMB_AUTH_SHA_512] = 64, + [IMB_AUTH_ZUC_EIA3_BITLEN] = 4, + [IMB_AUTH_SNOW3G_UIA2_BITLEN] = 4, + [IMB_AUTH_KASUMI_UIA1] = 4 +}; + +/** + * Get the IPsec specified truncated length in bytes of the HMAC digest for a + * specified authentication algorithm + * + * @Note: this function will not return a valid value for a non-valid + * authentication algorithm + */ +static inline unsigned int +get_truncated_digest_byte_length(IMB_HASH_ALG algo) +{ + return auth_truncated_digest_byte_lengths[algo]; +} + +static const unsigned int auth_digest_byte_lengths[] = { + [IMB_AUTH_MD5] = 16, + [IMB_AUTH_HMAC_SHA_1] = 20, + [IMB_AUTH_HMAC_SHA_224] = 28, + [IMB_AUTH_HMAC_SHA_256] = 32, + [IMB_AUTH_HMAC_SHA_384] = 48, + [IMB_AUTH_HMAC_SHA_512] = 64, + [IMB_AUTH_AES_XCBC] = 16, + [IMB_AUTH_AES_CMAC] = 16, + [IMB_AUTH_AES_CCM] = 16, + [IMB_AUTH_AES_GMAC] = 16, + [IMB_AUTH_NULL] = 0, + [IMB_AUTH_SHA_1] = 20, + [IMB_AUTH_SHA_224] = 28, + [IMB_AUTH_SHA_256] = 32, + [IMB_AUTH_SHA_384] = 48, + [IMB_AUTH_SHA_512] = 64, + [IMB_AUTH_ZUC_EIA3_BITLEN] = 4, + [IMB_AUTH_SNOW3G_UIA2_BITLEN] = 4, + [IMB_AUTH_KASUMI_UIA1] = 4 + /**< Vector mode dependent pointer table of the multi-buffer APIs */ + +}; + +/** + * Get the full digest size in bytes for a specified authentication algorithm + * (if available in the Multi-buffer library) + * + * @Note: this function will not return a valid value for a non-valid + * authentication algorithm + */ +static inline unsigned int +get_digest_byte_length(IMB_HASH_ALG algo) +{ + return auth_digest_byte_lengths[algo]; +} + +/** AES-NI multi-buffer private session structure */ +struct aesni_mb_session { + IMB_CIPHER_MODE cipher_mode; + IMB_CIPHER_DIRECTION cipher_direction; + IMB_HASH_ALG hash_alg; + IMB_CHAIN_ORDER chain_order; + /* common job fields */ + struct { + uint16_t length; + uint16_t offset; + } iv; + struct { + uint16_t length; + uint16_t offset; + } auth_iv; + /* *< IV parameters + */ + + /* * Cipher Parameters + */ + struct { + /* * Cipher direction - encrypt / decrypt */ + IMB_CIPHER_DIRECTION direction; + /* * Cipher mode - CBC / Counter */ + IMB_CIPHER_MODE mode; + + uint64_t key_length_in_bytes; + + union { + struct { + uint32_t encode[60] __rte_aligned(16); + /* *< encode key */ + uint32_t decode[60] __rte_aligned(16); + /* *< decode key */ + } expanded_aes_keys; + /* *< Expanded AES keys - Allocating space to + * contain the maximum expanded key size which + * is 240 bytes for 256 bit AES, calculate by: + * ((key size (bytes)) * + * ((number of rounds) + 1)) + */ + struct { + const void *ks_ptr[3]; + uint64_t key[3][16]; + } exp_3des_keys; + /* *< Expanded 3DES keys */ + + struct gcm_key_data gcm_key; + /* *< Expanded GCM key */ + uint8_t zuc_cipher_key[16]; + /* *< ZUC cipher key */ + snow3g_key_schedule_t pKeySched_snow3g_cipher; + /* *< SNOW3G scheduled cipher key */ + kasumi_key_sched_t pKeySched_kasumi_cipher; + /* *< KASUMI scheduled cipher key */ + }; + } cipher; + + /* *< Authentication Parameters */ + struct { + IMB_HASH_ALG algo; /* *< Authentication Algorithm */ + enum rte_crypto_auth_operation operation; + /* *< auth operation generate or verify */ + union { + struct { + uint8_t inner[128] __rte_aligned(16); + /* *< inner pad */ + uint8_t outer[128] __rte_aligned(16); + /* *< outer pad */ + } pads; + /* *< HMAC Authentication pads - + * allocating space for the maximum pad + * size supported which is 128 bytes for + * SHA512 + */ + + struct { + uint32_t k1_expanded[44] __rte_aligned(16); + /* *< k1 (expanded key). */ + uint8_t k2[16] __rte_aligned(16); + /* *< k2. */ + uint8_t k3[16] __rte_aligned(16); + /* *< k3. */ + } xcbc; + + struct { + uint32_t expkey[60] __rte_aligned(16); + /* *< k1 (expanded key). */ + uint32_t skey1[4] __rte_aligned(16); + /* *< k2. */ + uint32_t skey2[4] __rte_aligned(16); + /* *< k3. */ + } cmac; + /* *< Expanded XCBC authentication keys */ + uint8_t zuc_auth_key[16]; + /* *< ZUC authentication key */ + snow3g_key_schedule_t pKeySched_snow3g_auth; + /* *< SNOW3G scheduled authentication key */ + kasumi_key_sched_t pKeySched_kasumi_auth; + /* *< KASUMI scheduled authentication key */ + }; + /* * Generated digest size by the Multi-buffer library */ + uint16_t gen_digest_len; + /* * Requested digest size from Cryptodev */ + uint16_t req_digest_len; + + } auth; + struct { + /* * AAD data length */ + uint16_t aad_len; + } aead; +} __rte_cache_aligned; + +typedef void (*hash_one_block_t)(const void *data, void *digest); +typedef void (*aes_keyexp_t)(const void *key, void *enc_exp_keys, + void *dec_exp_keys); + #ifdef AESNI_MB_DOCSIS_SEC_ENABLED static const struct rte_cryptodev_capabilities aesni_mb_pmd_security_crypto_cap[] = { @@ -738,389 +966,4 @@ static const struct rte_security_capability aesni_mb_pmd_security_cap[] = { }; #endif -/** Configure device */ -static int -aesni_mb_pmd_config(__rte_unused struct rte_cryptodev *dev, - __rte_unused struct rte_cryptodev_config *config) -{ - return 0; -} - -/** Start device */ -static int -aesni_mb_pmd_start(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - -/** Stop device */ -static void -aesni_mb_pmd_stop(__rte_unused struct rte_cryptodev *dev) -{ -} - -/** Close device */ -static int -aesni_mb_pmd_close(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - - -/** Get device statistics */ -static void -aesni_mb_pmd_stats_get(struct rte_cryptodev *dev, - struct rte_cryptodev_stats *stats) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct aesni_mb_qp *qp = dev->data->queue_pairs[qp_id]; - - stats->enqueued_count += qp->stats.enqueued_count; - stats->dequeued_count += qp->stats.dequeued_count; - - stats->enqueue_err_count += qp->stats.enqueue_err_count; - stats->dequeue_err_count += qp->stats.dequeue_err_count; - } -} - -/** Reset device statistics */ -static void -aesni_mb_pmd_stats_reset(struct rte_cryptodev *dev) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct aesni_mb_qp *qp = dev->data->queue_pairs[qp_id]; - - memset(&qp->stats, 0, sizeof(qp->stats)); - } -} - - -/** Get device info */ -static void -aesni_mb_pmd_info_get(struct rte_cryptodev *dev, - struct rte_cryptodev_info *dev_info) -{ - struct aesni_mb_private *internals = dev->data->dev_private; - - if (dev_info != NULL) { - dev_info->driver_id = dev->driver_id; - dev_info->feature_flags = dev->feature_flags; - dev_info->capabilities = aesni_mb_pmd_capabilities; - dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; - /* No limit of number of sessions */ - dev_info->sym.max_nb_sessions = 0; - } -} - -/** Release queue pair */ -static int -aesni_mb_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) -{ - struct aesni_mb_qp *qp = dev->data->queue_pairs[qp_id]; - struct rte_ring *r = NULL; - - if (qp != NULL) { - r = rte_ring_lookup(qp->name); - if (r) - rte_ring_free(r); - if (qp->mb_mgr) - free_mb_mgr(qp->mb_mgr); - rte_free(qp); - dev->data->queue_pairs[qp_id] = NULL; - } - return 0; -} - -/** set a unique name for the queue pair based on it's name, dev_id and qp_id */ -static int -aesni_mb_pmd_qp_set_unique_name(struct rte_cryptodev *dev, - struct aesni_mb_qp *qp) -{ - unsigned n = snprintf(qp->name, sizeof(qp->name), - "aesni_mb_pmd_%u_qp_%u", - dev->data->dev_id, qp->id); - - if (n >= sizeof(qp->name)) - return -1; - - return 0; -} - -/** Create a ring to place processed operations on */ -static struct rte_ring * -aesni_mb_pmd_qp_create_processed_ops_ring(struct aesni_mb_qp *qp, - unsigned int ring_size, int socket_id) -{ - struct rte_ring *r; - char ring_name[RTE_CRYPTODEV_NAME_MAX_LEN]; - - unsigned int n = strlcpy(ring_name, qp->name, sizeof(ring_name)); - - if (n >= sizeof(ring_name)) - return NULL; - - r = rte_ring_lookup(ring_name); - if (r) { - if (rte_ring_get_size(r) >= ring_size) { - AESNI_MB_LOG(INFO, "Reusing existing ring %s for processed ops", - ring_name); - return r; - } - - AESNI_MB_LOG(ERR, "Unable to reuse existing ring %s for processed ops", - ring_name); - return NULL; - } - - return rte_ring_create(ring_name, ring_size, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); -} - -/** Setup a queue pair */ -static int -aesni_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, - const struct rte_cryptodev_qp_conf *qp_conf, - int socket_id) -{ - struct aesni_mb_qp *qp = NULL; - struct aesni_mb_private *internals = dev->data->dev_private; - int ret = -1; - - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - aesni_mb_pmd_qp_release(dev, qp_id); - - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("AES-NI PMD Queue Pair", sizeof(*qp), - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return -ENOMEM; - - qp->id = qp_id; - dev->data->queue_pairs[qp_id] = qp; - - if (aesni_mb_pmd_qp_set_unique_name(dev, qp)) - goto qp_setup_cleanup; - - - qp->mb_mgr = alloc_mb_mgr(0); - if (qp->mb_mgr == NULL) { - ret = -ENOMEM; - goto qp_setup_cleanup; - } - - switch (internals->vector_mode) { - case RTE_AESNI_MB_SSE: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - init_mb_mgr_sse(qp->mb_mgr); - break; - case RTE_AESNI_MB_AVX: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - init_mb_mgr_avx(qp->mb_mgr); - break; - case RTE_AESNI_MB_AVX2: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; - init_mb_mgr_avx2(qp->mb_mgr); - break; - case RTE_AESNI_MB_AVX512: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; - init_mb_mgr_avx512(qp->mb_mgr); - break; - default: - AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n", - internals->vector_mode); - goto qp_setup_cleanup; - } - - qp->ingress_queue = aesni_mb_pmd_qp_create_processed_ops_ring(qp, - qp_conf->nb_descriptors, socket_id); - if (qp->ingress_queue == NULL) { - ret = -1; - goto qp_setup_cleanup; - } - - qp->sess_mp = qp_conf->mp_session; - qp->sess_mp_priv = qp_conf->mp_session_private; - - memset(&qp->stats, 0, sizeof(qp->stats)); - - char mp_name[RTE_MEMPOOL_NAMESIZE]; - - snprintf(mp_name, RTE_MEMPOOL_NAMESIZE, - "digest_mp_%u_%u", dev->data->dev_id, qp_id); - return 0; - -qp_setup_cleanup: - if (qp) { - if (qp->mb_mgr) - free_mb_mgr(qp->mb_mgr); - rte_free(qp); - } - - return ret; -} - -/** Returns the size of the aesni multi-buffer session structure */ -static unsigned -aesni_mb_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) -{ - return sizeof(struct aesni_mb_session); -} - -/** Configure a aesni multi-buffer session from a crypto xform chain */ -static int -aesni_mb_pmd_sym_session_configure(struct rte_cryptodev *dev, - struct rte_crypto_sym_xform *xform, - struct rte_cryptodev_sym_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - struct aesni_mb_private *internals = dev->data->dev_private; - int ret; - - if (unlikely(sess == NULL)) { - AESNI_MB_LOG(ERR, "invalid session struct"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - AESNI_MB_LOG(ERR, - "Couldn't get object from session mempool"); - return -ENOMEM; - } - - ret = aesni_mb_set_session_parameters(internals->mb_mgr, - sess_private_data, xform); - if (ret != 0) { - AESNI_MB_LOG(ERR, "failed configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sym_session_private_data(sess, dev->driver_id, - sess_private_data); - - return 0; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static void -aesni_mb_pmd_sym_session_clear(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess) -{ - uint8_t index = dev->driver_id; - void *sess_priv = get_sym_session_private_data(sess, index); - - /* Zero out the whole structure */ - if (sess_priv) { - memset(sess_priv, 0, sizeof(struct aesni_mb_session)); - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - set_sym_session_private_data(sess, index, NULL); - rte_mempool_put(sess_mp, sess_priv); - } -} - -struct rte_cryptodev_ops aesni_mb_pmd_ops = { - .dev_configure = aesni_mb_pmd_config, - .dev_start = aesni_mb_pmd_start, - .dev_stop = aesni_mb_pmd_stop, - .dev_close = aesni_mb_pmd_close, - - .stats_get = aesni_mb_pmd_stats_get, - .stats_reset = aesni_mb_pmd_stats_reset, - - .dev_infos_get = aesni_mb_pmd_info_get, - - .queue_pair_setup = aesni_mb_pmd_qp_setup, - .queue_pair_release = aesni_mb_pmd_qp_release, - - .sym_cpu_process = aesni_mb_cpu_crypto_process_bulk, - - .sym_session_get_size = aesni_mb_pmd_sym_session_get_size, - .sym_session_configure = aesni_mb_pmd_sym_session_configure, - .sym_session_clear = aesni_mb_pmd_sym_session_clear -}; - -struct rte_cryptodev_ops *rte_aesni_mb_pmd_ops = &aesni_mb_pmd_ops; - -#ifdef AESNI_MB_DOCSIS_SEC_ENABLED -/** - * Configure a aesni multi-buffer session from a security session - * configuration - */ -static int -aesni_mb_pmd_sec_sess_create(void *dev, struct rte_security_session_conf *conf, - struct rte_security_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - struct rte_cryptodev *cdev = (struct rte_cryptodev *)dev; - int ret; - - if (conf->action_type != RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL || - conf->protocol != RTE_SECURITY_PROTOCOL_DOCSIS) { - AESNI_MB_LOG(ERR, "Invalid security protocol"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - AESNI_MB_LOG(ERR, "Couldn't get object from session mempool"); - return -ENOMEM; - } - - ret = aesni_mb_set_docsis_sec_session_parameters(cdev, conf, - sess_private_data); - - if (ret != 0) { - AESNI_MB_LOG(ERR, "Failed to configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sec_session_private_data(sess, sess_private_data); - - return ret; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static int -aesni_mb_pmd_sec_sess_destroy(void *dev __rte_unused, - struct rte_security_session *sess) -{ - void *sess_priv = get_sec_session_private_data(sess); - - if (sess_priv) { - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - memset(sess_priv, 0, sizeof(struct aesni_mb_session)); - set_sec_session_private_data(sess, NULL); - rte_mempool_put(sess_mp, sess_priv); - } - return 0; -} - -/** Get security capabilities for aesni multi-buffer */ -static const struct rte_security_capability * -aesni_mb_pmd_sec_capa_get(void *device __rte_unused) -{ - return aesni_mb_pmd_security_cap; -} - -static struct rte_security_ops aesni_mb_pmd_sec_ops = { - .session_create = aesni_mb_pmd_sec_sess_create, - .session_update = NULL, - .session_stats_get = NULL, - .session_destroy = aesni_mb_pmd_sec_sess_destroy, - .set_pkt_metadata = NULL, - .capabilities_get = aesni_mb_pmd_sec_capa_get -}; - -struct rte_security_ops *rte_aesni_mb_pmd_sec_ops = &aesni_mb_pmd_sec_ops; -#endif +#endif /* _PMD_AESNI_MB_PRIV_H_ */ diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index 4d93dcd216..d409251214 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -7,7 +7,6 @@ endif drivers = [ 'aesni_gcm', - 'aesni_mb', 'armv8', 'bcmfs', 'caam_jr', From patchwork Fri Oct 15 14:39:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101775 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6B38FA0C45; Fri, 15 Oct 2021 16:40:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 89EC24127C; Fri, 15 Oct 2021 16:40:29 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 7A8AF4126A for ; Fri, 15 Oct 2021 16:40:27 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816143" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816143" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542812" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:23 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu Date: Fri, 15 Oct 2021 14:39:47 +0000 Message-Id: <20211015143957.842499-5-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 04/14] crypto/ipsec_mb: support ZUC-256 for aesni_mb X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Pablo de Lara Add support for ZUC-EEA3-256 and ZUC-EIA3-256. Only 4-byte tags supported for now. Signed-off-by: Pablo de Lara Acked-by: Fan Zhang --- doc/guides/cryptodevs/aesni_mb.rst | 1 + doc/guides/rel_notes/release_21_11.rst | 4 ++++ drivers/crypto/ipsec_mb/pmd_aesni_mb.c | 21 ++++++++++++++++----- drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h | 16 ++++++++-------- 4 files changed, 29 insertions(+), 13 deletions(-) diff --git a/doc/guides/cryptodevs/aesni_mb.rst b/doc/guides/cryptodevs/aesni_mb.rst index 3551a0dbd7..948128ae9b 100644 --- a/doc/guides/cryptodevs/aesni_mb.rst +++ b/doc/guides/cryptodevs/aesni_mb.rst @@ -77,6 +77,7 @@ Limitations protocol. * RTE_CRYPTO_CIPHER_DES_DOCSISBPI is not supported for combined Crypto-CRC DOCSIS security protocol. +* The only tag size supported for ZUC-EIA3-256 is 4 bytes. Installation diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 7628c32052..46f8a22c8d 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -106,6 +106,10 @@ New Features * AESNI_MB PMD. +* **Updated the aesni_mb crypto PMD.** + + * Added support for ZUC-EEA3-256 and ZUC-EIA3-256. + * **Updated Marvell cnxk ethdev driver.** * Added rte_flow support for dual VLAN insert and strip actions. diff --git a/drivers/crypto/ipsec_mb/pmd_aesni_mb.c b/drivers/crypto/ipsec_mb/pmd_aesni_mb.c index c9c4906722..c38d85a366 100644 --- a/drivers/crypto/ipsec_mb/pmd_aesni_mb.c +++ b/drivers/crypto/ipsec_mb/pmd_aesni_mb.c @@ -179,7 +179,15 @@ aesni_mb_set_session_auth_parameters(const IMB_MGR *mb_mgr, } if (xform->auth.algo == RTE_CRYPTO_AUTH_ZUC_EIA3) { - sess->auth.algo = IMB_AUTH_ZUC_EIA3_BITLEN; + if (xform->auth.key.length == 16) { + sess->auth.algo = IMB_AUTH_ZUC_EIA3_BITLEN; + } else if (xform->auth.key.length == 32) { + sess->auth.algo = IMB_AUTH_ZUC256_EIA3_BITLEN; + } else { + IPSEC_MB_LOG(ERR, "Invalid authentication key length\n"); + return -EINVAL; + } + uint16_t zuc_eia3_digest_len = get_truncated_digest_byte_length( IMB_AUTH_ZUC_EIA3_BITLEN); @@ -189,7 +197,8 @@ aesni_mb_set_session_auth_parameters(const IMB_MGR *mb_mgr, } sess->auth.gen_digest_len = sess->auth.req_digest_len; - memcpy(sess->auth.zuc_auth_key, xform->auth.key.data, 16); + memcpy(sess->auth.zuc_auth_key, xform->auth.key.data, + xform->auth.key.length); return 0; } else if (xform->auth.algo == RTE_CRYPTO_AUTH_SNOW3G_UIA2) { sess->auth.algo = IMB_AUTH_SNOW3G_UIA2_BITLEN; @@ -522,13 +531,14 @@ aesni_mb_set_session_cipher_parameters(const IMB_MGR *mb_mgr, sess->cipher.key_length_in_bytes = 24; } else if (is_zuc) { - if (xform->cipher.key.length != 16) { + if (xform->cipher.key.length != 16 && + xform->cipher.key.length != 32) { IPSEC_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } - sess->cipher.key_length_in_bytes = 16; + sess->cipher.key_length_in_bytes = xform->cipher.key.length; memcpy(sess->cipher.zuc_cipher_key, xform->cipher.key.data, - 16); + xform->cipher.key.length); } else if (is_snow3g) { if (xform->cipher.key.length != 16) { IPSEC_MB_LOG(ERR, "Invalid cipher key length"); @@ -1150,6 +1160,7 @@ set_mb_job_params(IMB_JOB *job, struct ipsec_mb_qp *qp, job->dec_keys = &session->cipher.gcm_key; break; case IMB_AUTH_ZUC_EIA3_BITLEN: + case IMB_AUTH_ZUC256_EIA3_BITLEN: job->u.ZUC_EIA3._key = session->auth.zuc_auth_key; job->u.ZUC_EIA3._iv = rte_crypto_op_ctod_offset(op, uint8_t *, session->auth_iv.offset); diff --git a/drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h b/drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h index db7d283ca2..d37cc787a0 100644 --- a/drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h +++ b/drivers/crypto/ipsec_mb/pmd_aesni_mb_priv.h @@ -526,8 +526,8 @@ static const struct rte_cryptodev_capabilities aesni_mb_capabilities[] = { .block_size = 16, .key_size = { .min = 16, - .max = 16, - .increment = 0 + .max = 32, + .increment = 16 }, .digest_size = { .min = 4, @@ -536,8 +536,8 @@ static const struct rte_cryptodev_capabilities aesni_mb_capabilities[] = { }, .iv_size = { .min = 16, - .max = 16, - .increment = 0 + .max = 25, + .increment = 9 } }, } }, } @@ -551,13 +551,13 @@ static const struct rte_cryptodev_capabilities aesni_mb_capabilities[] = { .block_size = 16, .key_size = { .min = 16, - .max = 16, - .increment = 0 + .max = 32, + .increment = 16 }, .iv_size = { .min = 16, - .max = 16, - .increment = 0 + .max = 25, + .increment = 9 }, }, } }, } From patchwork Fri Oct 15 14:39:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101776 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 285BDA0C45; Fri, 15 Oct 2021 16:40:57 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9101241271; Fri, 15 Oct 2021 16:40:30 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 871F54126A for ; Fri, 15 Oct 2021 16:40:28 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816151" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816151" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542819" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:25 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Declan Doherty Date: Fri, 15 Oct 2021 14:39:48 +0000 Message-Id: <20211015143957.842499-6-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 05/14] test/crypto: check cipher parameters X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Pablo de Lara Check for cipher parameters in the transform to verify if a test case is supported by the crypto device under test. Signed-off-by: Pablo de Lara Acked-by: Fan Zhang --- app/test/test_cryptodev.c | 55 +++++++++++++++++++++++++++++---------- 1 file changed, 41 insertions(+), 14 deletions(-) diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c index 65b64e1af0..2c88c03337 100644 --- a/app/test/test_cryptodev.c +++ b/app/test/test_cryptodev.c @@ -5827,6 +5827,33 @@ test_kasumi_cipher_auth(const struct kasumi_test_data *tdata) return 0; } +static int +check_cipher_capability(const struct crypto_testsuite_params *ts_params, + const enum rte_crypto_cipher_algorithm cipher_algo, + const uint16_t key_size, const uint16_t iv_size) +{ + struct rte_cryptodev_sym_capability_idx cap_idx; + const struct rte_cryptodev_symmetric_capability *cap; + + /* Check if device supports the algorithm */ + cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER; + cap_idx.algo.cipher = cipher_algo; + + cap = rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], + &cap_idx); + + if (cap == NULL) + return -1; + + /* Check if device supports key size and IV size */ + if (rte_cryptodev_sym_capability_check_cipher(cap, key_size, + iv_size) < 0) { + return -1; + } + + return 0; +} + static int test_zuc_encryption(const struct wireless_test_data *tdata) { @@ -5851,14 +5878,9 @@ test_zuc_encryption(const struct wireless_test_data *tdata) if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO) return TEST_SKIPPED; - struct rte_cryptodev_sym_capability_idx cap_idx; - /* Check if device supports ZUC EEA3 */ - cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - cap_idx.algo.cipher = RTE_CRYPTO_CIPHER_ZUC_EEA3; - - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + if (check_cipher_capability(ts_params, RTE_CRYPTO_CIPHER_ZUC_EEA3, + tdata->key.len, tdata->cipher_iv.len) < 0) return TEST_SKIPPED; /* Create ZUC session */ @@ -5933,14 +5955,9 @@ test_zuc_encryption_sgl(const struct wireless_test_data *tdata) uint8_t ciphertext_buffer[2048]; struct rte_cryptodev_info dev_info; - struct rte_cryptodev_sym_capability_idx cap_idx; - /* Check if device supports ZUC EEA3 */ - cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - cap_idx.algo.cipher = RTE_CRYPTO_CIPHER_ZUC_EEA3; - - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + if (check_cipher_capability(ts_params, RTE_CRYPTO_CIPHER_ZUC_EEA3, + tdata->key.len, tdata->cipher_iv.len) < 0) return TEST_SKIPPED; if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO) @@ -6134,6 +6151,11 @@ test_zuc_auth_cipher(const struct wireless_test_data *tdata, struct rte_cryptodev_info dev_info; struct rte_cryptodev_sym_capability_idx cap_idx; + /* Check if device supports ZUC EEA3 */ + if (check_cipher_capability(ts_params, RTE_CRYPTO_CIPHER_ZUC_EEA3, + tdata->key.len, tdata->cipher_iv.len) < 0) + return TEST_SKIPPED; + /* Check if device supports ZUC EIA3 */ cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; cap_idx.algo.auth = RTE_CRYPTO_AUTH_ZUC_EIA3; @@ -6332,6 +6354,11 @@ test_zuc_auth_cipher_sgl(const struct wireless_test_data *tdata, struct rte_cryptodev_info dev_info; struct rte_cryptodev_sym_capability_idx cap_idx; + /* Check if device supports ZUC EEA3 */ + if (check_cipher_capability(ts_params, RTE_CRYPTO_CIPHER_ZUC_EEA3, + tdata->key.len, tdata->cipher_iv.len) < 0) + return TEST_SKIPPED; + /* Check if device supports ZUC EIA3 */ cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; cap_idx.algo.auth = RTE_CRYPTO_AUTH_ZUC_EIA3; From patchwork Fri Oct 15 14:39:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101777 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07487A0C45; Fri, 15 Oct 2021 16:41:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9A22441269; Fri, 15 Oct 2021 16:40:32 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 91BAF41285 for ; Fri, 15 Oct 2021 16:40:30 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816156" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816156" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542850" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:27 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Declan Doherty Date: Fri, 15 Oct 2021 14:39:49 +0000 Message-Id: <20211015143957.842499-7-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 06/14] test/crypto: check auth parameters X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Pablo de Lara Check for auth parameters in the transform to verify if a test case is supported by the crypto device under test. Signed-off-by: Pablo de Lara Acked-by: Fan Zhang --- app/test/test_cryptodev.c | 55 ++++++++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 18 deletions(-) diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c index 2c88c03337..4eab8269e2 100644 --- a/app/test/test_cryptodev.c +++ b/app/test/test_cryptodev.c @@ -5854,6 +5854,34 @@ check_cipher_capability(const struct crypto_testsuite_params *ts_params, return 0; } +static int +check_auth_capability(const struct crypto_testsuite_params *ts_params, + const enum rte_crypto_auth_algorithm auth_algo, + const uint16_t key_size, const uint16_t iv_size, + const uint16_t tag_size) +{ + struct rte_cryptodev_sym_capability_idx cap_idx; + const struct rte_cryptodev_symmetric_capability *cap; + + /* Check if device supports the algorithm */ + cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; + cap_idx.algo.auth = auth_algo; + + cap = rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], + &cap_idx); + + if (cap == NULL) + return -1; + + /* Check if device supports key size and IV size */ + if (rte_cryptodev_sym_capability_check_auth(cap, key_size, + tag_size, iv_size) < 0) { + return -1; + } + + return 0; +} + static int test_zuc_encryption(const struct wireless_test_data *tdata) { @@ -6051,7 +6079,6 @@ test_zuc_authentication(const struct wireless_test_data *tdata) unsigned plaintext_len; uint8_t *plaintext; - struct rte_cryptodev_sym_capability_idx cap_idx; struct rte_cryptodev_info dev_info; rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info); @@ -6073,11 +6100,9 @@ test_zuc_authentication(const struct wireless_test_data *tdata) return TEST_SKIPPED; /* Check if device supports ZUC EIA3 */ - cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; - cap_idx.algo.auth = RTE_CRYPTO_AUTH_ZUC_EIA3; - - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + if (check_auth_capability(ts_params, RTE_CRYPTO_AUTH_ZUC_EIA3, + tdata->key.len, tdata->auth_iv.len, + tdata->digest.len) < 0) return TEST_SKIPPED; /* Create ZUC session */ @@ -6149,7 +6174,6 @@ test_zuc_auth_cipher(const struct wireless_test_data *tdata, unsigned int ciphertext_len; struct rte_cryptodev_info dev_info; - struct rte_cryptodev_sym_capability_idx cap_idx; /* Check if device supports ZUC EEA3 */ if (check_cipher_capability(ts_params, RTE_CRYPTO_CIPHER_ZUC_EEA3, @@ -6157,11 +6181,9 @@ test_zuc_auth_cipher(const struct wireless_test_data *tdata, return TEST_SKIPPED; /* Check if device supports ZUC EIA3 */ - cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; - cap_idx.algo.auth = RTE_CRYPTO_AUTH_ZUC_EIA3; - - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + if (check_auth_capability(ts_params, RTE_CRYPTO_AUTH_ZUC_EIA3, + tdata->key.len, tdata->auth_iv.len, + tdata->digest.len) < 0) return TEST_SKIPPED; rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info); @@ -6352,7 +6374,6 @@ test_zuc_auth_cipher_sgl(const struct wireless_test_data *tdata, uint8_t digest_buffer[10000]; struct rte_cryptodev_info dev_info; - struct rte_cryptodev_sym_capability_idx cap_idx; /* Check if device supports ZUC EEA3 */ if (check_cipher_capability(ts_params, RTE_CRYPTO_CIPHER_ZUC_EEA3, @@ -6360,11 +6381,9 @@ test_zuc_auth_cipher_sgl(const struct wireless_test_data *tdata, return TEST_SKIPPED; /* Check if device supports ZUC EIA3 */ - cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; - cap_idx.algo.auth = RTE_CRYPTO_AUTH_ZUC_EIA3; - - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + if (check_auth_capability(ts_params, RTE_CRYPTO_AUTH_ZUC_EIA3, + tdata->key.len, tdata->auth_iv.len, + tdata->digest.len) < 0) return TEST_SKIPPED; rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info); From patchwork Fri Oct 15 14:39:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101778 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 63D82A0C45; Fri, 15 Oct 2021 16:41:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B21C741285; Fri, 15 Oct 2021 16:40:34 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id A562541285 for ; Fri, 15 Oct 2021 16:40:32 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816161" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816161" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542867" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:30 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Declan Doherty Date: Fri, 15 Oct 2021 14:39:50 +0000 Message-Id: <20211015143957.842499-8-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 07/14] test/crypto: add ZUC-256 vectors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Pablo de Lara Add extra ZUC-EIA3-256 and ZUC-EEA3-256 test vectors. Signed-off-by: Pablo de Lara Acked-by: Fan Zhang --- app/test/test_cryptodev.c | 37 +++ app/test/test_cryptodev_zuc_test_vectors.h | 286 +++++++++++++++++++++ 2 files changed, 323 insertions(+) diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c index 4eab8269e2..17444d8fb7 100644 --- a/app/test/test_cryptodev.c +++ b/app/test/test_cryptodev.c @@ -7195,6 +7195,30 @@ test_zuc_auth_cipher_verify_test_case_1_oop_sgl(void) &zuc_auth_cipher_test_case_1, OUT_OF_PLACE, 1); } +static int +test_zuc256_encryption_test_case_1(void) +{ + return test_zuc_encryption(&zuc256_test_case_cipher_1); +} + +static int +test_zuc256_encryption_test_case_2(void) +{ + return test_zuc_encryption(&zuc256_test_case_cipher_2); +} + +static int +test_zuc256_authentication_test_case_1(void) +{ + return test_zuc_authentication(&zuc256_test_case_auth_1); +} + +static int +test_zuc256_authentication_test_case_2(void) +{ + return test_zuc_authentication(&zuc256_test_case_auth_2); +} + static int test_mixed_check_if_unsupported(const struct mixed_cipher_auth_test_data *tdata) { @@ -14844,6 +14868,19 @@ static struct unit_test_suite cryptodev_zuc_testsuite = { test_zuc_auth_cipher_verify_test_case_1_sgl), TEST_CASE_ST(ut_setup, ut_teardown, test_zuc_auth_cipher_verify_test_case_1_oop_sgl), + + /** ZUC-256 encrypt only **/ + TEST_CASE_ST(ut_setup, ut_teardown, + test_zuc256_encryption_test_case_1), + TEST_CASE_ST(ut_setup, ut_teardown, + test_zuc256_encryption_test_case_2), + + /** ZUC-256 authentication only **/ + TEST_CASE_ST(ut_setup, ut_teardown, + test_zuc256_authentication_test_case_1), + TEST_CASE_ST(ut_setup, ut_teardown, + test_zuc256_authentication_test_case_2), + TEST_CASES_END() } }; diff --git a/app/test/test_cryptodev_zuc_test_vectors.h b/app/test/test_cryptodev_zuc_test_vectors.h index 2e330bc8d3..35bbe0f6c4 100644 --- a/app/test/test_cryptodev_zuc_test_vectors.h +++ b/app/test/test_cryptodev_zuc_test_vectors.h @@ -1299,4 +1299,290 @@ struct wireless_test_data zuc_auth_cipher_test_case_1 = { } }; +/** ZUC-256 vectors **/ +static struct wireless_test_data zuc256_test_case_cipher_1 = { + .key = { + .data = { + 0xf7, 0xb4, 0x04, 0x5a, 0x81, 0x5c, 0x1b, 0x01, + 0x82, 0xf9, 0xf4, 0x26, 0x80, 0xd4, 0x56, 0x26, + 0xd5, 0xf7, 0x4b, 0x68, 0x48, 0x6b, 0x92, 0x6a, + 0x34, 0x1f, 0x86, 0x66, 0x60, 0x0a, 0xfc, 0x57 + }, + .len = 32 + }, + .cipher_iv = { + .data = { + 0x8e, 0x5d, 0xbc, 0x3f, 0xb9, 0xae, 0x66, 0xa3, + 0xb9, 0x5c, 0x12, 0x14, 0xdb, 0xc5, 0xbc, 0x18, + 0x48, 0x92, 0x09, 0x86, 0x65, 0xb3, 0x2e, 0x92, + 0x12 + }, + .len = 25 + }, + .plaintext = { + .data = { + 0x36, 0xdb, 0x63, 0x68, 0xb5, 0x1f, 0x4e, 0x92, + 0x46, 0x1f, 0xde, 0xdb, 0xc2, 0xec, 0xfa, 0x7e, + 0x49, 0x85, 0x77, 0xaa, 0x46, 0x98, 0x30, 0x2d, + 0x3b, 0xc4, 0x11, 0x24, 0x98, 0x20, 0xa9, 0xce, + 0xfb, 0x0d, 0x36, 0xb0, 0x2c, 0x85, 0x42, 0x72, + 0xa4, 0x21, 0x4e, 0x66, 0x0d, 0x48, 0xe4, 0x57, + 0xce, 0x5b, 0x01, 0x14, 0xf3, 0x31, 0x42, 0x2e, + 0xf5, 0x53, 0x52, 0x8d, 0x73, 0xfc, 0x5c, 0x6e, + 0x09, 0x92, 0x1e, 0x35, 0x17, 0x60, 0xa8, 0xbb, + 0x81, 0xf6, 0x21, 0x8f, 0x3e, 0x05, 0xe6, 0x0c, + 0x60, 0xe7, 0x21, 0x53, 0x18, 0x63, 0x81, 0x0d, + 0xb6, 0xd4, 0x9a, 0x29, 0xd0, 0xf6, 0x97, 0xd9, + 0x89, 0xb5, 0x0e, 0xa0, 0x15, 0xb6, 0x5c, 0x97, + 0xac, 0x7d, 0x26, 0xeb, 0x83, 0x0c, 0xf7, 0xe3, + 0xf3, 0x18, 0x37, 0x0b, 0x7b, 0xb8, 0x18, 0x31, + 0x8c, 0xb2, 0x5a, 0x5c, 0xa9, 0xf1, 0x35, 0x32 + }, + .len = 1024 + }, + .ciphertext = { + .data = { + 0xa6, 0xe5, 0x71, 0x58, 0x5c, 0xcf, 0x5d, 0x0d, + 0x59, 0xb5, 0x51, 0xab, 0xf5, 0xfa, 0x31, 0xf9, + 0x8d, 0x4f, 0xf0, 0x3c, 0x7d, 0x61, 0x8d, 0x7a, + 0x6b, 0xcb, 0x2c, 0x79, 0xca, 0x99, 0x06, 0x6f, + 0xff, 0x5d, 0x12, 0x5f, 0x0e, 0x7a, 0x33, 0x6b, + 0x51, 0xbc, 0x58, 0x53, 0xff, 0xbd, 0x85, 0xc9, + 0xac, 0x5f, 0x33, 0xc2, 0xa2, 0xf1, 0x17, 0x7a, + 0xd9, 0x3f, 0x81, 0x82, 0x2f, 0x0a, 0xb0, 0xaf, + 0xb9, 0x19, 0x3b, 0xfa, 0xcd, 0xa4, 0x06, 0x81, + 0x2a, 0x7a, 0xbf, 0x2c, 0x07, 0xde, 0xc1, 0xa4, + 0x8c, 0x15, 0x85, 0x81, 0xa6, 0xd3, 0x73, 0x1c, + 0x29, 0x0b, 0xee, 0x3c, 0x57, 0xfa, 0x82, 0xad, + 0x6f, 0xe0, 0xa1, 0x54, 0x8d, 0xa4, 0x92, 0x29, + 0xf4, 0xfa, 0x6d, 0x01, 0xe3, 0x6c, 0xb9, 0x76, + 0x80, 0x53, 0xbb, 0x27, 0xb8, 0x18, 0x47, 0x6c, + 0xae, 0xb5, 0x44, 0x60, 0x43, 0x9d, 0xa7, 0x3f + }, + .len = 1024 + }, + .validDataLenInBits = { + .len = 1024 + }, + .validCipherLenInBits = { + .len = 1024 + } +}; + +static struct wireless_test_data zuc256_test_case_cipher_2 = { + .key = { + .data = { + 0x1d, 0x0f, 0x0e, 0x75, 0x86, 0xb3, 0xfc, 0x65, + 0x94, 0xbf, 0xaa, 0xa8, 0xf5, 0xd0, 0x0f, 0xe8, + 0x14, 0x7a, 0x96, 0x61, 0x15, 0x49, 0x79, 0x71, + 0x13, 0x82, 0xb4, 0xae, 0x34, 0x04, 0x75, 0x51 + }, + .len = 32 + }, + .cipher_iv = { + .data = { + 0x98, 0xcc, 0x89, 0x9f, 0xaf, 0x6d, 0x64, 0xb6, + 0xb1, 0xe8, 0x21, 0x72, 0xee, 0xb6, 0xcc, 0xe3, + 0xcf, 0xf2, 0x68, 0x61, 0xe1, 0x0d, 0x9e, 0xdc, + 0x74 + }, + .len = 25 + }, + .plaintext = { + .data = { + 0xa4, 0xcb, 0x6e, 0x76, 0x99, 0xfb, 0x0c, 0xab, + 0x6d, 0x57, 0xb1, 0x69, 0xc0, 0x47, 0x80, 0x63, + 0x00, 0xe1, 0xf9, 0x51, 0x10, 0xbe, 0xc0, 0x0f, + 0x99, 0x62, 0x2d, 0x71, 0xca, 0x75, 0xa0, 0x6e, + 0x41, 0x0e, 0xe4, 0xda, 0x09, 0xf1, 0x86, 0x76, + 0x48, 0x37, 0xe0, 0x08, 0x7e, 0x60, 0x6c, 0x7f, + 0x41, 0x65, 0xd0, 0x51, 0x24, 0x91, 0x61, 0xbd, + 0xf3, 0x8e, 0x2e, 0xbd, 0x04, 0xce, 0x2b, 0x45, + 0xdc, 0x0f, 0x1f, 0xe5, 0x00, 0xa5, 0x5c, 0x48, + 0xdd, 0x3c, 0x51, 0x5b, 0x9c, 0xbd, 0xda, 0xde, + 0x22, 0xab, 0x2f, 0x46, 0x3c, 0x90, 0x03, 0x2f, + 0x1f, 0x31, 0xec, 0x23, 0xff, 0x17, 0x68, 0xdb, + 0x26, 0x87, 0xc1, 0x27, 0x2d, 0x1d, 0x6f, 0x0a, + 0x59, 0xc0, 0x65, 0xf5, 0x7d, 0x40, 0xd3, 0xa0, + 0xeb, 0x03, 0xe6, 0x27, 0x93, 0xea, 0x56, 0xb2, + 0x1b, 0x42, 0xd5, 0x1b, 0x59, 0x3d, 0xf6, 0x7f, + 0xc5, 0xb7, 0xa6, 0xf2, 0xd4, 0x16, 0xfc, 0x2d, + 0xd6, 0x61, 0x23, 0x54, 0xa1, 0xf6, 0xf4, 0x8c, + 0xf9, 0xda, 0xb3, 0x8d, 0xc4, 0x09, 0x3f, 0xe0, + 0x4b, 0x15, 0xfb, 0xa4, 0x52, 0xf1, 0x24, 0x17, + 0xa9, 0xca, 0x09, 0x7d, 0xe0, 0x05, 0xab, 0xb7, + 0x67, 0xce, 0x0b, 0x08, 0xc4, 0xff, 0x95, 0xbe, + 0xd9, 0x48, 0x4b, 0x9e, 0x52, 0x8a, 0x7e, 0x9d, + 0x9f, 0x79, 0x42, 0xf2, 0x6a, 0x66, 0x09, 0x13, + 0x30, 0x13, 0x91, 0x11, 0x18, 0x3c, 0xc8, 0x7f, + 0x0a, 0xd3, 0x88, 0xce, 0xd2, 0x1d, 0x8c, 0xab, + 0x65, 0xd7, 0x49, 0xb7, 0x62, 0xc7, 0x55, 0x01, + 0x40, 0x97, 0xf3, 0xab, 0xfd, 0xfd, 0xbe, 0x2d, + 0x10, 0x4f, 0x3e, 0x28, 0x8b, 0x06, 0xa8, 0x95, + 0xd9, 0x30, 0x64, 0xab, 0x4d, 0xf0, 0x57, 0xb2, + 0xc8 + }, + .len = 1928 + }, + .ciphertext = { + .data = { + 0xd0, 0xf9, 0xff, 0xce, 0x03, 0x81, 0x14, 0x9c, + 0xd5, 0xf2, 0xbf, 0xe5, 0xff, 0xc8, 0x15, 0x4a, + 0x9c, 0x06, 0x2b, 0x17, 0x99, 0xe3, 0x48, 0x70, + 0x37, 0x01, 0x5e, 0x24, 0x80, 0x9a, 0x46, 0x4e, + 0xa8, 0xc0, 0x59, 0xd7, 0x03, 0x74, 0x28, 0x91, + 0x79, 0xb4, 0xb5, 0xd6, 0x52, 0x92, 0x04, 0x77, + 0x5b, 0x4f, 0x34, 0xd1, 0xbe, 0xaa, 0x74, 0xd9, + 0x01, 0x40, 0x24, 0xc7, 0x8c, 0x62, 0x2a, 0x51, + 0x5a, 0x58, 0x0e, 0xc8, 0x70, 0x12, 0x06, 0x1c, + 0x62, 0x7f, 0xf5, 0x23, 0xcb, 0x3c, 0xc1, 0xbe, + 0x8b, 0x7f, 0x9d, 0x12, 0xb8, 0x26, 0xc8, 0xa3, + 0x77, 0x7e, 0x83, 0xda, 0x83, 0xe1, 0x9f, 0xef, + 0x33, 0x62, 0x17, 0xa7, 0x74, 0x68, 0x34, 0x5e, + 0x16, 0xcc, 0xbc, 0x6c, 0x33, 0x2f, 0x73, 0xf0, + 0xfc, 0xe5, 0x2c, 0x2d, 0xfb, 0x81, 0xbe, 0x1e, + 0x6e, 0x4f, 0xf4, 0x14, 0x37, 0x7c, 0x97, 0xac, + 0xa9, 0xac, 0x68, 0x95, 0xf3, 0x55, 0xb3, 0xfb, + 0xf6, 0x64, 0xd9, 0x1b, 0xe1, 0x54, 0x79, 0x6e, + 0xfa, 0x21, 0xa4, 0x19, 0x9f, 0xb4, 0x4b, 0xb7, + 0xef, 0x52, 0xd8, 0x44, 0x75, 0x99, 0x07, 0x6d, + 0xa9, 0xcf, 0x32, 0xc5, 0xc1, 0x31, 0x0c, 0xa8, + 0x86, 0x40, 0x75, 0xeb, 0x12, 0xcf, 0x26, 0x5c, + 0x5f, 0xa3, 0x3c, 0xb6, 0x12, 0x45, 0xf3, 0x0a, + 0x38, 0x09, 0xa8, 0x36, 0x32, 0x4a, 0x2f, 0xad, + 0x50, 0x11, 0x38, 0xba, 0x8f, 0xdd, 0xd1, 0x58, + 0xd7, 0x3d, 0x3a, 0x40, 0x7c, 0x3f, 0xa7, 0x98, + 0xf3, 0x12, 0x7f, 0x9f, 0x89, 0xcf, 0x48, 0x58, + 0x01, 0xeb, 0x98, 0x7c, 0x59, 0x11, 0x9f, 0x57, + 0x74, 0x5f, 0x70, 0x72, 0x74, 0xa4, 0x82, 0x3c, + 0x36, 0xe6, 0x31, 0x9e, 0xba, 0x7b, 0x53, 0xfc, + 0x56 + }, + .len = 1928 + }, + .validDataLenInBits = { + .len = 1928 + }, + .validCipherLenInBits = { + .len = 1928 + } +}; + +static struct wireless_test_data zuc256_test_case_auth_1 = { + .key = { + .data = { + 0xe3, 0x8e, 0xaf, 0x08, 0xde, 0x8c, 0x08, 0x41, + 0x7f, 0x2b, 0x97, 0x20, 0x10, 0x87, 0xc7, 0xf7, + 0xbe, 0x3c, 0xd2, 0x68, 0x80, 0x10, 0x1e, 0x71, + 0xfd, 0xb2, 0xbb, 0xad, 0x25, 0x0f, 0x06, 0x08 + }, + .len = 32 + }, + .auth_iv = { + .data = { + 0xf5, 0x8d, 0x08, 0x26, 0x94, 0x14, 0xc7, 0x4d, + 0xf5, 0x7c, 0x9c, 0xaa, 0x45, 0x53, 0xfd, 0x85, + 0x23, 0x8b, 0xc0, 0x4e, 0xa6, 0x2b, 0xcf, 0x41, + 0xe6 + }, + .len = 25 + }, + .plaintext = { + .data = { + 0x08, 0xba, 0x8d, 0xf1, 0xf8, 0x62, 0xa6, 0xaf, + 0xf9, 0x03, 0x88, 0x9c, 0xa3, 0x68, 0x6b, 0x87, + 0xb6, 0x92, 0xd1, 0x47, 0x3e, 0x54, 0xaf, 0x46, + 0x07, 0x8f, 0x89, 0xea, 0x26, 0x9d, 0x0e, 0x2f, + 0x57, 0x9b, 0x20, 0x4f, 0xfe, 0xc7, 0xfe, 0xf7, + 0xca, 0x86, 0x93, 0x6d, 0xee + }, + .len = 360 + }, + .validAuthLenInBits = { + .len = 360 + }, + .digest = { + .data = { 0x58, 0x19, 0xab, 0xa5 }, + .len = 4 + } +}; + +static struct wireless_test_data zuc256_test_case_auth_2 = { + .key = { + .data = { + 0x6a, 0x7e, 0x4c, 0x7e, 0x51, 0x25, 0xb3, 0x48, + 0x84, 0x53, 0x3a, 0x94, 0xfb, 0x31, 0x99, 0x90, + 0x32, 0x57, 0x44, 0xee, 0x9b, 0xbc, 0xe9, 0xe5, + 0x25, 0xcf, 0x08, 0xf5, 0xe9, 0xe2, 0x5e, 0x53 + }, + .len = 32 + }, + .auth_iv = { + .data = { + 0x60, 0xaa, 0xd2, 0xb2, 0xd0, 0x85, 0xfa, 0x54, + 0xd8, 0x35, 0xe8, 0xd4, 0x66, 0x82, 0x64, 0x98, + 0xd9, 0x2a, 0x08, 0x1d, 0x35, 0x19, 0x17, 0x01, + 0x1A + }, + .len = 25 + }, + .plaintext = { + .data = { + 0xc6, 0x69, 0x73, 0x51, 0xff, 0x4a, 0xec, 0x29, + 0xcd, 0xba, 0xab, 0xf2, 0xfb, 0xe3, 0x46, 0x7c, + 0xc2, 0x54, 0xf8, 0x1b, 0xe8, 0xe7, 0x8d, 0x76, + 0x5a, 0x2e, 0x63, 0x33, 0x9f, 0xc9, 0x9a, 0x66, + 0x32, 0x0d, 0xb7, 0x31, 0x58, 0xa3, 0x5a, 0x25, + 0x5d, 0x05, 0x17, 0x58, 0xe9, 0x5e, 0xd4, 0xab, + 0xb2, 0xcd, 0xc6, 0x9b, 0xb4, 0x54, 0x11, 0x0e, + 0x82, 0x74, 0x41, 0x21, 0x3d, 0xdc, 0x87, 0x70, + 0xe9, 0x3e, 0xa1, 0x41, 0xe1, 0xfc, 0x67, 0x3e, + 0x01, 0x7e, 0x97, 0xea, 0xdc, 0x6b, 0x96, 0x8f, + 0x38, 0x5c, 0x2a, 0xec, 0xb0, 0x3b, 0xfb, 0x32, + 0xaf, 0x3c, 0x54, 0xec, 0x18, 0xdb, 0x5c, 0x02, + 0x1a, 0xfe, 0x43, 0xfb, 0xfa, 0xaa, 0x3a, 0xfb, + 0x29, 0xd1, 0xe6, 0x05, 0x3c, 0x7c, 0x94, 0x75, + 0xd8, 0xbe, 0x61, 0x89, 0xf9, 0x5c, 0xbb, 0xa8, + 0x99, 0x0f, 0x95, 0xb1, 0xeb, 0xf1, 0xb3, 0x05, + 0xef, 0xf7, 0x00, 0xe9, 0xa1, 0x3a, 0xe5, 0xca, + 0x0b, 0xcb, 0xd0, 0x48, 0x47, 0x64, 0xbd, 0x1f, + 0x23, 0x1e, 0xa8, 0x1c, 0x7b, 0x64, 0xc5, 0x14, + 0x73, 0x5a, 0xc5, 0x5e, 0x4b, 0x79, 0x63, 0x3b, + 0x70, 0x64, 0x24, 0x11, 0x9e, 0x09, 0xdc, 0xaa, + 0xd4, 0xac, 0xf2, 0x1b, 0x10, 0xaf, 0x3b, 0x33, + 0xcd, 0xe3, 0x50, 0x48, 0x47, 0x15, 0x5c, 0xbb, + 0x6f, 0x22, 0x19, 0xba, 0x9b, 0x7d, 0xf5, 0x0b, + 0xe1, 0x1a, 0x1c, 0x7f, 0x23, 0xf8, 0x29, 0xf8, + 0xa4, 0x1b, 0x13, 0xb5, 0xca, 0x4e, 0xe8, 0x98, + 0x32, 0x38, 0xe0, 0x79, 0x4d, 0x3d, 0x34, 0xbc, + 0x5f, 0x4e, 0x77, 0xfa, 0xcb, 0x6c, 0x05, 0xac, + 0x86, 0x21, 0x2b, 0xaa, 0x1a, 0x55, 0xa2, 0xbe, + 0x70, 0xb5, 0x73, 0x3b, 0x04, 0x5c, 0xd3, 0x36, + 0x94, 0xb3, 0xaf, 0xe2, 0xf0, 0xe4, 0x9e, 0x4f, + 0x32, 0x15, 0x49, 0xfd, 0x82, 0x4e, 0xa9, 0x08, + 0x70, 0xd4, 0xb2, 0x8a, 0x29, 0x54, 0x48, 0x9a, + 0x0a, 0xbc, 0xd5, 0x0e, 0x18, 0xa8, 0x44, 0xac, + 0x5b, 0xf3, 0x8e, 0x4c, 0xd7, 0x2d, 0x9b, 0x09, + 0x42, 0xe5, 0x06, 0xc4, 0x33, 0xaf, 0xcd, 0xa3, + 0x84, 0x7f, 0x2d, 0xad, 0xd4, 0x76, 0x47, 0xde, + 0x32, 0x1c, 0xec, 0x4a, 0xc4, 0x30, 0xf6, 0x20, + 0x23, 0x85, 0x6c, 0xfb, 0xb2, 0x07, 0x04, 0xf4, + 0xec, 0x0b, 0xb9, 0x20, 0xba, 0x86, 0xc3, 0x3e, + 0x05, 0xf1, 0xec, 0xd9, 0x67, 0x33, 0xb7, 0x99, + 0x50, 0xa3, 0xe3, 0x14, 0xd3, 0xd9, 0x34, 0xf7, + 0x5e, 0xa0, 0xf2, 0x10, 0xa8, 0xf6, 0x05, 0x94, + 0x01, 0xbe, 0xb4, 0xbc, 0x44, 0x78, 0xfa, 0x49, + 0x69, 0xe6, 0x23, 0xd0, 0x1a, 0xda, 0x69 + }, + .len = 2872 + }, + .validAuthLenInBits = { + .len = 2872 + }, + .digest = { + .data = { 0xd1, 0x1e, 0x33, 0xf6 }, + .len = 4 + } +}; + #endif /* TEST_CRYPTODEV_ZUC_TEST_VECTORS_H_ */ From patchwork Fri Oct 15 14:39:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101779 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 45830A0C45; Fri, 15 Oct 2021 16:41:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 301CC4128E; Fri, 15 Oct 2021 16:40:38 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id CEA6D41290 for ; Fri, 15 Oct 2021 16:40:35 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816169" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816169" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542875" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:32 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Ciara Power , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:51 +0000 Message-Id: <20211015143957.842499-9-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 08/14] drivers/crypto: move aesni-gcm PMD to IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch removes the crypto/aesni_gcm folder and gathers all aesni-gcm PMD implementation specific details into a single file, pmd_aesni_gcm.c in the crypto/ipsec_mb folder. A redundant check for iv length is removed. GCM ops are stored in the queue pair for multi process support, they are updated during queue pair setup for both primary and secondary processes. GCM ops are also set per lcore for the CPU crypto mode. Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - Fixed some function naming. - Added release note. - Fixed copyright date. - Split into private header file. - Remove security includes. v3: - Moved session GCM ops to queue pair. - Added GCM ops per lcore. - Fixed some formatting. v2: - Fixed enum for GCM key length. - Updated maintainers file. --- MAINTAINERS | 9 +- doc/guides/cryptodevs/aesni_gcm.rst | 4 +- doc/guides/rel_notes/release_21_11.rst | 1 + drivers/crypto/aesni_gcm/aesni_gcm_ops.h | 104 -- drivers/crypto/aesni_gcm/aesni_gcm_pmd.c | 984 ------------------ drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c | 333 ------ .../crypto/aesni_gcm/aesni_gcm_pmd_private.h | 123 --- drivers/crypto/aesni_gcm/meson.build | 24 - drivers/crypto/aesni_gcm/version.map | 3 - drivers/crypto/ipsec_mb/ipsec_mb_private.h | 7 + drivers/crypto/ipsec_mb/meson.build | 3 +- drivers/crypto/ipsec_mb/pmd_aesni_gcm.c | 837 +++++++++++++++ drivers/crypto/ipsec_mb/pmd_aesni_gcm_priv.h | 169 +++ drivers/crypto/meson.build | 1 - 14 files changed, 1021 insertions(+), 1581 deletions(-) delete mode 100644 drivers/crypto/aesni_gcm/aesni_gcm_ops.h delete mode 100644 drivers/crypto/aesni_gcm/aesni_gcm_pmd.c delete mode 100644 drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c delete mode 100644 drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h delete mode 100644 drivers/crypto/aesni_gcm/meson.build delete mode 100644 drivers/crypto/aesni_gcm/version.map create mode 100644 drivers/crypto/ipsec_mb/pmd_aesni_gcm.c create mode 100644 drivers/crypto/ipsec_mb/pmd_aesni_gcm_priv.h diff --git a/MAINTAINERS b/MAINTAINERS index 9aeb0d8e03..af062cf00d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1044,13 +1044,6 @@ M: Fan Zhang F: drivers/crypto/scheduler/ F: doc/guides/cryptodevs/scheduler.rst -Intel AES-NI GCM -M: Declan Doherty -M: Pablo de Lara -F: drivers/crypto/aesni_gcm/ -F: doc/guides/cryptodevs/aesni_gcm.rst -F: doc/guides/cryptodevs/features/aesni_gcm.ini - Intel QuickAssist M: John Griffin M: Fiona Trahe @@ -1064,7 +1057,9 @@ IPsec MB M: Fan Zhang M: Pablo de Lara F: drivers/crypto/ipsec_mb/ +F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/aesni_mb.rst +F: doc/guides/cryptodevs/features/aesni_gcm.ini F: doc/guides/cryptodevs/features/aesni_mb.ini KASUMI diff --git a/doc/guides/cryptodevs/aesni_gcm.rst b/doc/guides/cryptodevs/aesni_gcm.rst index 11b23958d5..bbe9d99840 100644 --- a/doc/guides/cryptodevs/aesni_gcm.rst +++ b/doc/guides/cryptodevs/aesni_gcm.rst @@ -83,7 +83,9 @@ and the external crypto libraries supported by them: 17.02 - 17.05 ISA-L Crypto v2.18 17.08 - 18.02 Multi-buffer library 0.46 - 0.48 18.05 - 19.02 Multi-buffer library 0.49 - 0.52 - 19.05+ Multi-buffer library 0.52 - 1.0* + 19.05 - 20.08 Multi-buffer library 0.52 - 0.55 + 20.11 - 21.08 Multi-buffer library 0.53 - 1.0* + 21.11+ Multi-buffer library 1.0* ============= ================================ \* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 46f8a22c8d..43814a2320 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -105,6 +105,7 @@ New Features however their usage and EAL options remain unchanged. * AESNI_MB PMD. + * AESNI_GCM PMD. * **Updated the aesni_mb crypto PMD.** diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_ops.h b/drivers/crypto/aesni_gcm/aesni_gcm_ops.h deleted file mode 100644 index 8a0d074b6e..0000000000 --- a/drivers/crypto/aesni_gcm/aesni_gcm_ops.h +++ /dev/null @@ -1,104 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2020 Intel Corporation - */ - -#ifndef _AESNI_GCM_OPS_H_ -#define _AESNI_GCM_OPS_H_ - -#ifndef LINUX -#define LINUX -#endif - -#include - -/** Supported vector modes */ -enum aesni_gcm_vector_mode { - RTE_AESNI_GCM_NOT_SUPPORTED = 0, - RTE_AESNI_GCM_SSE, - RTE_AESNI_GCM_AVX, - RTE_AESNI_GCM_AVX2, - RTE_AESNI_GCM_AVX512, - RTE_AESNI_GCM_VECTOR_NUM -}; - -enum aesni_gcm_key { - GCM_KEY_128 = 0, - GCM_KEY_192, - GCM_KEY_256, - GCM_KEY_NUM -}; - -typedef void (*aesni_gcm_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, uint8_t *out, - const uint8_t *in, uint64_t plaintext_len, const uint8_t *iv, - const uint8_t *aad, uint64_t aad_len, - uint8_t *auth_tag, uint64_t auth_tag_len); - -typedef void (*aesni_gcm_pre_t)(const void *key, struct gcm_key_data *gcm_data); - -typedef void (*aesni_gcm_init_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, - const uint8_t *iv, - uint8_t const *aad, - uint64_t aad_len); - -typedef void (*aesni_gcm_update_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, - uint8_t *out, - const uint8_t *in, - uint64_t plaintext_len); - -typedef void (*aesni_gcm_finalize_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, - uint8_t *auth_tag, - uint64_t auth_tag_len); - -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM -typedef void (*aesni_gmac_init_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, - const uint8_t *iv, - const uint64_t iv_len); - -typedef void (*aesni_gmac_update_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, - const uint8_t *in, - const uint64_t plaintext_len); - -typedef void (*aesni_gmac_finalize_t)(const struct gcm_key_data *gcm_key_data, - struct gcm_context_data *gcm_ctx_data, - uint8_t *auth_tag, - const uint64_t auth_tag_len); -#endif - -/** GCM library function pointer table */ -struct aesni_gcm_ops { - aesni_gcm_t enc; /**< GCM encode function pointer */ - aesni_gcm_t dec; /**< GCM decode function pointer */ - aesni_gcm_pre_t pre; /**< GCM pre-compute */ - aesni_gcm_init_t init; - aesni_gcm_update_t update_enc; - aesni_gcm_update_t update_dec; - aesni_gcm_finalize_t finalize_enc; - aesni_gcm_finalize_t finalize_dec; -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM - aesni_gmac_init_t gmac_init; - aesni_gmac_update_t gmac_update; - aesni_gmac_finalize_t gmac_finalize; -#endif -}; - -/** GCM per-session operation handlers */ -struct aesni_gcm_session_ops { - aesni_gcm_t cipher; - aesni_gcm_pre_t pre; - aesni_gcm_init_t init; - aesni_gcm_update_t update; - aesni_gcm_finalize_t finalize; -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM - aesni_gmac_init_t gmac_init; - aesni_gmac_update_t gmac_update; - aesni_gmac_finalize_t gmac_finalize; -#endif -}; - -#endif /* _AESNI_GCM_OPS_H_ */ diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c deleted file mode 100644 index 330aad8157..0000000000 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c +++ /dev/null @@ -1,984 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2020 Intel Corporation - */ - -#include -#include -#include -#include -#include -#include -#include -#include - -#include "aesni_gcm_pmd_private.h" - -static uint8_t cryptodev_driver_id; - -/* setup session handlers */ -static void -set_func_ops(struct aesni_gcm_session *s, const struct aesni_gcm_ops *gcm_ops) -{ - s->ops.pre = gcm_ops->pre; - s->ops.init = gcm_ops->init; - - switch (s->op) { - case AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION: - s->ops.cipher = gcm_ops->enc; - s->ops.update = gcm_ops->update_enc; - s->ops.finalize = gcm_ops->finalize_enc; - break; - case AESNI_GCM_OP_AUTHENTICATED_DECRYPTION: - s->ops.cipher = gcm_ops->dec; - s->ops.update = gcm_ops->update_dec; - s->ops.finalize = gcm_ops->finalize_dec; - break; - case AESNI_GMAC_OP_GENERATE: - case AESNI_GMAC_OP_VERIFY: - s->ops.finalize = gcm_ops->finalize_enc; - break; - } -} - -/** Parse crypto xform chain and set private session parameters */ -int -aesni_gcm_set_session_parameters(const struct aesni_gcm_ops *gcm_ops, - struct aesni_gcm_session *sess, - const struct rte_crypto_sym_xform *xform) -{ - const struct rte_crypto_sym_xform *auth_xform; - const struct rte_crypto_sym_xform *aead_xform; - uint8_t key_length; - const uint8_t *key; - - /* AES-GMAC */ - if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - auth_xform = xform; - if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_AES_GMAC) { - AESNI_GCM_LOG(ERR, "Only AES GMAC is supported as an " - "authentication only algorithm"); - return -ENOTSUP; - } - /* Set IV parameters */ - sess->iv.offset = auth_xform->auth.iv.offset; - sess->iv.length = auth_xform->auth.iv.length; - - /* Select Crypto operation */ - if (auth_xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) - sess->op = AESNI_GMAC_OP_GENERATE; - else - sess->op = AESNI_GMAC_OP_VERIFY; - - key_length = auth_xform->auth.key.length; - key = auth_xform->auth.key.data; - sess->req_digest_length = auth_xform->auth.digest_length; - - /* AES-GCM */ - } else if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { - aead_xform = xform; - - if (aead_xform->aead.algo != RTE_CRYPTO_AEAD_AES_GCM) { - AESNI_GCM_LOG(ERR, "The only combined operation " - "supported is AES GCM"); - return -ENOTSUP; - } - - /* Set IV parameters */ - sess->iv.offset = aead_xform->aead.iv.offset; - sess->iv.length = aead_xform->aead.iv.length; - - /* Select Crypto operation */ - if (aead_xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) - sess->op = AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION; - /* op == RTE_CRYPTO_AEAD_OP_DECRYPT */ - else - sess->op = AESNI_GCM_OP_AUTHENTICATED_DECRYPTION; - - key_length = aead_xform->aead.key.length; - key = aead_xform->aead.key.data; - - sess->aad_length = aead_xform->aead.aad_length; - sess->req_digest_length = aead_xform->aead.digest_length; - } else { - AESNI_GCM_LOG(ERR, "Wrong xform type, has to be AEAD or authentication"); - return -ENOTSUP; - } - - /* IV check */ - if (sess->iv.length != 16 && sess->iv.length != 12 && - sess->iv.length != 0) { - AESNI_GCM_LOG(ERR, "Wrong IV length"); - return -EINVAL; - } - - /* Check key length and calculate GCM pre-compute. */ - switch (key_length) { - case 16: - sess->key = GCM_KEY_128; - break; - case 24: - sess->key = GCM_KEY_192; - break; - case 32: - sess->key = GCM_KEY_256; - break; - default: - AESNI_GCM_LOG(ERR, "Invalid key length"); - return -EINVAL; - } - - /* setup session handlers */ - set_func_ops(sess, &gcm_ops[sess->key]); - - /* pre-generate key */ - gcm_ops[sess->key].pre(key, &sess->gdata_key); - - /* Digest check */ - if (sess->req_digest_length > 16) { - AESNI_GCM_LOG(ERR, "Invalid digest length"); - return -EINVAL; - } - /* - * Multi-buffer lib supports digest sizes from 4 to 16 bytes - * in version 0.50 and sizes of 8, 12 and 16 bytes, - * in version 0.49. - * If size requested is different, generate the full digest - * (16 bytes) in a temporary location and then memcpy - * the requested number of bytes. - */ -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) - if (sess->req_digest_length < 4) -#else - if (sess->req_digest_length != 16 && - sess->req_digest_length != 12 && - sess->req_digest_length != 8) -#endif - sess->gen_digest_length = 16; - else - sess->gen_digest_length = sess->req_digest_length; - - return 0; -} - -/** Get gcm session */ -static struct aesni_gcm_session * -aesni_gcm_get_session(struct aesni_gcm_qp *qp, struct rte_crypto_op *op) -{ - struct aesni_gcm_session *sess = NULL; - struct rte_crypto_sym_op *sym_op = op->sym; - - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { - if (likely(sym_op->session != NULL)) - sess = (struct aesni_gcm_session *) - get_sym_session_private_data( - sym_op->session, - cryptodev_driver_id); - } else { - void *_sess; - void *_sess_private_data = NULL; - - if (rte_mempool_get(qp->sess_mp, (void **)&_sess)) - return NULL; - - if (rte_mempool_get(qp->sess_mp_priv, - (void **)&_sess_private_data)) - return NULL; - - sess = (struct aesni_gcm_session *)_sess_private_data; - - if (unlikely(aesni_gcm_set_session_parameters(qp->ops, - sess, sym_op->xform) != 0)) { - rte_mempool_put(qp->sess_mp, _sess); - rte_mempool_put(qp->sess_mp_priv, _sess_private_data); - sess = NULL; - } - sym_op->session = (struct rte_cryptodev_sym_session *)_sess; - set_sym_session_private_data(sym_op->session, - cryptodev_driver_id, _sess_private_data); - } - - if (unlikely(sess == NULL)) - op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - - return sess; -} - -/** - * Process a crypto operation, calling - * the GCM API from the multi buffer library. - * - * @param qp queue pair - * @param op symmetric crypto operation - * @param session GCM session - * - * @return - * - */ -static int -process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, - struct aesni_gcm_session *session) -{ - uint8_t *src, *dst; - uint8_t *iv_ptr; - struct rte_crypto_sym_op *sym_op = op->sym; - struct rte_mbuf *m_src = sym_op->m_src; - uint32_t offset, data_offset, data_length; - uint32_t part_len, total_len, data_len; - uint8_t *tag; - unsigned int oop = 0; - - if (session->op == AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION || - session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION) { - offset = sym_op->aead.data.offset; - data_offset = offset; - data_length = sym_op->aead.data.length; - } else { - offset = sym_op->auth.data.offset; - data_offset = offset; - data_length = sym_op->auth.data.length; - } - - RTE_ASSERT(m_src != NULL); - - while (offset >= m_src->data_len && data_length != 0) { - offset -= m_src->data_len; - m_src = m_src->next; - - RTE_ASSERT(m_src != NULL); - } - - src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset); - - data_len = m_src->data_len - offset; - part_len = (data_len < data_length) ? data_len : - data_length; - - RTE_ASSERT((sym_op->m_dst == NULL) || - ((sym_op->m_dst != NULL) && - rte_pktmbuf_is_contiguous(sym_op->m_dst))); - - /* In-place */ - if (sym_op->m_dst == NULL || (sym_op->m_dst == sym_op->m_src)) - dst = src; - /* Out-of-place */ - else { - oop = 1; - /* Segmented destination buffer is not supported if operation is - * Out-of-place */ - RTE_ASSERT(rte_pktmbuf_is_contiguous(sym_op->m_dst)); - dst = rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *, - data_offset); - } - - iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, - session->iv.offset); - - if (session->op == AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION) { - qp->ops[session->key].init(&session->gdata_key, - &qp->gdata_ctx, - iv_ptr, - sym_op->aead.aad.data, - (uint64_t)session->aad_length); - - qp->ops[session->key].update_enc(&session->gdata_key, - &qp->gdata_ctx, dst, src, - (uint64_t)part_len); - total_len = data_length - part_len; - - while (total_len) { - m_src = m_src->next; - - RTE_ASSERT(m_src != NULL); - - src = rte_pktmbuf_mtod(m_src, uint8_t *); - if (oop) - dst += part_len; - else - dst = src; - part_len = (m_src->data_len < total_len) ? - m_src->data_len : total_len; - - qp->ops[session->key].update_enc(&session->gdata_key, - &qp->gdata_ctx, dst, src, - (uint64_t)part_len); - total_len -= part_len; - } - - if (session->req_digest_length != session->gen_digest_length) - tag = qp->temp_digest; - else - tag = sym_op->aead.digest.data; - - qp->ops[session->key].finalize_enc(&session->gdata_key, - &qp->gdata_ctx, - tag, - session->gen_digest_length); - } else if (session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION) { - qp->ops[session->key].init(&session->gdata_key, - &qp->gdata_ctx, - iv_ptr, - sym_op->aead.aad.data, - (uint64_t)session->aad_length); - - qp->ops[session->key].update_dec(&session->gdata_key, - &qp->gdata_ctx, dst, src, - (uint64_t)part_len); - total_len = data_length - part_len; - - while (total_len) { - m_src = m_src->next; - - RTE_ASSERT(m_src != NULL); - - src = rte_pktmbuf_mtod(m_src, uint8_t *); - if (oop) - dst += part_len; - else - dst = src; - part_len = (m_src->data_len < total_len) ? - m_src->data_len : total_len; - - qp->ops[session->key].update_dec(&session->gdata_key, - &qp->gdata_ctx, - dst, src, - (uint64_t)part_len); - total_len -= part_len; - } - - tag = qp->temp_digest; - qp->ops[session->key].finalize_dec(&session->gdata_key, - &qp->gdata_ctx, - tag, - session->gen_digest_length); -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM - } else if (session->op == AESNI_GMAC_OP_GENERATE) { - qp->ops[session->key].gmac_init(&session->gdata_key, - &qp->gdata_ctx, - iv_ptr, - session->iv.length); - - qp->ops[session->key].gmac_update(&session->gdata_key, - &qp->gdata_ctx, src, - (uint64_t)part_len); - total_len = data_length - part_len; - - while (total_len) { - m_src = m_src->next; - - RTE_ASSERT(m_src != NULL); - - src = rte_pktmbuf_mtod(m_src, uint8_t *); - part_len = (m_src->data_len < total_len) ? - m_src->data_len : total_len; - - qp->ops[session->key].gmac_update(&session->gdata_key, - &qp->gdata_ctx, src, - (uint64_t)part_len); - total_len -= part_len; - } - - if (session->req_digest_length != session->gen_digest_length) - tag = qp->temp_digest; - else - tag = sym_op->auth.digest.data; - - qp->ops[session->key].gmac_finalize(&session->gdata_key, - &qp->gdata_ctx, - tag, - session->gen_digest_length); - } else { /* AESNI_GMAC_OP_VERIFY */ - qp->ops[session->key].gmac_init(&session->gdata_key, - &qp->gdata_ctx, - iv_ptr, - session->iv.length); - - qp->ops[session->key].gmac_update(&session->gdata_key, - &qp->gdata_ctx, src, - (uint64_t)part_len); - total_len = data_length - part_len; - - while (total_len) { - m_src = m_src->next; - - RTE_ASSERT(m_src != NULL); - - src = rte_pktmbuf_mtod(m_src, uint8_t *); - part_len = (m_src->data_len < total_len) ? - m_src->data_len : total_len; - - qp->ops[session->key].gmac_update(&session->gdata_key, - &qp->gdata_ctx, src, - (uint64_t)part_len); - total_len -= part_len; - } - - tag = qp->temp_digest; - - qp->ops[session->key].gmac_finalize(&session->gdata_key, - &qp->gdata_ctx, - tag, - session->gen_digest_length); - } -#else - } else if (session->op == AESNI_GMAC_OP_GENERATE) { - qp->ops[session->key].init(&session->gdata_key, - &qp->gdata_ctx, - iv_ptr, - src, - (uint64_t)data_length); - if (session->req_digest_length != session->gen_digest_length) - tag = qp->temp_digest; - else - tag = sym_op->auth.digest.data; - qp->ops[session->key].finalize_enc(&session->gdata_key, - &qp->gdata_ctx, - tag, - session->gen_digest_length); - } else { /* AESNI_GMAC_OP_VERIFY */ - qp->ops[session->key].init(&session->gdata_key, - &qp->gdata_ctx, - iv_ptr, - src, - (uint64_t)data_length); - - /* - * Generate always 16 bytes and later compare only - * the bytes passed. - */ - tag = qp->temp_digest; - qp->ops[session->key].finalize_enc(&session->gdata_key, - &qp->gdata_ctx, - tag, - session->gen_digest_length); - } -#endif - - return 0; -} - -static inline void -aesni_gcm_fill_error_code(struct rte_crypto_sym_vec *vec, int32_t errnum) -{ - uint32_t i; - - for (i = 0; i < vec->num; i++) - vec->status[i] = errnum; -} - - -static inline int32_t -aesni_gcm_sgl_op_finalize_encryption(const struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, uint8_t *digest) -{ - if (s->req_digest_length != s->gen_digest_length) { - uint8_t tmpdigest[s->gen_digest_length]; - - s->ops.finalize(&s->gdata_key, gdata_ctx, tmpdigest, - s->gen_digest_length); - memcpy(digest, tmpdigest, s->req_digest_length); - } else { - s->ops.finalize(&s->gdata_key, gdata_ctx, digest, - s->gen_digest_length); - } - - return 0; -} - -static inline int32_t -aesni_gcm_sgl_op_finalize_decryption(const struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, uint8_t *digest) -{ - uint8_t tmpdigest[s->gen_digest_length]; - - s->ops.finalize(&s->gdata_key, gdata_ctx, tmpdigest, - s->gen_digest_length); - - return memcmp(digest, tmpdigest, s->req_digest_length) == 0 ? 0 : - EBADMSG; -} - -static inline void -aesni_gcm_process_gcm_sgl_op(const struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, struct rte_crypto_sgl *sgl, - void *iv, void *aad) -{ - uint32_t i; - - /* init crypto operation */ - s->ops.init(&s->gdata_key, gdata_ctx, iv, aad, - (uint64_t)s->aad_length); - - /* update with sgl data */ - for (i = 0; i < sgl->num; i++) { - struct rte_crypto_vec *vec = &sgl->vec[i]; - - s->ops.update(&s->gdata_key, gdata_ctx, vec->base, vec->base, - vec->len); - } -} - -static inline void -aesni_gcm_process_gmac_sgl_op(const struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, struct rte_crypto_sgl *sgl, - void *iv) -{ - s->ops.init(&s->gdata_key, gdata_ctx, iv, sgl->vec[0].base, - sgl->vec[0].len); -} - -static inline uint32_t -aesni_gcm_sgl_encrypt(struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, struct rte_crypto_sym_vec *vec) -{ - uint32_t i, processed; - - processed = 0; - for (i = 0; i < vec->num; ++i) { - aesni_gcm_process_gcm_sgl_op(s, gdata_ctx, - &vec->sgl[i], vec->iv[i].va, - vec->aad[i].va); - vec->status[i] = aesni_gcm_sgl_op_finalize_encryption(s, - gdata_ctx, vec->digest[i].va); - processed += (vec->status[i] == 0); - } - - return processed; -} - -static inline uint32_t -aesni_gcm_sgl_decrypt(struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, struct rte_crypto_sym_vec *vec) -{ - uint32_t i, processed; - - processed = 0; - for (i = 0; i < vec->num; ++i) { - aesni_gcm_process_gcm_sgl_op(s, gdata_ctx, - &vec->sgl[i], vec->iv[i].va, - vec->aad[i].va); - vec->status[i] = aesni_gcm_sgl_op_finalize_decryption(s, - gdata_ctx, vec->digest[i].va); - processed += (vec->status[i] == 0); - } - - return processed; -} - -static inline uint32_t -aesni_gmac_sgl_generate(struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, struct rte_crypto_sym_vec *vec) -{ - uint32_t i, processed; - - processed = 0; - for (i = 0; i < vec->num; ++i) { - if (vec->sgl[i].num != 1) { - vec->status[i] = ENOTSUP; - continue; - } - - aesni_gcm_process_gmac_sgl_op(s, gdata_ctx, - &vec->sgl[i], vec->iv[i].va); - vec->status[i] = aesni_gcm_sgl_op_finalize_encryption(s, - gdata_ctx, vec->digest[i].va); - processed += (vec->status[i] == 0); - } - - return processed; -} - -static inline uint32_t -aesni_gmac_sgl_verify(struct aesni_gcm_session *s, - struct gcm_context_data *gdata_ctx, struct rte_crypto_sym_vec *vec) -{ - uint32_t i, processed; - - processed = 0; - for (i = 0; i < vec->num; ++i) { - if (vec->sgl[i].num != 1) { - vec->status[i] = ENOTSUP; - continue; - } - - aesni_gcm_process_gmac_sgl_op(s, gdata_ctx, - &vec->sgl[i], vec->iv[i].va); - vec->status[i] = aesni_gcm_sgl_op_finalize_decryption(s, - gdata_ctx, vec->digest[i].va); - processed += (vec->status[i] == 0); - } - - return processed; -} - -/** Process CPU crypto bulk operations */ -uint32_t -aesni_gcm_pmd_cpu_crypto_process(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess, - __rte_unused union rte_crypto_sym_ofs ofs, - struct rte_crypto_sym_vec *vec) -{ - void *sess_priv; - struct aesni_gcm_session *s; - struct gcm_context_data gdata_ctx; - - sess_priv = get_sym_session_private_data(sess, dev->driver_id); - if (unlikely(sess_priv == NULL)) { - aesni_gcm_fill_error_code(vec, EINVAL); - return 0; - } - - s = sess_priv; - switch (s->op) { - case AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION: - return aesni_gcm_sgl_encrypt(s, &gdata_ctx, vec); - case AESNI_GCM_OP_AUTHENTICATED_DECRYPTION: - return aesni_gcm_sgl_decrypt(s, &gdata_ctx, vec); - case AESNI_GMAC_OP_GENERATE: - return aesni_gmac_sgl_generate(s, &gdata_ctx, vec); - case AESNI_GMAC_OP_VERIFY: - return aesni_gmac_sgl_verify(s, &gdata_ctx, vec); - default: - aesni_gcm_fill_error_code(vec, EINVAL); - return 0; - } -} - -/** - * Process a completed job and return rte_mbuf which job processed - * - * @param job JOB_AES_HMAC job to process - * - * @return - * - Returns processed mbuf which is trimmed of output digest used in - * verification of supplied digest in the case of a HASH_CIPHER operation - * - Returns NULL on invalid job - */ -static void -post_process_gcm_crypto_op(struct aesni_gcm_qp *qp, - struct rte_crypto_op *op, - struct aesni_gcm_session *session) -{ - op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - - /* Verify digest if required */ - if (session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION || - session->op == AESNI_GMAC_OP_VERIFY) { - uint8_t *digest; - - uint8_t *tag = qp->temp_digest; - - if (session->op == AESNI_GMAC_OP_VERIFY) - digest = op->sym->auth.digest.data; - else - digest = op->sym->aead.digest.data; - -#ifdef RTE_LIBRTE_PMD_AESNI_GCM_DEBUG - rte_hexdump(stdout, "auth tag (orig):", - digest, session->req_digest_length); - rte_hexdump(stdout, "auth tag (calc):", - tag, session->req_digest_length); -#endif - - if (memcmp(tag, digest, session->req_digest_length) != 0) - op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - } else { - if (session->req_digest_length != session->gen_digest_length) { - if (session->op == AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION) - memcpy(op->sym->aead.digest.data, qp->temp_digest, - session->req_digest_length); - else - memcpy(op->sym->auth.digest.data, qp->temp_digest, - session->req_digest_length); - } - } -} - -/** - * Process a completed GCM request - * - * @param qp Queue Pair to process - * @param op Crypto operation - * @param job JOB_AES_HMAC job - * - * @return - * - Number of processed jobs - */ -static void -handle_completed_gcm_crypto_op(struct aesni_gcm_qp *qp, - struct rte_crypto_op *op, - struct aesni_gcm_session *sess) -{ - post_process_gcm_crypto_op(qp, op, sess); - - /* Free session if a session-less crypto op */ - if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { - memset(sess, 0, sizeof(struct aesni_gcm_session)); - memset(op->sym->session, 0, - rte_cryptodev_sym_get_existing_header_session_size( - op->sym->session)); - rte_mempool_put(qp->sess_mp_priv, sess); - rte_mempool_put(qp->sess_mp, op->sym->session); - op->sym->session = NULL; - } -} - -static uint16_t -aesni_gcm_pmd_dequeue_burst(void *queue_pair, - struct rte_crypto_op **ops, uint16_t nb_ops) -{ - struct aesni_gcm_session *sess; - struct aesni_gcm_qp *qp = queue_pair; - - int retval = 0; - unsigned int i, nb_dequeued; - - nb_dequeued = rte_ring_dequeue_burst(qp->processed_pkts, - (void **)ops, nb_ops, NULL); - - for (i = 0; i < nb_dequeued; i++) { - - sess = aesni_gcm_get_session(qp, ops[i]); - if (unlikely(sess == NULL)) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - qp->qp_stats.dequeue_err_count++; - break; - } - - retval = process_gcm_crypto_op(qp, ops[i], sess); - if (retval < 0) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - qp->qp_stats.dequeue_err_count++; - break; - } - - handle_completed_gcm_crypto_op(qp, ops[i], sess); - } - - qp->qp_stats.dequeued_count += i; - - return i; -} - -static uint16_t -aesni_gcm_pmd_enqueue_burst(void *queue_pair, - struct rte_crypto_op **ops, uint16_t nb_ops) -{ - struct aesni_gcm_qp *qp = queue_pair; - - unsigned int nb_enqueued; - - nb_enqueued = rte_ring_enqueue_burst(qp->processed_pkts, - (void **)ops, nb_ops, NULL); - qp->qp_stats.enqueued_count += nb_enqueued; - - return nb_enqueued; -} - -static int aesni_gcm_remove(struct rte_vdev_device *vdev); - -static int -aesni_gcm_create(const char *name, - struct rte_vdev_device *vdev, - struct rte_cryptodev_pmd_init_params *init_params) -{ - struct rte_cryptodev *dev; - struct aesni_gcm_private *internals; - enum aesni_gcm_vector_mode vector_mode; - MB_MGR *mb_mgr; - - dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); - if (dev == NULL) { - AESNI_GCM_LOG(ERR, "driver %s: create failed", - init_params->name); - return -ENODEV; - } - - /* Check CPU for supported vector instruction set */ - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) - vector_mode = RTE_AESNI_GCM_AVX512; - else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) - vector_mode = RTE_AESNI_GCM_AVX2; - else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) - vector_mode = RTE_AESNI_GCM_AVX; - else - vector_mode = RTE_AESNI_GCM_SSE; - - dev->driver_id = cryptodev_driver_id; - dev->dev_ops = rte_aesni_gcm_pmd_ops; - - /* register rx/tx burst functions for data path */ - dev->dequeue_burst = aesni_gcm_pmd_dequeue_burst; - dev->enqueue_burst = aesni_gcm_pmd_enqueue_burst; - - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_IN_PLACE_SGL | - RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | - RTE_CRYPTODEV_FF_SYM_CPU_CRYPTO | - RTE_CRYPTODEV_FF_SYM_SESSIONLESS; - - /* Check CPU for support for AES instruction set */ - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AESNI; - else - AESNI_GCM_LOG(WARNING, "AES instructions not supported by CPU"); - - mb_mgr = alloc_mb_mgr(0); - if (mb_mgr == NULL) - return -ENOMEM; - - switch (vector_mode) { - case RTE_AESNI_GCM_SSE: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - init_mb_mgr_sse(mb_mgr); - break; - case RTE_AESNI_GCM_AVX: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - init_mb_mgr_avx(mb_mgr); - break; - case RTE_AESNI_GCM_AVX2: - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; - init_mb_mgr_avx2(mb_mgr); - break; - case RTE_AESNI_GCM_AVX512: - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_VAES)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; - init_mb_mgr_avx512(mb_mgr); - } else { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; - init_mb_mgr_avx2(mb_mgr); - vector_mode = RTE_AESNI_GCM_AVX2; - } - break; - default: - AESNI_GCM_LOG(ERR, "Unsupported vector mode %u\n", vector_mode); - goto error_exit; - } - - internals = dev->data->dev_private; - - internals->vector_mode = vector_mode; - internals->mb_mgr = mb_mgr; - - /* Set arch independent function pointers, based on key size */ - internals->ops[GCM_KEY_128].enc = mb_mgr->gcm128_enc; - internals->ops[GCM_KEY_128].dec = mb_mgr->gcm128_dec; - internals->ops[GCM_KEY_128].pre = mb_mgr->gcm128_pre; - internals->ops[GCM_KEY_128].init = mb_mgr->gcm128_init; - internals->ops[GCM_KEY_128].update_enc = mb_mgr->gcm128_enc_update; - internals->ops[GCM_KEY_128].update_dec = mb_mgr->gcm128_dec_update; - internals->ops[GCM_KEY_128].finalize_enc = mb_mgr->gcm128_enc_finalize; - internals->ops[GCM_KEY_128].finalize_dec = mb_mgr->gcm128_dec_finalize; -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM - internals->ops[GCM_KEY_128].gmac_init = mb_mgr->gmac128_init; - internals->ops[GCM_KEY_128].gmac_update = mb_mgr->gmac128_update; - internals->ops[GCM_KEY_128].gmac_finalize = mb_mgr->gmac128_finalize; -#endif - - internals->ops[GCM_KEY_192].enc = mb_mgr->gcm192_enc; - internals->ops[GCM_KEY_192].dec = mb_mgr->gcm192_dec; - internals->ops[GCM_KEY_192].pre = mb_mgr->gcm192_pre; - internals->ops[GCM_KEY_192].init = mb_mgr->gcm192_init; - internals->ops[GCM_KEY_192].update_enc = mb_mgr->gcm192_enc_update; - internals->ops[GCM_KEY_192].update_dec = mb_mgr->gcm192_dec_update; - internals->ops[GCM_KEY_192].finalize_enc = mb_mgr->gcm192_enc_finalize; - internals->ops[GCM_KEY_192].finalize_dec = mb_mgr->gcm192_dec_finalize; -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM - internals->ops[GCM_KEY_192].gmac_init = mb_mgr->gmac192_init; - internals->ops[GCM_KEY_192].gmac_update = mb_mgr->gmac192_update; - internals->ops[GCM_KEY_192].gmac_finalize = mb_mgr->gmac192_finalize; -#endif - - internals->ops[GCM_KEY_256].enc = mb_mgr->gcm256_enc; - internals->ops[GCM_KEY_256].dec = mb_mgr->gcm256_dec; - internals->ops[GCM_KEY_256].pre = mb_mgr->gcm256_pre; - internals->ops[GCM_KEY_256].init = mb_mgr->gcm256_init; - internals->ops[GCM_KEY_256].update_enc = mb_mgr->gcm256_enc_update; - internals->ops[GCM_KEY_256].update_dec = mb_mgr->gcm256_dec_update; - internals->ops[GCM_KEY_256].finalize_enc = mb_mgr->gcm256_enc_finalize; - internals->ops[GCM_KEY_256].finalize_dec = mb_mgr->gcm256_dec_finalize; -#if IMB_VERSION(0, 54, 0) < IMB_VERSION_NUM - internals->ops[GCM_KEY_256].gmac_init = mb_mgr->gmac256_init; - internals->ops[GCM_KEY_256].gmac_update = mb_mgr->gmac256_update; - internals->ops[GCM_KEY_256].gmac_finalize = mb_mgr->gmac256_finalize; -#endif - - internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; - -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) - AESNI_GCM_LOG(INFO, "IPSec Multi-buffer library version used: %s\n", - imb_get_version_str()); -#else - AESNI_GCM_LOG(INFO, "IPSec Multi-buffer library version used: 0.49.0\n"); -#endif - - return 0; - -error_exit: - if (mb_mgr) - free_mb_mgr(mb_mgr); - - rte_cryptodev_pmd_destroy(dev); - - return -1; -} - -static int -aesni_gcm_probe(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev_pmd_init_params init_params = { - "", - sizeof(struct aesni_gcm_private), - rte_socket_id(), - RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS - }; - const char *name; - const char *input_args; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - input_args = rte_vdev_device_args(vdev); - rte_cryptodev_pmd_parse_input_args(&init_params, input_args); - - return aesni_gcm_create(name, vdev, &init_params); -} - -static int -aesni_gcm_remove(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev *cryptodev; - struct aesni_gcm_private *internals; - const char *name; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - - cryptodev = rte_cryptodev_pmd_get_named_dev(name); - if (cryptodev == NULL) - return -ENODEV; - - internals = cryptodev->data->dev_private; - - free_mb_mgr(internals->mb_mgr); - - return rte_cryptodev_pmd_destroy(cryptodev); -} - -static struct rte_vdev_driver aesni_gcm_pmd_drv = { - .probe = aesni_gcm_probe, - .remove = aesni_gcm_remove -}; - -static struct cryptodev_driver aesni_gcm_crypto_drv; - -RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_AESNI_GCM_PMD, aesni_gcm_pmd_drv); -RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_AESNI_GCM_PMD, cryptodev_aesni_gcm_pmd); -RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_AESNI_GCM_PMD, - "max_nb_queue_pairs= " - "socket_id="); -RTE_PMD_REGISTER_CRYPTO_DRIVER(aesni_gcm_crypto_drv, aesni_gcm_pmd_drv.driver, - cryptodev_driver_id); -RTE_LOG_REGISTER_DEFAULT(aesni_gcm_logtype_driver, NOTICE); diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c deleted file mode 100644 index edb7275e76..0000000000 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c +++ /dev/null @@ -1,333 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2020 Intel Corporation - */ - -#include - -#include -#include -#include - -#include "aesni_gcm_pmd_private.h" - -static const struct rte_cryptodev_capabilities aesni_gcm_pmd_capabilities[] = { - { /* AES GMAC (AUTH) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_AES_GMAC, - .block_size = 16, - .key_size = { - .min = 16, - .max = 32, - .increment = 8 - }, - .digest_size = { - .min = 1, - .max = 16, - .increment = 1 - }, - .iv_size = { - .min = 12, - .max = 12, - .increment = 0 - } - }, } - }, } - }, - { /* AES GCM */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, - {.aead = { - .algo = RTE_CRYPTO_AEAD_AES_GCM, - .block_size = 16, - .key_size = { - .min = 16, - .max = 32, - .increment = 8 - }, - .digest_size = { - .min = 1, - .max = 16, - .increment = 1 - }, - .aad_size = { - .min = 0, - .max = 65535, - .increment = 1 - }, - .iv_size = { - .min = 12, - .max = 12, - .increment = 0 - } - }, } - }, } - }, - RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() -}; - -/** Configure device */ -static int -aesni_gcm_pmd_config(__rte_unused struct rte_cryptodev *dev, - __rte_unused struct rte_cryptodev_config *config) -{ - return 0; -} - -/** Start device */ -static int -aesni_gcm_pmd_start(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - -/** Stop device */ -static void -aesni_gcm_pmd_stop(__rte_unused struct rte_cryptodev *dev) -{ -} - -/** Close device */ -static int -aesni_gcm_pmd_close(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - - -/** Get device statistics */ -static void -aesni_gcm_pmd_stats_get(struct rte_cryptodev *dev, - struct rte_cryptodev_stats *stats) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct aesni_gcm_qp *qp = dev->data->queue_pairs[qp_id]; - - stats->enqueued_count += qp->qp_stats.enqueued_count; - stats->dequeued_count += qp->qp_stats.dequeued_count; - - stats->enqueue_err_count += qp->qp_stats.enqueue_err_count; - stats->dequeue_err_count += qp->qp_stats.dequeue_err_count; - } -} - -/** Reset device statistics */ -static void -aesni_gcm_pmd_stats_reset(struct rte_cryptodev *dev) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct aesni_gcm_qp *qp = dev->data->queue_pairs[qp_id]; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - } -} - - -/** Get device info */ -static void -aesni_gcm_pmd_info_get(struct rte_cryptodev *dev, - struct rte_cryptodev_info *dev_info) -{ - struct aesni_gcm_private *internals = dev->data->dev_private; - - if (dev_info != NULL) { - dev_info->driver_id = dev->driver_id; - dev_info->feature_flags = dev->feature_flags; - dev_info->capabilities = aesni_gcm_pmd_capabilities; - - dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; - /* No limit of number of sessions */ - dev_info->sym.max_nb_sessions = 0; - } -} - -/** Release queue pair */ -static int -aesni_gcm_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) -{ - if (dev->data->queue_pairs[qp_id] != NULL) { - struct aesni_gcm_qp *qp = dev->data->queue_pairs[qp_id]; - - if (qp->processed_pkts) - rte_ring_free(qp->processed_pkts); - - rte_free(dev->data->queue_pairs[qp_id]); - dev->data->queue_pairs[qp_id] = NULL; - } - return 0; -} - -/** set a unique name for the queue pair based on it's name, dev_id and qp_id */ -static int -aesni_gcm_pmd_qp_set_unique_name(struct rte_cryptodev *dev, - struct aesni_gcm_qp *qp) -{ - unsigned n = snprintf(qp->name, sizeof(qp->name), - "aesni_gcm_pmd_%u_qp_%u", - dev->data->dev_id, qp->id); - - if (n >= sizeof(qp->name)) - return -1; - - return 0; -} - -/** Create a ring to place process packets on */ -static struct rte_ring * -aesni_gcm_pmd_qp_create_processed_pkts_ring(struct aesni_gcm_qp *qp, - unsigned ring_size, int socket_id) -{ - struct rte_ring *r; - - r = rte_ring_lookup(qp->name); - if (r) { - if (rte_ring_get_size(r) >= ring_size) { - AESNI_GCM_LOG(INFO, "Reusing existing ring %s for processed" - " packets", qp->name); - return r; - } - AESNI_GCM_LOG(ERR, "Unable to reuse existing ring %s for processed" - " packets", qp->name); - return NULL; - } - - return rte_ring_create(qp->name, ring_size, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); -} - -/** Setup a queue pair */ -static int -aesni_gcm_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, - const struct rte_cryptodev_qp_conf *qp_conf, - int socket_id) -{ - struct aesni_gcm_qp *qp = NULL; - struct aesni_gcm_private *internals = dev->data->dev_private; - - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - aesni_gcm_pmd_qp_release(dev, qp_id); - - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("AES-NI PMD Queue Pair", sizeof(*qp), - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return (-ENOMEM); - - qp->id = qp_id; - dev->data->queue_pairs[qp_id] = qp; - - if (aesni_gcm_pmd_qp_set_unique_name(dev, qp)) - goto qp_setup_cleanup; - - qp->ops = (const struct aesni_gcm_ops *)internals->ops; - - qp->processed_pkts = aesni_gcm_pmd_qp_create_processed_pkts_ring(qp, - qp_conf->nb_descriptors, socket_id); - if (qp->processed_pkts == NULL) - goto qp_setup_cleanup; - - qp->sess_mp = qp_conf->mp_session; - qp->sess_mp_priv = qp_conf->mp_session_private; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - - return 0; - -qp_setup_cleanup: - if (qp) - rte_free(qp); - - return -1; -} - -/** Returns the size of the aesni gcm session structure */ -static unsigned -aesni_gcm_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) -{ - return sizeof(struct aesni_gcm_session); -} - -/** Configure a aesni gcm session from a crypto xform chain */ -static int -aesni_gcm_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, - struct rte_crypto_sym_xform *xform, - struct rte_cryptodev_sym_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - int ret; - struct aesni_gcm_private *internals = dev->data->dev_private; - - if (unlikely(sess == NULL)) { - AESNI_GCM_LOG(ERR, "invalid session struct"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - AESNI_GCM_LOG(ERR, - "Couldn't get object from session mempool"); - return -ENOMEM; - } - ret = aesni_gcm_set_session_parameters(internals->ops, - sess_private_data, xform); - if (ret != 0) { - AESNI_GCM_LOG(ERR, "failed configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sym_session_private_data(sess, dev->driver_id, - sess_private_data); - - return 0; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static void -aesni_gcm_pmd_sym_session_clear(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess) -{ - uint8_t index = dev->driver_id; - void *sess_priv = get_sym_session_private_data(sess, index); - - /* Zero out the whole structure */ - if (sess_priv) { - memset(sess_priv, 0, sizeof(struct aesni_gcm_session)); - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - set_sym_session_private_data(sess, index, NULL); - rte_mempool_put(sess_mp, sess_priv); - } -} - -struct rte_cryptodev_ops aesni_gcm_pmd_ops = { - .dev_configure = aesni_gcm_pmd_config, - .dev_start = aesni_gcm_pmd_start, - .dev_stop = aesni_gcm_pmd_stop, - .dev_close = aesni_gcm_pmd_close, - - .stats_get = aesni_gcm_pmd_stats_get, - .stats_reset = aesni_gcm_pmd_stats_reset, - - .dev_infos_get = aesni_gcm_pmd_info_get, - - .queue_pair_setup = aesni_gcm_pmd_qp_setup, - .queue_pair_release = aesni_gcm_pmd_qp_release, - - .sym_cpu_process = aesni_gcm_pmd_cpu_crypto_process, - - .sym_session_get_size = aesni_gcm_pmd_sym_session_get_size, - .sym_session_configure = aesni_gcm_pmd_sym_session_configure, - .sym_session_clear = aesni_gcm_pmd_sym_session_clear -}; - -struct rte_cryptodev_ops *rte_aesni_gcm_pmd_ops = &aesni_gcm_pmd_ops; diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h deleted file mode 100644 index 2763d1c492..0000000000 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h +++ /dev/null @@ -1,123 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2020 Intel Corporation - */ - -#ifndef _AESNI_GCM_PMD_PRIVATE_H_ -#define _AESNI_GCM_PMD_PRIVATE_H_ - -#include "aesni_gcm_ops.h" - -/* - * IMB_VERSION_NUM macro was introduced in version Multi-buffer 0.50, - * so if macro is not defined, it means that the version is 0.49. - */ -#if !defined(IMB_VERSION_NUM) -#define IMB_VERSION(a, b, c) (((a) << 16) + ((b) << 8) + (c)) -#define IMB_VERSION_NUM IMB_VERSION(0, 49, 0) -#endif - -#define CRYPTODEV_NAME_AESNI_GCM_PMD crypto_aesni_gcm -/**< AES-NI GCM PMD device name */ - -/** AES-NI GCM PMD LOGTYPE DRIVER */ -extern int aesni_gcm_logtype_driver; -#define AESNI_GCM_LOG(level, fmt, ...) \ - rte_log(RTE_LOG_ ## level, aesni_gcm_logtype_driver, \ - "%s() line %u: "fmt "\n", __func__, __LINE__, \ - ## __VA_ARGS__) - -/* Maximum length for digest */ -#define DIGEST_LENGTH_MAX 16 - -/** private data structure for each virtual AESNI GCM device */ -struct aesni_gcm_private { - enum aesni_gcm_vector_mode vector_mode; - /**< Vector mode */ - unsigned max_nb_queue_pairs; - /**< Max number of queue pairs supported by device */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ - struct aesni_gcm_ops ops[GCM_KEY_NUM]; - /**< Function pointer table of the gcm APIs */ -}; - -struct aesni_gcm_qp { - const struct aesni_gcm_ops *ops; - /**< Function pointer table of the gcm APIs */ - struct rte_ring *processed_pkts; - /**< Ring for placing process packets */ - struct gcm_context_data gdata_ctx; /* (16 * 5) + 8 = 88 B */ - /**< GCM parameters */ - struct rte_cryptodev_stats qp_stats; /* 8 * 4 = 32 B */ - /**< Queue pair statistics */ - struct rte_mempool *sess_mp; - /**< Session Mempool */ - struct rte_mempool *sess_mp_priv; - /**< Session Private Data Mempool */ - uint16_t id; - /**< Queue Pair Identifier */ - char name[RTE_CRYPTODEV_NAME_MAX_LEN]; - /**< Unique Queue Pair Name */ - uint8_t temp_digest[DIGEST_LENGTH_MAX]; - /**< Buffer used to store the digest generated - * by the driver when verifying a digest provided - * by the user (using authentication verify operation) - */ -} __rte_cache_aligned; - - -enum aesni_gcm_operation { - AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION, - AESNI_GCM_OP_AUTHENTICATED_DECRYPTION, - AESNI_GMAC_OP_GENERATE, - AESNI_GMAC_OP_VERIFY -}; - -/** AESNI GCM private session structure */ -struct aesni_gcm_session { - struct { - uint16_t length; - uint16_t offset; - } iv; - /**< IV parameters */ - uint16_t aad_length; - /**< AAD length */ - uint16_t req_digest_length; - /**< Requested digest length */ - uint16_t gen_digest_length; - /**< Generated digest length */ - enum aesni_gcm_operation op; - /**< GCM operation type */ - enum aesni_gcm_key key; - /**< GCM key type */ - struct gcm_key_data gdata_key; - /**< GCM parameters */ - struct aesni_gcm_session_ops ops; - /**< Session handlers */ -}; - - -/** - * Setup GCM session parameters - * @param sess aesni gcm session structure - * @param xform crypto transform chain - * - * @return - * - On success returns 0 - * - On failure returns error code < 0 - */ -extern int -aesni_gcm_set_session_parameters(const struct aesni_gcm_ops *ops, - struct aesni_gcm_session *sess, - const struct rte_crypto_sym_xform *xform); - -/* Device specific operations function pointer structure */ -extern struct rte_cryptodev_ops *rte_aesni_gcm_pmd_ops; - -/** CPU crypto bulk process handler */ -uint32_t -aesni_gcm_pmd_cpu_crypto_process(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs ofs, - struct rte_crypto_sym_vec *vec); - -#endif /* _AESNI_GCM_PMD_PRIVATE_H_ */ diff --git a/drivers/crypto/aesni_gcm/meson.build b/drivers/crypto/aesni_gcm/meson.build deleted file mode 100644 index 0fcac2a8eb..0000000000 --- a/drivers/crypto/aesni_gcm/meson.build +++ /dev/null @@ -1,24 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018 Intel Corporation - -IMB_required_ver = '0.52.0' -lib = cc.find_library('IPSec_MB', required: false) -if not lib.found() - build = false - reason = 'missing dependency, "libIPSec_MB"' -else - ext_deps += lib - - # version comes with quotes, so we split based on " and take the middle - imb_ver = cc.get_define('IMB_VERSION_STR', - prefix : '#include').split('"')[1] - - if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) - reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( - IMB_required_ver, imb_ver) - build = false - endif -endif - -sources = files('aesni_gcm_pmd.c', 'aesni_gcm_pmd_ops.c') -deps += ['bus_vdev'] diff --git a/drivers/crypto/aesni_gcm/version.map b/drivers/crypto/aesni_gcm/version.map deleted file mode 100644 index c2e0723b4c..0000000000 --- a/drivers/crypto/aesni_gcm/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_22 { - local: *; -}; diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index 1b7bf459ea..ed755dbf13 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -37,6 +37,9 @@ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); #define CRYPTODEV_NAME_AESNI_MB_PMD crypto_aesni_mb /**< IPSEC Multi buffer aesni_mb PMD device name */ +#define CRYPTODEV_NAME_AESNI_GCM_PMD crypto_aesni_gcm +/**< IPSEC Multi buffer PMD aesni_gcm device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -46,6 +49,7 @@ extern int ipsec_mb_logtype_driver; /** All supported device types */ enum ipsec_mb_pmd_types { IPSEC_MB_PMD_TYPE_AESNI_MB = 0, + IPSEC_MB_PMD_TYPE_AESNI_GCM, IPSEC_MB_N_PMD_TYPES }; @@ -65,6 +69,7 @@ enum ipsec_mb_operation { }; extern uint8_t pmd_driver_id_aesni_mb; +extern uint8_t pmd_driver_id_aesni_gcm; /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t @@ -73,6 +78,8 @@ ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) switch (pmd_type) { case IPSEC_MB_PMD_TYPE_AESNI_MB: return pmd_driver_id_aesni_mb; + case IPSEC_MB_PMD_TYPE_AESNI_GCM: + return pmd_driver_id_aesni_gcm; default: break; } diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index 14e654ba09..b8166c7e75 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -23,6 +23,7 @@ endif sources = files('ipsec_mb_private.c', 'ipsec_mb_ops.c', - 'pmd_aesni_mb.c' + 'pmd_aesni_mb.c', + 'pmd_aesni_gcm.c' ) deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/ipsec_mb/pmd_aesni_gcm.c b/drivers/crypto/ipsec_mb/pmd_aesni_gcm.c new file mode 100644 index 0000000000..ca7b63224c --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_aesni_gcm.c @@ -0,0 +1,837 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#include "pmd_aesni_gcm_priv.h" + +static void +aesni_gcm_set_ops(struct aesni_gcm_ops *ops, IMB_MGR *mb_mgr) +{ + /* Set 128 bit function pointers. */ + ops[GCM_KEY_128].pre = mb_mgr->gcm128_pre; + ops[GCM_KEY_128].init = mb_mgr->gcm128_init; + + ops[GCM_KEY_128].enc = mb_mgr->gcm128_enc; + ops[GCM_KEY_128].update_enc = mb_mgr->gcm128_enc_update; + ops[GCM_KEY_128].finalize_enc = mb_mgr->gcm128_enc_finalize; + + ops[GCM_KEY_128].dec = mb_mgr->gcm128_dec; + ops[GCM_KEY_128].update_dec = mb_mgr->gcm128_dec_update; + ops[GCM_KEY_128].finalize_dec = mb_mgr->gcm128_dec_finalize; + + ops[GCM_KEY_128].gmac_init = mb_mgr->gmac128_init; + ops[GCM_KEY_128].gmac_update = mb_mgr->gmac128_update; + ops[GCM_KEY_128].gmac_finalize = mb_mgr->gmac128_finalize; + + /* Set 192 bit function pointers. */ + ops[GCM_KEY_192].pre = mb_mgr->gcm192_pre; + ops[GCM_KEY_192].init = mb_mgr->gcm192_init; + + ops[GCM_KEY_192].enc = mb_mgr->gcm192_enc; + ops[GCM_KEY_192].update_enc = mb_mgr->gcm192_enc_update; + ops[GCM_KEY_192].finalize_enc = mb_mgr->gcm192_enc_finalize; + + ops[GCM_KEY_192].dec = mb_mgr->gcm192_dec; + ops[GCM_KEY_192].update_dec = mb_mgr->gcm192_dec_update; + ops[GCM_KEY_192].finalize_dec = mb_mgr->gcm192_dec_finalize; + + ops[GCM_KEY_192].gmac_init = mb_mgr->gmac192_init; + ops[GCM_KEY_192].gmac_update = mb_mgr->gmac192_update; + ops[GCM_KEY_192].gmac_finalize = mb_mgr->gmac192_finalize; + + /* Set 256 bit function pointers. */ + ops[GCM_KEY_256].pre = mb_mgr->gcm256_pre; + ops[GCM_KEY_256].init = mb_mgr->gcm256_init; + + ops[GCM_KEY_256].enc = mb_mgr->gcm256_enc; + ops[GCM_KEY_256].update_enc = mb_mgr->gcm256_enc_update; + ops[GCM_KEY_256].finalize_enc = mb_mgr->gcm256_enc_finalize; + + ops[GCM_KEY_256].dec = mb_mgr->gcm256_dec; + ops[GCM_KEY_256].update_dec = mb_mgr->gcm256_dec_update; + ops[GCM_KEY_256].finalize_dec = mb_mgr->gcm256_dec_finalize; + + ops[GCM_KEY_256].gmac_init = mb_mgr->gmac256_init; + ops[GCM_KEY_256].gmac_update = mb_mgr->gmac256_update; + ops[GCM_KEY_256].gmac_finalize = mb_mgr->gmac256_finalize; +} + +static int +aesni_gcm_session_configure(IMB_MGR *mb_mgr, void *session, + const struct rte_crypto_sym_xform *xform) +{ + struct aesni_gcm_session *sess = session; + const struct rte_crypto_sym_xform *auth_xform; + const struct rte_crypto_sym_xform *cipher_xform; + const struct rte_crypto_sym_xform *aead_xform; + + uint8_t key_length; + const uint8_t *key; + enum ipsec_mb_operation mode; + int ret = 0; + + ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, &aead_xform); + if (ret) + return ret; + + /**< GCM key type */ + + sess->op = mode; + + switch (sess->op) { + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + /* AES-GMAC + * auth_xform = xform; + */ + if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_AES_GMAC) { + IPSEC_MB_LOG(ERR, + "Only AES GMAC is supported as an authentication only algorithm"); + ret = -ENOTSUP; + goto error_exit; + } + /* Set IV parameters */ + sess->iv.offset = auth_xform->auth.iv.offset; + sess->iv.length = auth_xform->auth.iv.length; + key_length = auth_xform->auth.key.length; + key = auth_xform->auth.key.data; + sess->req_digest_length = auth_xform->auth.digest_length; + break; + case IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT: + case IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT: + /* AES-GCM + * aead_xform = xform; + */ + + if (aead_xform->aead.algo != RTE_CRYPTO_AEAD_AES_GCM) { + IPSEC_MB_LOG(ERR, + "The only combined operation supported is AES GCM"); + ret = -ENOTSUP; + goto error_exit; + } + /* Set IV parameters */ + sess->iv.offset = aead_xform->aead.iv.offset; + sess->iv.length = aead_xform->aead.iv.length; + key_length = aead_xform->aead.key.length; + key = aead_xform->aead.key.data; + sess->aad_length = aead_xform->aead.aad_length; + sess->req_digest_length = aead_xform->aead.digest_length; + break; + default: + IPSEC_MB_LOG( + ERR, "Wrong xform type, has to be AEAD or authentication"); + ret = -ENOTSUP; + goto error_exit; + } + + /* Check key length, and calculate GCM pre-compute. */ + switch (key_length) { + case 16: + sess->key_length = GCM_KEY_128; + mb_mgr->gcm128_pre(key, &sess->gdata_key); + break; + case 24: + sess->key_length = GCM_KEY_192; + mb_mgr->gcm192_pre(key, &sess->gdata_key); + break; + case 32: + sess->key_length = GCM_KEY_256; + mb_mgr->gcm256_pre(key, &sess->gdata_key); + break; + default: + IPSEC_MB_LOG(ERR, "Invalid key length"); + ret = -EINVAL; + goto error_exit; + } + + /* Digest check */ + if (sess->req_digest_length > 16) { + IPSEC_MB_LOG(ERR, "Invalid digest length"); + ret = -EINVAL; + goto error_exit; + } + /* + * If size requested is different, generate the full digest + * (16 bytes) in a temporary location and then memcpy + * the requested number of bytes. + */ + if (sess->req_digest_length < 4) + sess->gen_digest_length = 16; + else + sess->gen_digest_length = sess->req_digest_length; + +error_exit: + return ret; +} + +/** + * Process a completed job and return rte_mbuf which job processed + * + * @param job IMB_JOB job to process + * + * @return + * - Returns processed mbuf which is trimmed of output digest used in + * verification of supplied digest in the case of a HASH_CIPHER operation + * - Returns NULL on invalid job + */ +static void +post_process_gcm_crypto_op(struct ipsec_mb_qp *qp, + struct rte_crypto_op *op, + struct aesni_gcm_session *session) +{ + struct aesni_gcm_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); + + op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + /* Verify digest if required */ + if (session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT || + session->op == IPSEC_MB_OP_HASH_VERIFY_ONLY) { + uint8_t *digest; + + uint8_t *tag = qp_data->temp_digest; + + if (session->op == IPSEC_MB_OP_HASH_VERIFY_ONLY) + digest = op->sym->auth.digest.data; + else + digest = op->sym->aead.digest.data; + +#ifdef RTE_LIBRTE_PMD_AESNI_GCM_DEBUG + rte_hexdump(stdout, "auth tag (orig):", + digest, session->req_digest_length); + rte_hexdump(stdout, "auth tag (calc):", + tag, session->req_digest_length); +#endif + + if (memcmp(tag, digest, session->req_digest_length) != 0) + op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + } else { + if (session->req_digest_length != session->gen_digest_length) { + if (session->op == + IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT) + memcpy(op->sym->aead.digest.data, + qp_data->temp_digest, + session->req_digest_length); + else + memcpy(op->sym->auth.digest.data, + qp_data->temp_digest, + session->req_digest_length); + } + } +} + +/** + * Process a completed GCM request + * + * @param qp Queue Pair to process + * @param op Crypto operation + * @param sess AESNI-GCM session + * + */ +static void +handle_completed_gcm_crypto_op(struct ipsec_mb_qp *qp, + struct rte_crypto_op *op, + struct aesni_gcm_session *sess) +{ + post_process_gcm_crypto_op(qp, op, sess); + + /* Free session if a session-less crypto op */ + if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + memset(sess, 0, sizeof(struct aesni_gcm_session)); + memset(op->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + op->sym->session)); + rte_mempool_put(qp->sess_mp_priv, sess); + rte_mempool_put(qp->sess_mp, op->sym->session); + op->sym->session = NULL; + } +} + +/** + * Process a crypto operation, calling + * the GCM API from the multi buffer library. + * + * @param qp queue pair + * @param op symmetric crypto operation + * @param session GCM session + * + * @return + * 0 on success + */ +static int +process_gcm_crypto_op(struct ipsec_mb_qp *qp, struct rte_crypto_op *op, + struct aesni_gcm_session *session) +{ + struct aesni_gcm_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); + uint8_t *src, *dst; + uint8_t *iv_ptr; + struct rte_crypto_sym_op *sym_op = op->sym; + struct rte_mbuf *m_src = sym_op->m_src; + uint32_t offset, data_offset, data_length; + uint32_t part_len, total_len, data_len; + uint8_t *tag; + unsigned int oop = 0; + struct aesni_gcm_ops *ops = &qp_data->ops[session->key_length]; + + if (session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT || + session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT) { + offset = sym_op->aead.data.offset; + data_offset = offset; + data_length = sym_op->aead.data.length; + } else { + offset = sym_op->auth.data.offset; + data_offset = offset; + data_length = sym_op->auth.data.length; + } + + RTE_ASSERT(m_src != NULL); + + while (offset >= m_src->data_len && data_length != 0) { + offset -= m_src->data_len; + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + } + + src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset); + + data_len = m_src->data_len - offset; + part_len = (data_len < data_length) ? data_len : + data_length; + + RTE_ASSERT((sym_op->m_dst == NULL) || + ((sym_op->m_dst != NULL) && + rte_pktmbuf_is_contiguous(sym_op->m_dst))); + + /* In-place */ + if (sym_op->m_dst == NULL || (sym_op->m_dst == sym_op->m_src)) + dst = src; + /* Out-of-place */ + else { + oop = 1; + /* Segmented destination buffer is not supported + * if operation is Out-of-place + */ + RTE_ASSERT(rte_pktmbuf_is_contiguous(sym_op->m_dst)); + dst = rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *, + data_offset); + } + + iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + session->iv.offset); + + if (session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT) { + ops->init(&session->gdata_key, &qp_data->gcm_ctx_data, iv_ptr, + sym_op->aead.aad.data, + (uint64_t)session->aad_length); + + ops->update_enc(&session->gdata_key, &qp_data->gcm_ctx_data, + dst, src, (uint64_t)part_len); + total_len = data_length - part_len; + + while (total_len) { + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + + src = rte_pktmbuf_mtod(m_src, uint8_t *); + if (oop) + dst += part_len; + else + dst = src; + part_len = (m_src->data_len < total_len) ? + m_src->data_len : total_len; + + ops->update_enc(&session->gdata_key, + &qp_data->gcm_ctx_data, + dst, src, (uint64_t)part_len); + total_len -= part_len; + } + + if (session->req_digest_length != session->gen_digest_length) + tag = qp_data->temp_digest; + else + tag = sym_op->aead.digest.data; + + ops->finalize_enc(&session->gdata_key, &qp_data->gcm_ctx_data, + tag, session->gen_digest_length); + } else if (session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT) { + ops->init(&session->gdata_key, &qp_data->gcm_ctx_data, iv_ptr, + sym_op->aead.aad.data, + (uint64_t)session->aad_length); + + ops->update_dec(&session->gdata_key, &qp_data->gcm_ctx_data, + dst, src, (uint64_t)part_len); + total_len = data_length - part_len; + + while (total_len) { + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + + src = rte_pktmbuf_mtod(m_src, uint8_t *); + if (oop) + dst += part_len; + else + dst = src; + part_len = (m_src->data_len < total_len) ? + m_src->data_len : total_len; + + ops->update_dec(&session->gdata_key, + &qp_data->gcm_ctx_data, + dst, src, (uint64_t)part_len); + total_len -= part_len; + } + + tag = qp_data->temp_digest; + ops->finalize_dec(&session->gdata_key, &qp_data->gcm_ctx_data, + tag, session->gen_digest_length); + } else if (session->op == IPSEC_MB_OP_HASH_GEN_ONLY) { + ops->gmac_init(&session->gdata_key, &qp_data->gcm_ctx_data, + iv_ptr, session->iv.length); + + ops->gmac_update(&session->gdata_key, &qp_data->gcm_ctx_data, + src, (uint64_t)part_len); + total_len = data_length - part_len; + + while (total_len) { + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + + src = rte_pktmbuf_mtod(m_src, uint8_t *); + part_len = (m_src->data_len < total_len) ? + m_src->data_len : total_len; + + ops->gmac_update(&session->gdata_key, + &qp_data->gcm_ctx_data, src, + (uint64_t)part_len); + total_len -= part_len; + } + + if (session->req_digest_length != session->gen_digest_length) + tag = qp_data->temp_digest; + else + tag = sym_op->auth.digest.data; + + ops->gmac_finalize(&session->gdata_key, &qp_data->gcm_ctx_data, + tag, session->gen_digest_length); + } else { /* IPSEC_MB_OP_HASH_VERIFY_ONLY */ + ops->gmac_init(&session->gdata_key, &qp_data->gcm_ctx_data, + iv_ptr, session->iv.length); + + ops->gmac_update(&session->gdata_key, &qp_data->gcm_ctx_data, + src, (uint64_t)part_len); + total_len = data_length - part_len; + + while (total_len) { + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + + src = rte_pktmbuf_mtod(m_src, uint8_t *); + part_len = (m_src->data_len < total_len) ? + m_src->data_len : total_len; + + ops->gmac_update(&session->gdata_key, + &qp_data->gcm_ctx_data, src, + (uint64_t)part_len); + total_len -= part_len; + } + + tag = qp_data->temp_digest; + + ops->gmac_finalize(&session->gdata_key, &qp_data->gcm_ctx_data, + tag, session->gen_digest_length); + } + return 0; +} + +/** Get gcm session */ +static inline struct aesni_gcm_session * +aesni_gcm_get_session(struct ipsec_mb_qp *qp, + struct rte_crypto_op *op) +{ + struct aesni_gcm_session *sess = NULL; + uint32_t driver_id = + ipsec_mb_get_driver_id(IPSEC_MB_PMD_TYPE_AESNI_GCM); + struct rte_crypto_sym_op *sym_op = op->sym; + + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + if (likely(sym_op->session != NULL)) + sess = (struct aesni_gcm_session *) + get_sym_session_private_data(sym_op->session, + driver_id); + } else { + void *_sess; + void *_sess_private_data = NULL; + + if (rte_mempool_get(qp->sess_mp, (void **)&_sess)) + return NULL; + + if (rte_mempool_get(qp->sess_mp_priv, + (void **)&_sess_private_data)) + return NULL; + + sess = (struct aesni_gcm_session *)_sess_private_data; + + if (unlikely(aesni_gcm_session_configure(qp->mb_mgr, + _sess_private_data, sym_op->xform) != 0)) { + rte_mempool_put(qp->sess_mp, _sess); + rte_mempool_put(qp->sess_mp_priv, _sess_private_data); + sess = NULL; + } + sym_op->session = (struct rte_cryptodev_sym_session *)_sess; + set_sym_session_private_data(sym_op->session, driver_id, + _sess_private_data); + } + + if (unlikely(sess == NULL)) + op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; + + return sess; +} + +static uint16_t +aesni_gcm_pmd_dequeue_burst(void *queue_pair, + struct rte_crypto_op **ops, uint16_t nb_ops) +{ + struct aesni_gcm_session *sess; + struct ipsec_mb_qp *qp = queue_pair; + + int retval = 0; + unsigned int i, nb_dequeued; + + nb_dequeued = rte_ring_dequeue_burst(qp->ingress_queue, + (void **)ops, nb_ops, NULL); + + for (i = 0; i < nb_dequeued; i++) { + + sess = aesni_gcm_get_session(qp, ops[i]); + if (unlikely(sess == NULL)) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + qp->stats.dequeue_err_count++; + break; + } + + retval = process_gcm_crypto_op(qp, ops[i], sess); + if (retval < 0) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + qp->stats.dequeue_err_count++; + break; + } + + handle_completed_gcm_crypto_op(qp, ops[i], sess); + } + + qp->stats.dequeued_count += i; + + return i; +} + +static inline void +aesni_gcm_fill_error_code(struct rte_crypto_sym_vec *vec, + int32_t errnum) +{ + uint32_t i; + + for (i = 0; i < vec->num; i++) + vec->status[i] = errnum; +} + +static inline int32_t +aesni_gcm_sgl_op_finalize_encryption(const struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + uint8_t *digest, struct aesni_gcm_ops ops) +{ + if (s->req_digest_length != s->gen_digest_length) { + uint8_t tmpdigest[s->gen_digest_length]; + + ops.finalize_enc(&s->gdata_key, gdata_ctx, tmpdigest, + s->gen_digest_length); + memcpy(digest, tmpdigest, s->req_digest_length); + } else { + ops.finalize_enc(&s->gdata_key, gdata_ctx, digest, + s->gen_digest_length); + } + + return 0; +} + +static inline int32_t +aesni_gcm_sgl_op_finalize_decryption(const struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + uint8_t *digest, struct aesni_gcm_ops ops) +{ + uint8_t tmpdigest[s->gen_digest_length]; + + ops.finalize_dec(&s->gdata_key, gdata_ctx, tmpdigest, + s->gen_digest_length); + + return memcmp(digest, tmpdigest, s->req_digest_length) == 0 ? 0 + : EBADMSG; +} + +static inline void +aesni_gcm_process_gcm_sgl_op(const struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + struct rte_crypto_sgl *sgl, void *iv, void *aad, + struct aesni_gcm_ops ops) +{ + uint32_t i; + + /* init crypto operation */ + ops.init(&s->gdata_key, gdata_ctx, iv, aad, + (uint64_t)s->aad_length); + + /* update with sgl data */ + for (i = 0; i < sgl->num; i++) { + struct rte_crypto_vec *vec = &sgl->vec[i]; + + switch (s->op) { + case IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT: + ops.update_enc(&s->gdata_key, gdata_ctx, + vec->base, vec->base, vec->len); + break; + case IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT: + ops.update_dec(&s->gdata_key, gdata_ctx, + vec->base, vec->base, vec->len); + break; + default: + IPSEC_MB_LOG(ERR, "Invalid session op"); + break; + } + + } +} + +static inline void +aesni_gcm_process_gmac_sgl_op(const struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + struct rte_crypto_sgl *sgl, void *iv, + struct aesni_gcm_ops ops) +{ + ops.init(&s->gdata_key, gdata_ctx, iv, sgl->vec[0].base, + sgl->vec[0].len); +} + +static inline uint32_t +aesni_gcm_sgl_encrypt(struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + struct rte_crypto_sym_vec *vec, + struct aesni_gcm_ops ops) +{ + uint32_t i, processed; + + processed = 0; + for (i = 0; i < vec->num; ++i) { + aesni_gcm_process_gcm_sgl_op(s, gdata_ctx, &vec->sgl[i], + vec->iv[i].va, vec->aad[i].va, + ops); + vec->status[i] = aesni_gcm_sgl_op_finalize_encryption( + s, gdata_ctx, vec->digest[i].va, ops); + processed += (vec->status[i] == 0); + } + + return processed; +} + +static inline uint32_t +aesni_gcm_sgl_decrypt(struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + struct rte_crypto_sym_vec *vec, + struct aesni_gcm_ops ops) +{ + uint32_t i, processed; + + processed = 0; + for (i = 0; i < vec->num; ++i) { + aesni_gcm_process_gcm_sgl_op(s, gdata_ctx, &vec->sgl[i], + vec->iv[i].va, vec->aad[i].va, + ops); + vec->status[i] = aesni_gcm_sgl_op_finalize_decryption( + s, gdata_ctx, vec->digest[i].va, ops); + processed += (vec->status[i] == 0); + } + + return processed; +} + +static inline uint32_t +aesni_gmac_sgl_generate(struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + struct rte_crypto_sym_vec *vec, + struct aesni_gcm_ops ops) +{ + uint32_t i, processed; + + processed = 0; + for (i = 0; i < vec->num; ++i) { + if (vec->sgl[i].num != 1) { + vec->status[i] = ENOTSUP; + continue; + } + + aesni_gcm_process_gmac_sgl_op(s, gdata_ctx, &vec->sgl[i], + vec->iv[i].va, ops); + vec->status[i] = aesni_gcm_sgl_op_finalize_encryption( + s, gdata_ctx, vec->digest[i].va, ops); + processed += (vec->status[i] == 0); + } + + return processed; +} + +static inline uint32_t +aesni_gmac_sgl_verify(struct aesni_gcm_session *s, + struct gcm_context_data *gdata_ctx, + struct rte_crypto_sym_vec *vec, + struct aesni_gcm_ops ops) +{ + uint32_t i, processed; + + processed = 0; + for (i = 0; i < vec->num; ++i) { + if (vec->sgl[i].num != 1) { + vec->status[i] = ENOTSUP; + continue; + } + + aesni_gcm_process_gmac_sgl_op(s, gdata_ctx, &vec->sgl[i], + vec->iv[i].va, ops); + vec->status[i] = aesni_gcm_sgl_op_finalize_decryption( + s, gdata_ctx, vec->digest[i].va, ops); + processed += (vec->status[i] == 0); + } + + return processed; +} + +/** Process CPU crypto bulk operations */ +static uint32_t +aesni_gcm_process_bulk(struct rte_cryptodev *dev, + struct rte_cryptodev_sym_session *sess, + __rte_unused union rte_crypto_sym_ofs ofs, + struct rte_crypto_sym_vec *vec) +{ + void *sess_priv; + struct aesni_gcm_session *s; + struct gcm_context_data gdata_ctx; + IMB_MGR *mb_mgr; + + sess_priv = get_sym_session_private_data(sess, dev->driver_id); + if (unlikely(sess_priv == NULL)) { + aesni_gcm_fill_error_code(vec, EINVAL); + return 0; + } + + s = sess_priv; + + /* get per-thread MB MGR, create one if needed */ + mb_mgr = get_per_thread_mb_mgr(); + if (unlikely(mb_mgr == NULL)) + return 0; + + /* Check if function pointers have been set for this thread ops. */ + if (unlikely(RTE_PER_LCORE(gcm_ops)[s->key_length].init == NULL)) + aesni_gcm_set_ops(RTE_PER_LCORE(gcm_ops), mb_mgr); + + switch (s->op) { + case IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT: + return aesni_gcm_sgl_encrypt(s, &gdata_ctx, vec, + RTE_PER_LCORE(gcm_ops)[s->key_length]); + case IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT: + return aesni_gcm_sgl_decrypt(s, &gdata_ctx, vec, + RTE_PER_LCORE(gcm_ops)[s->key_length]); + case IPSEC_MB_OP_HASH_GEN_ONLY: + return aesni_gmac_sgl_generate(s, &gdata_ctx, vec, + RTE_PER_LCORE(gcm_ops)[s->key_length]); + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + return aesni_gmac_sgl_verify(s, &gdata_ctx, vec, + RTE_PER_LCORE(gcm_ops)[s->key_length]); + default: + aesni_gcm_fill_error_code(vec, EINVAL); + return 0; + } +} + +static int +aesni_gcm_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, + const struct rte_cryptodev_qp_conf *qp_conf, + int socket_id) +{ + int ret = ipsec_mb_qp_setup(dev, qp_id, qp_conf, socket_id); + if (ret < 0) + return ret; + + struct ipsec_mb_qp *qp = dev->data->queue_pairs[qp_id]; + struct aesni_gcm_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); + aesni_gcm_set_ops(qp_data->ops, qp->mb_mgr); + return 0; +} + +struct rte_cryptodev_ops aesni_gcm_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, + + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, + + .dev_infos_get = ipsec_mb_info_get, + + .queue_pair_setup = aesni_gcm_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, + + .sym_cpu_process = aesni_gcm_process_bulk, + + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; + +static int +aesni_gcm_probe(struct rte_vdev_device *vdev) +{ + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_AESNI_GCM); +} + +static struct rte_vdev_driver cryptodev_aesni_gcm_pmd_drv = { + .probe = aesni_gcm_probe, + .remove = ipsec_mb_remove +}; + +static struct cryptodev_driver aesni_gcm_crypto_drv; + +RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_AESNI_GCM_PMD, + cryptodev_aesni_gcm_pmd_drv); +RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_AESNI_GCM_PMD, cryptodev_aesni_gcm_pmd); +RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_AESNI_GCM_PMD, + "max_nb_queue_pairs= socket_id="); +RTE_PMD_REGISTER_CRYPTO_DRIVER(aesni_gcm_crypto_drv, + cryptodev_aesni_gcm_pmd_drv.driver, + pmd_driver_id_aesni_gcm); + +/* Constructor function to register aesni-gcm PMD */ +RTE_INIT(ipsec_mb_register_aesni_gcm) +{ + struct ipsec_mb_internals *aesni_gcm_data = + &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_AESNI_GCM]; + + aesni_gcm_data->caps = aesni_gcm_capabilities; + aesni_gcm_data->dequeue_burst = aesni_gcm_pmd_dequeue_burst; + aesni_gcm_data->feature_flags = + RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | + RTE_CRYPTODEV_FF_IN_PLACE_SGL | + RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | + RTE_CRYPTODEV_FF_SYM_CPU_CRYPTO | + RTE_CRYPTODEV_FF_SYM_SESSIONLESS; + aesni_gcm_data->internals_priv_size = 0; + aesni_gcm_data->ops = &aesni_gcm_pmd_ops; + aesni_gcm_data->qp_priv_size = sizeof(struct aesni_gcm_qp_data); + aesni_gcm_data->queue_pair_configure = NULL; + aesni_gcm_data->session_configure = aesni_gcm_session_configure; + aesni_gcm_data->session_priv_size = sizeof(struct aesni_gcm_session); +} diff --git a/drivers/crypto/ipsec_mb/pmd_aesni_gcm_priv.h b/drivers/crypto/ipsec_mb/pmd_aesni_gcm_priv.h new file mode 100644 index 0000000000..0f6bd7a2a5 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_aesni_gcm_priv.h @@ -0,0 +1,169 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#ifndef _PMD_AESNI_GCM_PRIV_H_ +#define _PMD_AESNI_GCM_PRIV_H_ + +#include + +#include "ipsec_mb_private.h" + +#define AESNI_GCM_IV_LENGTH 12 + +static const struct rte_cryptodev_capabilities aesni_gcm_capabilities[] = { + { /* AES GMAC (AUTH) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_AES_GMAC, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .digest_size = { + .min = 1, + .max = 16, + .increment = 1 + }, + .iv_size = { + .min = AESNI_GCM_IV_LENGTH, + .max = AESNI_GCM_IV_LENGTH, + .increment = 0 + } + }, } + }, } + }, + { /* AES GCM */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, + {.aead = { + .algo = RTE_CRYPTO_AEAD_AES_GCM, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .digest_size = { + .min = 1, + .max = 16, + .increment = 1 + }, + .aad_size = { + .min = 0, + .max = 65535, + .increment = 1 + }, + .iv_size = { + .min = AESNI_GCM_IV_LENGTH, + .max = AESNI_GCM_IV_LENGTH, + .increment = 0 + } + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +uint8_t pmd_driver_id_aesni_gcm; + +enum aesni_gcm_key_length { + GCM_KEY_128 = 0, + GCM_KEY_192, + GCM_KEY_256, + GCM_NUM_KEY_TYPES +}; + +typedef void (*aesni_gcm_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + uint8_t *out, const uint8_t *in, + uint64_t plaintext_len, const uint8_t *iv, + const uint8_t *aad, uint64_t aad_len, + uint8_t *auth_tag, uint64_t auth_tag_len); + +typedef void (*aesni_gcm_pre_t)(const void *key, + struct gcm_key_data *gcm_data); + +typedef void (*aesni_gcm_init_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + const uint8_t *iv, uint8_t const *aad, + uint64_t aad_len); + +typedef void (*aesni_gcm_update_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + uint8_t *out, const uint8_t *in, + uint64_t plaintext_len); + +typedef void (*aesni_gcm_finalize_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + uint8_t *auth_tag, uint64_t auth_tag_len); + +typedef void (*aesni_gmac_init_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + const uint8_t *iv, const uint64_t iv_len); + +typedef void (*aesni_gmac_update_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + const uint8_t *in, + const uint64_t plaintext_len); + +typedef void (*aesni_gmac_finalize_t)(const struct gcm_key_data *gcm_key_data, + struct gcm_context_data *gcm_ctx_data, + uint8_t *auth_tag, + const uint64_t auth_tag_len); + +/** GCM operation handlers */ +struct aesni_gcm_ops { + aesni_gcm_t enc; + aesni_gcm_t dec; + aesni_gcm_pre_t pre; + aesni_gcm_init_t init; + aesni_gcm_update_t update_enc; + aesni_gcm_update_t update_dec; + aesni_gcm_finalize_t finalize_enc; + aesni_gcm_finalize_t finalize_dec; + aesni_gmac_init_t gmac_init; + aesni_gmac_update_t gmac_update; + aesni_gmac_finalize_t gmac_finalize; +}; + +RTE_DEFINE_PER_LCORE(struct aesni_gcm_ops[GCM_NUM_KEY_TYPES], gcm_ops); + +struct aesni_gcm_qp_data { + struct gcm_context_data gcm_ctx_data; + uint8_t temp_digest[DIGEST_LENGTH_MAX]; + /* *< Buffers used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ + struct aesni_gcm_ops ops[GCM_NUM_KEY_TYPES]; + /**< Operation Handlers */ +}; + +/** AESNI GCM private session structure */ +struct aesni_gcm_session { + struct { + uint16_t length; + uint16_t offset; + } iv; + /**< IV parameters */ + uint16_t aad_length; + /**< AAD length */ + uint16_t req_digest_length; + /**< Requested digest length */ + uint16_t gen_digest_length; + /**< Generated digest length */ + enum ipsec_mb_operation op; + /**< GCM operation type */ + struct gcm_key_data gdata_key; + /**< GCM parameters */ + enum aesni_gcm_key_length key_length; + /** Key Length */ +}; + +#endif /* _PMD_AESNI_GCM_PRIV_H_ */ diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index d409251214..00a31d52f6 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -6,7 +6,6 @@ if is_windows endif drivers = [ - 'aesni_gcm', 'armv8', 'bcmfs', 'caam_jr', From patchwork Fri Oct 15 14:39:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101780 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EEE63A0C45; Fri, 15 Oct 2021 16:41:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A4C8241277; Fri, 15 Oct 2021 16:40:40 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 788FB41296 for ; Fri, 15 Oct 2021 16:40:38 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816179" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816179" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542886" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:35 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, =?utf-8?q?Ciara=C2=A0Power?= , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:52 +0000 Message-Id: <20211015143957.842499-10-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 09/14] drivers/crypto: move kasumi PMD to IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch removes the crypto/kasumi folder and gathers all kasumi PMD implementation specific details into a single file, pmd_kasumi.c in the crypto/ipsec_mb folder. Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - Fixed some function names. - Removed unnecessary enqueued counts from process_ops. - Added release note. - Split into private header file. - Fixed some checkpatch errors. v2: Updated maintainers file. --- MAINTAINERS | 6 +- doc/guides/cryptodevs/kasumi.rst | 3 +- doc/guides/rel_notes/release_21_11.rst | 1 + drivers/crypto/ipsec_mb/ipsec_mb_private.h | 7 + drivers/crypto/ipsec_mb/meson.build | 3 +- drivers/crypto/ipsec_mb/pmd_kasumi.c | 476 +++++++++++++++ drivers/crypto/ipsec_mb/pmd_kasumi_priv.h | 81 +++ drivers/crypto/kasumi/kasumi_pmd_private.h | 81 --- drivers/crypto/kasumi/meson.build | 24 - drivers/crypto/kasumi/rte_kasumi_pmd.c | 642 --------------------- drivers/crypto/kasumi/rte_kasumi_pmd_ops.c | 316 ---------- drivers/crypto/kasumi/version.map | 3 - drivers/crypto/meson.build | 1 - 13 files changed, 570 insertions(+), 1074 deletions(-) create mode 100644 drivers/crypto/ipsec_mb/pmd_kasumi.c create mode 100644 drivers/crypto/ipsec_mb/pmd_kasumi_priv.h delete mode 100644 drivers/crypto/kasumi/kasumi_pmd_private.h delete mode 100644 drivers/crypto/kasumi/meson.build delete mode 100644 drivers/crypto/kasumi/rte_kasumi_pmd.c delete mode 100644 drivers/crypto/kasumi/rte_kasumi_pmd_ops.c delete mode 100644 drivers/crypto/kasumi/version.map diff --git a/MAINTAINERS b/MAINTAINERS index af062cf00d..12b354a304 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1059,13 +1059,9 @@ M: Pablo de Lara F: drivers/crypto/ipsec_mb/ F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/aesni_mb.rst +F: doc/guides/cryptodevs/kasumi.rst F: doc/guides/cryptodevs/features/aesni_gcm.ini F: doc/guides/cryptodevs/features/aesni_mb.ini - -KASUMI -M: Pablo de Lara -F: drivers/crypto/kasumi/ -F: doc/guides/cryptodevs/kasumi.rst F: doc/guides/cryptodevs/features/kasumi.ini Marvell cnxk crypto diff --git a/doc/guides/cryptodevs/kasumi.rst b/doc/guides/cryptodevs/kasumi.rst index 35c5941317..bc82744fcc 100644 --- a/doc/guides/cryptodevs/kasumi.rst +++ b/doc/guides/cryptodevs/kasumi.rst @@ -78,7 +78,8 @@ and the external crypto libraries supported by them: DPDK version Crypto library version ============= ================================ 16.11 - 19.11 LibSSO KASUMI - 20.02+ Multi-buffer library 0.53 - 1.0* + 20.02 - 21.08 Multi-buffer library 0.53 - 1.0* + 21.11+ Multi-buffer library 1.0* ============= ================================ \* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 43814a2320..7b4016f33a 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -106,6 +106,7 @@ New Features * AESNI_MB PMD. * AESNI_GCM PMD. + * KASUMI PMD. * **Updated the aesni_mb crypto PMD.** diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index ed755dbf13..a545296580 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -40,6 +40,9 @@ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); #define CRYPTODEV_NAME_AESNI_GCM_PMD crypto_aesni_gcm /**< IPSEC Multi buffer PMD aesni_gcm device name */ +#define CRYPTODEV_NAME_KASUMI_PMD crypto_kasumi +/**< IPSEC Multi buffer PMD kasumi device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -50,6 +53,7 @@ extern int ipsec_mb_logtype_driver; enum ipsec_mb_pmd_types { IPSEC_MB_PMD_TYPE_AESNI_MB = 0, IPSEC_MB_PMD_TYPE_AESNI_GCM, + IPSEC_MB_PMD_TYPE_KASUMI, IPSEC_MB_N_PMD_TYPES }; @@ -70,6 +74,7 @@ enum ipsec_mb_operation { extern uint8_t pmd_driver_id_aesni_mb; extern uint8_t pmd_driver_id_aesni_gcm; +extern uint8_t pmd_driver_id_kasumi; /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t @@ -80,6 +85,8 @@ ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) return pmd_driver_id_aesni_mb; case IPSEC_MB_PMD_TYPE_AESNI_GCM: return pmd_driver_id_aesni_gcm; + case IPSEC_MB_PMD_TYPE_KASUMI: + return pmd_driver_id_kasumi; default: break; } diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index b8166c7e75..54d2b69480 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -24,6 +24,7 @@ endif sources = files('ipsec_mb_private.c', 'ipsec_mb_ops.c', 'pmd_aesni_mb.c', - 'pmd_aesni_gcm.c' + 'pmd_aesni_gcm.c', + 'pmd_kasumi.c' ) deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/ipsec_mb/pmd_kasumi.c b/drivers/crypto/ipsec_mb/pmd_kasumi.c new file mode 100644 index 0000000000..c9d4f9d0ae --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_kasumi.c @@ -0,0 +1,476 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#include +#include +#include +#include +#include +#include + +#include "pmd_kasumi_priv.h" + +/** Parse crypto xform chain and set private session parameters. */ +static int +kasumi_session_configure(IMB_MGR *mgr, void *priv_sess, + const struct rte_crypto_sym_xform *xform) +{ + const struct rte_crypto_sym_xform *auth_xform = NULL; + const struct rte_crypto_sym_xform *cipher_xform = NULL; + enum ipsec_mb_operation mode; + struct kasumi_session *sess = (struct kasumi_session *)priv_sess; + /* Select Crypto operation - hash then cipher / cipher then hash */ + int ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, NULL); + + if (ret) + return ret; + + if (cipher_xform) { + /* Only KASUMI F8 supported */ + if (cipher_xform->cipher.algo != RTE_CRYPTO_CIPHER_KASUMI_F8) { + IPSEC_MB_LOG(ERR, "Unsupported cipher algorithm "); + return -ENOTSUP; + } + + sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; + if (cipher_xform->cipher.iv.length != KASUMI_IV_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong IV length"); + return -EINVAL; + } + + /* Initialize key */ + IMB_KASUMI_INIT_F8_KEY_SCHED(mgr, + cipher_xform->cipher.key.data, + &sess->pKeySched_cipher); + } + + if (auth_xform) { + /* Only KASUMI F9 supported */ + if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_KASUMI_F9) { + IPSEC_MB_LOG(ERR, "Unsupported authentication"); + return -ENOTSUP; + } + + if (auth_xform->auth.digest_length != KASUMI_DIGEST_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong digest length"); + return -EINVAL; + } + + sess->auth_op = auth_xform->auth.op; + + /* Initialize key */ + IMB_KASUMI_INIT_F9_KEY_SCHED(mgr, auth_xform->auth.key.data, + &sess->pKeySched_hash); + } + + sess->op = mode; + return ret; +} + +/** Encrypt/decrypt mbufs with same cipher key. */ +static uint8_t +process_kasumi_cipher_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, + struct kasumi_session *session, uint8_t num_ops) +{ + unsigned int i; + uint8_t processed_ops = 0; + const void *src[num_ops]; + void *dst[num_ops]; + uint8_t *iv_ptr; + uint64_t iv[num_ops]; + uint32_t num_bytes[num_ops]; + + for (i = 0; i < num_ops; i++) { + src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3); + dst[i] = ops[i]->sym->m_dst + ? rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3) + : rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3); + iv_ptr = rte_crypto_op_ctod_offset(ops[i], uint8_t *, + session->cipher_iv_offset); + iv[i] = *((uint64_t *)(iv_ptr)); + num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; + + processed_ops++; + } + + if (processed_ops != 0) + IMB_KASUMI_F8_N_BUFFER(qp->mb_mgr, &session->pKeySched_cipher, + iv, src, dst, num_bytes, + processed_ops); + + return processed_ops; +} + +/** Encrypt/decrypt mbuf (bit level function). */ +static uint8_t +process_kasumi_cipher_op_bit(struct ipsec_mb_qp *qp, struct rte_crypto_op *op, + struct kasumi_session *session) +{ + uint8_t *src, *dst; + uint8_t *iv_ptr; + uint64_t iv; + uint32_t length_in_bits, offset_in_bits; + + offset_in_bits = op->sym->cipher.data.offset; + src = rte_pktmbuf_mtod(op->sym->m_src, uint8_t *); + if (op->sym->m_dst == NULL) + dst = src; + else + dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *); + iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + session->cipher_iv_offset); + iv = *((uint64_t *)(iv_ptr)); + length_in_bits = op->sym->cipher.data.length; + + IMB_KASUMI_F8_1_BUFFER_BIT(qp->mb_mgr, &session->pKeySched_cipher, iv, + src, dst, length_in_bits, offset_in_bits); + + return 1; +} + +/** Generate/verify hash from mbufs with same hash key. */ +static int +process_kasumi_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, + struct kasumi_session *session, uint8_t num_ops) +{ + unsigned int i; + uint8_t processed_ops = 0; + uint8_t *src, *dst; + uint32_t length_in_bits; + uint32_t num_bytes; + struct kasumi_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); + + for (i = 0; i < num_ops; i++) { + /* Data must be byte aligned */ + if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + IPSEC_MB_LOG(ERR, "Invalid Offset"); + break; + } + + length_in_bits = ops[i]->sym->auth.data.length; + + src = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->auth.data.offset >> 3); + /* Direction from next bit after end of message */ + num_bytes = length_in_bits >> 3; + + if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { + dst = qp_data->temp_digest; + IMB_KASUMI_F9_1_BUFFER(qp->mb_mgr, + &session->pKeySched_hash, src, + num_bytes, dst); + + /* Verify digest. */ + if (memcmp(dst, ops[i]->sym->auth.digest.data, + KASUMI_DIGEST_LENGTH) + != 0) + ops[i]->status + = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + } else { + dst = ops[i]->sym->auth.digest.data; + + IMB_KASUMI_F9_1_BUFFER(qp->mb_mgr, + &session->pKeySched_hash, src, + num_bytes, dst); + } + processed_ops++; + } + + return processed_ops; +} + +/** Process a batch of crypto ops which shares the same session. */ +static int +process_ops(struct rte_crypto_op **ops, struct kasumi_session *session, + struct ipsec_mb_qp *qp, uint8_t num_ops) +{ + unsigned int i; + unsigned int processed_ops; + + switch (session->op) { + case IPSEC_MB_OP_ENCRYPT_ONLY: + case IPSEC_MB_OP_DECRYPT_ONLY: + processed_ops + = process_kasumi_cipher_op(qp, ops, session, num_ops); + break; + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + processed_ops + = process_kasumi_hash_op(qp, ops, session, num_ops); + break; + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + case IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY: + processed_ops + = process_kasumi_cipher_op(qp, ops, session, num_ops); + process_kasumi_hash_op(qp, ops, session, processed_ops); + break; + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + case IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT: + processed_ops + = process_kasumi_hash_op(qp, ops, session, num_ops); + process_kasumi_cipher_op(qp, ops, session, processed_ops); + break; + default: + /* Operation not supported. */ + processed_ops = 0; + } + + for (i = 0; i < num_ops; i++) { + /* + * If there was no error/authentication failure, + * change status to successful. + */ + if (ops[i]->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) + ops[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + /* Free session if a session-less crypto op. */ + if (ops[i]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + memset(session, 0, sizeof(struct kasumi_session)); + memset( + ops[i]->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + ops[i]->sym->session)); + rte_mempool_put(qp->sess_mp_priv, session); + rte_mempool_put(qp->sess_mp, ops[i]->sym->session); + ops[i]->sym->session = NULL; + } + } + return processed_ops; +} + +/** Process a crypto op with length/offset in bits. */ +static int +process_op_bit(struct rte_crypto_op *op, struct kasumi_session *session, + struct ipsec_mb_qp *qp) +{ + unsigned int processed_op; + + switch (session->op) { + /* case KASUMI_OP_ONLY_CIPHER: */ + case IPSEC_MB_OP_ENCRYPT_ONLY: + case IPSEC_MB_OP_DECRYPT_ONLY: + processed_op = process_kasumi_cipher_op_bit(qp, op, session); + break; + /* case KASUMI_OP_ONLY_AUTH: */ + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + processed_op = process_kasumi_hash_op(qp, &op, session, 1); + break; + /* case KASUMI_OP_CIPHER_AUTH: */ + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + processed_op = process_kasumi_cipher_op_bit(qp, op, session); + if (processed_op == 1) + process_kasumi_hash_op(qp, &op, session, 1); + break; + /* case KASUMI_OP_AUTH_CIPHER: */ + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + processed_op = process_kasumi_hash_op(qp, &op, session, 1); + if (processed_op == 1) + process_kasumi_cipher_op_bit(qp, op, session); + break; + default: + /* Operation not supported. */ + processed_op = 0; + } + + /* + * If there was no error/authentication failure, + * change status to successful. + */ + if (op->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) + op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + + /* Free session if a session-less crypto op. */ + if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + memset(op->sym->session, 0, sizeof(struct kasumi_session)); + rte_cryptodev_sym_session_free(op->sym->session); + op->sym->session = NULL; + } + return processed_op; +} + +static uint16_t +kasumi_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, + uint16_t nb_ops) +{ + struct rte_crypto_op *c_ops[nb_ops]; + struct rte_crypto_op *curr_c_op = NULL; + + struct kasumi_session *prev_sess = NULL, *curr_sess = NULL; + struct ipsec_mb_qp *qp = queue_pair; + unsigned int i; + uint8_t burst_size = 0; + uint8_t processed_ops; + unsigned int nb_dequeued; + + nb_dequeued = rte_ring_dequeue_burst(qp->ingress_queue, + (void **)ops, nb_ops, NULL); + for (i = 0; i < nb_dequeued; i++) { + curr_c_op = ops[i]; + +#ifdef RTE_LIBRTE_PMD_KASUMI_DEBUG + if (!rte_pktmbuf_is_contiguous(curr_c_op->sym->m_src) + || (curr_c_op->sym->m_dst != NULL + && !rte_pktmbuf_is_contiguous( + curr_c_op->sym->m_dst))) { + IPSEC_MB_LOG(ERR, + "PMD supports only contiguous mbufs, op (%p) provides noncontiguous mbuf as source/destination buffer.", + curr_c_op); + curr_c_op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + break; + } +#endif + + /* Set status as enqueued (not processed yet) by default. */ + curr_c_op->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED; + + curr_sess = (struct kasumi_session *) + ipsec_mb_get_session_private(qp, curr_c_op); + if (unlikely(curr_sess == NULL + || curr_sess->op == IPSEC_MB_OP_NOT_SUPPORTED)) { + curr_c_op->status + = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; + break; + } + + /* If length/offset is at bit-level, process this buffer alone. + */ + if (((curr_c_op->sym->cipher.data.length % BYTE_LEN) != 0) + || ((ops[i]->sym->cipher.data.offset % BYTE_LEN) != 0)) { + /* Process the ops of the previous session. */ + if (prev_sess != NULL) { + processed_ops = process_ops(c_ops, prev_sess, + qp, burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + prev_sess = NULL; + } + + processed_ops = process_op_bit(curr_c_op, + curr_sess, qp); + if (processed_ops != 1) + break; + + continue; + } + + /* Batch ops that share the same session. */ + if (prev_sess == NULL) { + prev_sess = curr_sess; + c_ops[burst_size++] = curr_c_op; + } else if (curr_sess == prev_sess) { + c_ops[burst_size++] = curr_c_op; + /* + * When there are enough ops to process in a batch, + * process them, and start a new batch. + */ + if (burst_size == KASUMI_MAX_BURST) { + processed_ops = process_ops(c_ops, prev_sess, + qp, burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + prev_sess = NULL; + } + } else { + /* + * Different session, process the ops + * of the previous session. + */ + processed_ops = process_ops(c_ops, prev_sess, qp, + burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + prev_sess = curr_sess; + + c_ops[burst_size++] = curr_c_op; + } + } + + if (burst_size != 0) { + /* Process the crypto ops of the last session. */ + processed_ops = process_ops(c_ops, prev_sess, qp, burst_size); + } + + qp->stats.dequeued_count += i; + return i; +} + +struct rte_cryptodev_ops kasumi_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, + + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, + + .dev_infos_get = ipsec_mb_info_get, + + .queue_pair_setup = ipsec_mb_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, + + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; + +struct rte_cryptodev_ops *rte_kasumi_pmd_ops = &kasumi_pmd_ops; + +static int +kasumi_probe(struct rte_vdev_device *vdev) +{ + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_KASUMI); +} + +static struct rte_vdev_driver cryptodev_kasumi_pmd_drv = { + .probe = kasumi_probe, + .remove = ipsec_mb_remove +}; + +static struct cryptodev_driver kasumi_crypto_drv; + +RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_KASUMI_PMD, cryptodev_kasumi_pmd_drv); +RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_KASUMI_PMD, cryptodev_kasumi_pmd); +RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_KASUMI_PMD, + "max_nb_queue_pairs= socket_id="); +RTE_PMD_REGISTER_CRYPTO_DRIVER(kasumi_crypto_drv, + cryptodev_kasumi_pmd_drv.driver, + pmd_driver_id_kasumi); + +/* Constructor function to register kasumi PMD */ +RTE_INIT(ipsec_mb_register_kasumi) +{ + struct ipsec_mb_internals *kasumi_data + = &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_KASUMI]; + + kasumi_data->caps = kasumi_capabilities; + kasumi_data->dequeue_burst = kasumi_pmd_dequeue_burst; + kasumi_data->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO + | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING + | RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA + | RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT + | RTE_CRYPTODEV_FF_SYM_SESSIONLESS + | RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + kasumi_data->internals_priv_size = 0; + kasumi_data->ops = &kasumi_pmd_ops; + kasumi_data->qp_priv_size = sizeof(struct kasumi_qp_data); + kasumi_data->session_configure = kasumi_session_configure; + kasumi_data->session_priv_size = sizeof(struct kasumi_session); +} diff --git a/drivers/crypto/ipsec_mb/pmd_kasumi_priv.h b/drivers/crypto/ipsec_mb/pmd_kasumi_priv.h new file mode 100644 index 0000000000..8db1d1cc5b --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_kasumi_priv.h @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#ifndef _PMD_KASUMI_PRIV_H_ +#define _PMD_KASUMI_PRIV_H_ + +#include "ipsec_mb_private.h" + +#define KASUMI_KEY_LENGTH 16 +#define KASUMI_IV_LENGTH 8 +#define KASUMI_MAX_BURST 4 +#define BYTE_LEN 8 +#define KASUMI_DIGEST_LENGTH 4 + +uint8_t pmd_driver_id_kasumi; + +static const struct rte_cryptodev_capabilities kasumi_capabilities[] = { + { /* KASUMI (F9) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_KASUMI_F9, + .block_size = 8, + .key_size = { + .min = KASUMI_KEY_LENGTH, + .max = KASUMI_KEY_LENGTH, + .increment = 0 + }, + .digest_size = { + .min = KASUMI_DIGEST_LENGTH, + .max = KASUMI_DIGEST_LENGTH, + .increment = 0 + }, + .iv_size = { 0 } + }, } + }, } + }, + { /* KASUMI (F8) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, + {.cipher = { + .algo = RTE_CRYPTO_CIPHER_KASUMI_F8, + .block_size = 8, + .key_size = { + .min = KASUMI_KEY_LENGTH, + .max = KASUMI_KEY_LENGTH, + .increment = 0 + }, + .iv_size = { + .min = KASUMI_IV_LENGTH, + .max = KASUMI_IV_LENGTH, + .increment = 0 + } + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +/** KASUMI private session structure */ +struct kasumi_session { + /* Keys have to be 16-byte aligned */ + kasumi_key_sched_t pKeySched_cipher; + kasumi_key_sched_t pKeySched_hash; + enum ipsec_mb_operation op; + enum rte_crypto_auth_operation auth_op; + uint16_t cipher_iv_offset; +} __rte_cache_aligned; + +struct kasumi_qp_data { + uint8_t temp_digest[KASUMI_DIGEST_LENGTH]; + /* *< Buffers used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ +}; + +#endif /* _PMD_KASUMI_PRIV_H_ */ diff --git a/drivers/crypto/kasumi/kasumi_pmd_private.h b/drivers/crypto/kasumi/kasumi_pmd_private.h deleted file mode 100644 index abedcd616d..0000000000 --- a/drivers/crypto/kasumi/kasumi_pmd_private.h +++ /dev/null @@ -1,81 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#ifndef _KASUMI_PMD_PRIVATE_H_ -#define _KASUMI_PMD_PRIVATE_H_ - -#include - -#define CRYPTODEV_NAME_KASUMI_PMD crypto_kasumi -/**< KASUMI PMD device name */ - -/** KASUMI PMD LOGTYPE DRIVER */ -extern int kasumi_logtype_driver; - -#define KASUMI_LOG(level, fmt, ...) \ - rte_log(RTE_LOG_ ## level, kasumi_logtype_driver, \ - "%s() line %u: " fmt "\n", __func__, __LINE__, \ - ## __VA_ARGS__) - -#define KASUMI_DIGEST_LENGTH 4 - -/** private data structure for each virtual KASUMI device */ -struct kasumi_private { - unsigned max_nb_queue_pairs; - /**< Max number of queue pairs supported by device */ - MB_MGR *mgr; - /**< Multi-buffer instance */ -}; - -/** KASUMI buffer queue pair */ -struct kasumi_qp { - uint16_t id; - /**< Queue Pair Identifier */ - char name[RTE_CRYPTODEV_NAME_MAX_LEN]; - /**< Unique Queue Pair Name */ - struct rte_ring *processed_ops; - /**< Ring for placing processed ops */ - struct rte_mempool *sess_mp; - /**< Session Mempool */ - struct rte_mempool *sess_mp_priv; - /**< Session Private Data Mempool */ - struct rte_cryptodev_stats qp_stats; - /**< Queue pair statistics */ - uint8_t temp_digest[KASUMI_DIGEST_LENGTH]; - /**< Buffer used to store the digest generated - * by the driver when verifying a digest provided - * by the user (using authentication verify operation) - */ - MB_MGR *mgr; - /**< Multi-buffer instance */ -} __rte_cache_aligned; - -enum kasumi_operation { - KASUMI_OP_ONLY_CIPHER, - KASUMI_OP_ONLY_AUTH, - KASUMI_OP_CIPHER_AUTH, - KASUMI_OP_AUTH_CIPHER, - KASUMI_OP_NOT_SUPPORTED -}; - -/** KASUMI private session structure */ -struct kasumi_session { - /* Keys have to be 16-byte aligned */ - kasumi_key_sched_t pKeySched_cipher; - kasumi_key_sched_t pKeySched_hash; - enum kasumi_operation op; - enum rte_crypto_auth_operation auth_op; - uint16_t cipher_iv_offset; -} __rte_cache_aligned; - - -int -kasumi_set_session_parameters(MB_MGR *mgr, struct kasumi_session *sess, - const struct rte_crypto_sym_xform *xform); - - -/** device specific operations function pointer structure */ -extern struct rte_cryptodev_ops *rte_kasumi_pmd_ops; - -#endif /* _KASUMI_PMD_PRIVATE_H_ */ diff --git a/drivers/crypto/kasumi/meson.build b/drivers/crypto/kasumi/meson.build deleted file mode 100644 index e6e0f08c3d..0000000000 --- a/drivers/crypto/kasumi/meson.build +++ /dev/null @@ -1,24 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018-2020 Intel Corporation - -IMB_required_ver = '0.53.0' -lib = cc.find_library('IPSec_MB', required: false) -if not lib.found() - build = false - reason = 'missing dependency, "libIPSec_MB"' -else - # version comes with quotes, so we split based on " and take the middle - imb_ver = cc.get_define('IMB_VERSION_STR', - prefix : '#include').split('"')[1] - - if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) - reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( - IMB_required_ver, imb_ver) - build = false - endif - -endif - -ext_deps += lib -sources = files('rte_kasumi_pmd.c', 'rte_kasumi_pmd_ops.c') -deps += ['bus_vdev'] diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd.c b/drivers/crypto/kasumi/rte_kasumi_pmd.c deleted file mode 100644 index d6f927417a..0000000000 --- a/drivers/crypto/kasumi/rte_kasumi_pmd.c +++ /dev/null @@ -1,642 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include -#include -#include -#include -#include -#include -#include - -#include "kasumi_pmd_private.h" - -#define KASUMI_KEY_LENGTH 16 -#define KASUMI_IV_LENGTH 8 -#define KASUMI_MAX_BURST 4 -#define BYTE_LEN 8 - -static uint8_t cryptodev_driver_id; - -/** Get xform chain order. */ -static enum kasumi_operation -kasumi_get_mode(const struct rte_crypto_sym_xform *xform) -{ - if (xform == NULL) - return KASUMI_OP_NOT_SUPPORTED; - - if (xform->next) - if (xform->next->next != NULL) - return KASUMI_OP_NOT_SUPPORTED; - - if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - if (xform->next == NULL) - return KASUMI_OP_ONLY_AUTH; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_CIPHER) - return KASUMI_OP_AUTH_CIPHER; - else - return KASUMI_OP_NOT_SUPPORTED; - } - - if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { - if (xform->next == NULL) - return KASUMI_OP_ONLY_CIPHER; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) - return KASUMI_OP_CIPHER_AUTH; - else - return KASUMI_OP_NOT_SUPPORTED; - } - - return KASUMI_OP_NOT_SUPPORTED; -} - - -/** Parse crypto xform chain and set private session parameters. */ -int -kasumi_set_session_parameters(MB_MGR *mgr, struct kasumi_session *sess, - const struct rte_crypto_sym_xform *xform) -{ - const struct rte_crypto_sym_xform *auth_xform = NULL; - const struct rte_crypto_sym_xform *cipher_xform = NULL; - enum kasumi_operation mode; - - /* Select Crypto operation - hash then cipher / cipher then hash */ - mode = kasumi_get_mode(xform); - - switch (mode) { - case KASUMI_OP_CIPHER_AUTH: - auth_xform = xform->next; - /* Fall-through */ - case KASUMI_OP_ONLY_CIPHER: - cipher_xform = xform; - break; - case KASUMI_OP_AUTH_CIPHER: - cipher_xform = xform->next; - /* Fall-through */ - case KASUMI_OP_ONLY_AUTH: - auth_xform = xform; - break; - case KASUMI_OP_NOT_SUPPORTED: - default: - KASUMI_LOG(ERR, "Unsupported operation chain order parameter"); - return -ENOTSUP; - } - - if (cipher_xform) { - /* Only KASUMI F8 supported */ - if (cipher_xform->cipher.algo != RTE_CRYPTO_CIPHER_KASUMI_F8) { - KASUMI_LOG(ERR, "Unsupported cipher algorithm "); - return -ENOTSUP; - } - - sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; - if (cipher_xform->cipher.iv.length != KASUMI_IV_LENGTH) { - KASUMI_LOG(ERR, "Wrong IV length"); - return -EINVAL; - } - - /* Initialize key */ - IMB_KASUMI_INIT_F8_KEY_SCHED(mgr, cipher_xform->cipher.key.data, - &sess->pKeySched_cipher); - } - - if (auth_xform) { - /* Only KASUMI F9 supported */ - if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_KASUMI_F9) { - KASUMI_LOG(ERR, "Unsupported authentication"); - return -ENOTSUP; - } - - if (auth_xform->auth.digest_length != KASUMI_DIGEST_LENGTH) { - KASUMI_LOG(ERR, "Wrong digest length"); - return -EINVAL; - } - - sess->auth_op = auth_xform->auth.op; - - /* Initialize key */ - IMB_KASUMI_INIT_F9_KEY_SCHED(mgr, auth_xform->auth.key.data, - &sess->pKeySched_hash); - } - - - sess->op = mode; - - return 0; -} - -/** Get KASUMI session. */ -static struct kasumi_session * -kasumi_get_session(struct kasumi_qp *qp, struct rte_crypto_op *op) -{ - struct kasumi_session *sess = NULL; - - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { - if (likely(op->sym->session != NULL)) - sess = (struct kasumi_session *) - get_sym_session_private_data( - op->sym->session, - cryptodev_driver_id); - } else { - void *_sess = NULL; - void *_sess_private_data = NULL; - - if (rte_mempool_get(qp->sess_mp, (void **)&_sess)) - return NULL; - - if (rte_mempool_get(qp->sess_mp_priv, - (void **)&_sess_private_data)) - return NULL; - - sess = (struct kasumi_session *)_sess_private_data; - - if (unlikely(kasumi_set_session_parameters(qp->mgr, sess, - op->sym->xform) != 0)) { - rte_mempool_put(qp->sess_mp, _sess); - rte_mempool_put(qp->sess_mp_priv, _sess_private_data); - sess = NULL; - } - op->sym->session = (struct rte_cryptodev_sym_session *)_sess; - set_sym_session_private_data(op->sym->session, - cryptodev_driver_id, _sess_private_data); - } - - if (unlikely(sess == NULL)) - op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - - return sess; -} - -/** Encrypt/decrypt mbufs with same cipher key. */ -static uint8_t -process_kasumi_cipher_op(struct kasumi_qp *qp, struct rte_crypto_op **ops, - struct kasumi_session *session, uint8_t num_ops) -{ - unsigned i; - uint8_t processed_ops = 0; - const void *src[num_ops]; - void *dst[num_ops]; - uint8_t *iv_ptr; - uint64_t iv[num_ops]; - uint32_t num_bytes[num_ops]; - - for (i = 0; i < num_ops; i++) { - src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - dst[i] = ops[i]->sym->m_dst ? - rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3) : - rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - iv_ptr = rte_crypto_op_ctod_offset(ops[i], uint8_t *, - session->cipher_iv_offset); - iv[i] = *((uint64_t *)(iv_ptr)); - num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; - - processed_ops++; - } - - if (processed_ops != 0) - IMB_KASUMI_F8_N_BUFFER(qp->mgr, &session->pKeySched_cipher, iv, - src, dst, num_bytes, processed_ops); - - return processed_ops; -} - -/** Encrypt/decrypt mbuf (bit level function). */ -static uint8_t -process_kasumi_cipher_op_bit(struct kasumi_qp *qp, struct rte_crypto_op *op, - struct kasumi_session *session) -{ - uint8_t *src, *dst; - uint8_t *iv_ptr; - uint64_t iv; - uint32_t length_in_bits, offset_in_bits; - - offset_in_bits = op->sym->cipher.data.offset; - src = rte_pktmbuf_mtod(op->sym->m_src, uint8_t *); - if (op->sym->m_dst == NULL) - dst = src; - else - dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *); - iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, - session->cipher_iv_offset); - iv = *((uint64_t *)(iv_ptr)); - length_in_bits = op->sym->cipher.data.length; - - IMB_KASUMI_F8_1_BUFFER_BIT(qp->mgr, &session->pKeySched_cipher, iv, - src, dst, length_in_bits, offset_in_bits); - - return 1; -} - -/** Generate/verify hash from mbufs with same hash key. */ -static int -process_kasumi_hash_op(struct kasumi_qp *qp, struct rte_crypto_op **ops, - struct kasumi_session *session, - uint8_t num_ops) -{ - unsigned i; - uint8_t processed_ops = 0; - uint8_t *src, *dst; - uint32_t length_in_bits; - uint32_t num_bytes; - - for (i = 0; i < num_ops; i++) { - /* Data must be byte aligned */ - if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - KASUMI_LOG(ERR, "Invalid Offset"); - break; - } - - length_in_bits = ops[i]->sym->auth.data.length; - - src = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->auth.data.offset >> 3); - /* Direction from next bit after end of message */ - num_bytes = length_in_bits >> 3; - - if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { - dst = qp->temp_digest; - IMB_KASUMI_F9_1_BUFFER(qp->mgr, - &session->pKeySched_hash, src, - num_bytes, dst); - - /* Verify digest. */ - if (memcmp(dst, ops[i]->sym->auth.digest.data, - KASUMI_DIGEST_LENGTH) != 0) - ops[i]->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - } else { - dst = ops[i]->sym->auth.digest.data; - - IMB_KASUMI_F9_1_BUFFER(qp->mgr, - &session->pKeySched_hash, src, - num_bytes, dst); - } - processed_ops++; - } - - return processed_ops; -} - -/** Process a batch of crypto ops which shares the same session. */ -static int -process_ops(struct rte_crypto_op **ops, struct kasumi_session *session, - struct kasumi_qp *qp, uint8_t num_ops, - uint16_t *accumulated_enqueued_ops) -{ - unsigned i; - unsigned enqueued_ops, processed_ops; - - switch (session->op) { - case KASUMI_OP_ONLY_CIPHER: - processed_ops = process_kasumi_cipher_op(qp, ops, - session, num_ops); - break; - case KASUMI_OP_ONLY_AUTH: - processed_ops = process_kasumi_hash_op(qp, ops, session, - num_ops); - break; - case KASUMI_OP_CIPHER_AUTH: - processed_ops = process_kasumi_cipher_op(qp, ops, session, - num_ops); - process_kasumi_hash_op(qp, ops, session, processed_ops); - break; - case KASUMI_OP_AUTH_CIPHER: - processed_ops = process_kasumi_hash_op(qp, ops, session, - num_ops); - process_kasumi_cipher_op(qp, ops, session, processed_ops); - break; - default: - /* Operation not supported. */ - processed_ops = 0; - } - - for (i = 0; i < num_ops; i++) { - /* - * If there was no error/authentication failure, - * change status to successful. - */ - if (ops[i]->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) - ops[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - /* Free session if a session-less crypto op. */ - if (ops[i]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { - memset(session, 0, sizeof(struct kasumi_session)); - memset(ops[i]->sym->session, 0, - rte_cryptodev_sym_get_existing_header_session_size( - ops[i]->sym->session)); - rte_mempool_put(qp->sess_mp_priv, session); - rte_mempool_put(qp->sess_mp, ops[i]->sym->session); - ops[i]->sym->session = NULL; - } - } - - enqueued_ops = rte_ring_enqueue_burst(qp->processed_ops, - (void **)ops, processed_ops, NULL); - qp->qp_stats.enqueued_count += enqueued_ops; - *accumulated_enqueued_ops += enqueued_ops; - - return enqueued_ops; -} - -/** Process a crypto op with length/offset in bits. */ -static int -process_op_bit(struct rte_crypto_op *op, struct kasumi_session *session, - struct kasumi_qp *qp, uint16_t *accumulated_enqueued_ops) -{ - unsigned enqueued_op, processed_op; - - switch (session->op) { - case KASUMI_OP_ONLY_CIPHER: - processed_op = process_kasumi_cipher_op_bit(qp, op, - session); - break; - case KASUMI_OP_ONLY_AUTH: - processed_op = process_kasumi_hash_op(qp, &op, session, 1); - break; - case KASUMI_OP_CIPHER_AUTH: - processed_op = process_kasumi_cipher_op_bit(qp, op, session); - if (processed_op == 1) - process_kasumi_hash_op(qp, &op, session, 1); - break; - case KASUMI_OP_AUTH_CIPHER: - processed_op = process_kasumi_hash_op(qp, &op, session, 1); - if (processed_op == 1) - process_kasumi_cipher_op_bit(qp, op, session); - break; - default: - /* Operation not supported. */ - processed_op = 0; - } - - /* - * If there was no error/authentication failure, - * change status to successful. - */ - if (op->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) - op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - - /* Free session if a session-less crypto op. */ - if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { - memset(op->sym->session, 0, sizeof(struct kasumi_session)); - rte_cryptodev_sym_session_free(op->sym->session); - op->sym->session = NULL; - } - - enqueued_op = rte_ring_enqueue_burst(qp->processed_ops, (void **)&op, - processed_op, NULL); - qp->qp_stats.enqueued_count += enqueued_op; - *accumulated_enqueued_ops += enqueued_op; - - return enqueued_op; -} - -static uint16_t -kasumi_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, - uint16_t nb_ops) -{ - struct rte_crypto_op *c_ops[nb_ops]; - struct rte_crypto_op *curr_c_op; - - struct kasumi_session *prev_sess = NULL, *curr_sess = NULL; - struct kasumi_qp *qp = queue_pair; - unsigned i; - uint8_t burst_size = 0; - uint16_t enqueued_ops = 0; - uint8_t processed_ops; - - for (i = 0; i < nb_ops; i++) { - curr_c_op = ops[i]; - -#ifdef RTE_LIBRTE_PMD_KASUMI_DEBUG - if (!rte_pktmbuf_is_contiguous(curr_c_op->sym->m_src) || - (curr_c_op->sym->m_dst != NULL && - !rte_pktmbuf_is_contiguous( - curr_c_op->sym->m_dst))) { - KASUMI_LOG(ERR, "PMD supports only contiguous mbufs, " - "op (%p) provides noncontiguous mbuf as " - "source/destination buffer.", curr_c_op); - curr_c_op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - break; - } -#endif - - /* Set status as enqueued (not processed yet) by default. */ - curr_c_op->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED; - - curr_sess = kasumi_get_session(qp, curr_c_op); - if (unlikely(curr_sess == NULL || - curr_sess->op == KASUMI_OP_NOT_SUPPORTED)) { - curr_c_op->status = - RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - break; - } - - /* If length/offset is at bit-level, process this buffer alone. */ - if (((curr_c_op->sym->cipher.data.length % BYTE_LEN) != 0) - || ((ops[i]->sym->cipher.data.offset - % BYTE_LEN) != 0)) { - /* Process the ops of the previous session. */ - if (prev_sess != NULL) { - processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - prev_sess = NULL; - } - - processed_ops = process_op_bit(curr_c_op, curr_sess, - qp, &enqueued_ops); - if (processed_ops != 1) - break; - - continue; - } - - /* Batch ops that share the same session. */ - if (prev_sess == NULL) { - prev_sess = curr_sess; - c_ops[burst_size++] = curr_c_op; - } else if (curr_sess == prev_sess) { - c_ops[burst_size++] = curr_c_op; - /* - * When there are enough ops to process in a batch, - * process them, and start a new batch. - */ - if (burst_size == KASUMI_MAX_BURST) { - processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - prev_sess = NULL; - } - } else { - /* - * Different session, process the ops - * of the previous session. - */ - processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - prev_sess = curr_sess; - - c_ops[burst_size++] = curr_c_op; - } - } - - if (burst_size != 0) { - /* Process the crypto ops of the last session. */ - processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); - } - - qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops; - return enqueued_ops; -} - -static uint16_t -kasumi_pmd_dequeue_burst(void *queue_pair, - struct rte_crypto_op **c_ops, uint16_t nb_ops) -{ - struct kasumi_qp *qp = queue_pair; - - unsigned nb_dequeued; - - nb_dequeued = rte_ring_dequeue_burst(qp->processed_ops, - (void **)c_ops, nb_ops, NULL); - qp->qp_stats.dequeued_count += nb_dequeued; - - return nb_dequeued; -} - -static int cryptodev_kasumi_remove(struct rte_vdev_device *vdev); - -static int -cryptodev_kasumi_create(const char *name, - struct rte_vdev_device *vdev, - struct rte_cryptodev_pmd_init_params *init_params) -{ - struct rte_cryptodev *dev; - struct kasumi_private *internals; - MB_MGR *mgr; - - dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); - if (dev == NULL) { - KASUMI_LOG(ERR, "failed to create cryptodev vdev"); - goto init_error; - } - - dev->driver_id = cryptodev_driver_id; - dev->dev_ops = rte_kasumi_pmd_ops; - - /* Register RX/TX burst functions for data path. */ - dev->dequeue_burst = kasumi_pmd_dequeue_burst; - dev->enqueue_burst = kasumi_pmd_enqueue_burst; - - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | - RTE_CRYPTODEV_FF_SYM_SESSIONLESS | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; - - mgr = alloc_mb_mgr(0); - if (mgr == NULL) - return -ENOMEM; - - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - init_mb_mgr_avx(mgr); - } else { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - init_mb_mgr_sse(mgr); - } - - internals = dev->data->dev_private; - - internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; - internals->mgr = mgr; - - return 0; -init_error: - KASUMI_LOG(ERR, "driver %s: failed", - init_params->name); - - cryptodev_kasumi_remove(vdev); - return -EFAULT; -} - -static int -cryptodev_kasumi_probe(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev_pmd_init_params init_params = { - "", - sizeof(struct kasumi_private), - rte_socket_id(), - RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS - }; - const char *name; - const char *input_args; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - input_args = rte_vdev_device_args(vdev); - - rte_cryptodev_pmd_parse_input_args(&init_params, input_args); - - return cryptodev_kasumi_create(name, vdev, &init_params); -} - -static int -cryptodev_kasumi_remove(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev *cryptodev; - const char *name; - struct kasumi_private *internals; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - - cryptodev = rte_cryptodev_pmd_get_named_dev(name); - if (cryptodev == NULL) - return -ENODEV; - - internals = cryptodev->data->dev_private; - - free_mb_mgr(internals->mgr); - - return rte_cryptodev_pmd_destroy(cryptodev); -} - -static struct rte_vdev_driver cryptodev_kasumi_pmd_drv = { - .probe = cryptodev_kasumi_probe, - .remove = cryptodev_kasumi_remove -}; - -static struct cryptodev_driver kasumi_crypto_drv; - -RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_KASUMI_PMD, cryptodev_kasumi_pmd_drv); -RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_KASUMI_PMD, cryptodev_kasumi_pmd); -RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_KASUMI_PMD, - "max_nb_queue_pairs= " - "socket_id="); -RTE_PMD_REGISTER_CRYPTO_DRIVER(kasumi_crypto_drv, - cryptodev_kasumi_pmd_drv.driver, cryptodev_driver_id); - -RTE_LOG_REGISTER_DEFAULT(kasumi_logtype_driver, NOTICE); diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c b/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c deleted file mode 100644 index f075054807..0000000000 --- a/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c +++ /dev/null @@ -1,316 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include - -#include -#include -#include - -#include "kasumi_pmd_private.h" - -static const struct rte_cryptodev_capabilities kasumi_pmd_capabilities[] = { - { /* KASUMI (F9) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_KASUMI_F9, - .block_size = 8, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .digest_size = { - .min = 4, - .max = 4, - .increment = 0 - }, - .iv_size = { 0 } - }, } - }, } - }, - { /* KASUMI (F8) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, - {.cipher = { - .algo = RTE_CRYPTO_CIPHER_KASUMI_F8, - .block_size = 8, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .iv_size = { - .min = 8, - .max = 8, - .increment = 0 - } - }, } - }, } - }, - RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() -}; - -/** Configure device */ -static int -kasumi_pmd_config(__rte_unused struct rte_cryptodev *dev, - __rte_unused struct rte_cryptodev_config *config) -{ - return 0; -} - -/** Start device */ -static int -kasumi_pmd_start(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - -/** Stop device */ -static void -kasumi_pmd_stop(__rte_unused struct rte_cryptodev *dev) -{ -} - -/** Close device */ -static int -kasumi_pmd_close(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - - -/** Get device statistics */ -static void -kasumi_pmd_stats_get(struct rte_cryptodev *dev, - struct rte_cryptodev_stats *stats) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct kasumi_qp *qp = dev->data->queue_pairs[qp_id]; - - stats->enqueued_count += qp->qp_stats.enqueued_count; - stats->dequeued_count += qp->qp_stats.dequeued_count; - - stats->enqueue_err_count += qp->qp_stats.enqueue_err_count; - stats->dequeue_err_count += qp->qp_stats.dequeue_err_count; - } -} - -/** Reset device statistics */ -static void -kasumi_pmd_stats_reset(struct rte_cryptodev *dev) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct kasumi_qp *qp = dev->data->queue_pairs[qp_id]; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - } -} - - -/** Get device info */ -static void -kasumi_pmd_info_get(struct rte_cryptodev *dev, - struct rte_cryptodev_info *dev_info) -{ - struct kasumi_private *internals = dev->data->dev_private; - - if (dev_info != NULL) { - dev_info->driver_id = dev->driver_id; - dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; - /* No limit of number of sessions */ - dev_info->sym.max_nb_sessions = 0; - dev_info->feature_flags = dev->feature_flags; - dev_info->capabilities = kasumi_pmd_capabilities; - } -} - -/** Release queue pair */ -static int -kasumi_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) -{ - struct kasumi_qp *qp = dev->data->queue_pairs[qp_id]; - - if (qp != NULL) { - rte_ring_free(qp->processed_ops); - rte_free(qp); - dev->data->queue_pairs[qp_id] = NULL; - } - return 0; -} - -/** set a unique name for the queue pair based on its name, dev_id and qp_id */ -static int -kasumi_pmd_qp_set_unique_name(struct rte_cryptodev *dev, - struct kasumi_qp *qp) -{ - unsigned n = snprintf(qp->name, sizeof(qp->name), - "kasumi_pmd_%u_qp_%u", - dev->data->dev_id, qp->id); - - if (n >= sizeof(qp->name)) - return -1; - - return 0; -} - -/** Create a ring to place processed ops on */ -static struct rte_ring * -kasumi_pmd_qp_create_processed_ops_ring(struct kasumi_qp *qp, - unsigned ring_size, int socket_id) -{ - struct rte_ring *r; - - r = rte_ring_lookup(qp->name); - if (r) { - if (rte_ring_get_size(r) == ring_size) { - KASUMI_LOG(INFO, "Reusing existing ring %s" - " for processed packets", - qp->name); - return r; - } - - KASUMI_LOG(ERR, "Unable to reuse existing ring %s" - " for processed packets", - qp->name); - return NULL; - } - - return rte_ring_create(qp->name, ring_size, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); -} - -/** Setup a queue pair */ -static int -kasumi_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, - const struct rte_cryptodev_qp_conf *qp_conf, - int socket_id) -{ - struct kasumi_qp *qp = NULL; - struct kasumi_private *internals = dev->data->dev_private; - - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - kasumi_pmd_qp_release(dev, qp_id); - - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("KASUMI PMD Queue Pair", sizeof(*qp), - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return (-ENOMEM); - - qp->id = qp_id; - dev->data->queue_pairs[qp_id] = qp; - - if (kasumi_pmd_qp_set_unique_name(dev, qp)) - goto qp_setup_cleanup; - - qp->processed_ops = kasumi_pmd_qp_create_processed_ops_ring(qp, - qp_conf->nb_descriptors, socket_id); - if (qp->processed_ops == NULL) - goto qp_setup_cleanup; - - qp->mgr = internals->mgr; - qp->sess_mp = qp_conf->mp_session; - qp->sess_mp_priv = qp_conf->mp_session_private; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - - return 0; - -qp_setup_cleanup: - rte_free(qp); - - return -1; -} - -/** Returns the size of the KASUMI session structure */ -static unsigned -kasumi_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) -{ - return sizeof(struct kasumi_session); -} - -/** Configure a KASUMI session from a crypto xform chain */ -static int -kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev, - struct rte_crypto_sym_xform *xform, - struct rte_cryptodev_sym_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - int ret; - struct kasumi_private *internals = dev->data->dev_private; - - if (unlikely(sess == NULL)) { - KASUMI_LOG(ERR, "invalid session struct"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - KASUMI_LOG(ERR, - "Couldn't get object from session mempool"); - return -ENOMEM; - } - - ret = kasumi_set_session_parameters(internals->mgr, - sess_private_data, xform); - if (ret != 0) { - KASUMI_LOG(ERR, "failed configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sym_session_private_data(sess, dev->driver_id, - sess_private_data); - - return 0; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static void -kasumi_pmd_sym_session_clear(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess) -{ - uint8_t index = dev->driver_id; - void *sess_priv = get_sym_session_private_data(sess, index); - - /* Zero out the whole structure */ - if (sess_priv) { - memset(sess_priv, 0, sizeof(struct kasumi_session)); - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - set_sym_session_private_data(sess, index, NULL); - rte_mempool_put(sess_mp, sess_priv); - } -} - -struct rte_cryptodev_ops kasumi_pmd_ops = { - .dev_configure = kasumi_pmd_config, - .dev_start = kasumi_pmd_start, - .dev_stop = kasumi_pmd_stop, - .dev_close = kasumi_pmd_close, - - .stats_get = kasumi_pmd_stats_get, - .stats_reset = kasumi_pmd_stats_reset, - - .dev_infos_get = kasumi_pmd_info_get, - - .queue_pair_setup = kasumi_pmd_qp_setup, - .queue_pair_release = kasumi_pmd_qp_release, - - .sym_session_get_size = kasumi_pmd_sym_session_get_size, - .sym_session_configure = kasumi_pmd_sym_session_configure, - .sym_session_clear = kasumi_pmd_sym_session_clear -}; - -struct rte_cryptodev_ops *rte_kasumi_pmd_ops = &kasumi_pmd_ops; diff --git a/drivers/crypto/kasumi/version.map b/drivers/crypto/kasumi/version.map deleted file mode 100644 index c2e0723b4c..0000000000 --- a/drivers/crypto/kasumi/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_22 { - local: *; -}; diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index 00a31d52f6..effaf729ec 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -14,7 +14,6 @@ drivers = [ 'dpaa_sec', 'dpaa2_sec', 'ipsec_mb', - 'kasumi', 'mlx5', 'mvsam', 'nitrox', From patchwork Fri Oct 15 14:39:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101781 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5DAFAA0C45; Fri, 15 Oct 2021 16:41:37 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 18505412A7; Fri, 15 Oct 2021 16:40:44 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 5B9E24129A for ; Fri, 15 Oct 2021 16:40:41 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816187" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816187" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542899" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:38 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, =?utf-8?q?Ciara=C2=A0Power?= , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:53 +0000 Message-Id: <20211015143957.842499-11-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 10/14] drivers/crypto: move snow3g PMD to IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch removes the crypto/snow3g folder and gathers all snow3g PMD implementation specific details into a single file, pmd_snow3g.c in the crypto/ipsec_mb folder. Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - Removed extra empty lines. - Removed call to enqueue when processing ops, no longer needed. - Fixed some function names. - Removed unnecessary enqueued counts from process_ops function. - Added release note. - Removed security includes. - Split into private header file. v3: Removed extra empty lines. v2: Updated maintainers file. --- MAINTAINERS | 8 +- doc/guides/cryptodevs/snow3g.rst | 3 +- doc/guides/rel_notes/release_21_11.rst | 1 + drivers/crypto/ipsec_mb/ipsec_mb_private.h | 7 + drivers/crypto/ipsec_mb/meson.build | 3 +- .../pmd_snow3g.c} | 405 +++++------------- drivers/crypto/ipsec_mb/pmd_snow3g_priv.h | 85 ++++ drivers/crypto/meson.build | 1 - drivers/crypto/snow3g/meson.build | 24 -- drivers/crypto/snow3g/rte_snow3g_pmd_ops.c | 323 -------------- drivers/crypto/snow3g/snow3g_pmd_private.h | 84 ---- drivers/crypto/snow3g/version.map | 3 - 12 files changed, 218 insertions(+), 729 deletions(-) rename drivers/crypto/{snow3g/rte_snow3g_pmd.c => ipsec_mb/pmd_snow3g.c} (54%) create mode 100644 drivers/crypto/ipsec_mb/pmd_snow3g_priv.h delete mode 100644 drivers/crypto/snow3g/meson.build delete mode 100644 drivers/crypto/snow3g/rte_snow3g_pmd_ops.c delete mode 100644 drivers/crypto/snow3g/snow3g_pmd_private.h delete mode 100644 drivers/crypto/snow3g/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 12b354a304..cff55e4525 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1060,9 +1060,11 @@ F: drivers/crypto/ipsec_mb/ F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/aesni_mb.rst F: doc/guides/cryptodevs/kasumi.rst +F: doc/guides/cryptodevs/snow3g.rst F: doc/guides/cryptodevs/features/aesni_gcm.ini F: doc/guides/cryptodevs/features/aesni_mb.ini F: doc/guides/cryptodevs/features/kasumi.ini +F: doc/guides/cryptodevs/features/snow3g.ini Marvell cnxk crypto M: Ankur Dwivedi @@ -1133,12 +1135,6 @@ F: drivers/crypto/openssl/ F: doc/guides/cryptodevs/openssl.rst F: doc/guides/cryptodevs/features/openssl.ini -SNOW 3G -M: Pablo de Lara -F: drivers/crypto/snow3g/ -F: doc/guides/cryptodevs/snow3g.rst -F: doc/guides/cryptodevs/features/snow3g.ini - Virtio M: Jay Zhou F: drivers/crypto/virtio/ diff --git a/doc/guides/cryptodevs/snow3g.rst b/doc/guides/cryptodevs/snow3g.rst index 0258b71bb4..4ba71d66ce 100644 --- a/doc/guides/cryptodevs/snow3g.rst +++ b/doc/guides/cryptodevs/snow3g.rst @@ -77,7 +77,8 @@ and the external crypto libraries supported by them: DPDK version Crypto library version ============= ================================ 16.04 - 19.11 LibSSO SNOW3G - 20.02+ Multi-buffer library 0.53 - 1.0* + 20.02 - 21.08 Multi-buffer library 0.53 - 1.0* + 21.11+ Multi-buffer library 1.0* ============= ================================ \* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 7b4016f33a..d6d65d6623 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -107,6 +107,7 @@ New Features * AESNI_MB PMD. * AESNI_GCM PMD. * KASUMI PMD. + * SNOW3G PMD. * **Updated the aesni_mb crypto PMD.** diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index a545296580..9ee23d8fb5 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -43,6 +43,9 @@ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); #define CRYPTODEV_NAME_KASUMI_PMD crypto_kasumi /**< IPSEC Multi buffer PMD kasumi device name */ +#define CRYPTODEV_NAME_SNOW3G_PMD crypto_snow3g +/**< IPSEC Multi buffer PMD snow3g device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -54,6 +57,7 @@ enum ipsec_mb_pmd_types { IPSEC_MB_PMD_TYPE_AESNI_MB = 0, IPSEC_MB_PMD_TYPE_AESNI_GCM, IPSEC_MB_PMD_TYPE_KASUMI, + IPSEC_MB_PMD_TYPE_SNOW3G, IPSEC_MB_N_PMD_TYPES }; @@ -75,6 +79,7 @@ enum ipsec_mb_operation { extern uint8_t pmd_driver_id_aesni_mb; extern uint8_t pmd_driver_id_aesni_gcm; extern uint8_t pmd_driver_id_kasumi; +extern uint8_t pmd_driver_id_snow3g; /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t @@ -87,6 +92,8 @@ ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) return pmd_driver_id_aesni_gcm; case IPSEC_MB_PMD_TYPE_KASUMI: return pmd_driver_id_kasumi; + case IPSEC_MB_PMD_TYPE_SNOW3G: + return pmd_driver_id_snow3g; default: break; } diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index 54d2b69480..0c074d789d 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -25,6 +25,7 @@ sources = files('ipsec_mb_private.c', 'ipsec_mb_ops.c', 'pmd_aesni_mb.c', 'pmd_aesni_gcm.c', - 'pmd_kasumi.c' + 'pmd_kasumi.c', + 'pmd_snow3g.c' ) deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd.c b/drivers/crypto/ipsec_mb/pmd_snow3g.c similarity index 54% rename from drivers/crypto/snow3g/rte_snow3g_pmd.c rename to drivers/crypto/ipsec_mb/pmd_snow3g.c index 8284ac0b66..3b76c602dc 100644 --- a/drivers/crypto/snow3g/rte_snow3g_pmd.c +++ b/drivers/crypto/ipsec_mb/pmd_snow3g.c @@ -1,87 +1,24 @@ /* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation + * Copyright(c) 2016-2021 Intel Corporation */ -#include -#include -#include -#include -#include -#include -#include - -#include "snow3g_pmd_private.h" - -#define SNOW3G_IV_LENGTH 16 -#define SNOW3G_MAX_BURST 8 -#define BYTE_LEN 8 - -static uint8_t cryptodev_driver_id; - -/** Get xform chain order. */ -static enum snow3g_operation -snow3g_get_mode(const struct rte_crypto_sym_xform *xform) -{ - if (xform == NULL) - return SNOW3G_OP_NOT_SUPPORTED; - - if (xform->next) - if (xform->next->next != NULL) - return SNOW3G_OP_NOT_SUPPORTED; - - if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - if (xform->next == NULL) - return SNOW3G_OP_ONLY_AUTH; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_CIPHER) - return SNOW3G_OP_AUTH_CIPHER; - else - return SNOW3G_OP_NOT_SUPPORTED; - } - - if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { - if (xform->next == NULL) - return SNOW3G_OP_ONLY_CIPHER; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) - return SNOW3G_OP_CIPHER_AUTH; - else - return SNOW3G_OP_NOT_SUPPORTED; - } - - return SNOW3G_OP_NOT_SUPPORTED; -} - +#include "pmd_snow3g_priv.h" /** Parse crypto xform chain and set private session parameters. */ -int -snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, +static int +snow3g_session_configure(IMB_MGR *mgr, void *priv_sess, const struct rte_crypto_sym_xform *xform) { + struct snow3g_session *sess = (struct snow3g_session *)priv_sess; const struct rte_crypto_sym_xform *auth_xform = NULL; const struct rte_crypto_sym_xform *cipher_xform = NULL; - enum snow3g_operation mode; + enum ipsec_mb_operation mode; /* Select Crypto operation - hash then cipher / cipher then hash */ - mode = snow3g_get_mode(xform); - - switch (mode) { - case SNOW3G_OP_CIPHER_AUTH: - auth_xform = xform->next; - - /* Fall-through */ - case SNOW3G_OP_ONLY_CIPHER: - cipher_xform = xform; - break; - case SNOW3G_OP_AUTH_CIPHER: - cipher_xform = xform->next; - /* Fall-through */ - case SNOW3G_OP_ONLY_AUTH: - auth_xform = xform; - break; - case SNOW3G_OP_NOT_SUPPORTED: - default: - SNOW3G_LOG(ERR, "Unsupported operation chain order parameter"); - return -ENOTSUP; - } + int ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, NULL); + if (ret) + return ret; if (cipher_xform) { /* Only SNOW 3G UEA2 supported */ @@ -89,11 +26,11 @@ snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, return -ENOTSUP; if (cipher_xform->cipher.iv.length != SNOW3G_IV_LENGTH) { - SNOW3G_LOG(ERR, "Wrong IV length"); + IPSEC_MB_LOG(ERR, "Wrong IV length"); return -EINVAL; } if (cipher_xform->cipher.key.length > SNOW3G_MAX_KEY_SIZE) { - SNOW3G_LOG(ERR, "Not enough memory to store the key"); + IPSEC_MB_LOG(ERR, "Not enough memory to store the key"); return -ENOMEM; } @@ -110,18 +47,18 @@ snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, return -ENOTSUP; if (auth_xform->auth.digest_length != SNOW3G_DIGEST_LENGTH) { - SNOW3G_LOG(ERR, "Wrong digest length"); + IPSEC_MB_LOG(ERR, "Wrong digest length"); return -EINVAL; } if (auth_xform->auth.key.length > SNOW3G_MAX_KEY_SIZE) { - SNOW3G_LOG(ERR, "Not enough memory to store the key"); + IPSEC_MB_LOG(ERR, "Not enough memory to store the key"); return -ENOMEM; } sess->auth_op = auth_xform->auth.op; if (auth_xform->auth.iv.length != SNOW3G_IV_LENGTH) { - SNOW3G_LOG(ERR, "Wrong IV length"); + IPSEC_MB_LOG(ERR, "Wrong IV length"); return -EINVAL; } sess->auth_iv_offset = auth_xform->auth.iv.offset; @@ -136,56 +73,13 @@ snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, return 0; } -/** Get SNOW 3G session. */ -static struct snow3g_session * -snow3g_get_session(struct snow3g_qp *qp, struct rte_crypto_op *op) -{ - struct snow3g_session *sess = NULL; - - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { - if (likely(op->sym->session != NULL)) - sess = (struct snow3g_session *) - get_sym_session_private_data( - op->sym->session, - cryptodev_driver_id); - } else { - void *_sess = NULL; - void *_sess_private_data = NULL; - - if (rte_mempool_get(qp->sess_mp, (void **)&_sess)) - return NULL; - - if (rte_mempool_get(qp->sess_mp_priv, - (void **)&_sess_private_data)) - return NULL; - - sess = (struct snow3g_session *)_sess_private_data; - - if (unlikely(snow3g_set_session_parameters(qp->mgr, sess, - op->sym->xform) != 0)) { - rte_mempool_put(qp->sess_mp, _sess); - rte_mempool_put(qp->sess_mp_priv, _sess_private_data); - sess = NULL; - } - op->sym->session = (struct rte_cryptodev_sym_session *)_sess; - set_sym_session_private_data(op->sym->session, - cryptodev_driver_id, _sess_private_data); - } - - if (unlikely(sess == NULL)) - op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - - - return sess; -} - /** Encrypt/decrypt mbufs with same cipher key. */ static uint8_t -process_snow3g_cipher_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, +process_snow3g_cipher_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, struct snow3g_session *session, uint8_t num_ops) { - unsigned i; + uint32_t i; uint8_t processed_ops = 0; const void *src[SNOW3G_MAX_BURST]; void *dst[SNOW3G_MAX_BURST]; @@ -207,7 +101,7 @@ process_snow3g_cipher_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, processed_ops++; } - IMB_SNOW3G_F8_N_BUFFER(qp->mgr, &session->pKeySched_cipher, iv, + IMB_SNOW3G_F8_N_BUFFER(qp->mb_mgr, &session->pKeySched_cipher, iv, src, dst, num_bytes, processed_ops); return processed_ops; @@ -215,7 +109,7 @@ process_snow3g_cipher_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, /** Encrypt/decrypt mbuf (bit level function). */ static uint8_t -process_snow3g_cipher_op_bit(struct snow3g_qp *qp, +process_snow3g_cipher_op_bit(struct ipsec_mb_qp *qp, struct rte_crypto_op *op, struct snow3g_session *session) { @@ -227,7 +121,7 @@ process_snow3g_cipher_op_bit(struct snow3g_qp *qp, src = rte_pktmbuf_mtod(op->sym->m_src, uint8_t *); if (op->sym->m_dst == NULL) { op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - SNOW3G_LOG(ERR, "bit-level in-place not supported\n"); + IPSEC_MB_LOG(ERR, "bit-level in-place not supported\n"); return 0; } dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *); @@ -235,7 +129,7 @@ process_snow3g_cipher_op_bit(struct snow3g_qp *qp, session->cipher_iv_offset); length_in_bits = op->sym->cipher.data.length; - IMB_SNOW3G_F8_1_BUFFER_BIT(qp->mgr, &session->pKeySched_cipher, iv, + IMB_SNOW3G_F8_1_BUFFER_BIT(qp->mb_mgr, &session->pKeySched_cipher, iv, src, dst, length_in_bits, offset_in_bits); return 1; @@ -243,21 +137,22 @@ process_snow3g_cipher_op_bit(struct snow3g_qp *qp, /** Generate/verify hash from mbufs with same hash key. */ static int -process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, +process_snow3g_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, struct snow3g_session *session, uint8_t num_ops) { - unsigned i; + uint32_t i; uint8_t processed_ops = 0; uint8_t *src, *dst; uint32_t length_in_bits; uint8_t *iv; + struct snow3g_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); for (i = 0; i < num_ops; i++) { /* Data must be byte aligned */ if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - SNOW3G_LOG(ERR, "Offset"); + IPSEC_MB_LOG(ERR, "Offset"); break; } @@ -269,19 +164,20 @@ process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, session->auth_iv_offset); if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { - dst = qp->temp_digest; + dst = qp_data->temp_digest; - IMB_SNOW3G_F9_1_BUFFER(qp->mgr, + IMB_SNOW3G_F9_1_BUFFER(qp->mb_mgr, &session->pKeySched_hash, iv, src, length_in_bits, dst); /* Verify digest. */ if (memcmp(dst, ops[i]->sym->auth.digest.data, SNOW3G_DIGEST_LENGTH) != 0) - ops[i]->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + ops[i]->status = + RTE_CRYPTO_OP_STATUS_AUTH_FAILED; } else { dst = ops[i]->sym->auth.digest.data; - IMB_SNOW3G_F9_1_BUFFER(qp->mgr, + IMB_SNOW3G_F9_1_BUFFER(qp->mb_mgr, &session->pKeySched_hash, iv, src, length_in_bits, dst); } @@ -294,11 +190,10 @@ process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, /** Process a batch of crypto ops which shares the same session. */ static int process_ops(struct rte_crypto_op **ops, struct snow3g_session *session, - struct snow3g_qp *qp, uint8_t num_ops, - uint16_t *accumulated_enqueued_ops) + struct ipsec_mb_qp *qp, uint8_t num_ops) { - unsigned i; - unsigned enqueued_ops, processed_ops; + uint32_t i; + uint32_t processed_ops; #ifdef RTE_LIBRTE_PMD_SNOW3G_DEBUG for (i = 0; i < num_ops; i++) { @@ -306,7 +201,8 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session, (ops[i]->sym->m_dst != NULL && !rte_pktmbuf_is_contiguous( ops[i]->sym->m_dst))) { - SNOW3G_LOG(ERR, "PMD supports only contiguous mbufs, " + IPSEC_MB_LOG(ERR, + "PMD supports only contiguous mbufs, " "op (%p) provides noncontiguous mbuf as " "source/destination buffer.\n", ops[i]); ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; @@ -316,20 +212,24 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session, #endif switch (session->op) { - case SNOW3G_OP_ONLY_CIPHER: + case IPSEC_MB_OP_ENCRYPT_ONLY: + case IPSEC_MB_OP_DECRYPT_ONLY: processed_ops = process_snow3g_cipher_op(qp, ops, session, num_ops); break; - case SNOW3G_OP_ONLY_AUTH: + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: processed_ops = process_snow3g_hash_op(qp, ops, session, num_ops); break; - case SNOW3G_OP_CIPHER_AUTH: + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + case IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY: processed_ops = process_snow3g_cipher_op(qp, ops, session, num_ops); process_snow3g_hash_op(qp, ops, session, processed_ops); break; - case SNOW3G_OP_AUTH_CIPHER: + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + case IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT: processed_ops = process_snow3g_hash_op(qp, ops, session, num_ops); process_snow3g_cipher_op(qp, ops, session, processed_ops); @@ -357,36 +257,35 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session, ops[i]->sym->session = NULL; } } - - enqueued_ops = rte_ring_enqueue_burst(qp->processed_ops, - (void **)ops, processed_ops, NULL); - qp->qp_stats.enqueued_count += enqueued_ops; - *accumulated_enqueued_ops += enqueued_ops; - - return enqueued_ops; + return processed_ops; } /** Process a crypto op with length/offset in bits. */ static int process_op_bit(struct rte_crypto_op *op, struct snow3g_session *session, - struct snow3g_qp *qp, uint16_t *accumulated_enqueued_ops) + struct ipsec_mb_qp *qp, uint16_t *accumulated_enqueued_ops) { - unsigned enqueued_op, processed_op; + uint32_t enqueued_op, processed_op; switch (session->op) { - case SNOW3G_OP_ONLY_CIPHER: + case IPSEC_MB_OP_ENCRYPT_ONLY: + case IPSEC_MB_OP_DECRYPT_ONLY: + processed_op = process_snow3g_cipher_op_bit(qp, op, session); break; - case SNOW3G_OP_ONLY_AUTH: + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: processed_op = process_snow3g_hash_op(qp, &op, session, 1); break; - case SNOW3G_OP_CIPHER_AUTH: + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + case IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY: processed_op = process_snow3g_cipher_op_bit(qp, op, session); if (processed_op == 1) process_snow3g_hash_op(qp, &op, session, 1); break; - case SNOW3G_OP_AUTH_CIPHER: + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + case IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT: processed_op = process_snow3g_hash_op(qp, &op, session, 1); if (processed_op == 1) process_snow3g_cipher_op_bit(qp, op, session); @@ -410,50 +309,56 @@ process_op_bit(struct rte_crypto_op *op, struct snow3g_session *session, op->sym->session = NULL; } - enqueued_op = rte_ring_enqueue_burst(qp->processed_ops, + enqueued_op = rte_ring_enqueue_burst(qp->ingress_queue, (void **)&op, processed_op, NULL); - qp->qp_stats.enqueued_count += enqueued_op; + qp->stats.enqueued_count += enqueued_op; *accumulated_enqueued_ops += enqueued_op; return enqueued_op; } static uint16_t -snow3g_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, - uint16_t nb_ops) +snow3g_pmd_dequeue_burst(void *queue_pair, + struct rte_crypto_op **ops, uint16_t nb_ops) { + struct ipsec_mb_qp *qp = queue_pair; struct rte_crypto_op *c_ops[SNOW3G_MAX_BURST]; struct rte_crypto_op *curr_c_op; struct snow3g_session *prev_sess = NULL, *curr_sess = NULL; - struct snow3g_qp *qp = queue_pair; - unsigned i; + uint32_t i; uint8_t burst_size = 0; uint16_t enqueued_ops = 0; uint8_t processed_ops; + uint32_t nb_dequeued; - for (i = 0; i < nb_ops; i++) { + nb_dequeued = rte_ring_dequeue_burst(qp->ingress_queue, + (void **)ops, nb_ops, NULL); + + for (i = 0; i < nb_dequeued; i++) { curr_c_op = ops[i]; /* Set status as enqueued (not processed yet) by default. */ curr_c_op->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED; - curr_sess = snow3g_get_session(qp, curr_c_op); + curr_sess = ipsec_mb_get_session_private(qp, curr_c_op); if (unlikely(curr_sess == NULL || - curr_sess->op == SNOW3G_OP_NOT_SUPPORTED)) { + curr_sess->op == IPSEC_MB_OP_NOT_SUPPORTED)) { curr_c_op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; break; } - /* If length/offset is at bit-level, process this buffer alone. */ + /* If length/offset is at bit-level, + * process this buffer alone. + */ if (((curr_c_op->sym->cipher.data.length % BYTE_LEN) != 0) || ((curr_c_op->sym->cipher.data.offset % BYTE_LEN) != 0)) { /* Process the ops of the previous session. */ if (prev_sess != NULL) { processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); + qp, burst_size); if (processed_ops < burst_size) { burst_size = 0; break; @@ -483,7 +388,7 @@ snow3g_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, */ if (burst_size == SNOW3G_MAX_BURST) { processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); + qp, burst_size); if (processed_ops < burst_size) { burst_size = 0; break; @@ -498,7 +403,7 @@ snow3g_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, * of the previous session. */ processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); + qp, burst_size); if (processed_ops < burst_size) { burst_size = 0; break; @@ -514,134 +419,43 @@ snow3g_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, if (burst_size != 0) { /* Process the crypto ops of the last session. */ processed_ops = process_ops(c_ops, prev_sess, - qp, burst_size, &enqueued_ops); + qp, burst_size); } - qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops; - return enqueued_ops; + qp->stats.dequeued_count += i; + return i; } -static uint16_t -snow3g_pmd_dequeue_burst(void *queue_pair, - struct rte_crypto_op **c_ops, uint16_t nb_ops) -{ - struct snow3g_qp *qp = queue_pair; +struct rte_cryptodev_ops snow3g_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, - unsigned nb_dequeued; + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, - nb_dequeued = rte_ring_dequeue_burst(qp->processed_ops, - (void **)c_ops, nb_ops, NULL); - qp->qp_stats.dequeued_count += nb_dequeued; + .dev_infos_get = ipsec_mb_info_get, - return nb_dequeued; -} + .queue_pair_setup = ipsec_mb_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, -static int cryptodev_snow3g_remove(struct rte_vdev_device *vdev); - -static int -cryptodev_snow3g_create(const char *name, - struct rte_vdev_device *vdev, - struct rte_cryptodev_pmd_init_params *init_params) -{ - struct rte_cryptodev *dev; - struct snow3g_private *internals; - MB_MGR *mgr; - - dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); - if (dev == NULL) { - SNOW3G_LOG(ERR, "failed to create cryptodev vdev"); - goto init_error; - } - - dev->driver_id = cryptodev_driver_id; - dev->dev_ops = rte_snow3g_pmd_ops; - - /* Register RX/TX burst functions for data path. */ - dev->dequeue_burst = snow3g_pmd_dequeue_burst; - dev->enqueue_burst = snow3g_pmd_enqueue_burst; - - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | - RTE_CRYPTODEV_FF_SYM_SESSIONLESS | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; - - mgr = alloc_mb_mgr(0); - if (mgr == NULL) - return -ENOMEM; - - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; - init_mb_mgr_avx2(mgr); - } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - init_mb_mgr_avx(mgr); - } else { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - init_mb_mgr_sse(mgr); - } - - internals = dev->data->dev_private; - internals->mgr = mgr; - - internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; - - return 0; -init_error: - SNOW3G_LOG(ERR, "driver %s: cryptodev_snow3g_create failed", - init_params->name); - - cryptodev_snow3g_remove(vdev); - return -EFAULT; -} + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; -static int -cryptodev_snow3g_probe(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev_pmd_init_params init_params = { - "", - sizeof(struct snow3g_private), - rte_socket_id(), - RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS - }; - const char *name; - const char *input_args; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - input_args = rte_vdev_device_args(vdev); - - rte_cryptodev_pmd_parse_input_args(&init_params, input_args); - - return cryptodev_snow3g_create(name, vdev, &init_params); -} +struct rte_cryptodev_ops *rte_snow3g_pmd_ops = &snow3g_pmd_ops; static int -cryptodev_snow3g_remove(struct rte_vdev_device *vdev) +snow3g_probe(struct rte_vdev_device *vdev) { - struct rte_cryptodev *cryptodev; - const char *name; - struct snow3g_private *internals; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - - cryptodev = rte_cryptodev_pmd_get_named_dev(name); - if (cryptodev == NULL) - return -ENODEV; - - internals = cryptodev->data->dev_private; - - free_mb_mgr(internals->mgr); - - return rte_cryptodev_pmd_destroy(cryptodev); + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_SNOW3G); } static struct rte_vdev_driver cryptodev_snow3g_pmd_drv = { - .probe = cryptodev_snow3g_probe, - .remove = cryptodev_snow3g_remove + .probe = snow3g_probe, + .remove = ipsec_mb_remove }; static struct cryptodev_driver snow3g_crypto_drv; @@ -649,8 +463,27 @@ static struct cryptodev_driver snow3g_crypto_drv; RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_SNOW3G_PMD, cryptodev_snow3g_pmd_drv); RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_SNOW3G_PMD, cryptodev_snow3g_pmd); RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_SNOW3G_PMD, - "max_nb_queue_pairs= " - "socket_id="); + "max_nb_queue_pairs= socket_id="); RTE_PMD_REGISTER_CRYPTO_DRIVER(snow3g_crypto_drv, - cryptodev_snow3g_pmd_drv.driver, cryptodev_driver_id); -RTE_LOG_REGISTER_DEFAULT(snow3g_logtype_driver, INFO); + cryptodev_snow3g_pmd_drv.driver, + pmd_driver_id_snow3g); + +/* Constructor function to register snow3g PMD */ +RTE_INIT(ipsec_mb_register_snow3g) +{ + struct ipsec_mb_internals *snow3g_data + = &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_SNOW3G]; + + snow3g_data->caps = snow3g_capabilities; + snow3g_data->dequeue_burst = snow3g_pmd_dequeue_burst; + snow3g_data->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | + RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | + RTE_CRYPTODEV_FF_SYM_SESSIONLESS | + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + snow3g_data->internals_priv_size = 0; + snow3g_data->ops = &snow3g_pmd_ops; + snow3g_data->qp_priv_size = sizeof(struct snow3g_qp_data); + snow3g_data->session_configure = snow3g_session_configure; + snow3g_data->session_priv_size = sizeof(struct snow3g_session); +} diff --git a/drivers/crypto/ipsec_mb/pmd_snow3g_priv.h b/drivers/crypto/ipsec_mb/pmd_snow3g_priv.h new file mode 100644 index 0000000000..ca1ce7f9d6 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_snow3g_priv.h @@ -0,0 +1,85 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#ifndef _PMD_SNOW3G_PRIV_H_ +#define _PMD_SNOW3G_PRIV_H_ + +#include "ipsec_mb_private.h" + +#define SNOW3G_IV_LENGTH 16 +#define SNOW3G_MAX_BURST 8 +#define BYTE_LEN 8 +#define SNOW3G_DIGEST_LENGTH 4 +#define SNOW3G_MAX_KEY_SIZE 128 + +uint8_t pmd_driver_id_snow3g; + +static const struct rte_cryptodev_capabilities snow3g_capabilities[] = { + { /* SNOW 3G (UIA2) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SNOW3G_UIA2, + .block_size = 16, + .key_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .digest_size = { + .min = SNOW3G_DIGEST_LENGTH, + .max = SNOW3G_DIGEST_LENGTH, + .increment = 0 + }, + .iv_size = { + .min = SNOW3G_IV_LENGTH, + .max = SNOW3G_IV_LENGTH, + .increment = 0 + } + }, } + }, } + }, + { /* SNOW 3G (UEA2) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, + {.cipher = { + .algo = RTE_CRYPTO_CIPHER_SNOW3G_UEA2, + .block_size = 16, + .key_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .iv_size = { + .min = SNOW3G_IV_LENGTH, + .max = SNOW3G_IV_LENGTH, + .increment = 0 + } + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +/** SNOW 3G private session structure */ +struct snow3g_session { + enum ipsec_mb_operation op; + enum rte_crypto_auth_operation auth_op; + snow3g_key_schedule_t pKeySched_cipher; + snow3g_key_schedule_t pKeySched_hash; + uint16_t cipher_iv_offset; + uint16_t auth_iv_offset; +} __rte_cache_aligned; + +struct snow3g_qp_data { + uint8_t temp_digest[SNOW3G_DIGEST_LENGTH]; + /**< Buffer used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ +}; + +#endif /* _PMD_SNOW3G_PRIV_H_ */ diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index effaf729ec..5ee3f68c89 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -22,7 +22,6 @@ drivers = [ 'octeontx2', 'openssl', 'scheduler', - 'snow3g', 'virtio', 'zuc', ] diff --git a/drivers/crypto/snow3g/meson.build b/drivers/crypto/snow3g/meson.build deleted file mode 100644 index 0c087baa2a..0000000000 --- a/drivers/crypto/snow3g/meson.build +++ /dev/null @@ -1,24 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2019-2020 Intel Corporation - -IMB_required_ver = '0.53.0' -lib = cc.find_library('IPSec_MB', required: false) -if not lib.found() - build = false - reason = 'missing dependency, "libIPSec_MB"' -else - # version comes with quotes, so we split based on " and take the middle - imb_ver = cc.get_define('IMB_VERSION_STR', - prefix : '#include').split('"')[1] - - if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) - reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( - IMB_required_ver, imb_ver) - build = false - endif - -endif - -ext_deps += lib -sources = files('rte_snow3g_pmd.c', 'rte_snow3g_pmd_ops.c') -deps += ['bus_vdev', 'cryptodev'] diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c b/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c deleted file mode 100644 index 3f46014b7d..0000000000 --- a/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c +++ /dev/null @@ -1,323 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include - -#include -#include -#include - -#include "snow3g_pmd_private.h" - -static const struct rte_cryptodev_capabilities snow3g_pmd_capabilities[] = { - { /* SNOW 3G (UIA2) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_SNOW3G_UIA2, - .block_size = 16, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .digest_size = { - .min = 4, - .max = 4, - .increment = 0 - }, - .iv_size = { - .min = 16, - .max = 16, - .increment = 0 - } - }, } - }, } - }, - { /* SNOW 3G (UEA2) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, - {.cipher = { - .algo = RTE_CRYPTO_CIPHER_SNOW3G_UEA2, - .block_size = 16, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .iv_size = { - .min = 16, - .max = 16, - .increment = 0 - } - }, } - }, } - }, - RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() -}; - -/** Configure device */ -static int -snow3g_pmd_config(__rte_unused struct rte_cryptodev *dev, - __rte_unused struct rte_cryptodev_config *config) -{ - return 0; -} - -/** Start device */ -static int -snow3g_pmd_start(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - -/** Stop device */ -static void -snow3g_pmd_stop(__rte_unused struct rte_cryptodev *dev) -{ -} - -/** Close device */ -static int -snow3g_pmd_close(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - - -/** Get device statistics */ -static void -snow3g_pmd_stats_get(struct rte_cryptodev *dev, - struct rte_cryptodev_stats *stats) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct snow3g_qp *qp = dev->data->queue_pairs[qp_id]; - - stats->enqueued_count += qp->qp_stats.enqueued_count; - stats->dequeued_count += qp->qp_stats.dequeued_count; - - stats->enqueue_err_count += qp->qp_stats.enqueue_err_count; - stats->dequeue_err_count += qp->qp_stats.dequeue_err_count; - } -} - -/** Reset device statistics */ -static void -snow3g_pmd_stats_reset(struct rte_cryptodev *dev) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct snow3g_qp *qp = dev->data->queue_pairs[qp_id]; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - } -} - - -/** Get device info */ -static void -snow3g_pmd_info_get(struct rte_cryptodev *dev, - struct rte_cryptodev_info *dev_info) -{ - struct snow3g_private *internals = dev->data->dev_private; - - if (dev_info != NULL) { - dev_info->driver_id = dev->driver_id; - dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; - /* No limit of number of sessions */ - dev_info->sym.max_nb_sessions = 0; - dev_info->feature_flags = dev->feature_flags; - dev_info->capabilities = snow3g_pmd_capabilities; - } -} - -/** Release queue pair */ -static int -snow3g_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) -{ - if (dev->data->queue_pairs[qp_id] != NULL) { - struct snow3g_qp *qp = dev->data->queue_pairs[qp_id]; - - if (qp->processed_ops) - rte_ring_free(qp->processed_ops); - - rte_free(dev->data->queue_pairs[qp_id]); - dev->data->queue_pairs[qp_id] = NULL; - } - return 0; -} - -/** set a unique name for the queue pair based on its name, dev_id and qp_id */ -static int -snow3g_pmd_qp_set_unique_name(struct rte_cryptodev *dev, - struct snow3g_qp *qp) -{ - unsigned n = snprintf(qp->name, sizeof(qp->name), - "snow3g_pmd_%u_qp_%u", - dev->data->dev_id, qp->id); - - if (n >= sizeof(qp->name)) - return -1; - - return 0; -} - -/** Create a ring to place processed ops on */ -static struct rte_ring * -snow3g_pmd_qp_create_processed_ops_ring(struct snow3g_qp *qp, - unsigned ring_size, int socket_id) -{ - struct rte_ring *r; - - r = rte_ring_lookup(qp->name); - if (r) { - if (rte_ring_get_size(r) >= ring_size) { - SNOW3G_LOG(INFO, "Reusing existing ring %s" - " for processed packets", - qp->name); - return r; - } - - SNOW3G_LOG(ERR, "Unable to reuse existing ring %s" - " for processed packets", - qp->name); - return NULL; - } - - return rte_ring_create(qp->name, ring_size, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); -} - -/** Setup a queue pair */ -static int -snow3g_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, - const struct rte_cryptodev_qp_conf *qp_conf, - int socket_id) -{ - struct snow3g_qp *qp = NULL; - struct snow3g_private *internals = dev->data->dev_private; - - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - snow3g_pmd_qp_release(dev, qp_id); - - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("SNOW 3G PMD Queue Pair", sizeof(*qp), - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return (-ENOMEM); - - qp->id = qp_id; - dev->data->queue_pairs[qp_id] = qp; - - if (snow3g_pmd_qp_set_unique_name(dev, qp)) - goto qp_setup_cleanup; - - qp->processed_ops = snow3g_pmd_qp_create_processed_ops_ring(qp, - qp_conf->nb_descriptors, socket_id); - if (qp->processed_ops == NULL) - goto qp_setup_cleanup; - - qp->mgr = internals->mgr; - qp->sess_mp = qp_conf->mp_session; - qp->sess_mp_priv = qp_conf->mp_session_private; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - - return 0; - -qp_setup_cleanup: - if (qp) - rte_free(qp); - - return -1; -} - -/** Returns the size of the SNOW 3G session structure */ -static unsigned -snow3g_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) -{ - return sizeof(struct snow3g_session); -} - -/** Configure a SNOW 3G session from a crypto xform chain */ -static int -snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev, - struct rte_crypto_sym_xform *xform, - struct rte_cryptodev_sym_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - int ret; - struct snow3g_private *internals = dev->data->dev_private; - - if (unlikely(sess == NULL)) { - SNOW3G_LOG(ERR, "invalid session struct"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - SNOW3G_LOG(ERR, - "Couldn't get object from session mempool"); - return -ENOMEM; - } - - ret = snow3g_set_session_parameters(internals->mgr, - sess_private_data, xform); - if (ret != 0) { - SNOW3G_LOG(ERR, "failed configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sym_session_private_data(sess, dev->driver_id, - sess_private_data); - - return 0; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static void -snow3g_pmd_sym_session_clear(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess) -{ - uint8_t index = dev->driver_id; - void *sess_priv = get_sym_session_private_data(sess, index); - - /* Zero out the whole structure */ - if (sess_priv) { - memset(sess_priv, 0, sizeof(struct snow3g_session)); - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - set_sym_session_private_data(sess, index, NULL); - rte_mempool_put(sess_mp, sess_priv); - } -} - -struct rte_cryptodev_ops snow3g_pmd_ops = { - .dev_configure = snow3g_pmd_config, - .dev_start = snow3g_pmd_start, - .dev_stop = snow3g_pmd_stop, - .dev_close = snow3g_pmd_close, - - .stats_get = snow3g_pmd_stats_get, - .stats_reset = snow3g_pmd_stats_reset, - - .dev_infos_get = snow3g_pmd_info_get, - - .queue_pair_setup = snow3g_pmd_qp_setup, - .queue_pair_release = snow3g_pmd_qp_release, - - .sym_session_get_size = snow3g_pmd_sym_session_get_size, - .sym_session_configure = snow3g_pmd_sym_session_configure, - .sym_session_clear = snow3g_pmd_sym_session_clear -}; - -struct rte_cryptodev_ops *rte_snow3g_pmd_ops = &snow3g_pmd_ops; diff --git a/drivers/crypto/snow3g/snow3g_pmd_private.h b/drivers/crypto/snow3g/snow3g_pmd_private.h deleted file mode 100644 index 23cf078a9c..0000000000 --- a/drivers/crypto/snow3g/snow3g_pmd_private.h +++ /dev/null @@ -1,84 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2019 Intel Corporation - */ - -#ifndef _SNOW3G_PMD_PRIVATE_H_ -#define _SNOW3G_PMD_PRIVATE_H_ - -#include - -#define CRYPTODEV_NAME_SNOW3G_PMD crypto_snow3g -/**< SNOW 3G PMD device name */ - -/** SNOW 3G PMD LOGTYPE DRIVER */ -extern int snow3g_logtype_driver; - -#define SNOW3G_LOG(level, fmt, ...) \ - rte_log(RTE_LOG_ ## level, snow3g_logtype_driver, \ - "%s() line %u: " fmt "\n", __func__, __LINE__, \ - ## __VA_ARGS__) - -#define SNOW3G_DIGEST_LENGTH 4 -#define SNOW3G_MAX_KEY_SIZE 128 - -/** private data structure for each virtual SNOW 3G device */ -struct snow3g_private { - unsigned max_nb_queue_pairs; - /**< Max number of queue pairs supported by device */ - MB_MGR *mgr; - /**< Multi-buffer instance */ -}; - -/** SNOW 3G buffer queue pair */ -struct snow3g_qp { - uint16_t id; - /**< Queue Pair Identifier */ - char name[RTE_CRYPTODEV_NAME_MAX_LEN]; - /**< Unique Queue Pair Name */ - struct rte_ring *processed_ops; - /**< Ring for placing processed ops */ - struct rte_mempool *sess_mp; - /**< Session Mempool */ - struct rte_mempool *sess_mp_priv; - /**< Session Private Data Mempool */ - struct rte_cryptodev_stats qp_stats; - /**< Queue pair statistics */ - uint8_t temp_digest[SNOW3G_DIGEST_LENGTH]; - /**< Buffer used to store the digest generated - * by the driver when verifying a digest provided - * by the user (using authentication verify operation) - */ - MB_MGR *mgr; - /**< Multi-buffer instance */ -} __rte_cache_aligned; - -enum snow3g_operation { - SNOW3G_OP_ONLY_CIPHER, - SNOW3G_OP_ONLY_AUTH, - SNOW3G_OP_CIPHER_AUTH, - SNOW3G_OP_AUTH_CIPHER, - SNOW3G_OP_NOT_SUPPORTED -}; - -/** SNOW 3G private session structure */ -struct snow3g_session { - enum snow3g_operation op; - enum rte_crypto_auth_operation auth_op; - snow3g_key_schedule_t pKeySched_cipher; - snow3g_key_schedule_t pKeySched_hash; - uint16_t cipher_iv_offset; - uint16_t auth_iv_offset; -} __rte_cache_aligned; - - -extern int -snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, - const struct rte_crypto_sym_xform *xform); - - -/** device specific operations function pointer structure */ -extern struct rte_cryptodev_ops *rte_snow3g_pmd_ops; - - - -#endif /* _SNOW3G_PMD_PRIVATE_H_ */ diff --git a/drivers/crypto/snow3g/version.map b/drivers/crypto/snow3g/version.map deleted file mode 100644 index c2e0723b4c..0000000000 --- a/drivers/crypto/snow3g/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_22 { - local: *; -}; From patchwork Fri Oct 15 14:39:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101782 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A57B3A0C45; Fri, 15 Oct 2021 16:41:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7054D412AB; Fri, 15 Oct 2021 16:40:47 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id A4CB3412A6 for ; Fri, 15 Oct 2021 16:40:43 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816192" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816192" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542905" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:41 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Damian Nowak , Kai Ji , =?utf-8?q?Ciara=C2=A0Power?= Date: Fri, 15 Oct 2021 14:39:54 +0000 Message-Id: <20211015143957.842499-12-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 11/14] crypto/ipsec_mb: add snow3g digest appended ops support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch enables out-of-place auth-cipher operations where digest should be encrypted along with the rest of raw data. It also adds support for partially encrypted digest when using auth-cipher operations. Signed-off-by: Damian Nowak Signed-off-by: Kai Ji Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v2: Added release note. --- doc/guides/cryptodevs/features/snow3g.ini | 1 + doc/guides/rel_notes/release_21_11.rst | 7 + drivers/crypto/ipsec_mb/pmd_snow3g.c | 148 +++++++++++++++++++--- 3 files changed, 139 insertions(+), 17 deletions(-) diff --git a/doc/guides/cryptodevs/features/snow3g.ini b/doc/guides/cryptodevs/features/snow3g.ini index 14ac7e4b6d..4d4c5b579b 100644 --- a/doc/guides/cryptodevs/features/snow3g.ini +++ b/doc/guides/cryptodevs/features/snow3g.ini @@ -8,6 +8,7 @@ Symmetric crypto = Y Sym operation chaining = Y Symmetric sessionless = Y Non-Byte aligned data = Y +Digest encrypted = Y OOP LB In LB Out = Y ; diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index d6d65d6623..3ba3aad180 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -113,6 +113,13 @@ New Features * Added support for ZUC-EEA3-256 and ZUC-EIA3-256. +* **Added digest appended ops support for Snow3G PMD.** + + * Added support for out-of-place auth-cipher operations that encrypt + the digest along with the rest of the raw data. + * Added support for partially encrypted digest when using auth-cipher + operations. + * **Updated Marvell cnxk ethdev driver.** * Added rte_flow support for dual VLAN insert and strip actions. diff --git a/drivers/crypto/ipsec_mb/pmd_snow3g.c b/drivers/crypto/ipsec_mb/pmd_snow3g.c index 3b76c602dc..ebc9a0b562 100644 --- a/drivers/crypto/ipsec_mb/pmd_snow3g.c +++ b/drivers/crypto/ipsec_mb/pmd_snow3g.c @@ -73,6 +73,24 @@ snow3g_session_configure(IMB_MGR *mgr, void *priv_sess, return 0; } +/** Check if conditions are met for digest-appended operations */ +static uint8_t * +snow3g_digest_appended_in_src(struct rte_crypto_op *op) +{ + unsigned int auth_size, cipher_size; + + auth_size = (op->sym->auth.data.offset >> 3) + + (op->sym->auth.data.length >> 3); + cipher_size = (op->sym->cipher.data.offset >> 3) + + (op->sym->cipher.data.length >> 3); + + if (auth_size < cipher_size) + return rte_pktmbuf_mtod_offset(op->sym->m_src, + uint8_t *, auth_size); + + return NULL; +} + /** Encrypt/decrypt mbufs with same cipher key. */ static uint8_t process_snow3g_cipher_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, @@ -81,29 +99,75 @@ process_snow3g_cipher_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, { uint32_t i; uint8_t processed_ops = 0; - const void *src[SNOW3G_MAX_BURST]; - void *dst[SNOW3G_MAX_BURST]; - const void *iv[SNOW3G_MAX_BURST]; - uint32_t num_bytes[SNOW3G_MAX_BURST]; + const void *src[SNOW3G_MAX_BURST] = {NULL}; + void *dst[SNOW3G_MAX_BURST] = {NULL}; + uint8_t *digest_appended[SNOW3G_MAX_BURST] = {NULL}; + const void *iv[SNOW3G_MAX_BURST] = {NULL}; + uint32_t num_bytes[SNOW3G_MAX_BURST] = {0}; + uint32_t cipher_off, cipher_len; + int unencrypted_bytes = 0; for (i = 0; i < num_ops; i++) { - src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - dst[i] = ops[i]->sym->m_dst ? - rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3) : - rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); + + cipher_off = ops[i]->sym->cipher.data.offset >> 3; + cipher_len = ops[i]->sym->cipher.data.length >> 3; + src[i] = rte_pktmbuf_mtod_offset( + ops[i]->sym->m_src, uint8_t *, cipher_off); + + /* If out-of-place operation */ + if (ops[i]->sym->m_dst && + ops[i]->sym->m_src != ops[i]->sym->m_dst) { + dst[i] = rte_pktmbuf_mtod_offset( + ops[i]->sym->m_dst, uint8_t *, cipher_off); + + /* In case of out-of-place, auth-cipher operation + * with partial encryption of the digest, copy + * the remaining, unencrypted part. + */ + if (session->op == IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT + || session->op == IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT) + unencrypted_bytes = + (ops[i]->sym->auth.data.offset >> 3) + + (ops[i]->sym->auth.data.length >> 3) + + (SNOW3G_DIGEST_LENGTH) - + cipher_off - cipher_len; + if (unencrypted_bytes > 0) + rte_memcpy( + rte_pktmbuf_mtod_offset( + ops[i]->sym->m_dst, uint8_t *, + cipher_off + cipher_len), + rte_pktmbuf_mtod_offset( + ops[i]->sym->m_src, uint8_t *, + cipher_off + cipher_len), + unencrypted_bytes); + } else + dst[i] = rte_pktmbuf_mtod_offset(ops[i]->sym->m_src, + uint8_t *, cipher_off); + iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, session->cipher_iv_offset); - num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; - + num_bytes[i] = cipher_len; processed_ops++; } IMB_SNOW3G_F8_N_BUFFER(qp->mb_mgr, &session->pKeySched_cipher, iv, src, dst, num_bytes, processed_ops); + /* Take care of the raw digest data in src buffer */ + for (i = 0; i < num_ops; i++) { + if ((session->op == IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT || + session->op == IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT) && + ops[i]->sym->m_dst != NULL) { + digest_appended[i] = + snow3g_digest_appended_in_src(ops[i]); + /* Clear unencrypted digest from + * the src buffer + */ + if (digest_appended[i] != NULL) + memset(digest_appended[i], + 0, SNOW3G_DIGEST_LENGTH); + } + } return processed_ops; } @@ -116,6 +180,7 @@ process_snow3g_cipher_op_bit(struct ipsec_mb_qp *qp, uint8_t *src, *dst; uint8_t *iv; uint32_t length_in_bits, offset_in_bits; + int unencrypted_bytes = 0; offset_in_bits = op->sym->cipher.data.offset; src = rte_pktmbuf_mtod(op->sym->m_src, uint8_t *); @@ -124,10 +189,32 @@ process_snow3g_cipher_op_bit(struct ipsec_mb_qp *qp, IPSEC_MB_LOG(ERR, "bit-level in-place not supported\n"); return 0; } + length_in_bits = op->sym->cipher.data.length; dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *); + /* In case of out-of-place, auth-cipher operation + * with partial encryption of the digest, copy + * the remaining, unencrypted part. + */ + if (session->op == IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT || + session->op == IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT) + unencrypted_bytes = + (op->sym->auth.data.offset >> 3) + + (op->sym->auth.data.length >> 3) + + (SNOW3G_DIGEST_LENGTH) - + (offset_in_bits >> 3) - + (length_in_bits >> 3); + if (unencrypted_bytes > 0) + rte_memcpy( + rte_pktmbuf_mtod_offset( + op->sym->m_dst, uint8_t *, + (length_in_bits >> 3)), + rte_pktmbuf_mtod_offset( + op->sym->m_src, uint8_t *, + (length_in_bits >> 3)), + unencrypted_bytes); + iv = rte_crypto_op_ctod_offset(op, uint8_t *, session->cipher_iv_offset); - length_in_bits = op->sym->cipher.data.length; IMB_SNOW3G_F8_1_BUFFER_BIT(qp->mb_mgr, &session->pKeySched_cipher, iv, src, dst, length_in_bits, offset_in_bits); @@ -146,6 +233,7 @@ process_snow3g_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, uint8_t *src, *dst; uint32_t length_in_bits; uint8_t *iv; + uint8_t digest_appended = 0; struct snow3g_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); for (i = 0; i < num_ops; i++) { @@ -156,6 +244,8 @@ process_snow3g_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, break; } + dst = NULL; + length_in_bits = ops[i]->sym->auth.data.length; src = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + @@ -165,6 +255,15 @@ process_snow3g_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { dst = qp_data->temp_digest; + /* Handle auth cipher verify oop case*/ + if ((session->op == + IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN || + session->op == + IPSEC_MB_OP_DECRYPT_THEN_HASH_VERIFY) && + ops[i]->sym->m_dst != NULL) + src = rte_pktmbuf_mtod_offset( + ops[i]->sym->m_dst, uint8_t *, + ops[i]->sym->auth.data.offset >> 3); IMB_SNOW3G_F9_1_BUFFER(qp->mb_mgr, &session->pKeySched_hash, @@ -174,12 +273,26 @@ process_snow3g_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, SNOW3G_DIGEST_LENGTH) != 0) ops[i]->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - } else { - dst = ops[i]->sym->auth.digest.data; + } else { + if (session->op == + IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT || + session->op == + IPSEC_MB_OP_HASH_GEN_THEN_ENCRYPT) + dst = snow3g_digest_appended_in_src(ops[i]); + + if (dst != NULL) + digest_appended = 1; + else + dst = ops[i]->sym->auth.digest.data; IMB_SNOW3G_F9_1_BUFFER(qp->mb_mgr, &session->pKeySched_hash, iv, src, length_in_bits, dst); + + /* Copy back digest from src to auth.digest.data */ + if (digest_appended) + rte_memcpy(ops[i]->sym->auth.digest.data, + dst, SNOW3G_DIGEST_LENGTH); } processed_ops++; } @@ -480,7 +593,8 @@ RTE_INIT(ipsec_mb_register_snow3g) RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | RTE_CRYPTODEV_FF_SYM_SESSIONLESS | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | + RTE_CRYPTODEV_FF_DIGEST_ENCRYPTED; snow3g_data->internals_priv_size = 0; snow3g_data->ops = &snow3g_pmd_ops; snow3g_data->qp_priv_size = sizeof(struct snow3g_qp_data); From patchwork Fri Oct 15 14:39:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101783 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 97273A0C45; Fri, 15 Oct 2021 16:41:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 83A13412A2; Fri, 15 Oct 2021 16:40:49 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id DBD144129B for ; Fri, 15 Oct 2021 16:40:46 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816198" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816198" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542910" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:43 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, =?utf-8?q?Ciara=C2=A0Power?= , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:55 +0000 Message-Id: <20211015143957.842499-13-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 12/14] drivers/crypto: move zuc PMD to IPsec-mb framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Piotr Bronowski This patch removes the crypto/zuc folder and gathers all zuc PMD implementation specific details into two files, pmd_zuc.c and pmd_zuc_priv.h in the crypto/ipsec_mb folder. Signed-off-by: Piotr Bronowski Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - Fixed some function names. - Removed unnecessary enqueued counts from process_ops function. - Added release note. - Removed security includes. - Split into private header file. v2: Updated maintainers file. --- MAINTAINERS | 8 +- doc/guides/cryptodevs/zuc.rst | 3 +- doc/guides/rel_notes/release_21_11.rst | 1 + drivers/crypto/ipsec_mb/ipsec_mb_private.h | 7 + drivers/crypto/ipsec_mb/meson.build | 3 +- drivers/crypto/ipsec_mb/pmd_zuc.c | 403 ++++++++++++++ drivers/crypto/ipsec_mb/pmd_zuc_priv.h | 85 +++ drivers/crypto/meson.build | 1 - drivers/crypto/zuc/meson.build | 24 - drivers/crypto/zuc/rte_zuc_pmd.c | 583 --------------------- drivers/crypto/zuc/rte_zuc_pmd_ops.c | 322 ------------ drivers/crypto/zuc/version.map | 3 - drivers/crypto/zuc/zuc_pmd_private.h | 83 --- 13 files changed, 502 insertions(+), 1024 deletions(-) create mode 100644 drivers/crypto/ipsec_mb/pmd_zuc.c create mode 100644 drivers/crypto/ipsec_mb/pmd_zuc_priv.h delete mode 100644 drivers/crypto/zuc/meson.build delete mode 100644 drivers/crypto/zuc/rte_zuc_pmd.c delete mode 100644 drivers/crypto/zuc/rte_zuc_pmd_ops.c delete mode 100644 drivers/crypto/zuc/version.map delete mode 100644 drivers/crypto/zuc/zuc_pmd_private.h diff --git a/MAINTAINERS b/MAINTAINERS index cff55e4525..1c269bd991 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1061,10 +1061,12 @@ F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/aesni_mb.rst F: doc/guides/cryptodevs/kasumi.rst F: doc/guides/cryptodevs/snow3g.rst +F: doc/guides/cryptodevs/zuc.rst F: doc/guides/cryptodevs/features/aesni_gcm.ini F: doc/guides/cryptodevs/features/aesni_mb.ini F: doc/guides/cryptodevs/features/kasumi.ini F: doc/guides/cryptodevs/features/snow3g.ini +F: doc/guides/cryptodevs/features/zuc.ini Marvell cnxk crypto M: Ankur Dwivedi @@ -1141,12 +1143,6 @@ F: drivers/crypto/virtio/ F: doc/guides/cryptodevs/virtio.rst F: doc/guides/cryptodevs/features/virtio.ini -ZUC -M: Pablo de Lara -F: drivers/crypto/zuc/ -F: doc/guides/cryptodevs/zuc.rst -F: doc/guides/cryptodevs/features/zuc.ini - Compression Drivers ------------------- diff --git a/doc/guides/cryptodevs/zuc.rst b/doc/guides/cryptodevs/zuc.rst index 988a79bc26..8c925665b5 100644 --- a/doc/guides/cryptodevs/zuc.rst +++ b/doc/guides/cryptodevs/zuc.rst @@ -77,7 +77,8 @@ and the external crypto libraries supported by them: DPDK version Crypto library version ============= ================================ 16.11 - 19.11 LibSSO ZUC - 20.02+ Multi-buffer library 0.53 - 1.0* + 20.02 - 21.08 Multi-buffer library 0.53 - 1.0* + 21.11+ Multi-buffer library 1.0* ============= ================================ \* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 3ba3aad180..52ac148884 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -108,6 +108,7 @@ New Features * AESNI_GCM PMD. * KASUMI PMD. * SNOW3G PMD. + * ZUC PMD. * **Updated the aesni_mb crypto PMD.** diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index 9ee23d8fb5..fef3fcab68 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -46,6 +46,9 @@ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); #define CRYPTODEV_NAME_SNOW3G_PMD crypto_snow3g /**< IPSEC Multi buffer PMD snow3g device name */ +#define CRYPTODEV_NAME_ZUC_PMD crypto_zuc +/**< IPSEC Multi buffer PMD zuc device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -58,6 +61,7 @@ enum ipsec_mb_pmd_types { IPSEC_MB_PMD_TYPE_AESNI_GCM, IPSEC_MB_PMD_TYPE_KASUMI, IPSEC_MB_PMD_TYPE_SNOW3G, + IPSEC_MB_PMD_TYPE_ZUC, IPSEC_MB_N_PMD_TYPES }; @@ -80,6 +84,7 @@ extern uint8_t pmd_driver_id_aesni_mb; extern uint8_t pmd_driver_id_aesni_gcm; extern uint8_t pmd_driver_id_kasumi; extern uint8_t pmd_driver_id_snow3g; +extern uint8_t pmd_driver_id_zuc; /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t @@ -94,6 +99,8 @@ ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) return pmd_driver_id_kasumi; case IPSEC_MB_PMD_TYPE_SNOW3G: return pmd_driver_id_snow3g; + case IPSEC_MB_PMD_TYPE_ZUC: + return pmd_driver_id_zuc; default: break; } diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index 0c074d789d..c712c2adb1 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -26,6 +26,7 @@ sources = files('ipsec_mb_private.c', 'pmd_aesni_mb.c', 'pmd_aesni_gcm.c', 'pmd_kasumi.c', - 'pmd_snow3g.c' + 'pmd_snow3g.c', + 'pmd_zuc.c' ) deps += ['bus_vdev', 'net', 'security'] diff --git a/drivers/crypto/ipsec_mb/pmd_zuc.c b/drivers/crypto/ipsec_mb/pmd_zuc.c new file mode 100644 index 0000000000..b542264069 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_zuc.c @@ -0,0 +1,403 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#include "pmd_zuc_priv.h" + +/** Parse crypto xform chain and set private session parameters. */ +static int +zuc_session_configure(__rte_unused IMB_MGR * mgr, void *zuc_sess, + const struct rte_crypto_sym_xform *xform) +{ + struct zuc_session *sess = (struct zuc_session *) zuc_sess; + const struct rte_crypto_sym_xform *auth_xform = NULL; + const struct rte_crypto_sym_xform *cipher_xform = NULL; + enum ipsec_mb_operation mode; + /* Select Crypto operation - hash then cipher / cipher then hash */ + int ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, NULL); + + if (ret) + return ret; + + if (cipher_xform) { + /* Only ZUC EEA3 supported */ + if (cipher_xform->cipher.algo != RTE_CRYPTO_CIPHER_ZUC_EEA3) + return -ENOTSUP; + + if (cipher_xform->cipher.iv.length != ZUC_IV_KEY_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong IV length"); + return -EINVAL; + } + sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; + + /* Copy the key */ + memcpy(sess->pKey_cipher, cipher_xform->cipher.key.data, + ZUC_IV_KEY_LENGTH); + } + + if (auth_xform) { + /* Only ZUC EIA3 supported */ + if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_ZUC_EIA3) + return -ENOTSUP; + + if (auth_xform->auth.digest_length != ZUC_DIGEST_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong digest length"); + return -EINVAL; + } + + sess->auth_op = auth_xform->auth.op; + + if (auth_xform->auth.iv.length != ZUC_IV_KEY_LENGTH) { + IPSEC_MB_LOG(ERR, "Wrong IV length"); + return -EINVAL; + } + sess->auth_iv_offset = auth_xform->auth.iv.offset; + + /* Copy the key */ + memcpy(sess->pKey_hash, auth_xform->auth.key.data, + ZUC_IV_KEY_LENGTH); + } + + sess->op = mode; + return 0; +} + +/** Encrypt/decrypt mbufs. */ +static uint8_t +process_zuc_cipher_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, + struct zuc_session **sessions, + uint8_t num_ops) +{ + unsigned int i; + uint8_t processed_ops = 0; + const void *src[ZUC_MAX_BURST]; + void *dst[ZUC_MAX_BURST]; + const void *iv[ZUC_MAX_BURST]; + uint32_t num_bytes[ZUC_MAX_BURST]; + const void *cipher_keys[ZUC_MAX_BURST]; + struct zuc_session *sess; + + for (i = 0; i < num_ops; i++) { + if (((ops[i]->sym->cipher.data.length % BYTE_LEN) != 0) + || ((ops[i]->sym->cipher.data.offset + % BYTE_LEN) != 0)) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + IPSEC_MB_LOG(ERR, "Data Length or offset"); + break; + } + + sess = sessions[i]; + +#ifdef RTE_LIBRTE_PMD_ZUC_DEBUG + if (!rte_pktmbuf_is_contiguous(ops[i]->sym->m_src) || + (ops[i]->sym->m_dst != NULL && + !rte_pktmbuf_is_contiguous( + ops[i]->sym->m_dst))) { + IPSEC_MB_LOG(ERR, "PMD supports only " + " contiguous mbufs, op (%p) " + "provides noncontiguous mbuf " + "as source/destination buffer.\n", + "PMD supports only contiguous mbufs, " + "op (%p) provides noncontiguous mbuf " + "as source/destination buffer.\n", + ops[i]); + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + break; + } +#endif + + src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3); + dst[i] = ops[i]->sym->m_dst ? + rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3) : + rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->cipher.data.offset >> 3); + iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, + sess->cipher_iv_offset); + num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; + + cipher_keys[i] = sess->pKey_cipher; + + processed_ops++; + } + + IMB_ZUC_EEA3_N_BUFFER(qp->mb_mgr, (const void **)cipher_keys, + (const void **)iv, (const void **)src, (void **)dst, + num_bytes, processed_ops); + + return processed_ops; +} + +/** Generate/verify hash from mbufs. */ +static int +process_zuc_hash_op(struct ipsec_mb_qp *qp, struct rte_crypto_op **ops, + struct zuc_session **sessions, + uint8_t num_ops) +{ + unsigned int i; + uint8_t processed_ops = 0; + uint8_t *src[ZUC_MAX_BURST]; + uint32_t *dst[ZUC_MAX_BURST]; + uint32_t length_in_bits[ZUC_MAX_BURST]; + uint8_t *iv[ZUC_MAX_BURST]; + const void *hash_keys[ZUC_MAX_BURST]; + struct zuc_session *sess; + struct zuc_qp_data *qp_data = ipsec_mb_get_qp_private_data(qp); + + + for (i = 0; i < num_ops; i++) { + /* Data must be byte aligned */ + if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + IPSEC_MB_LOG(ERR, "Offset"); + break; + } + + sess = sessions[i]; + + length_in_bits[i] = ops[i]->sym->auth.data.length; + + src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + + (ops[i]->sym->auth.data.offset >> 3); + iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, + sess->auth_iv_offset); + + hash_keys[i] = sess->pKey_hash; + if (sess->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) + dst[i] = (uint32_t *)qp_data->temp_digest; + else + dst[i] = (uint32_t *)ops[i]->sym->auth.digest.data; + + processed_ops++; + } + + IMB_ZUC_EIA3_N_BUFFER(qp->mb_mgr, (const void **)hash_keys, + (const void * const *)iv, (const void * const *)src, + length_in_bits, dst, processed_ops); + + /* + * If tag needs to be verified, compare generated tag + * with attached tag + */ + for (i = 0; i < processed_ops; i++) + if (sessions[i]->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) + if (memcmp(dst[i], ops[i]->sym->auth.digest.data, + ZUC_DIGEST_LENGTH) != 0) + ops[i]->status = + RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + + return processed_ops; +} + +/** Process a batch of crypto ops which shares the same operation type. */ +static int +process_ops(struct rte_crypto_op **ops, enum ipsec_mb_operation op_type, + struct zuc_session **sessions, + struct ipsec_mb_qp *qp, uint8_t num_ops) +{ + unsigned int i; + unsigned int processed_ops; + + switch (op_type) { + case IPSEC_MB_OP_ENCRYPT_ONLY: + case IPSEC_MB_OP_DECRYPT_ONLY: + processed_ops = process_zuc_cipher_op(qp, ops, + sessions, num_ops); + break; + case IPSEC_MB_OP_HASH_GEN_ONLY: + case IPSEC_MB_OP_HASH_VERIFY_ONLY: + processed_ops = process_zuc_hash_op(qp, ops, sessions, + num_ops); + break; + case IPSEC_MB_OP_ENCRYPT_THEN_HASH_GEN: + processed_ops = process_zuc_cipher_op(qp, ops, sessions, + num_ops); + process_zuc_hash_op(qp, ops, sessions, processed_ops); + break; + case IPSEC_MB_OP_HASH_VERIFY_THEN_DECRYPT: + processed_ops = process_zuc_hash_op(qp, ops, sessions, + num_ops); + process_zuc_cipher_op(qp, ops, sessions, processed_ops); + break; + default: + /* Operation not supported. */ + processed_ops = 0; + } + + for (i = 0; i < num_ops; i++) { + /* + * If there was no error/authentication failure, + * change status to successful. + */ + if (ops[i]->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) + ops[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + /* Free session if a session-less crypto op. */ + if (ops[i]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + memset(sessions[i], 0, sizeof(struct zuc_session)); + memset(ops[i]->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + ops[i]->sym->session)); + rte_mempool_put(qp->sess_mp_priv, sessions[i]); + rte_mempool_put(qp->sess_mp, ops[i]->sym->session); + ops[i]->sym->session = NULL; + } + } + return processed_ops; +} + +static uint16_t +zuc_pmd_dequeue_burst(void *queue_pair, + struct rte_crypto_op **c_ops, uint16_t nb_ops) +{ + + struct rte_crypto_op *curr_c_op; + + struct zuc_session *curr_sess; + struct zuc_session *sessions[ZUC_MAX_BURST]; + enum ipsec_mb_operation prev_zuc_op = IPSEC_MB_OP_NOT_SUPPORTED; + enum ipsec_mb_operation curr_zuc_op; + struct ipsec_mb_qp *qp = queue_pair; + unsigned int nb_dequeued; + unsigned int i; + uint8_t burst_size = 0; + uint8_t processed_ops; + + nb_dequeued = rte_ring_dequeue_burst(qp->ingress_queue, + (void **)c_ops, nb_ops, NULL); + + + for (i = 0; i < nb_dequeued; i++) { + curr_c_op = c_ops[i]; + + curr_sess = (struct zuc_session *) + ipsec_mb_get_session_private(qp, curr_c_op); + if (unlikely(curr_sess == NULL)) { + curr_c_op->status = + RTE_CRYPTO_OP_STATUS_INVALID_SESSION; + break; + } + + curr_zuc_op = curr_sess->op; + + /* + * Batch ops that share the same operation type + * (cipher only, auth only...). + */ + if (burst_size == 0) { + prev_zuc_op = curr_zuc_op; + c_ops[0] = curr_c_op; + sessions[0] = curr_sess; + burst_size++; + } else if (curr_zuc_op == prev_zuc_op) { + c_ops[burst_size] = curr_c_op; + sessions[burst_size] = curr_sess; + burst_size++; + /* + * When there are enough ops to process in a batch, + * process them, and start a new batch. + */ + if (burst_size == ZUC_MAX_BURST) { + processed_ops = process_ops(c_ops, curr_zuc_op, + sessions, qp, burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + } + } else { + /* + * Different operation type, process the ops + * of the previous type. + */ + processed_ops = process_ops(c_ops, prev_zuc_op, + sessions, qp, burst_size); + if (processed_ops < burst_size) { + burst_size = 0; + break; + } + + burst_size = 0; + prev_zuc_op = curr_zuc_op; + + c_ops[0] = curr_c_op; + sessions[0] = curr_sess; + burst_size++; + } + } + + if (burst_size != 0) { + /* Process the crypto ops of the last operation type. */ + processed_ops = process_ops(c_ops, prev_zuc_op, + sessions, qp, burst_size); + } + + qp->stats.dequeued_count += i; + return i; +} + +struct rte_cryptodev_ops zuc_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, + + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, + + .dev_infos_get = ipsec_mb_info_get, + + .queue_pair_setup = ipsec_mb_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, + + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; + +struct rte_cryptodev_ops *rte_zuc_pmd_ops = &zuc_pmd_ops; + +static int +zuc_probe(struct rte_vdev_device *vdev) +{ + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_ZUC); +} + +static struct rte_vdev_driver cryptodev_zuc_pmd_drv = { + .probe = zuc_probe, + .remove = ipsec_mb_remove + +}; + +static struct cryptodev_driver zuc_crypto_drv; + +RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_ZUC_PMD, cryptodev_zuc_pmd_drv); +RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_ZUC_PMD, cryptodev_zuc_pmd); +RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_ZUC_PMD, + "max_nb_queue_pairs= socket_id="); +RTE_PMD_REGISTER_CRYPTO_DRIVER(zuc_crypto_drv, cryptodev_zuc_pmd_drv.driver, + pmd_driver_id_zuc); + +/* Constructor function to register zuc PMD */ +RTE_INIT(ipsec_mb_register_zuc) +{ + struct ipsec_mb_internals *zuc_data + = &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_ZUC]; + + zuc_data->caps = zuc_capabilities; + zuc_data->dequeue_burst = zuc_pmd_dequeue_burst; + zuc_data->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO + | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING + | RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA + | RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT + | RTE_CRYPTODEV_FF_SYM_SESSIONLESS + | RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + zuc_data->internals_priv_size = 0; + zuc_data->ops = &zuc_pmd_ops; + zuc_data->qp_priv_size = sizeof(struct zuc_qp_data); + zuc_data->session_configure = zuc_session_configure; + zuc_data->session_priv_size = sizeof(struct zuc_session); +} diff --git a/drivers/crypto/ipsec_mb/pmd_zuc_priv.h b/drivers/crypto/ipsec_mb/pmd_zuc_priv.h new file mode 100644 index 0000000000..46d5bfae37 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_zuc_priv.h @@ -0,0 +1,85 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2016-2021 Intel Corporation + */ + +#ifndef _PMD_ZUC_PRIV_H_ +#define _PMD_ZUC_PRIV_H_ + +#include "ipsec_mb_private.h" + +#define ZUC_IV_KEY_LENGTH 16 +#define ZUC_DIGEST_LENGTH 4 +#define ZUC_MAX_BURST 16 +#define BYTE_LEN 8 + +uint8_t pmd_driver_id_zuc; + +static const struct rte_cryptodev_capabilities zuc_capabilities[] = { + { /* ZUC (EIA3) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_ZUC_EIA3, + .block_size = 16, + .key_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .digest_size = { + .min = ZUC_DIGEST_LENGTH, + .max = ZUC_DIGEST_LENGTH, + .increment = 0 + }, + .iv_size = { + .min = ZUC_IV_KEY_LENGTH, + .max = ZUC_IV_KEY_LENGTH, + .increment = 0 + } + }, } + }, } + }, + { /* ZUC (EEA3) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, + {.cipher = { + .algo = RTE_CRYPTO_CIPHER_ZUC_EEA3, + .block_size = 16, + .key_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .iv_size = { + .min = ZUC_IV_KEY_LENGTH, + .max = ZUC_IV_KEY_LENGTH, + .increment = 0 + }, + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +/** ZUC private session structure */ +struct zuc_session { + enum ipsec_mb_operation op; + enum rte_crypto_auth_operation auth_op; + uint8_t pKey_cipher[ZUC_IV_KEY_LENGTH]; + uint8_t pKey_hash[ZUC_IV_KEY_LENGTH]; + uint16_t cipher_iv_offset; + uint16_t auth_iv_offset; +} __rte_cache_aligned; + +struct zuc_qp_data { + + uint8_t temp_digest[ZUC_DIGEST_LENGTH]; + /* *< Buffers used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ +}; + +#endif /* _PMD_ZUC_PRIV_H_ */ diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index 5ee3f68c89..2585471e93 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -23,7 +23,6 @@ drivers = [ 'openssl', 'scheduler', 'virtio', - 'zuc', ] std_deps = ['cryptodev'] # cryptodev pulls in all other needed deps diff --git a/drivers/crypto/zuc/meson.build b/drivers/crypto/zuc/meson.build deleted file mode 100644 index a5f77a22d8..0000000000 --- a/drivers/crypto/zuc/meson.build +++ /dev/null @@ -1,24 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018-2020 Intel Corporation - -IMB_required_ver = '0.53.0' -lib = cc.find_library('IPSec_MB', required: false) -if not lib.found() - build = false - reason = 'missing dependency, "libIPSec_MB"' -else - # version comes with quotes, so we split based on " and take the middle - imb_ver = cc.get_define('IMB_VERSION_STR', - prefix : '#include').split('"')[1] - - if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) - reason = 'IPSec_MB version >= @0@ is required, found version @1@'.format( - IMB_required_ver, imb_ver) - build = false - endif - -endif - -ext_deps += lib -sources = files('rte_zuc_pmd.c', 'rte_zuc_pmd_ops.c') -deps += ['bus_vdev'] diff --git a/drivers/crypto/zuc/rte_zuc_pmd.c b/drivers/crypto/zuc/rte_zuc_pmd.c deleted file mode 100644 index d4b343a7af..0000000000 --- a/drivers/crypto/zuc/rte_zuc_pmd.c +++ /dev/null @@ -1,583 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include -#include -#include -#include -#include -#include -#include - -#include "zuc_pmd_private.h" -#define ZUC_MAX_BURST 16 -#define BYTE_LEN 8 - -static uint8_t cryptodev_driver_id; - -/** Get xform chain order. */ -static enum zuc_operation -zuc_get_mode(const struct rte_crypto_sym_xform *xform) -{ - if (xform == NULL) - return ZUC_OP_NOT_SUPPORTED; - - if (xform->next) - if (xform->next->next != NULL) - return ZUC_OP_NOT_SUPPORTED; - - if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - if (xform->next == NULL) - return ZUC_OP_ONLY_AUTH; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_CIPHER) - return ZUC_OP_AUTH_CIPHER; - else - return ZUC_OP_NOT_SUPPORTED; - } - - if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { - if (xform->next == NULL) - return ZUC_OP_ONLY_CIPHER; - else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) - return ZUC_OP_CIPHER_AUTH; - else - return ZUC_OP_NOT_SUPPORTED; - } - - return ZUC_OP_NOT_SUPPORTED; -} - - -/** Parse crypto xform chain and set private session parameters. */ -int -zuc_set_session_parameters(struct zuc_session *sess, - const struct rte_crypto_sym_xform *xform) -{ - const struct rte_crypto_sym_xform *auth_xform = NULL; - const struct rte_crypto_sym_xform *cipher_xform = NULL; - enum zuc_operation mode; - - /* Select Crypto operation - hash then cipher / cipher then hash */ - mode = zuc_get_mode(xform); - - switch (mode) { - case ZUC_OP_CIPHER_AUTH: - auth_xform = xform->next; - - /* Fall-through */ - case ZUC_OP_ONLY_CIPHER: - cipher_xform = xform; - break; - case ZUC_OP_AUTH_CIPHER: - cipher_xform = xform->next; - /* Fall-through */ - case ZUC_OP_ONLY_AUTH: - auth_xform = xform; - break; - case ZUC_OP_NOT_SUPPORTED: - default: - ZUC_LOG(ERR, "Unsupported operation chain order parameter"); - return -ENOTSUP; - } - - if (cipher_xform) { - /* Only ZUC EEA3 supported */ - if (cipher_xform->cipher.algo != RTE_CRYPTO_CIPHER_ZUC_EEA3) - return -ENOTSUP; - - if (cipher_xform->cipher.iv.length != ZUC_IV_KEY_LENGTH) { - ZUC_LOG(ERR, "Wrong IV length"); - return -EINVAL; - } - sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; - - /* Copy the key */ - memcpy(sess->pKey_cipher, cipher_xform->cipher.key.data, - ZUC_IV_KEY_LENGTH); - } - - if (auth_xform) { - /* Only ZUC EIA3 supported */ - if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_ZUC_EIA3) - return -ENOTSUP; - - if (auth_xform->auth.digest_length != ZUC_DIGEST_LENGTH) { - ZUC_LOG(ERR, "Wrong digest length"); - return -EINVAL; - } - - sess->auth_op = auth_xform->auth.op; - - if (auth_xform->auth.iv.length != ZUC_IV_KEY_LENGTH) { - ZUC_LOG(ERR, "Wrong IV length"); - return -EINVAL; - } - sess->auth_iv_offset = auth_xform->auth.iv.offset; - - /* Copy the key */ - memcpy(sess->pKey_hash, auth_xform->auth.key.data, - ZUC_IV_KEY_LENGTH); - } - - - sess->op = mode; - - return 0; -} - -/** Get ZUC session. */ -static struct zuc_session * -zuc_get_session(struct zuc_qp *qp, struct rte_crypto_op *op) -{ - struct zuc_session *sess = NULL; - - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { - if (likely(op->sym->session != NULL)) - sess = (struct zuc_session *)get_sym_session_private_data( - op->sym->session, - cryptodev_driver_id); - } else { - void *_sess = NULL; - void *_sess_private_data = NULL; - - if (rte_mempool_get(qp->sess_mp, (void **)&_sess)) - return NULL; - - if (rte_mempool_get(qp->sess_mp_priv, - (void **)&_sess_private_data)) - return NULL; - - sess = (struct zuc_session *)_sess_private_data; - - if (unlikely(zuc_set_session_parameters(sess, - op->sym->xform) != 0)) { - rte_mempool_put(qp->sess_mp, _sess); - rte_mempool_put(qp->sess_mp_priv, _sess_private_data); - sess = NULL; - } - op->sym->session = (struct rte_cryptodev_sym_session *)_sess; - set_sym_session_private_data(op->sym->session, - cryptodev_driver_id, _sess_private_data); - } - - if (unlikely(sess == NULL)) - op->status = RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - - - return sess; -} - -/** Encrypt/decrypt mbufs. */ -static uint8_t -process_zuc_cipher_op(struct zuc_qp *qp, struct rte_crypto_op **ops, - struct zuc_session **sessions, - uint8_t num_ops) -{ - unsigned i; - uint8_t processed_ops = 0; - const void *src[ZUC_MAX_BURST]; - void *dst[ZUC_MAX_BURST]; - const void *iv[ZUC_MAX_BURST]; - uint32_t num_bytes[ZUC_MAX_BURST]; - const void *cipher_keys[ZUC_MAX_BURST]; - struct zuc_session *sess; - - for (i = 0; i < num_ops; i++) { - if (((ops[i]->sym->cipher.data.length % BYTE_LEN) != 0) - || ((ops[i]->sym->cipher.data.offset - % BYTE_LEN) != 0)) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - ZUC_LOG(ERR, "Data Length or offset"); - break; - } - - sess = sessions[i]; - -#ifdef RTE_LIBRTE_PMD_ZUC_DEBUG - if (!rte_pktmbuf_is_contiguous(ops[i]->sym->m_src) || - (ops[i]->sym->m_dst != NULL && - !rte_pktmbuf_is_contiguous( - ops[i]->sym->m_dst))) { - ZUC_LOG(ERR, "PMD supports only contiguous mbufs, " - "op (%p) provides noncontiguous mbuf as " - "source/destination buffer.\n", ops[i]); - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - break; - } -#endif - - src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - dst[i] = ops[i]->sym->m_dst ? - rte_pktmbuf_mtod(ops[i]->sym->m_dst, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3) : - rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->cipher.data.offset >> 3); - iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, - sess->cipher_iv_offset); - num_bytes[i] = ops[i]->sym->cipher.data.length >> 3; - - cipher_keys[i] = sess->pKey_cipher; - - processed_ops++; - } - - IMB_ZUC_EEA3_N_BUFFER(qp->mb_mgr, (const void **)cipher_keys, - (const void **)iv, (const void **)src, (void **)dst, - num_bytes, processed_ops); - - return processed_ops; -} - -/** Generate/verify hash from mbufs. */ -static int -process_zuc_hash_op(struct zuc_qp *qp, struct rte_crypto_op **ops, - struct zuc_session **sessions, - uint8_t num_ops) -{ - unsigned int i; - uint8_t processed_ops = 0; - uint8_t *src[ZUC_MAX_BURST] = { 0 }; - uint32_t *dst[ZUC_MAX_BURST]; - uint32_t length_in_bits[ZUC_MAX_BURST] = { 0 }; - uint8_t *iv[ZUC_MAX_BURST] = { 0 }; - const void *hash_keys[ZUC_MAX_BURST] = { 0 }; - struct zuc_session *sess; - - for (i = 0; i < num_ops; i++) { - /* Data must be byte aligned */ - if ((ops[i]->sym->auth.data.offset % BYTE_LEN) != 0) { - ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - ZUC_LOG(ERR, "Offset"); - break; - } - - sess = sessions[i]; - - length_in_bits[i] = ops[i]->sym->auth.data.length; - - src[i] = rte_pktmbuf_mtod(ops[i]->sym->m_src, uint8_t *) + - (ops[i]->sym->auth.data.offset >> 3); - iv[i] = rte_crypto_op_ctod_offset(ops[i], uint8_t *, - sess->auth_iv_offset); - - hash_keys[i] = sess->pKey_hash; - if (sess->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) - dst[i] = (uint32_t *)qp->temp_digest; - else - dst[i] = (uint32_t *)ops[i]->sym->auth.digest.data; - -#if IMB_VERSION_NUM < IMB_VERSION(0, 53, 3) - IMB_ZUC_EIA3_1_BUFFER(qp->mb_mgr, hash_keys[i], - iv[i], src[i], length_in_bits[i], dst[i]); -#endif - processed_ops++; - } - -#if IMB_VERSION_NUM >= IMB_VERSION(0, 53, 3) - IMB_ZUC_EIA3_N_BUFFER(qp->mb_mgr, (const void **)hash_keys, - (const void * const *)iv, (const void * const *)src, - length_in_bits, dst, processed_ops); -#endif - - /* - * If tag needs to be verified, compare generated tag - * with attached tag - */ - for (i = 0; i < processed_ops; i++) - if (sessions[i]->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) - if (memcmp(dst[i], ops[i]->sym->auth.digest.data, - ZUC_DIGEST_LENGTH) != 0) - ops[i]->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - - return processed_ops; -} - -/** Process a batch of crypto ops which shares the same operation type. */ -static int -process_ops(struct rte_crypto_op **ops, enum zuc_operation op_type, - struct zuc_session **sessions, - struct zuc_qp *qp, uint8_t num_ops, - uint16_t *accumulated_enqueued_ops) -{ - unsigned i; - unsigned enqueued_ops, processed_ops; - - switch (op_type) { - case ZUC_OP_ONLY_CIPHER: - processed_ops = process_zuc_cipher_op(qp, ops, - sessions, num_ops); - break; - case ZUC_OP_ONLY_AUTH: - processed_ops = process_zuc_hash_op(qp, ops, sessions, - num_ops); - break; - case ZUC_OP_CIPHER_AUTH: - processed_ops = process_zuc_cipher_op(qp, ops, sessions, - num_ops); - process_zuc_hash_op(qp, ops, sessions, processed_ops); - break; - case ZUC_OP_AUTH_CIPHER: - processed_ops = process_zuc_hash_op(qp, ops, sessions, - num_ops); - process_zuc_cipher_op(qp, ops, sessions, processed_ops); - break; - default: - /* Operation not supported. */ - processed_ops = 0; - } - - for (i = 0; i < num_ops; i++) { - /* - * If there was no error/authentication failure, - * change status to successful. - */ - if (ops[i]->status == RTE_CRYPTO_OP_STATUS_NOT_PROCESSED) - ops[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - /* Free session if a session-less crypto op. */ - if (ops[i]->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { - memset(sessions[i], 0, sizeof(struct zuc_session)); - memset(ops[i]->sym->session, 0, - rte_cryptodev_sym_get_existing_header_session_size( - ops[i]->sym->session)); - rte_mempool_put(qp->sess_mp_priv, sessions[i]); - rte_mempool_put(qp->sess_mp, ops[i]->sym->session); - ops[i]->sym->session = NULL; - } - } - - enqueued_ops = rte_ring_enqueue_burst(qp->processed_ops, - (void **)ops, processed_ops, NULL); - qp->qp_stats.enqueued_count += enqueued_ops; - *accumulated_enqueued_ops += enqueued_ops; - - return enqueued_ops; -} - -static uint16_t -zuc_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, - uint16_t nb_ops) -{ - struct rte_crypto_op *c_ops[ZUC_MAX_BURST]; - struct rte_crypto_op *curr_c_op; - - struct zuc_session *curr_sess; - struct zuc_session *sessions[ZUC_MAX_BURST]; - enum zuc_operation prev_zuc_op = ZUC_OP_NOT_SUPPORTED; - enum zuc_operation curr_zuc_op; - struct zuc_qp *qp = queue_pair; - unsigned i; - uint8_t burst_size = 0; - uint16_t enqueued_ops = 0; - uint8_t processed_ops; - - for (i = 0; i < nb_ops; i++) { - curr_c_op = ops[i]; - - curr_sess = zuc_get_session(qp, curr_c_op); - if (unlikely(curr_sess == NULL)) { - curr_c_op->status = - RTE_CRYPTO_OP_STATUS_INVALID_SESSION; - break; - } - - curr_zuc_op = curr_sess->op; - - /* - * Batch ops that share the same operation type - * (cipher only, auth only...). - */ - if (burst_size == 0) { - prev_zuc_op = curr_zuc_op; - c_ops[0] = curr_c_op; - sessions[0] = curr_sess; - burst_size++; - } else if (curr_zuc_op == prev_zuc_op) { - c_ops[burst_size] = curr_c_op; - sessions[burst_size] = curr_sess; - burst_size++; - /* - * When there are enough ops to process in a batch, - * process them, and start a new batch. - */ - if (burst_size == ZUC_MAX_BURST) { - processed_ops = process_ops(c_ops, curr_zuc_op, - sessions, qp, burst_size, - &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - } - } else { - /* - * Different operation type, process the ops - * of the previous type. - */ - processed_ops = process_ops(c_ops, prev_zuc_op, - sessions, qp, burst_size, - &enqueued_ops); - if (processed_ops < burst_size) { - burst_size = 0; - break; - } - - burst_size = 0; - prev_zuc_op = curr_zuc_op; - - c_ops[0] = curr_c_op; - sessions[0] = curr_sess; - burst_size++; - } - } - - if (burst_size != 0) { - /* Process the crypto ops of the last operation type. */ - processed_ops = process_ops(c_ops, prev_zuc_op, - sessions, qp, burst_size, - &enqueued_ops); - } - - qp->qp_stats.enqueue_err_count += nb_ops - enqueued_ops; - return enqueued_ops; -} - -static uint16_t -zuc_pmd_dequeue_burst(void *queue_pair, - struct rte_crypto_op **c_ops, uint16_t nb_ops) -{ - struct zuc_qp *qp = queue_pair; - - unsigned nb_dequeued; - - nb_dequeued = rte_ring_dequeue_burst(qp->processed_ops, - (void **)c_ops, nb_ops, NULL); - qp->qp_stats.dequeued_count += nb_dequeued; - - return nb_dequeued; -} - -static int cryptodev_zuc_remove(struct rte_vdev_device *vdev); - -static int -cryptodev_zuc_create(const char *name, - struct rte_vdev_device *vdev, - struct rte_cryptodev_pmd_init_params *init_params) -{ - struct rte_cryptodev *dev; - struct zuc_private *internals; - MB_MGR *mb_mgr; - - dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); - if (dev == NULL) { - ZUC_LOG(ERR, "failed to create cryptodev vdev"); - goto init_error; - } - - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA | - RTE_CRYPTODEV_FF_SYM_SESSIONLESS | - RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; - - mb_mgr = alloc_mb_mgr(0); - if (mb_mgr == NULL) - return -ENOMEM; - - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; - init_mb_mgr_avx512(mb_mgr); - } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; - init_mb_mgr_avx2(mb_mgr); - } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - init_mb_mgr_avx(mb_mgr); - } else { - dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - init_mb_mgr_sse(mb_mgr); - } - - dev->driver_id = cryptodev_driver_id; - dev->dev_ops = rte_zuc_pmd_ops; - - /* Register RX/TX burst functions for data path. */ - dev->dequeue_burst = zuc_pmd_dequeue_burst; - dev->enqueue_burst = zuc_pmd_enqueue_burst; - - internals = dev->data->dev_private; - internals->mb_mgr = mb_mgr; - - internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; - - return 0; -init_error: - ZUC_LOG(ERR, "driver %s: failed", - init_params->name); - - cryptodev_zuc_remove(vdev); - return -EFAULT; -} - -static int -cryptodev_zuc_probe(struct rte_vdev_device *vdev) -{ - struct rte_cryptodev_pmd_init_params init_params = { - "", - sizeof(struct zuc_private), - rte_socket_id(), - RTE_CRYPTODEV_PMD_DEFAULT_MAX_NB_QUEUE_PAIRS - }; - const char *name; - const char *input_args; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - input_args = rte_vdev_device_args(vdev); - - rte_cryptodev_pmd_parse_input_args(&init_params, input_args); - - return cryptodev_zuc_create(name, vdev, &init_params); -} - -static int -cryptodev_zuc_remove(struct rte_vdev_device *vdev) -{ - - struct rte_cryptodev *cryptodev; - const char *name; - struct zuc_private *internals; - - name = rte_vdev_device_name(vdev); - if (name == NULL) - return -EINVAL; - - cryptodev = rte_cryptodev_pmd_get_named_dev(name); - if (cryptodev == NULL) - return -ENODEV; - - internals = cryptodev->data->dev_private; - - free_mb_mgr(internals->mb_mgr); - - return rte_cryptodev_pmd_destroy(cryptodev); -} - -static struct rte_vdev_driver cryptodev_zuc_pmd_drv = { - .probe = cryptodev_zuc_probe, - .remove = cryptodev_zuc_remove -}; - -static struct cryptodev_driver zuc_crypto_drv; - -RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_ZUC_PMD, cryptodev_zuc_pmd_drv); -RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_ZUC_PMD, - "max_nb_queue_pairs= " - "socket_id="); -RTE_PMD_REGISTER_CRYPTO_DRIVER(zuc_crypto_drv, cryptodev_zuc_pmd_drv.driver, - cryptodev_driver_id); -RTE_LOG_REGISTER_DEFAULT(zuc_logtype_driver, INFO); diff --git a/drivers/crypto/zuc/rte_zuc_pmd_ops.c b/drivers/crypto/zuc/rte_zuc_pmd_ops.c deleted file mode 100644 index 38642d45ab..0000000000 --- a/drivers/crypto/zuc/rte_zuc_pmd_ops.c +++ /dev/null @@ -1,322 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#include - -#include -#include -#include - -#include "zuc_pmd_private.h" - -static const struct rte_cryptodev_capabilities zuc_pmd_capabilities[] = { - { /* ZUC (EIA3) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_ZUC_EIA3, - .block_size = 16, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .digest_size = { - .min = 4, - .max = 4, - .increment = 0 - }, - .iv_size = { - .min = 16, - .max = 16, - .increment = 0 - } - }, } - }, } - }, - { /* ZUC (EEA3) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, - {.cipher = { - .algo = RTE_CRYPTO_CIPHER_ZUC_EEA3, - .block_size = 16, - .key_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - .iv_size = { - .min = 16, - .max = 16, - .increment = 0 - }, - }, } - }, } - }, - RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() -}; - -/** Configure device */ -static int -zuc_pmd_config(__rte_unused struct rte_cryptodev *dev, - __rte_unused struct rte_cryptodev_config *config) -{ - return 0; -} - -/** Start device */ -static int -zuc_pmd_start(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - -/** Stop device */ -static void -zuc_pmd_stop(__rte_unused struct rte_cryptodev *dev) -{ -} - -/** Close device */ -static int -zuc_pmd_close(__rte_unused struct rte_cryptodev *dev) -{ - return 0; -} - - -/** Get device statistics */ -static void -zuc_pmd_stats_get(struct rte_cryptodev *dev, - struct rte_cryptodev_stats *stats) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct zuc_qp *qp = dev->data->queue_pairs[qp_id]; - - stats->enqueued_count += qp->qp_stats.enqueued_count; - stats->dequeued_count += qp->qp_stats.dequeued_count; - - stats->enqueue_err_count += qp->qp_stats.enqueue_err_count; - stats->dequeue_err_count += qp->qp_stats.dequeue_err_count; - } -} - -/** Reset device statistics */ -static void -zuc_pmd_stats_reset(struct rte_cryptodev *dev) -{ - int qp_id; - - for (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) { - struct zuc_qp *qp = dev->data->queue_pairs[qp_id]; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - } -} - - -/** Get device info */ -static void -zuc_pmd_info_get(struct rte_cryptodev *dev, - struct rte_cryptodev_info *dev_info) -{ - struct zuc_private *internals = dev->data->dev_private; - - if (dev_info != NULL) { - dev_info->driver_id = dev->driver_id; - dev_info->max_nb_queue_pairs = internals->max_nb_queue_pairs; - /* No limit of number of sessions */ - dev_info->sym.max_nb_sessions = 0; - dev_info->feature_flags = dev->feature_flags; - dev_info->capabilities = zuc_pmd_capabilities; - } -} - -/** Release queue pair */ -static int -zuc_pmd_qp_release(struct rte_cryptodev *dev, uint16_t qp_id) -{ - if (dev->data->queue_pairs[qp_id] != NULL) { - struct zuc_qp *qp = dev->data->queue_pairs[qp_id]; - - if (qp->processed_ops) - rte_ring_free(qp->processed_ops); - - rte_free(dev->data->queue_pairs[qp_id]); - dev->data->queue_pairs[qp_id] = NULL; - } - return 0; -} - -/** set a unique name for the queue pair based on its name, dev_id and qp_id */ -static int -zuc_pmd_qp_set_unique_name(struct rte_cryptodev *dev, - struct zuc_qp *qp) -{ - unsigned n = snprintf(qp->name, sizeof(qp->name), - "zuc_pmd_%u_qp_%u", - dev->data->dev_id, qp->id); - - if (n >= sizeof(qp->name)) - return -1; - - return 0; -} - -/** Create a ring to place processed ops on */ -static struct rte_ring * -zuc_pmd_qp_create_processed_ops_ring(struct zuc_qp *qp, - unsigned ring_size, int socket_id) -{ - struct rte_ring *r; - - r = rte_ring_lookup(qp->name); - if (r) { - if (rte_ring_get_size(r) >= ring_size) { - ZUC_LOG(INFO, "Reusing existing ring %s" - " for processed packets", - qp->name); - return r; - } - - ZUC_LOG(ERR, "Unable to reuse existing ring %s" - " for processed packets", - qp->name); - return NULL; - } - - return rte_ring_create(qp->name, ring_size, socket_id, - RING_F_SP_ENQ | RING_F_SC_DEQ); -} - -/** Setup a queue pair */ -static int -zuc_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, - const struct rte_cryptodev_qp_conf *qp_conf, - int socket_id) -{ - struct zuc_qp *qp = NULL; - struct zuc_private *internals = dev->data->dev_private; - - /* Free memory prior to re-allocation if needed. */ - if (dev->data->queue_pairs[qp_id] != NULL) - zuc_pmd_qp_release(dev, qp_id); - - /* Allocate the queue pair data structure. */ - qp = rte_zmalloc_socket("ZUC PMD Queue Pair", sizeof(*qp), - RTE_CACHE_LINE_SIZE, socket_id); - if (qp == NULL) - return (-ENOMEM); - - qp->id = qp_id; - dev->data->queue_pairs[qp_id] = qp; - - if (zuc_pmd_qp_set_unique_name(dev, qp)) - goto qp_setup_cleanup; - - qp->processed_ops = zuc_pmd_qp_create_processed_ops_ring(qp, - qp_conf->nb_descriptors, socket_id); - if (qp->processed_ops == NULL) - goto qp_setup_cleanup; - - qp->mb_mgr = internals->mb_mgr; - qp->sess_mp = qp_conf->mp_session; - qp->sess_mp_priv = qp_conf->mp_session_private; - - memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); - - return 0; - -qp_setup_cleanup: - if (qp) - rte_free(qp); - - return -1; -} - -/** Returns the size of the ZUC session structure */ -static unsigned -zuc_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) -{ - return sizeof(struct zuc_session); -} - -/** Configure a ZUC session from a crypto xform chain */ -static int -zuc_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, - struct rte_crypto_sym_xform *xform, - struct rte_cryptodev_sym_session *sess, - struct rte_mempool *mempool) -{ - void *sess_private_data; - int ret; - - if (unlikely(sess == NULL)) { - ZUC_LOG(ERR, "invalid session struct"); - return -EINVAL; - } - - if (rte_mempool_get(mempool, &sess_private_data)) { - ZUC_LOG(ERR, - "Couldn't get object from session mempool"); - - return -ENOMEM; - } - - ret = zuc_set_session_parameters(sess_private_data, xform); - if (ret != 0) { - ZUC_LOG(ERR, "failed configure session parameters"); - - /* Return session to mempool */ - rte_mempool_put(mempool, sess_private_data); - return ret; - } - - set_sym_session_private_data(sess, dev->driver_id, - sess_private_data); - - return 0; -} - -/** Clear the memory of session so it doesn't leave key material behind */ -static void -zuc_pmd_sym_session_clear(struct rte_cryptodev *dev, - struct rte_cryptodev_sym_session *sess) -{ - uint8_t index = dev->driver_id; - void *sess_priv = get_sym_session_private_data(sess, index); - - /* Zero out the whole structure */ - if (sess_priv) { - memset(sess_priv, 0, sizeof(struct zuc_session)); - struct rte_mempool *sess_mp = rte_mempool_from_obj(sess_priv); - set_sym_session_private_data(sess, index, NULL); - rte_mempool_put(sess_mp, sess_priv); - } -} - -struct rte_cryptodev_ops zuc_pmd_ops = { - .dev_configure = zuc_pmd_config, - .dev_start = zuc_pmd_start, - .dev_stop = zuc_pmd_stop, - .dev_close = zuc_pmd_close, - - .stats_get = zuc_pmd_stats_get, - .stats_reset = zuc_pmd_stats_reset, - - .dev_infos_get = zuc_pmd_info_get, - - .queue_pair_setup = zuc_pmd_qp_setup, - .queue_pair_release = zuc_pmd_qp_release, - - .sym_session_get_size = zuc_pmd_sym_session_get_size, - .sym_session_configure = zuc_pmd_sym_session_configure, - .sym_session_clear = zuc_pmd_sym_session_clear -}; - -struct rte_cryptodev_ops *rte_zuc_pmd_ops = &zuc_pmd_ops; diff --git a/drivers/crypto/zuc/version.map b/drivers/crypto/zuc/version.map deleted file mode 100644 index c2e0723b4c..0000000000 --- a/drivers/crypto/zuc/version.map +++ /dev/null @@ -1,3 +0,0 @@ -DPDK_22 { - local: *; -}; diff --git a/drivers/crypto/zuc/zuc_pmd_private.h b/drivers/crypto/zuc/zuc_pmd_private.h deleted file mode 100644 index d8684891ee..0000000000 --- a/drivers/crypto/zuc/zuc_pmd_private.h +++ /dev/null @@ -1,83 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016-2018 Intel Corporation - */ - -#ifndef _ZUC_PMD_PRIVATE_H_ -#define _ZUC_PMD_PRIVATE_H_ - -#include - -#define CRYPTODEV_NAME_ZUC_PMD crypto_zuc -/**< ZUC PMD device name */ - -/** ZUC PMD LOGTYPE DRIVER */ -extern int zuc_logtype_driver; -#define ZUC_LOG(level, fmt, ...) \ - rte_log(RTE_LOG_ ## level, zuc_logtype_driver, \ - "%s()... line %u: " fmt "\n", __func__, __LINE__, \ - ## __VA_ARGS__) - -#define ZUC_IV_KEY_LENGTH 16 -#define ZUC_DIGEST_LENGTH 4 - -/** private data structure for each virtual ZUC device */ -struct zuc_private { - unsigned max_nb_queue_pairs; - /**< Max number of queue pairs supported by device */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ -}; - -/** ZUC buffer queue pair */ -struct zuc_qp { - uint16_t id; - /**< Queue Pair Identifier */ - char name[RTE_CRYPTODEV_NAME_MAX_LEN]; - /**< Unique Queue Pair Name */ - struct rte_ring *processed_ops; - /**< Ring for placing processed ops */ - struct rte_mempool *sess_mp; - /**< Session Mempool */ - struct rte_mempool *sess_mp_priv; - /**< Session Private Data Mempool */ - struct rte_cryptodev_stats qp_stats; - /**< Queue pair statistics */ - uint8_t temp_digest[ZUC_DIGEST_LENGTH]; - /**< Buffer used to store the digest generated - * by the driver when verifying a digest provided - * by the user (using authentication verify operation) - */ - MB_MGR *mb_mgr; - /**< Multi-buffer instance */ -} __rte_cache_aligned; - -enum zuc_operation { - ZUC_OP_ONLY_CIPHER, - ZUC_OP_ONLY_AUTH, - ZUC_OP_CIPHER_AUTH, - ZUC_OP_AUTH_CIPHER, - ZUC_OP_NOT_SUPPORTED -}; - -/** ZUC private session structure */ -struct zuc_session { - enum zuc_operation op; - enum rte_crypto_auth_operation auth_op; - uint8_t pKey_cipher[ZUC_IV_KEY_LENGTH]; - uint8_t pKey_hash[ZUC_IV_KEY_LENGTH]; - uint16_t cipher_iv_offset; - uint16_t auth_iv_offset; -} __rte_cache_aligned; - - -extern int -zuc_set_session_parameters(struct zuc_session *sess, - const struct rte_crypto_sym_xform *xform); - - -/** device specific operations function pointer structure */ -extern struct rte_cryptodev_ops *rte_zuc_pmd_ops; - - - -#endif /* _ZUC_PMD_PRIVATE_H_ */ From patchwork Fri Oct 15 14:39:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101784 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 706C6A0C45; Fri, 15 Oct 2021 16:41:59 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 19415412C0; Fri, 15 Oct 2021 16:40:52 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 1A27B412A0 for ; Fri, 15 Oct 2021 16:40:48 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816206" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816206" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542922" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:46 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Kai Ji , Ciara Power , Thomas Monjalon Date: Fri, 15 Oct 2021 14:39:56 +0000 Message-Id: <20211015143957.842499-14-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 13/14] crypto/ipsec_mb: add chacha20-poly1305 PMD to framework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Kai Ji Add in new chacha20_poly1305 PMD to the ipsec_mb framework. Signed-off-by: Kai Ji Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - Fixed some function names. - Removed unnecessary enqueued counts from process_ops function. - Fixed table label in doc. - Split into private header file. - Removed security includes. - Added sub release note under ipsec_mb note. - Split out test app changes to new patch. - Fixed copyright years. - Added docs to maintainers file. v3: - Fixed some formatting. - Removed unnecessary get session function. v2: - Added unused tag to session configure parameter. - Added release note. - Added documentation for PMD. --- MAINTAINERS | 2 + doc/guides/cryptodevs/chacha20_poly1305.rst | 99 +++++ .../cryptodevs/features/chacha20_poly1305.ini | 35 ++ doc/guides/cryptodevs/index.rst | 1 + doc/guides/rel_notes/release_21_11.rst | 1 + drivers/crypto/ipsec_mb/ipsec_mb_private.h | 7 + drivers/crypto/ipsec_mb/meson.build | 1 + drivers/crypto/ipsec_mb/pmd_chacha_poly.c | 408 ++++++++++++++++++ .../crypto/ipsec_mb/pmd_chacha_poly_priv.h | 74 ++++ 9 files changed, 628 insertions(+) create mode 100644 doc/guides/cryptodevs/chacha20_poly1305.rst create mode 100644 doc/guides/cryptodevs/features/chacha20_poly1305.ini create mode 100644 drivers/crypto/ipsec_mb/pmd_chacha_poly.c create mode 100644 drivers/crypto/ipsec_mb/pmd_chacha_poly_priv.h diff --git a/MAINTAINERS b/MAINTAINERS index 1c269bd991..9d11c76482 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1059,11 +1059,13 @@ M: Pablo de Lara F: drivers/crypto/ipsec_mb/ F: doc/guides/cryptodevs/aesni_gcm.rst F: doc/guides/cryptodevs/aesni_mb.rst +F: doc/guides/cryptodevs/chacha20_poly1305.rst F: doc/guides/cryptodevs/kasumi.rst F: doc/guides/cryptodevs/snow3g.rst F: doc/guides/cryptodevs/zuc.rst F: doc/guides/cryptodevs/features/aesni_gcm.ini F: doc/guides/cryptodevs/features/aesni_mb.ini +F: doc/guides/cryptodevs/features/chacha20_poly1305.ini F: doc/guides/cryptodevs/features/kasumi.ini F: doc/guides/cryptodevs/features/snow3g.ini F: doc/guides/cryptodevs/features/zuc.ini diff --git a/doc/guides/cryptodevs/chacha20_poly1305.rst b/doc/guides/cryptodevs/chacha20_poly1305.rst new file mode 100644 index 0000000000..4db260818e --- /dev/null +++ b/doc/guides/cryptodevs/chacha20_poly1305.rst @@ -0,0 +1,99 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +Chacha20-poly1305 Crypto Poll Mode Driver +========================================= + +The Chacha20-poly1305 PMD provides poll mode crypto driver support for +utilizing `Intel IPSec Multi-buffer library `_. + +Features +-------- + +Chacha20-poly1305 PMD has support for: + +AEAD algorithms: + +* RTE_CRYPTO_AEAD_CHACHA20_POLY1305 + + +Installation +------------ + +To build DPDK with the Chacha20-poly1305 PMD the user is required to download +the multi-buffer library from `here `_ +and compile it on their user system before building DPDK. +The latest version of the library supported by this PMD is v1.0, which +can be downloaded from ``_. + +After downloading the library, the user needs to unpack and compile it +on their system before building DPDK: + +.. code-block:: console + + make + make install + +The library requires NASM to be built. Depending on the library version, it might +require a minimum NASM version (e.g. v0.54 requires at least NASM 2.14). + +NASM is packaged for different OS. However, on some OS the version is too old, +so a manual installation is required. In that case, NASM can be downloaded from +`NASM website `_. +Once it is downloaded, extract it and follow these steps: + +.. code-block:: console + + ./configure + make + make install + +.. note:: + + Compilation of the Multi-Buffer library is broken when GCC < 5.0, if library <= v0.53. + If a lower GCC version than 5.0, the workaround proposed by the following link + should be used: ``_. + +As a reference, the following table shows a mapping between the past DPDK versions +and the external crypto libraries supported by them: + +.. _table_chacha20_poly1305_versions: + +.. table:: DPDK and external crypto library version compatibility + + ============= ================================ + DPDK version Crypto library version + ============= ================================ + 21.11+ Multi-buffer library 1.0* + ============= ================================ + +\* Multi-buffer library 1.0 or newer only works for Meson but not Make build system. + +Initialization +-------------- + +In order to enable this virtual crypto PMD, user must: + +* Build the multi buffer library (explained in Installation section). + +To use the PMD in an application, user must: + +* Call rte_vdev_init("crypto_chacha20_poly1305") within the application. + +* Use --vdev="crypto_chacha20_poly1305" in the EAL options, which will call + rte_vdev_init() internally. + +The following parameters (all optional) can be provided in the previous two calls: + +* socket_id: Specify the socket where the memory for the device is going to be allocated + (by default, socket_id will be the socket where the core that is creating the PMD is running on). + +* max_nb_queue_pairs: Specify the maximum number of queue pairs in the device (8 by default). + +* max_nb_sessions: Specify the maximum number of sessions that can be created (2048 by default). + +Example: + +.. code-block:: console + + --vdev="crypto_chacha20_poly1305,socket_id=0,max_nb_sessions=128" diff --git a/doc/guides/cryptodevs/features/chacha20_poly1305.ini b/doc/guides/cryptodevs/features/chacha20_poly1305.ini new file mode 100644 index 0000000000..3353e031c9 --- /dev/null +++ b/doc/guides/cryptodevs/features/chacha20_poly1305.ini @@ -0,0 +1,35 @@ +; +; Supported features of the 'chacha20_poly1305' crypto driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Symmetric crypto = Y +Sym operation chaining = Y +Symmetric sessionless = Y +Non-Byte aligned data = Y +In Place SGL = Y +OOP SGL In LB Out = Y +OOP LB In LB Out = Y +CPU crypto = Y + +; +; Supported crypto algorithms of the 'chacha20_poly1305' crypto driver. +; +[Cipher] + +; +; Supported authentication algorithms of the 'chacha20_poly1305' crypto driver. +; +[Auth] + +; +; Supported AEAD algorithms of the 'chacha20_poly1305' crypto driver. +; +[AEAD] +CHACHA20-POLY1305 = Y + +; +; Supported Asymmetric algorithms of the 'chacha20_poly1305' crypto driver. +; +[Asymmetric] diff --git a/doc/guides/cryptodevs/index.rst b/doc/guides/cryptodevs/index.rst index 0f981c77b5..3dcc2ecd2e 100644 --- a/doc/guides/cryptodevs/index.rst +++ b/doc/guides/cryptodevs/index.rst @@ -16,6 +16,7 @@ Crypto Device Drivers bcmfs caam_jr ccp + chacha20_poly1305 cnxk dpaa2_sec dpaa_sec diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 52ac148884..0f216c7b4f 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -109,6 +109,7 @@ New Features * KASUMI PMD. * SNOW3G PMD. * ZUC PMD. + * CHACHA20_POLY1305 PMD. * **Updated the aesni_mb crypto PMD.** diff --git a/drivers/crypto/ipsec_mb/ipsec_mb_private.h b/drivers/crypto/ipsec_mb/ipsec_mb_private.h index fef3fcab68..866722d6f4 100644 --- a/drivers/crypto/ipsec_mb/ipsec_mb_private.h +++ b/drivers/crypto/ipsec_mb/ipsec_mb_private.h @@ -49,6 +49,9 @@ extern RTE_DEFINE_PER_LCORE(IMB_MGR *, mb_mgr); #define CRYPTODEV_NAME_ZUC_PMD crypto_zuc /**< IPSEC Multi buffer PMD zuc device name */ +#define CRYPTODEV_NAME_CHACHA20_POLY1305_PMD crypto_chacha20_poly1305 +/**< IPSEC Multi buffer PMD chacha20_poly1305 device name */ + /** PMD LOGTYPE DRIVER, common to all PMDs */ extern int ipsec_mb_logtype_driver; #define IPSEC_MB_LOG(level, fmt, ...) \ @@ -62,6 +65,7 @@ enum ipsec_mb_pmd_types { IPSEC_MB_PMD_TYPE_KASUMI, IPSEC_MB_PMD_TYPE_SNOW3G, IPSEC_MB_PMD_TYPE_ZUC, + IPSEC_MB_PMD_TYPE_CHACHA20_POLY1305, IPSEC_MB_N_PMD_TYPES }; @@ -85,6 +89,7 @@ extern uint8_t pmd_driver_id_aesni_gcm; extern uint8_t pmd_driver_id_kasumi; extern uint8_t pmd_driver_id_snow3g; extern uint8_t pmd_driver_id_zuc; +extern uint8_t pmd_driver_id_chacha20_poly1305; /** Helper function. Gets driver ID based on PMD type */ static __rte_always_inline uint8_t @@ -101,6 +106,8 @@ ipsec_mb_get_driver_id(enum ipsec_mb_pmd_types pmd_type) return pmd_driver_id_snow3g; case IPSEC_MB_PMD_TYPE_ZUC: return pmd_driver_id_zuc; + case IPSEC_MB_PMD_TYPE_CHACHA20_POLY1305: + return pmd_driver_id_chacha20_poly1305; default: break; } diff --git a/drivers/crypto/ipsec_mb/meson.build b/drivers/crypto/ipsec_mb/meson.build index c712c2adb1..d7037daea1 100644 --- a/drivers/crypto/ipsec_mb/meson.build +++ b/drivers/crypto/ipsec_mb/meson.build @@ -25,6 +25,7 @@ sources = files('ipsec_mb_private.c', 'ipsec_mb_ops.c', 'pmd_aesni_mb.c', 'pmd_aesni_gcm.c', + 'pmd_chacha_poly.c', 'pmd_kasumi.c', 'pmd_snow3g.c', 'pmd_zuc.c' diff --git a/drivers/crypto/ipsec_mb/pmd_chacha_poly.c b/drivers/crypto/ipsec_mb/pmd_chacha_poly.c new file mode 100644 index 0000000000..d953d6e5f5 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_chacha_poly.c @@ -0,0 +1,408 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include "pmd_chacha_poly_priv.h" + +/** Parse crypto xform chain and set private session parameters. */ +static int +chacha20_poly1305_session_configure(IMB_MGR * mb_mgr __rte_unused, + void *priv_sess, const struct rte_crypto_sym_xform *xform) +{ + struct chacha20_poly1305_session *sess = priv_sess; + const struct rte_crypto_sym_xform *auth_xform; + const struct rte_crypto_sym_xform *cipher_xform; + const struct rte_crypto_sym_xform *aead_xform; + + uint8_t key_length; + const uint8_t *key; + enum ipsec_mb_operation mode; + int ret = 0; + + ret = ipsec_mb_parse_xform(xform, &mode, &auth_xform, + &cipher_xform, &aead_xform); + if (ret) + return ret; + + sess->op = mode; + + switch (sess->op) { + case IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT: + case IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT: + if (aead_xform->aead.algo != + RTE_CRYPTO_AEAD_CHACHA20_POLY1305) { + IPSEC_MB_LOG(ERR, + "The only combined operation supported is CHACHA20 POLY1305"); + ret = -ENOTSUP; + goto error_exit; + } + /* Set IV parameters */ + sess->iv.offset = aead_xform->aead.iv.offset; + sess->iv.length = aead_xform->aead.iv.length; + key_length = aead_xform->aead.key.length; + key = aead_xform->aead.key.data; + sess->aad_length = aead_xform->aead.aad_length; + sess->req_digest_length = aead_xform->aead.digest_length; + break; + default: + IPSEC_MB_LOG( + ERR, "Wrong xform type, has to be AEAD or authentication"); + ret = -ENOTSUP; + goto error_exit; + } + + /* IV check */ + if (sess->iv.length != CHACHA20_POLY1305_IV_LENGTH && + sess->iv.length != 0) { + IPSEC_MB_LOG(ERR, "Wrong IV length"); + ret = -EINVAL; + goto error_exit; + } + + /* Check key length */ + if (key_length != CHACHA20_POLY1305_KEY_SIZE) { + IPSEC_MB_LOG(ERR, "Invalid key length"); + ret = -EINVAL; + goto error_exit; + } else { + memcpy(sess->key, key, CHACHA20_POLY1305_KEY_SIZE); + } + + /* Digest check */ + if (sess->req_digest_length != CHACHA20_POLY1305_DIGEST_LENGTH) { + IPSEC_MB_LOG(ERR, "Invalid digest length"); + ret = -EINVAL; + goto error_exit; + } else { + sess->gen_digest_length = CHACHA20_POLY1305_DIGEST_LENGTH; + } + +error_exit: + return ret; +} + +/** + * Process a crypto operation, calling + * the direct chacha poly API from the multi buffer library. + * + * @param qp queue pair + * @param op symmetric crypto operation + * @param session chacha poly session + * + * @return + * - Return 0 if success + */ +static int +chacha20_poly1305_crypto_op(struct ipsec_mb_qp *qp, struct rte_crypto_op *op, + struct chacha20_poly1305_session *session) +{ + struct chacha20_poly1305_qp_data *qp_data = + ipsec_mb_get_qp_private_data(qp); + uint8_t *src, *dst; + uint8_t *iv_ptr; + struct rte_crypto_sym_op *sym_op = op->sym; + struct rte_mbuf *m_src = sym_op->m_src; + uint32_t offset, data_offset, data_length; + uint32_t part_len, data_len; + int total_len; + uint8_t *tag; + unsigned int oop = 0; + + offset = sym_op->aead.data.offset; + data_offset = offset; + data_length = sym_op->aead.data.length; + RTE_ASSERT(m_src != NULL); + + while (offset >= m_src->data_len && data_length != 0) { + offset -= m_src->data_len; + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + } + + src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset); + + data_len = m_src->data_len - offset; + part_len = (data_len < data_length) ? data_len : + data_length; + + /* In-place */ + if (sym_op->m_dst == NULL || (sym_op->m_dst == sym_op->m_src)) + dst = src; + /* Out-of-place */ + else { + oop = 1; + /* Segmented destination buffer is not supported + * if operation is Out-of-place + */ + RTE_ASSERT(rte_pktmbuf_is_contiguous(sym_op->m_dst)); + dst = rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *, + data_offset); + } + + iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + session->iv.offset); + + IMB_CHACHA20_POLY1305_INIT(qp->mb_mgr, session->key, + &qp_data->chacha20_poly1305_ctx_data, + iv_ptr, sym_op->aead.aad.data, + (uint64_t)session->aad_length); + + if (session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_ENCRYPT) { + IMB_CHACHA20_POLY1305_ENC_UPDATE(qp->mb_mgr, + session->key, + &qp_data->chacha20_poly1305_ctx_data, + dst, src, (uint64_t)part_len); + total_len = data_length - part_len; + + while (total_len) { + m_src = m_src->next; + RTE_ASSERT(m_src != NULL); + + src = rte_pktmbuf_mtod(m_src, uint8_t *); + if (oop) + dst += part_len; + else + dst = src; + part_len = (m_src->data_len < total_len) ? + m_src->data_len : total_len; + + if (dst == NULL || src == NULL) { + IPSEC_MB_LOG(ERR, "Invalid src or dst input"); + return -EINVAL; + } + IMB_CHACHA20_POLY1305_ENC_UPDATE(qp->mb_mgr, + session->key, + &qp_data->chacha20_poly1305_ctx_data, + dst, src, (uint64_t)part_len); + total_len -= part_len; + if (total_len < 0) { + IPSEC_MB_LOG(ERR, "Invalid part len"); + return -EINVAL; + } + } + + tag = sym_op->aead.digest.data; + IMB_CHACHA20_POLY1305_ENC_FINALIZE(qp->mb_mgr, + &qp_data->chacha20_poly1305_ctx_data, + tag, session->gen_digest_length); + + } else { + IMB_CHACHA20_POLY1305_DEC_UPDATE(qp->mb_mgr, + session->key, + &qp_data->chacha20_poly1305_ctx_data, + dst, src, (uint64_t)part_len); + + total_len = data_length - part_len; + + while (total_len) { + m_src = m_src->next; + + RTE_ASSERT(m_src != NULL); + + src = rte_pktmbuf_mtod(m_src, uint8_t *); + if (oop) + dst += part_len; + else + dst = src; + part_len = (m_src->data_len < total_len) ? + m_src->data_len : total_len; + + if (dst == NULL || src == NULL) { + IPSEC_MB_LOG(ERR, "Invalid src or dst input"); + return -EINVAL; + } + IMB_CHACHA20_POLY1305_DEC_UPDATE(qp->mb_mgr, + session->key, + &qp_data->chacha20_poly1305_ctx_data, + dst, src, (uint64_t)part_len); + total_len -= part_len; + if (total_len < 0) { + IPSEC_MB_LOG(ERR, "Invalid part len"); + return -EINVAL; + } + } + + tag = qp_data->temp_digest; + IMB_CHACHA20_POLY1305_DEC_FINALIZE(qp->mb_mgr, + &qp_data->chacha20_poly1305_ctx_data, + tag, session->gen_digest_length); + } + + return 0; +} + +/** + * Process a completed chacha poly op + * + * @param qp Queue Pair to process + * @param op Crypto operation + * @param sess Crypto session + * + * @return + * - void + */ +static void +post_process_chacha20_poly1305_crypto_op(struct ipsec_mb_qp *qp, + struct rte_crypto_op *op, + struct chacha20_poly1305_session *session) +{ + struct chacha20_poly1305_qp_data *qp_data = + ipsec_mb_get_qp_private_data(qp); + + op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + /* Verify digest if required */ + if (session->op == IPSEC_MB_OP_AEAD_AUTHENTICATED_DECRYPT || + session->op == IPSEC_MB_OP_HASH_VERIFY_ONLY) { + uint8_t *digest = op->sym->aead.digest.data; + uint8_t *tag = qp_data->temp_digest; + +#ifdef RTE_LIBRTE_PMD_CHACHA20_POLY1305_DEBUG + rte_hexdump(stdout, "auth tag (orig):", + digest, session->req_digest_length); + rte_hexdump(stdout, "auth tag (calc):", + tag, session->req_digest_length); +#endif + if (memcmp(tag, digest, session->req_digest_length) != 0) + op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + + } + +} + +/** + * Process a completed Chacha20_poly1305 request + * + * @param qp Queue Pair to process + * @param op Crypto operation + * @param sess Crypto session + * + * @return + * - void + */ +static void +handle_completed_chacha20_poly1305_crypto_op(struct ipsec_mb_qp *qp, + struct rte_crypto_op *op, + struct chacha20_poly1305_session *sess) +{ + post_process_chacha20_poly1305_crypto_op(qp, op, sess); + + /* Free session if a session-less crypto op */ + if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + memset(sess, 0, sizeof(struct chacha20_poly1305_session)); + memset(op->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + op->sym->session)); + rte_mempool_put(qp->sess_mp_priv, sess); + rte_mempool_put(qp->sess_mp, op->sym->session); + op->sym->session = NULL; + } +} + +static uint16_t +chacha20_poly1305_pmd_dequeue_burst(void *queue_pair, + struct rte_crypto_op **ops, uint16_t nb_ops) +{ + struct chacha20_poly1305_session *sess; + struct ipsec_mb_qp *qp = queue_pair; + + int retval = 0; + unsigned int i = 0, nb_dequeued; + + nb_dequeued = rte_ring_dequeue_burst(qp->ingress_queue, + (void **)ops, nb_ops, NULL); + + for (i = 0; i < nb_dequeued; i++) { + + sess = ipsec_mb_get_session_private(qp, ops[i]); + if (unlikely(sess == NULL)) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + qp->stats.dequeue_err_count++; + break; + } + + retval = chacha20_poly1305_crypto_op(qp, ops[i], sess); + if (retval < 0) { + ops[i]->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + qp->stats.dequeue_err_count++; + break; + } + + handle_completed_chacha20_poly1305_crypto_op(qp, ops[i], sess); + } + + qp->stats.dequeued_count += i; + + return i; +} + +struct rte_cryptodev_ops chacha20_poly1305_pmd_ops = { + .dev_configure = ipsec_mb_config, + .dev_start = ipsec_mb_start, + .dev_stop = ipsec_mb_stop, + .dev_close = ipsec_mb_close, + + .stats_get = ipsec_mb_stats_get, + .stats_reset = ipsec_mb_stats_reset, + + .dev_infos_get = ipsec_mb_info_get, + + .queue_pair_setup = ipsec_mb_qp_setup, + .queue_pair_release = ipsec_mb_qp_release, + + .sym_session_get_size = ipsec_mb_sym_session_get_size, + .sym_session_configure = ipsec_mb_sym_session_configure, + .sym_session_clear = ipsec_mb_sym_session_clear +}; + +struct rte_cryptodev_ops *rte_chacha20_poly1305_pmd_ops = + &chacha20_poly1305_pmd_ops; + +static int +chacha20_poly1305_probe(struct rte_vdev_device *vdev) +{ + return ipsec_mb_create(vdev, IPSEC_MB_PMD_TYPE_CHACHA20_POLY1305); +} + +static struct rte_vdev_driver cryptodev_chacha20_poly1305_pmd_drv = { + .probe = chacha20_poly1305_probe, + .remove = ipsec_mb_remove +}; + +static struct cryptodev_driver chacha20_poly1305_crypto_drv; + +RTE_PMD_REGISTER_VDEV(CRYPTODEV_NAME_CHACHA20_POLY1305_PMD, + cryptodev_chacha20_poly1305_pmd_drv); +RTE_PMD_REGISTER_ALIAS(CRYPTODEV_NAME_CHACHA20_POLY1305_PMD, + cryptodev_chacha20_poly1305_pmd); +RTE_PMD_REGISTER_PARAM_STRING(CRYPTODEV_NAME_CHACHA20_POLY1305_PMD, + "max_nb_queue_pairs= socket_id="); +RTE_PMD_REGISTER_CRYPTO_DRIVER(chacha20_poly1305_crypto_drv, + cryptodev_chacha20_poly1305_pmd_drv.driver, + pmd_driver_id_chacha20_poly1305); + +/* Constructor function to register chacha20_poly1305 PMD */ +RTE_INIT(ipsec_mb_register_chacha20_poly1305) +{ + struct ipsec_mb_internals *chacha_poly_data + = &ipsec_mb_pmds[IPSEC_MB_PMD_TYPE_CHACHA20_POLY1305]; + + chacha_poly_data->caps = chacha20_poly1305_capabilities; + chacha_poly_data->dequeue_burst = chacha20_poly1305_pmd_dequeue_burst; + chacha_poly_data->feature_flags = + RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | + RTE_CRYPTODEV_FF_IN_PLACE_SGL | + RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | + RTE_CRYPTODEV_FF_SYM_CPU_CRYPTO | + RTE_CRYPTODEV_FF_SYM_SESSIONLESS; + chacha_poly_data->internals_priv_size = 0; + chacha_poly_data->ops = &chacha20_poly1305_pmd_ops; + chacha_poly_data->qp_priv_size = + sizeof(struct chacha20_poly1305_qp_data); + chacha_poly_data->session_configure = + chacha20_poly1305_session_configure; + chacha_poly_data->session_priv_size = + sizeof(struct chacha20_poly1305_session); +} diff --git a/drivers/crypto/ipsec_mb/pmd_chacha_poly_priv.h b/drivers/crypto/ipsec_mb/pmd_chacha_poly_priv.h new file mode 100644 index 0000000000..842f62f5d1 --- /dev/null +++ b/drivers/crypto/ipsec_mb/pmd_chacha_poly_priv.h @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef _PMD_CHACHA_POLY_PRIV_H_ +#define _PMD_CHACHA_POLY_PRIV_H_ + +#include "ipsec_mb_private.h" + +#define CHACHA20_POLY1305_IV_LENGTH 12 +#define CHACHA20_POLY1305_DIGEST_LENGTH 16 +#define CHACHA20_POLY1305_KEY_SIZE 32 + +static const +struct rte_cryptodev_capabilities chacha20_poly1305_capabilities[] = { + {/* CHACHA20-POLY1305 */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, + {.aead = { + .algo = RTE_CRYPTO_AEAD_CHACHA20_POLY1305, + .block_size = 64, + .key_size = { + .min = 32, + .max = 32, + .increment = 0}, + .digest_size = { + .min = 16, + .max = 16, + .increment = 0}, + .aad_size = { + .min = 0, + .max = 240, + .increment = 1}, + .iv_size = { + .min = 12, + .max = 12, + .increment = 0}, + }, + } + },} + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +uint8_t pmd_driver_id_chacha20_poly1305; + +/** CHACHA20 POLY1305 private session structure */ +struct chacha20_poly1305_session { + struct { + uint16_t length; + uint16_t offset; + } iv; + /**< IV parameters */ + uint16_t aad_length; + /**< AAD length */ + uint16_t req_digest_length; + /**< Requested digest length */ + uint16_t gen_digest_length; + /**< Generated digest length */ + uint8_t key[CHACHA20_POLY1305_KEY_SIZE]; + enum ipsec_mb_operation op; +} __rte_cache_aligned; + +struct chacha20_poly1305_qp_data { + struct chacha20_poly1305_context_data chacha20_poly1305_ctx_data; + uint8_t temp_digest[CHACHA20_POLY1305_DIGEST_LENGTH]; + /**< Buffer used to store the digest generated + * by the driver when verifying a digest provided + * by the user (using authentication verify operation) + */ +}; + +#endif /* _PMD_CHACHA_POLY_PRIV_H_ */ From patchwork Fri Oct 15 14:39:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 101785 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6BFDBA0C45; Fri, 15 Oct 2021 16:42:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 182CD41250; Fri, 15 Oct 2021 16:40:55 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 6B9DF412B8 for ; Fri, 15 Oct 2021 16:40:51 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="227816208" X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="227816208" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 07:40:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,376,1624345200"; d="scan'208";a="442542930" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.87]) by orsmga006.jf.intel.com with ESMTP; 15 Oct 2021 07:40:48 -0700 From: Ciara Power To: dev@dpdk.org Cc: roy.fan.zhang@intel.com, piotrx.bronowski@intel.com, gakhil@marvell.com, pablo.de.lara.guarch@intel.com, mdr@ashroe.eu, Kai Ji , Ciara Power , Declan Doherty Date: Fri, 15 Oct 2021 14:39:57 +0000 Message-Id: <20211015143957.842499-15-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211015143957.842499-1-ciara.power@intel.com> References: <20210618121803.1189857-1-piotrx.bronowski@intel.com> <20211015143957.842499-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 14/14] test/crypto: add test for chacha20_poly1305 PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Kai Ji An autotest is added for the new chacha20_poly1305 PMD. A new test case is also added for SGL test. Signed-off-by: Kai Ji Signed-off-by: Ciara Power Acked-by: Ray Kinsella Acked-by: Fan Zhang --- v4: - This patch is newly created, having been split from the previous patch that adds chacha20_poly1305 PMD. --- app/test/test_cryptodev.c | 23 ++++ app/test/test_cryptodev.h | 1 + app/test/test_cryptodev_aead_test_vectors.h | 114 ++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 3 + 4 files changed, 141 insertions(+) diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c index 17444d8fb7..718269cf5f 100644 --- a/app/test/test_cryptodev.c +++ b/app/test/test_cryptodev.c @@ -13998,6 +13998,14 @@ test_chacha20_poly1305_decrypt_test_case_rfc8439(void) return test_authenticated_decryption(&chacha20_poly1305_case_rfc8439); } +static int +test_chacha20_poly1305_encrypt_SGL_out_of_place(void) +{ + return test_authenticated_encryption_SGL( + &chacha20_poly1305_case_2, OUT_OF_PLACE, 32, + chacha20_poly1305_case_2.plaintext.len); +} + #ifdef RTE_CRYPTO_SCHEDULER /* global AESNI worker IDs for the scheduler test */ @@ -14682,6 +14690,8 @@ static struct unit_test_suite cryptodev_chacha20_poly1305_testsuite = { test_chacha20_poly1305_encrypt_test_case_rfc8439), TEST_CASE_ST(ut_setup, ut_teardown, test_chacha20_poly1305_decrypt_test_case_rfc8439), + TEST_CASE_ST(ut_setup, ut_teardown, + test_chacha20_poly1305_encrypt_SGL_out_of_place), TEST_CASES_END() } }; @@ -15264,6 +15274,17 @@ test_cryptodev_cpu_aesni_mb(void) return rc; } +static int +test_cryptodev_chacha_poly_mb(void) +{ + int32_t rc; + enum rte_security_session_action_type at = gbl_action_type; + rc = run_cryptodev_testsuite( + RTE_STR(CRYPTODEV_NAME_CHACHA20_POLY1305_PMD)); + gbl_action_type = at; + return rc; +} + static int test_cryptodev_openssl(void) { @@ -15523,6 +15544,8 @@ REGISTER_TEST_COMMAND(cryptodev_qat_autotest, test_cryptodev_qat); REGISTER_TEST_COMMAND(cryptodev_aesni_mb_autotest, test_cryptodev_aesni_mb); REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_mb_autotest, test_cryptodev_cpu_aesni_mb); +REGISTER_TEST_COMMAND(cryptodev_chacha_poly_mb_autotest, + test_cryptodev_chacha_poly_mb); REGISTER_TEST_COMMAND(cryptodev_openssl_autotest, test_cryptodev_openssl); REGISTER_TEST_COMMAND(cryptodev_aesni_gcm_autotest, test_cryptodev_aesni_gcm); REGISTER_TEST_COMMAND(cryptodev_cpu_aesni_gcm_autotest, diff --git a/app/test/test_cryptodev.h b/app/test/test_cryptodev.h index 1cdd84d01f..90c8287365 100644 --- a/app/test/test_cryptodev.h +++ b/app/test/test_cryptodev.h @@ -59,6 +59,7 @@ #define CRYPTODEV_NAME_SNOW3G_PMD crypto_snow3g #define CRYPTODEV_NAME_KASUMI_PMD crypto_kasumi #define CRYPTODEV_NAME_ZUC_PMD crypto_zuc +#define CRYPTODEV_NAME_CHACHA20_POLY1305_PMD crypto_chacha20_poly1305 #define CRYPTODEV_NAME_ARMV8_PMD crypto_armv8 #define CRYPTODEV_NAME_DPAA_SEC_PMD crypto_dpaa_sec #define CRYPTODEV_NAME_DPAA2_SEC_PMD crypto_dpaa2_sec diff --git a/app/test/test_cryptodev_aead_test_vectors.h b/app/test/test_cryptodev_aead_test_vectors.h index 73cc143f10..07292620a4 100644 --- a/app/test/test_cryptodev_aead_test_vectors.h +++ b/app/test/test_cryptodev_aead_test_vectors.h @@ -3930,4 +3930,118 @@ static const struct aead_test_data chacha20_poly1305_case_rfc8439 = { .len = 16 } }; + +static uint8_t chacha_aad_2[] = { + 0xf3, 0x33, 0x88, 0x86, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x4e, 0x91 +}; + +static const struct aead_test_data chacha20_poly1305_case_2 = { + .algo = RTE_CRYPTO_AEAD_CHACHA20_POLY1305, + .key = { + .data = { + 0x1c, 0x92, 0x40, 0xa5, 0xeb, 0x55, 0xd3, 0x8a, + 0xf3, 0x33, 0x88, 0x86, 0x04, 0xf6, 0xb5, 0xf0, + 0x47, 0x39, 0x17, 0xc1, 0x40, 0x2b, 0x80, 0x09, + 0x9d, 0xca, 0x5c, 0xbc, 0x20, 0x70, 0x75, 0xc0 + }, + .len = 32 + }, + .iv = { + .data = { + 0x00, 0x00, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, + 0x05, 0x06, 0x07, 0x08 + }, + .len = 12 + }, + .aad = { + .data = chacha_aad_2, + .len = 12 + }, + .plaintext = { + .data = { + 0x49, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x65, 0x74, + 0x2d, 0x44, 0x72, 0x61, 0x66, 0x74, 0x73, 0x20, + 0x61, 0x72, 0x65, 0x20, 0x64, 0x72, 0x61, 0x66, + 0x74, 0x20, 0x64, 0x6f, 0x63, 0x75, 0x6d, 0x65, + 0x6e, 0x74, 0x73, 0x20, 0x76, 0x61, 0x6c, 0x69, + 0x64, 0x20, 0x66, 0x6f, 0x72, 0x20, 0x61, 0x20, + 0x6d, 0x61, 0x78, 0x69, 0x6d, 0x75, 0x6d, 0x20, + 0x6f, 0x66, 0x20, 0x73, 0x69, 0x78, 0x20, 0x6d, + 0x6f, 0x6e, 0x74, 0x68, 0x73, 0x20, 0x61, 0x6e, + 0x64, 0x20, 0x6d, 0x61, 0x79, 0x20, 0x62, 0x65, + 0x20, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x64, + 0x2c, 0x20, 0x72, 0x65, 0x70, 0x6c, 0x61, 0x63, + 0x65, 0x64, 0x2c, 0x20, 0x6f, 0x72, 0x20, 0x6f, + 0x62, 0x73, 0x6f, 0x6c, 0x65, 0x74, 0x65, 0x64, + 0x20, 0x62, 0x79, 0x20, 0x6f, 0x74, 0x68, 0x65, + 0x72, 0x20, 0x64, 0x6f, 0x63, 0x75, 0x6d, 0x65, + 0x6e, 0x74, 0x73, 0x20, 0x61, 0x74, 0x20, 0x61, + 0x6e, 0x79, 0x20, 0x74, 0x69, 0x6d, 0x65, 0x2e, + 0x20, 0x49, 0x74, 0x20, 0x69, 0x73, 0x20, 0x69, + 0x6e, 0x61, 0x70, 0x70, 0x72, 0x6f, 0x70, 0x72, + 0x69, 0x61, 0x74, 0x65, 0x20, 0x74, 0x6f, 0x20, + 0x75, 0x73, 0x65, 0x20, 0x49, 0x6e, 0x74, 0x65, + 0x72, 0x6e, 0x65, 0x74, 0x2d, 0x44, 0x72, 0x61, + 0x66, 0x74, 0x73, 0x20, 0x61, 0x73, 0x20, 0x72, + 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, + 0x20, 0x6d, 0x61, 0x74, 0x65, 0x72, 0x69, 0x61, + 0x6c, 0x20, 0x6f, 0x72, 0x20, 0x74, 0x6f, 0x20, + 0x63, 0x69, 0x74, 0x65, 0x20, 0x74, 0x68, 0x65, + 0x6d, 0x20, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x20, + 0x74, 0x68, 0x61, 0x6e, 0x20, 0x61, 0x73, 0x20, + 0x2f, 0xe2, 0x80, 0x9c, 0x77, 0x6f, 0x72, 0x6b, + 0x20, 0x69, 0x6e, 0x20, 0x70, 0x72, 0x6f, 0x67, + 0x72, 0x65, 0x73, 0x73, 0x2e, 0x2f, 0xe2, 0x80, + 0x9d + }, + .len = 265 + }, + .ciphertext = { + .data = { + 0x64, 0xa0, 0x86, 0x15, 0x75, 0x86, 0x1a, 0xf4, + 0x60, 0xf0, 0x62, 0xc7, 0x9b, 0xe6, 0x43, 0xbd, + 0x5e, 0x80, 0x5c, 0xfd, 0x34, 0x5c, 0xf3, 0x89, + 0xf1, 0x08, 0x67, 0x0a, 0xc7, 0x6c, 0x8c, 0xb2, + 0x4c, 0x6c, 0xfc, 0x18, 0x75, 0x5d, 0x43, 0xee, + 0xa0, 0x9e, 0xe9, 0x4e, 0x38, 0x2d, 0x26, 0xb0, + 0xbd, 0xb7, 0xb7, 0x3c, 0x32, 0x1b, 0x01, 0x00, + 0xd4, 0xf0, 0x3b, 0x7f, 0x35, 0x58, 0x94, 0xcf, + 0x33, 0x2f, 0x83, 0x0e, 0x71, 0x0b, 0x97, 0xce, + 0x98, 0xc8, 0xa8, 0x4a, 0xbd, 0x0b, 0x94, 0x81, + 0x14, 0xad, 0x17, 0x6e, 0x00, 0x8d, 0x33, 0xbd, + 0x60, 0xf9, 0x82, 0xb1, 0xff, 0x37, 0xc8, 0x55, + 0x97, 0x97, 0xa0, 0x6e, 0xf4, 0xf0, 0xef, 0x61, + 0xc1, 0x86, 0x32, 0x4e, 0x2b, 0x35, 0x06, 0x38, + 0x36, 0x06, 0x90, 0x7b, 0x6a, 0x7c, 0x02, 0xb0, + 0xf9, 0xf6, 0x15, 0x7b, 0x53, 0xc8, 0x67, 0xe4, + 0xb9, 0x16, 0x6c, 0x76, 0x7b, 0x80, 0x4d, 0x46, + 0xa5, 0x9b, 0x52, 0x16, 0xcd, 0xe7, 0xa4, 0xe9, + 0x90, 0x40, 0xc5, 0xa4, 0x04, 0x33, 0x22, 0x5e, + 0xe2, 0x82, 0xa1, 0xb0, 0xa0, 0x6c, 0x52, 0x3e, + 0xaf, 0x45, 0x34, 0xd7, 0xf8, 0x3f, 0xa1, 0x15, + 0x5b, 0x00, 0x47, 0x71, 0x8c, 0xbc, 0x54, 0x6a, + 0x0d, 0x07, 0x2b, 0x04, 0xb3, 0x56, 0x4e, 0xea, + 0x1b, 0x42, 0x22, 0x73, 0xf5, 0x48, 0x27, 0x1a, + 0x0b, 0xb2, 0x31, 0x60, 0x53, 0xfa, 0x76, 0x99, + 0x19, 0x55, 0xeb, 0xd6, 0x31, 0x59, 0x43, 0x4e, + 0xce, 0xbb, 0x4e, 0x46, 0x6d, 0xae, 0x5a, 0x10, + 0x73, 0xa6, 0x72, 0x76, 0x27, 0x09, 0x7a, 0x10, + 0x49, 0xe6, 0x17, 0xd9, 0x1d, 0x36, 0x10, 0x94, + 0xfa, 0x68, 0xf0, 0xff, 0x77, 0x98, 0x71, 0x30, + 0x30, 0x5b, 0xea, 0xba, 0x2e, 0xda, 0x04, 0xdf, + 0x99, 0x7b, 0x71, 0x4d, 0x6c, 0x6f, 0x2c, 0x29, + 0xa6, 0xad, 0x5c, 0xb4, 0x02, 0x2b, 0x02, 0x70, + 0x9b + }, + .len = 265 + }, + .auth_tag = { + .data = { + 0xee, 0xad, 0x9d, 0x67, 0x89, 0x0c, 0xbb, 0x22, + 0x39, 0x23, 0x36, 0xfe, 0xa1, 0x85, 0x1f, 0x38 + }, + .len = 16 + } +}; #endif /* TEST_CRYPTODEV_AEAD_TEST_VECTORS_H_ */ diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 0f216c7b4f..4990cf558e 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -111,6 +111,9 @@ New Features * ZUC PMD. * CHACHA20_POLY1305 PMD. + * An autotest was added for the CHACHA20_POLY1305 PMD, + including a new testcase for SGL OOP. + * **Updated the aesni_mb crypto PMD.** * Added support for ZUC-EEA3-256 and ZUC-EIA3-256.