From patchwork Wed Feb 28 14:00:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nayak, Nishikanta" X-Patchwork-Id: 137432 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 26CD943C2D; Wed, 28 Feb 2024 15:01:05 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 126F24335A; Wed, 28 Feb 2024 15:00:54 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by mails.dpdk.org (Postfix) with ESMTP id 8C38043285 for ; Wed, 28 Feb 2024 15:00:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709128850; x=1740664850; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=c1EbL8Sd0WSH8aLXdxrMEW9ocrYCTuAXxMl4UG1kJKo=; b=UeDaEXlfExsFgbTKuVfygcWAfuU6c9Sav8NonwaVnalSkTnT7+ycBnnO 6CZrhvzVbnY5ESqQ9E4SwFfiX+ZchgzKcw3NROzbiwch+ZboN/V8a7San 7y6UKDwkQ6Q+y+tG6H1HuUKitqGLRWcGonE2XssjNut2eWNxlt4XutlZA MAvtZ2CRrdoyVSsF5kb6JfQO9Dux/AJmgQg+dyMf+icnE+YVdzYP7RhxX RAvdEPXDAzbAUFnVuxnFLUN9LHcF/V63uUxYr48Nodq4gMVIow7NKEYed n0n0+8hJed/Ct18jjhAHV6uSovpdwEMvifUaVPx6a5GlZvaNoGHqvJAtw A==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="3374015" X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="3374015" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2024 06:00:44 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="12157148" Received: from silpixa00401797.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.113]) by orviesa005.jf.intel.com with ESMTP; 28 Feb 2024 06:00:40 -0800 From: Nishikant Nayak To: dev@dpdk.org Cc: ciara.power@intel.com, kai.ji@intel.com, arkadiuszx.kusztal@intel.com, rakesh.s.joshi@intel.com, Nishikant Nayak , Thomas Monjalon , Anatoly Burakov Subject: [PATCH v6 1/4] common/qat: add files specific to GEN LCE Date: Wed, 28 Feb 2024 14:00:33 +0000 Message-Id: <20240228140036.1996629-2-nishikanta.nayak@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240228140036.1996629-1-nishikanta.nayak@intel.com> References: <20231220132616.318983-1-nishikanta.nayak@intel.com> <20240228140036.1996629-1-nishikanta.nayak@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Adding GEN5 files for handling GEN LCE specific operations. These files are inherited from the existing files/APIs which has some changes specific GEN5 requirements Also updated the mailmap file. Signed-off-by: Nishikant Nayak Acked-by: Ciara Power --- v5: - Replaced usage of __u8 with uint8_t. v3: - Removed use of linux/kernel.h macro to fix ARM compilation. - Fixed typo in commit body and code comment. v2: - Renamed device from GEN 5 to GEN LCE. - Removed unused code. - Updated macro names. --- --- .mailmap | 1 + drivers/common/qat/dev/qat_dev_gen_lce.c | 306 ++++++++++++++++ drivers/common/qat/meson.build | 2 + .../adf_transport_access_macros_gen_lce.h | 51 +++ .../adf_transport_access_macros_gen_lcevf.h | 48 +++ drivers/common/qat/qat_adf/icp_qat_fw_la.h | 14 + drivers/common/qat/qat_common.h | 1 + .../crypto/qat/dev/qat_crypto_pmd_gen_lce.c | 329 ++++++++++++++++++ drivers/crypto/qat/qat_sym.h | 6 + 9 files changed, 758 insertions(+) create mode 100644 drivers/common/qat/dev/qat_dev_gen_lce.c create mode 100644 drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lce.h create mode 100644 drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lcevf.h create mode 100644 drivers/crypto/qat/dev/qat_crypto_pmd_gen_lce.c diff --git a/.mailmap b/.mailmap index 58cca13ef6..8008e5a899 100644 --- a/.mailmap +++ b/.mailmap @@ -1036,6 +1036,7 @@ Ning Li Nipun Gupta Nir Efrati Nirmoy Das +Nishikant Nayak Nithin Dabilpuram Nitin Saxena Nitzan Weller diff --git a/drivers/common/qat/dev/qat_dev_gen_lce.c b/drivers/common/qat/dev/qat_dev_gen_lce.c new file mode 100644 index 0000000000..5faaefc2d8 --- /dev/null +++ b/drivers/common/qat/dev/qat_dev_gen_lce.c @@ -0,0 +1,306 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2024 Intel Corporation + */ + +#include +#include + +#include "qat_device.h" +#include "qat_qp.h" +#include "adf_transport_access_macros_gen_lcevf.h" +#include "adf_pf2vf_msg.h" +#include "qat_pf2vf.h" + +#include +#include +#include + +#define BITS_PER_ULONG (sizeof(unsigned long) * 8) + +#define VFIO_PCI_LCE_DEVICE_CFG_REGION_INDEX VFIO_PCI_NUM_REGIONS +#define VFIO_PCI_LCE_CY_CFG_REGION_INDEX (VFIO_PCI_NUM_REGIONS + 2) +#define VFIO_PCI_LCE_RING_CFG_REGION_INDEX (VFIO_PCI_NUM_REGIONS + 4) +#define LCE_DEVICE_NAME_SIZE 64 +#define LCE_DEVICE_MAX_BANKS 2080 +#define LCE_DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d)) +#define LCE_DEVICE_BITMAP_SIZE \ + LCE_DIV_ROUND_UP(LCE_DEVICE_MAX_BANKS, BITS_PER_ULONG) + +/* QAT GEN_LCE specific macros */ +#define QAT_GEN_LCE_BUNDLE_NUM LCE_DEVICE_MAX_BANKS +#define QAT_GEN4_QPS_PER_BUNDLE_NUM 1 + +/** + * struct lce_vfio_dev_cap - LCE device capabilities + * + * Device level capabilities and service level capabilities + */ +struct lce_vfio_dev_cap { + uint16_t device_num; + uint16_t device_type; + uint32_t capability_mask; + uint32_t extended_capabilities; + uint16_t max_banks; + uint16_t max_rings_per_bank; + uint16_t arb_mask; + uint16_t services; + uint16_t pkg_id; + uint16_t node_id; + uint8_t device_name[LCE_DEVICE_NAME_SIZE]; +}; + +/* struct lce_vfio_dev_cy_cap - CY capabilities of LCE device */ +struct lce_vfio_dev_cy_cap { + uint32_t nr_banks; + unsigned long bitmap[LCE_DEVICE_BITMAP_SIZE]; +}; + +struct lce_qat_domain { + uint32_t nid :3; + uint32_t fid :7; + uint32_t ftype :2; + uint32_t vfid :13; + uint32_t rid :4; + uint32_t vld :1; + uint32_t desc_over :1; + uint32_t pasid_vld :1; + uint32_t pasid :20; +}; + +struct lce_qat_buf_domain { + uint32_t bank_id: 20; + uint32_t type: 4; + uint32_t resv: 8; + struct lce_qat_domain dom; +}; + +struct qat_dev_gen_lce_extra { + struct qat_qp_hw_data + qp_gen_lce_data[QAT_GEN_LCE_BUNDLE_NUM][QAT_GEN4_QPS_PER_BUNDLE_NUM]; +}; + +static struct qat_pf2vf_dev qat_pf2vf_gen_lce = { + .pf2vf_offset = ADF_4XXXIOV_PF2VM_OFFSET, + .vf2pf_offset = ADF_4XXXIOV_VM2PF_OFFSET, + .pf2vf_type_shift = ADF_PFVF_2X_MSGTYPE_SHIFT, + .pf2vf_type_mask = ADF_PFVF_2X_MSGTYPE_MASK, + .pf2vf_data_shift = ADF_PFVF_2X_MSGDATA_SHIFT, + .pf2vf_data_mask = ADF_PFVF_2X_MSGDATA_MASK, +}; + +static int +qat_select_valid_queue_gen_lce(struct qat_pci_device *qat_dev, int qp_id, + enum qat_service_type service_type) +{ + int i = 0, valid_qps = 0; + struct qat_dev_gen_lce_extra *dev_extra = qat_dev->dev_private; + + for (; i < QAT_GEN_LCE_BUNDLE_NUM; i++) { + if (dev_extra->qp_gen_lce_data[i][0].service_type == + service_type) { + if (valid_qps == qp_id) + return i; + ++valid_qps; + } + } + return -1; +} + +static const struct qat_qp_hw_data * +qat_qp_get_hw_data_gen_lce(struct qat_pci_device *qat_dev, + enum qat_service_type service_type, uint16_t qp_id) +{ + struct qat_dev_gen_lce_extra *dev_extra = qat_dev->dev_private; + int ring_pair = qat_select_valid_queue_gen_lce(qat_dev, qp_id, + service_type); + + if (ring_pair < 0) + return NULL; + + return &dev_extra->qp_gen_lce_data[ring_pair][0]; +} + +static int +qat_qp_rings_per_service_gen_lce(struct qat_pci_device *qat_dev, + enum qat_service_type service) +{ + int i = 0, count = 0, max_ops_per_srv = 0; + struct qat_dev_gen_lce_extra *dev_extra = qat_dev->dev_private; + + max_ops_per_srv = QAT_GEN_LCE_BUNDLE_NUM; + for (i = 0, count = 0; i < max_ops_per_srv; i++) + if (dev_extra->qp_gen_lce_data[i][0].service_type == service) + count++; + return count; +} + +static int qat_dev_read_config_gen_lce(struct qat_pci_device *qat_dev) +{ + struct qat_dev_gen_lce_extra *dev_extra = qat_dev->dev_private; + struct qat_qp_hw_data *hw_data; + + /** Enable only crypto ring: RP-0 */ + hw_data = &dev_extra->qp_gen_lce_data[0][0]; + memset(hw_data, 0, sizeof(*hw_data)); + + hw_data->service_type = QAT_SERVICE_SYMMETRIC; + hw_data->tx_msg_size = 128; + hw_data->rx_msg_size = 32; + + hw_data->tx_ring_num = 0; + hw_data->rx_ring_num = 1; + + hw_data->hw_bundle_num = 0; + + return 0; +} + +static void qat_qp_build_ring_base_gen_lce(void *io_addr, struct qat_queue *queue) +{ + uint64_t queue_base; + + queue_base = BUILD_RING_BASE_ADDR_GEN_LCE(queue->base_phys_addr, + queue->queue_size); + WRITE_CSR_RING_BASE_GEN_LCEVF(io_addr, queue->hw_bundle_number, + queue->hw_queue_number, queue_base); +} + +static void +qat_qp_adf_arb_enable_gen_lce(const struct qat_queue *txq, + void *base_addr, rte_spinlock_t *lock) +{ + uint32_t arb_csr_offset = 0, value; + + rte_spinlock_lock(lock); + arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET + + (ADF_RING_BUNDLE_SIZE_GEN_LCE * + txq->hw_bundle_number); + value = ADF_CSR_RD(base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, + arb_csr_offset); + value |= 0x01; + ADF_CSR_WR(base_addr, arb_csr_offset, value); + rte_spinlock_unlock(lock); +} + +static void +qat_qp_adf_arb_disable_gen_lce(const struct qat_queue *txq, + void *base_addr, rte_spinlock_t *lock) +{ + uint32_t arb_csr_offset = 0, value; + + rte_spinlock_lock(lock); + arb_csr_offset = ADF_ARB_RINGSRVARBEN_OFFSET + (ADF_RING_BUNDLE_SIZE_GEN_LCE * + txq->hw_bundle_number); + value = ADF_CSR_RD(base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, + arb_csr_offset); + value &= ~(0x01); + ADF_CSR_WR(base_addr, arb_csr_offset, value); + rte_spinlock_unlock(lock); +} + +static void +qat_qp_adf_configure_queues_gen_lce(struct qat_qp *qp) +{ + uint32_t q_tx_config, q_resp_config; + struct qat_queue *q_tx = &qp->tx_q, *q_rx = &qp->rx_q; + + /* q_tx/rx->queue_size is initialized as per bundle config register */ + q_tx_config = BUILD_RING_CONFIG(q_tx->queue_size); + + q_resp_config = BUILD_RESP_RING_CONFIG(q_rx->queue_size, + ADF_RING_NEAR_WATERMARK_512, + ADF_RING_NEAR_WATERMARK_0); + + WRITE_CSR_RING_CONFIG_GEN_LCEVF(qp->mmap_bar_addr, q_tx->hw_bundle_number, + q_tx->hw_queue_number, q_tx_config); + WRITE_CSR_RING_CONFIG_GEN_LCEVF(qp->mmap_bar_addr, q_rx->hw_bundle_number, + q_rx->hw_queue_number, q_resp_config); +} + +static void +qat_qp_csr_write_tail_gen_lce(struct qat_qp *qp, struct qat_queue *q) +{ + WRITE_CSR_RING_TAIL_GEN_LCEVF(qp->mmap_bar_addr, q->hw_bundle_number, + q->hw_queue_number, q->tail); +} + +static void +qat_qp_csr_write_head_gen_lce(struct qat_qp *qp, struct qat_queue *q, + uint32_t new_head) +{ + WRITE_CSR_RING_HEAD_GEN_LCEVF(qp->mmap_bar_addr, q->hw_bundle_number, + q->hw_queue_number, new_head); +} + +static void +qat_qp_csr_setup_gen_lce(struct qat_pci_device *qat_dev, void *io_addr, + struct qat_qp *qp) +{ + qat_qp_build_ring_base_gen_lce(io_addr, &qp->tx_q); + qat_qp_build_ring_base_gen_lce(io_addr, &qp->rx_q); + qat_qp_adf_configure_queues_gen_lce(qp); + qat_qp_adf_arb_enable_gen_lce(&qp->tx_q, qp->mmap_bar_addr, + &qat_dev->arb_csr_lock); +} + +static struct qat_qp_hw_spec_funcs qat_qp_hw_spec_gen_lce = { + .qat_qp_rings_per_service = qat_qp_rings_per_service_gen_lce, + .qat_qp_build_ring_base = qat_qp_build_ring_base_gen_lce, + .qat_qp_adf_arb_enable = qat_qp_adf_arb_enable_gen_lce, + .qat_qp_adf_arb_disable = qat_qp_adf_arb_disable_gen_lce, + .qat_qp_adf_configure_queues = qat_qp_adf_configure_queues_gen_lce, + .qat_qp_csr_write_tail = qat_qp_csr_write_tail_gen_lce, + .qat_qp_csr_write_head = qat_qp_csr_write_head_gen_lce, + .qat_qp_csr_setup = qat_qp_csr_setup_gen_lce, + .qat_qp_get_hw_data = qat_qp_get_hw_data_gen_lce, +}; + +static int +qat_reset_ring_pairs_gen_lce(struct qat_pci_device *qat_pci_dev __rte_unused) +{ + return 0; +} + +static const struct rte_mem_resource* +qat_dev_get_transport_bar_gen_lce(struct rte_pci_device *pci_dev) +{ + return &pci_dev->mem_resource[0]; +} + +static int +qat_dev_get_misc_bar_gen_lce(struct rte_mem_resource **mem_resource, + struct rte_pci_device *pci_dev) +{ + *mem_resource = &pci_dev->mem_resource[2]; + return 0; +} + +static int +qat_dev_get_extra_size_gen_lce(void) +{ + return sizeof(struct qat_dev_gen_lce_extra); +} + +static int +qat_dev_get_slice_map_gen_lce(uint32_t *map __rte_unused, + const struct rte_pci_device *pci_dev __rte_unused) +{ + return 0; +} + +static struct qat_dev_hw_spec_funcs qat_dev_hw_spec_gen_lce = { + .qat_dev_reset_ring_pairs = qat_reset_ring_pairs_gen_lce, + .qat_dev_get_transport_bar = qat_dev_get_transport_bar_gen_lce, + .qat_dev_get_misc_bar = qat_dev_get_misc_bar_gen_lce, + .qat_dev_read_config = qat_dev_read_config_gen_lce, + .qat_dev_get_extra_size = qat_dev_get_extra_size_gen_lce, + .qat_dev_get_slice_map = qat_dev_get_slice_map_gen_lce, +}; + +RTE_INIT(qat_dev_gen_lce_init) +{ + qat_qp_hw_spec[QAT_GEN_LCE] = &qat_qp_hw_spec_gen_lce; + qat_dev_hw_spec[QAT_GEN_LCE] = &qat_dev_hw_spec_gen_lce; + qat_gen_config[QAT_GEN_LCE].dev_gen = QAT_GEN_LCE; + qat_gen_config[QAT_GEN_LCE].pf2vf_dev = &qat_pf2vf_gen_lce; +} diff --git a/drivers/common/qat/meson.build b/drivers/common/qat/meson.build index 62abcb6fe3..bc7c3e5b85 100644 --- a/drivers/common/qat/meson.build +++ b/drivers/common/qat/meson.build @@ -82,6 +82,7 @@ sources += files( 'dev/qat_dev_gen2.c', 'dev/qat_dev_gen3.c', 'dev/qat_dev_gen4.c', + 'dev/qat_dev_gen_lce.c', ) includes += include_directories( 'qat_adf', @@ -108,6 +109,7 @@ if qat_crypto 'dev/qat_crypto_pmd_gen2.c', 'dev/qat_crypto_pmd_gen3.c', 'dev/qat_crypto_pmd_gen4.c', + 'dev/qat_crypto_pmd_gen_lce.c', ] sources += files(join_paths(qat_crypto_relpath, f)) endforeach diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lce.h b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lce.h new file mode 100644 index 0000000000..c9df8f5dd2 --- /dev/null +++ b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lce.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0) + * Copyright(c) 2021 Intel Corporation + */ + +#ifndef ADF_TRANSPORT_ACCESS_MACROS_GEN_LCE_H +#define ADF_TRANSPORT_ACCESS_MACROS_GEN_LCE_H + +#include "adf_transport_access_macros.h" + +#define ADF_RINGS_PER_INT_SRCSEL_GEN4 2 +#define ADF_BANK_INT_SRC_SEL_MASK_GEN4 0x44UL +#define ADF_BANK_INT_FLAG_CLEAR_MASK_GEN4 0x3 +#define ADF_RING_BUNDLE_SIZE_GEN_LCE 0x2000 +#define ADF_RING_CSR_RING_CONFIG_GEN_LCE 0x1000 +#define ADF_RING_CSR_RING_LBASE_GEN_LCE 0x1040 +#define ADF_RING_CSR_RING_UBASE_GEN_LCE 0x1080 + +#define BUILD_RING_BASE_ADDR_GEN_LCE(addr, size) \ + ((((addr) >> 6) & (0xFFFFFFFFFFFFFFFFULL << (size))) << 6) + +#define WRITE_CSR_RING_BASE_GEN_LCE(csr_base_addr, bank, ring, value) \ +do { \ + uint32_t l_base = 0, u_base = 0; \ + l_base = (uint32_t)(value & 0xFFFFFFFF); \ + u_base = (uint32_t)((value & 0xFFFFFFFF00000000ULL) >> 32); \ + ADF_CSR_WR(csr_base_addr, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * bank) + \ + ADF_RING_CSR_RING_LBASE_GEN_LCE + (ring << 2), \ + l_base); \ + ADF_CSR_WR(csr_base_addr, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * bank) + \ + ADF_RING_CSR_RING_UBASE_GEN_LCE + (ring << 2), \ + u_base); \ +} while (0) + +#define WRITE_CSR_RING_CONFIG_GEN_LCE(csr_base_addr, bank, ring, value) \ + ADF_CSR_WR(csr_base_addr, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * bank) + \ + ADF_RING_CSR_RING_CONFIG_GEN_LCE + (ring << 2), value) + +#define WRITE_CSR_RING_TAIL_GEN_LCE(csr_base_addr, bank, ring, value) \ + ADF_CSR_WR((u8 *)(csr_base_addr), \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * (bank)) + \ + ADF_RING_CSR_RING_TAIL + ((ring) << 2), value) + +#define WRITE_CSR_RING_HEAD_GEN_LCE(csr_base_addr, bank, ring, value) \ + ADF_CSR_WR((u8 *)(csr_base_addr), \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * (bank)) + \ + ADF_RING_CSR_RING_HEAD + ((ring) << 2), value) + +#endif diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lcevf.h b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lcevf.h new file mode 100644 index 0000000000..3c7232de12 --- /dev/null +++ b/drivers/common/qat/qat_adf/adf_transport_access_macros_gen_lcevf.h @@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: (BSD-3-Clause OR GPL-2.0) + * Copyright(c) 2024 Intel Corporation + */ + +#ifndef ADF_TRANSPORT_ACCESS_MACROS_GEN_LCEVF_H +#define ADF_TRANSPORT_ACCESS_MACROS_GEN_LCEVF_H + +#include "adf_transport_access_macros.h" +#include "adf_transport_access_macros_gen_lce.h" + +#define ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF 0x0 + +#define WRITE_CSR_RING_BASE_GEN_LCEVF(csr_base_addr, bank, ring, value) \ +do { \ + uint32_t l_base = 0, u_base = 0; \ + l_base = (uint32_t)(value & 0xFFFFFFFF); \ + u_base = (uint32_t)((value & 0xFFFFFFFF00000000ULL) >> 32); \ + ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * bank) + \ + ADF_RING_CSR_RING_LBASE_GEN_LCE + (ring << 2), \ + l_base); \ + ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * bank) + \ + ADF_RING_CSR_RING_UBASE_GEN_LCE + (ring << 2), \ + u_base); \ +} while (0) + +#define WRITE_CSR_RING_CONFIG_GEN_LCEVF(csr_base_addr, bank, ring, value) \ + ADF_CSR_WR(csr_base_addr + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * bank) + \ + ADF_RING_CSR_RING_CONFIG_GEN_LCE + (ring << 2), value) + +#define WRITE_CSR_RING_TAIL_GEN_LCEVF(csr_base_addr, bank, ring, value) \ + ADF_CSR_WR((csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * (bank)) + \ + ADF_RING_CSR_RING_TAIL + ((ring) << 2), (value)) + +#define WRITE_CSR_RING_HEAD_GEN_LCEVF(csr_base_addr, bank, ring, value) \ + ADF_CSR_WR((csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * (bank)) + \ + ADF_RING_CSR_RING_HEAD + ((ring) << 2), (value)) + +#define WRITE_CSR_RING_SRV_ARB_EN_GEN_LCEVF(csr_base_addr, bank, value) \ + ADF_CSR_WR((csr_base_addr) + ADF_RING_CSR_ADDR_OFFSET_GEN_LCEVF, \ + (ADF_RING_BUNDLE_SIZE_GEN_LCE * (bank)) + \ + ADF_RING_CSR_RING_SRV_ARB_EN, (value)) + +#endif diff --git a/drivers/common/qat/qat_adf/icp_qat_fw_la.h b/drivers/common/qat/qat_adf/icp_qat_fw_la.h index 70f0effa62..215b291b74 100644 --- a/drivers/common/qat/qat_adf/icp_qat_fw_la.h +++ b/drivers/common/qat/qat_adf/icp_qat_fw_la.h @@ -410,4 +410,18 @@ struct icp_qat_fw_la_cipher_20_req_params { uint8_t spc_auth_res_sz; }; +struct icp_qat_fw_la_cipher_30_req_params { + uint32_t spc_aad_sz; + uint8_t cipher_length; + uint8_t reserved[2]; + uint8_t spc_auth_res_sz; + union { + uint32_t cipher_IV_array[ICP_QAT_FW_NUM_LONGWORDS_4]; + struct { + uint64_t cipher_IV_ptr; + uint64_t resrvd1; + } s; + + } u; +}; #endif diff --git a/drivers/common/qat/qat_common.h b/drivers/common/qat/qat_common.h index 9411a79301..642e009f28 100644 --- a/drivers/common/qat/qat_common.h +++ b/drivers/common/qat/qat_common.h @@ -21,6 +21,7 @@ enum qat_device_gen { QAT_GEN2, QAT_GEN3, QAT_GEN4, + QAT_GEN_LCE, QAT_N_GENS }; diff --git a/drivers/crypto/qat/dev/qat_crypto_pmd_gen_lce.c b/drivers/crypto/qat/dev/qat_crypto_pmd_gen_lce.c new file mode 100644 index 0000000000..3f1668b3d3 --- /dev/null +++ b/drivers/crypto/qat/dev/qat_crypto_pmd_gen_lce.c @@ -0,0 +1,329 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2024 Intel Corporation + */ + +#include +#include +#include "qat_sym_session.h" +#include "qat_sym.h" +#include "qat_asym.h" +#include "qat_crypto.h" +#include "qat_crypto_pmd_gens.h" + +static struct rte_cryptodev_capabilities qat_sym_crypto_caps_gen_lce[] = { + QAT_SYM_AEAD_CAP(AES_GCM, + CAP_SET(block_size, 16), + CAP_RNG(key_size, 32, 32, 0), CAP_RNG(digest_size, 16, 16, 0), + CAP_RNG(aad_size, 0, 240, 1), CAP_RNG(iv_size, 12, 12, 0)), + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +static int +qat_sgl_add_buffer_gen_lce(void *list_in, uint64_t addr, uint32_t len) +{ + struct qat_sgl *list = (struct qat_sgl *)list_in; + uint32_t nr; + + nr = list->num_bufs; + + if (nr >= QAT_SYM_SGL_MAX_NUMBER) { + QAT_DP_LOG(ERR, "Adding %d entry failed, no empty SGL buffer", nr); + return -EINVAL; + } + + list->buffers[nr].len = len; + list->buffers[nr].resrvd = 0; + list->buffers[nr].addr = addr; + + list->num_bufs++; +#if RTE_LOG_DP_LEVEL >= RTE_LOG_DEBUG + QAT_DP_LOG(INFO, "SGL with %d buffers:", list->num_bufs); + QAT_DP_LOG(INFO, "QAT SGL buf %d, len = %d, iova = 0x%012"PRIx64, + nr, list->buffers[nr].len, list->buffers[nr].addr); +#endif + return 0; +} + +static int +qat_sgl_fill_array_with_mbuf(struct rte_mbuf *buf, int64_t offset, + void *list_in, uint32_t data_len) +{ + struct qat_sgl *list = (struct qat_sgl *)list_in; + uint32_t nr, buf_len; + int res = -EINVAL; +#if RTE_LOG_DP_LEVEL >= RTE_LOG_DEBUG + uint32_t start_idx; + start_idx = list->num_bufs; +#endif + + /* Append to the existing list */ + nr = list->num_bufs; + + for (buf_len = 0; buf && nr < QAT_SYM_SGL_MAX_NUMBER; buf = buf->next) { + if (offset >= rte_pktmbuf_data_len(buf)) { + offset -= rte_pktmbuf_data_len(buf); + /* Jump to next mbuf */ + continue; + } + + list->buffers[nr].len = rte_pktmbuf_data_len(buf) - offset; + list->buffers[nr].resrvd = 0; + list->buffers[nr].addr = rte_pktmbuf_iova_offset(buf, offset); + + offset = 0; + buf_len += list->buffers[nr].len; + + if (buf_len >= data_len) { + list->buffers[nr].len -= buf_len - data_len; + res = 0; + break; + } + ++nr; + } + + if (unlikely(res != 0)) { + if (nr == QAT_SYM_SGL_MAX_NUMBER) + QAT_DP_LOG(ERR, "Exceeded max segments in QAT SGL (%u)", + QAT_SYM_SGL_MAX_NUMBER); + else + QAT_DP_LOG(ERR, "Mbuf chain is too short"); + } else { + + list->num_bufs = ++nr; +#if RTE_LOG_DP_LEVEL >= RTE_LOG_DEBUG + QAT_DP_LOG(INFO, "SGL with %d buffers:", list->num_bufs); + for (nr = start_idx; nr < list->num_bufs; nr++) { + QAT_DP_LOG(INFO, "QAT SGL buf %d, len = %d, iova = 0x%012"PRIx64, + nr, list->buffers[nr].len, + list->buffers[nr].addr); + } +#endif + } + + return res; +} + +static int +qat_sym_build_op_aead_gen_lce(void *in_op, struct qat_sym_session *ctx, + uint8_t *out_msg, void *op_cookie) +{ + struct qat_sym_op_cookie *cookie = op_cookie; + struct rte_crypto_op *op = in_op; + uint64_t digest_phys_addr, aad_phys_addr; + uint16_t iv_len, aad_len, digest_len, key_len; + uint32_t cipher_ofs, iv_offset, cipher_len; + register struct icp_qat_fw_la_bulk_req *qat_req; + struct icp_qat_fw_la_cipher_30_req_params *cipher_param; + enum icp_qat_hw_cipher_dir dir; + bool is_digest_adjacent = false; + + if (ctx->qat_cmd != ICP_QAT_FW_LA_CMD_CIPHER || + ctx->qat_cipher_alg != ICP_QAT_HW_CIPHER_ALGO_AES256 || + ctx->qat_mode != ICP_QAT_HW_CIPHER_AEAD_MODE) { + + QAT_DP_LOG(ERR, "Not supported (cmd: %d, alg: %d, mode: %d). " + "GEN_LCE PMD only supports AES-256 AEAD mode", + ctx->qat_cmd, ctx->qat_cipher_alg, ctx->qat_mode); + return -EINVAL; + } + + qat_req = (struct icp_qat_fw_la_bulk_req *)out_msg; + rte_mov128((uint8_t *)qat_req, (const uint8_t *)&(ctx->fw_req)); + qat_req->comn_mid.opaque_data = (uint64_t)(uintptr_t)op; + cipher_param = (void *)&qat_req->serv_specif_rqpars; + + dir = ctx->qat_dir; + + aad_phys_addr = op->sym->aead.aad.phys_addr; + aad_len = ctx->aad_len; + + iv_offset = ctx->cipher_iv.offset; + iv_len = ctx->cipher_iv.length; + + cipher_ofs = op->sym->aead.data.offset; + cipher_len = op->sym->aead.data.length; + + digest_phys_addr = op->sym->aead.digest.phys_addr; + digest_len = ctx->digest_length; + + /* Up to 16B IV can be directly embedded in descriptor. + * GCM supports only 12B IV for GEN LCE + */ + if (iv_len != GCM_IV_LENGTH_GEN_LCE) { + QAT_DP_LOG(ERR, "iv_len: %d not supported. Must be 12B.", + iv_len); + return -EINVAL; + } + + rte_memcpy(cipher_param->u.cipher_IV_array, + rte_crypto_op_ctod_offset(op, uint8_t*, iv_offset), + iv_len); + + /* Always SGL */ + RTE_ASSERT((qat_req->comn_hdr.comn_req_flags & + ICP_QAT_FW_SYM_COMM_ADDR_SGL) == 1); + /* Always inplace */ + RTE_ASSERT(op->sym->m_dst == NULL); + + /* Key buffer address is already programmed by reusing the + * content-descriptor buffer + */ + key_len = ctx->auth_key_length; + + cipher_param->spc_aad_sz = aad_len; + cipher_param->cipher_length = key_len; + cipher_param->spc_auth_res_sz = digest_len; + + /* Knowing digest is contiguous to cipher-text helps optimizing SGL */ + if (rte_pktmbuf_iova_offset(op->sym->m_src, cipher_ofs + cipher_len) + == digest_phys_addr) + is_digest_adjacent = true; + + /* SRC-SGL: 3 entries: + * a) AAD + * b) cipher + * c) digest (only for decrypt and buffer is_NOT_adjacent) + * + */ + cookie->qat_sgl_src.num_bufs = 0; + if (aad_len) + qat_sgl_add_buffer_gen_lce(&cookie->qat_sgl_src, aad_phys_addr, + aad_len); + + if (is_digest_adjacent && dir == ICP_QAT_HW_CIPHER_DECRYPT) { + qat_sgl_fill_array_with_mbuf(op->sym->m_src, cipher_ofs, + &cookie->qat_sgl_src, + cipher_len + digest_len); + } else { + qat_sgl_fill_array_with_mbuf(op->sym->m_src, cipher_ofs, + &cookie->qat_sgl_src, + cipher_len); + + /* Digest buffer in decrypt job */ + if (dir == ICP_QAT_HW_CIPHER_DECRYPT) + qat_sgl_add_buffer_gen_lce(&cookie->qat_sgl_src, + digest_phys_addr, digest_len); + } + + /* (in-place) DST-SGL: 2 entries: + * a) cipher + * b) digest (only for encrypt and buffer is_NOT_adjacent) + */ + cookie->qat_sgl_dst.num_bufs = 0; + + if (is_digest_adjacent && dir == ICP_QAT_HW_CIPHER_ENCRYPT) { + qat_sgl_fill_array_with_mbuf(op->sym->m_src, cipher_ofs, + &cookie->qat_sgl_dst, + cipher_len + digest_len); + } else { + qat_sgl_fill_array_with_mbuf(op->sym->m_src, cipher_ofs, + &cookie->qat_sgl_dst, + cipher_len); + + /* Digest buffer in Encrypt job */ + if (dir == ICP_QAT_HW_CIPHER_ENCRYPT) + qat_sgl_add_buffer_gen_lce(&cookie->qat_sgl_dst, + digest_phys_addr, digest_len); + } + + /* Length values in 128B descriptor */ + qat_req->comn_mid.src_length = cipher_len; + qat_req->comn_mid.dst_length = cipher_len; + + if (dir == ICP_QAT_HW_CIPHER_ENCRYPT) /* Digest buffer in Encrypt job */ + qat_req->comn_mid.dst_length += GCM_256_DIGEST_LEN; + + /* src & dst SGL addresses in 128B descriptor */ + qat_req->comn_mid.src_data_addr = cookie->qat_sgl_src_phys_addr; + qat_req->comn_mid.dest_data_addr = cookie->qat_sgl_dst_phys_addr; + +#if RTE_LOG_DP_LEVEL >= RTE_LOG_DEBUG + QAT_DP_HEXDUMP_LOG(DEBUG, "qat_req:", qat_req, + sizeof(struct icp_qat_fw_la_bulk_req)); + QAT_DP_HEXDUMP_LOG(DEBUG, "src_data:", + rte_pktmbuf_mtod(op->sym->m_src, uint8_t*), + rte_pktmbuf_data_len(op->sym->m_src)); + QAT_DP_HEXDUMP_LOG(DEBUG, "digest:", op->sym->aead.digest.data, + digest_len); + QAT_DP_HEXDUMP_LOG(DEBUG, "aad:", op->sym->aead.aad.data, aad_len); +#endif + return 0; +} + +static int +qat_sym_crypto_set_session_gen_lce(void *cdev __rte_unused, void *session) +{ + struct qat_sym_session *ctx = session; + qat_sym_build_request_t build_request = NULL; + enum rte_proc_type_t proc_type = rte_eal_process_type(); + + if (proc_type == RTE_PROC_AUTO || proc_type == RTE_PROC_INVALID) + return -EINVAL; + + /* build request for aead */ + if (ctx->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_AES256 && + ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_128) { + build_request = qat_sym_build_op_aead_gen_lce; + ctx->build_request[proc_type] = build_request; + } + return 0; +} + + +static int +qat_sym_crypto_cap_get_gen_lce(struct qat_cryptodev_private *internals, + const char *capa_memz_name, + const uint16_t __rte_unused slice_map) +{ + const uint32_t size = sizeof(qat_sym_crypto_caps_gen_lce); + uint32_t i; + + internals->capa_mz = rte_memzone_lookup(capa_memz_name); + if (internals->capa_mz == NULL) { + internals->capa_mz = rte_memzone_reserve(capa_memz_name, + size, rte_socket_id(), 0); + if (internals->capa_mz == NULL) { + QAT_LOG(DEBUG, + "Error allocating memzone for capabilities"); + return -1; + } + } + + struct rte_cryptodev_capabilities *addr = + (struct rte_cryptodev_capabilities *) + internals->capa_mz->addr; + const struct rte_cryptodev_capabilities *capabilities = + qat_sym_crypto_caps_gen_lce; + const uint32_t capa_num = + size / sizeof(struct rte_cryptodev_capabilities); + uint32_t curr_capa = 0; + + for (i = 0; i < capa_num; i++) { + memcpy(addr + curr_capa, capabilities + i, + sizeof(struct rte_cryptodev_capabilities)); + curr_capa++; + } + internals->qat_dev_capabilities = internals->capa_mz->addr; + + return 0; +} + +RTE_INIT(qat_sym_crypto_gen_lce_init) +{ + qat_sym_gen_dev_ops[QAT_GEN_LCE].cryptodev_ops = &qat_sym_crypto_ops_gen1; + qat_sym_gen_dev_ops[QAT_GEN_LCE].get_capabilities = + qat_sym_crypto_cap_get_gen_lce; + qat_sym_gen_dev_ops[QAT_GEN_LCE].set_session = + qat_sym_crypto_set_session_gen_lce; + qat_sym_gen_dev_ops[QAT_GEN_LCE].set_raw_dp_ctx = NULL; + qat_sym_gen_dev_ops[QAT_GEN_LCE].get_feature_flags = + qat_sym_crypto_feature_flags_get_gen1; +} + +RTE_INIT(qat_asym_crypto_gen_lce_init) +{ + qat_asym_gen_dev_ops[QAT_GEN_LCE].cryptodev_ops = NULL; + qat_asym_gen_dev_ops[QAT_GEN_LCE].get_capabilities = NULL; + qat_asym_gen_dev_ops[QAT_GEN_LCE].get_feature_flags = NULL; + qat_asym_gen_dev_ops[QAT_GEN_LCE].set_session = NULL; +} diff --git a/drivers/crypto/qat/qat_sym.h b/drivers/crypto/qat/qat_sym.h index f976009bf2..f2f197d050 100644 --- a/drivers/crypto/qat/qat_sym.h +++ b/drivers/crypto/qat/qat_sym.h @@ -95,6 +95,12 @@ /* Maximum data length for single pass GMAC: 2^14-1 */ #define QAT_AES_GMAC_SPC_MAX_SIZE 16383 +/* Digest length for GCM Algo is 16 bytes */ +#define GCM_256_DIGEST_LEN 16 + +/* IV length for GCM algo is 12 bytes */ +#define GCM_IV_LENGTH_GEN_LCE 12 + struct qat_sym_session; struct qat_sym_sgl { From patchwork Wed Feb 28 14:00:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nayak, Nishikanta" X-Patchwork-Id: 137431 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3B45443C2D; Wed, 28 Feb 2024 15:00:56 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E99AC43353; Wed, 28 Feb 2024 15:00:52 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by mails.dpdk.org (Postfix) with ESMTP id DB16B4328B for ; Wed, 28 Feb 2024 15:00:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709128850; x=1740664850; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0H3hgIUBKzbVxvC6b24oPQySdSxb+NGK5ZmnInebBps=; b=BeHrGqYdUmGyGPOQ9pJZn96mIjwUYHQHOXRq1OuoAPZDbQnMiTgQjHfJ SZC4cTEdbOS0Py34/iJGRjyAejIv+eO/MoNSQbPK+jo8d4rkBSuS2uO8Y N4noi7WUjf3P8jFTzbs0vlalIKFdSd9IrJQ+sMvFG/7I3rqHU6LJmUYhB rCijbBizkc7VFrCLpjzbxufCS5w9uAUXvOvMeRXmKoUDawWwPHxTUfSnr 0Okfow2210R9qQY6cgBk6tSGqyJsL1/HcEsI8nn7U3hyXwz2ZMEWiGFIb JXEkqYbURvzEi1k9hX1Xb3P5PGxLYCEZuhLsYgKV7kw+kAtQV7LuOhqn6 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="3374025" X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="3374025" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2024 06:00:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="12157166" Received: from silpixa00401797.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.113]) by orviesa005.jf.intel.com with ESMTP; 28 Feb 2024 06:00:43 -0800 From: Nishikant Nayak To: dev@dpdk.org Cc: ciara.power@intel.com, kai.ji@intel.com, arkadiuszx.kusztal@intel.com, rakesh.s.joshi@intel.com, Nishikant Nayak Subject: [PATCH v6 2/4] common/qat: update common driver to support GEN LCE Date: Wed, 28 Feb 2024 14:00:34 +0000 Message-Id: <20240228140036.1996629-3-nishikanta.nayak@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240228140036.1996629-1-nishikanta.nayak@intel.com> References: <20231220132616.318983-1-nishikanta.nayak@intel.com> <20240228140036.1996629-1-nishikanta.nayak@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Adding GEN LCE specific macros which is required for updating the support for GEN LCE features. Also this patch adds other macros which is being used by GEN LCE Specific APIs. Signed-off-by: Nishikant Nayak Acked-by: Ciara Power --- v6: - Removed unused PCI device IDs from the device list. - Updated documentation and release note. v2: - Renamed device from GEN 5 to GEN LCE. - Removed unused code. - Updated macro names. - Fixed code formatting --- --- doc/guides/cryptodevs/qat.rst | 1 + doc/guides/rel_notes/release_24_03.rst | 4 ++ .../qat/qat_adf/adf_transport_access_macros.h | 1 + drivers/common/qat/qat_adf/icp_qat_fw.h | 34 ++++++++++++++ drivers/common/qat/qat_adf/icp_qat_fw_la.h | 45 ++++++++++++++++++- drivers/common/qat/qat_device.c | 5 +++ 6 files changed, 89 insertions(+), 1 deletion(-) diff --git a/doc/guides/cryptodevs/qat.rst b/doc/guides/cryptodevs/qat.rst index dc6b95165d..d9adbfc71e 100644 --- a/doc/guides/cryptodevs/qat.rst +++ b/doc/guides/cryptodevs/qat.rst @@ -26,6 +26,7 @@ poll mode crypto driver support for the following hardware accelerator devices: * ``Intel QuickAssist Technology D15xx`` * ``Intel QuickAssist Technology C4xxx`` * ``Intel QuickAssist Technology 4xxx`` +* ``Intel QuickAssist Technology apfxx`` Features diff --git a/doc/guides/rel_notes/release_24_03.rst b/doc/guides/rel_notes/release_24_03.rst index 879bb4944c..41dccbb0c1 100644 --- a/doc/guides/rel_notes/release_24_03.rst +++ b/doc/guides/rel_notes/release_24_03.rst @@ -131,6 +131,10 @@ New Features * Added support for comparing result between packet fields or value. * Added support for accumulating value of field into another one. +* **Updated Intel QuickAssist Technology driver.** + + * Added support for GEN LCE (1454) device, for AES-GCM only. + * **Updated Marvell cnxk crypto driver.** * Added support for Rx inject in crypto_cn10k. diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros.h b/drivers/common/qat/qat_adf/adf_transport_access_macros.h index 12a7258c60..19bd812419 100644 --- a/drivers/common/qat/qat_adf/adf_transport_access_macros.h +++ b/drivers/common/qat/qat_adf/adf_transport_access_macros.h @@ -47,6 +47,7 @@ #define ADF_RING_SIZE_512 0x03 #define ADF_RING_SIZE_4K 0x06 #define ADF_RING_SIZE_16K 0x08 +#define ADF_RING_SIZE_64K 0x0A #define ADF_RING_SIZE_4M 0x10 #define ADF_MIN_RING_SIZE ADF_RING_SIZE_128 #define ADF_MAX_RING_SIZE ADF_RING_SIZE_4M diff --git a/drivers/common/qat/qat_adf/icp_qat_fw.h b/drivers/common/qat/qat_adf/icp_qat_fw.h index 3aa17ae041..b78158e01d 100644 --- a/drivers/common/qat/qat_adf/icp_qat_fw.h +++ b/drivers/common/qat/qat_adf/icp_qat_fw.h @@ -57,6 +57,12 @@ struct icp_qat_fw_comn_req_hdr_cd_pars { } u; }; +struct lce_key_buff_desc { + uint64_t keybuff; + uint32_t keybuff_resrvd1; + uint32_t keybuff_resrvd2; +}; + struct icp_qat_fw_comn_req_mid { uint64_t opaque_data; uint64_t src_data_addr; @@ -123,6 +129,12 @@ struct icp_qat_fw_comn_resp { #define ICP_QAT_FW_COMN_NULL_VERSION_FLAG_BITPOS 0 #define ICP_QAT_FW_COMN_NULL_VERSION_FLAG_MASK 0x1 +/* GEN_LCE specific Common Header fields */ +#define ICP_QAT_FW_COMN_DESC_LAYOUT_BITPOS 5 +#define ICP_QAT_FW_COMN_DESC_LAYOUT_MASK 0x3 +#define ICP_QAT_FW_COMN_GEN_LCE_DESC_LAYOUT 3 +#define ICP_QAT_FW_COMN_GEN_LCE_STATUS_FLAG_ERROR 0 + #define ICP_QAT_FW_COMN_OV_SRV_TYPE_GET(icp_qat_fw_comn_req_hdr_t) \ icp_qat_fw_comn_req_hdr_t.service_type @@ -168,6 +180,12 @@ struct icp_qat_fw_comn_resp { (((valid) & ICP_QAT_FW_COMN_VALID_FLAG_MASK) << \ ICP_QAT_FW_COMN_VALID_FLAG_BITPOS) +#define ICP_QAT_FW_COMN_HDR_FLAGS_BUILD_GEN_LCE(valid, desc_layout) \ + ((((valid) & ICP_QAT_FW_COMN_VALID_FLAG_MASK) << \ + ICP_QAT_FW_COMN_VALID_FLAG_BITPOS) | \ + (((desc_layout) & ICP_QAT_FW_COMN_DESC_LAYOUT_MASK) << \ + ICP_QAT_FW_COMN_DESC_LAYOUT_BITPOS)) + #define QAT_COMN_PTR_TYPE_BITPOS 0 #define QAT_COMN_PTR_TYPE_MASK 0x1 #define QAT_COMN_CD_FLD_TYPE_BITPOS 1 @@ -180,10 +198,20 @@ struct icp_qat_fw_comn_resp { #define QAT_COMN_EXT_FLAGS_MASK 0x1 #define QAT_COMN_EXT_FLAGS_USED 0x1 +/* GEN_LCE specific Common Request Flags fields */ +#define QAT_COMN_KEYBUF_USAGE_BITPOS 1 +#define QAT_COMN_KEYBUF_USAGE_MASK 0x1 +#define QAT_COMN_KEY_BUFFER_USED 1 + #define ICP_QAT_FW_COMN_FLAGS_BUILD(cdt, ptr) \ ((((cdt) & QAT_COMN_CD_FLD_TYPE_MASK) << QAT_COMN_CD_FLD_TYPE_BITPOS) \ | (((ptr) & QAT_COMN_PTR_TYPE_MASK) << QAT_COMN_PTR_TYPE_BITPOS)) +#define ICP_QAT_FW_COMN_FLAGS_BUILD_GEN_LCE(ptr, keybuf) \ + ((((ptr) & QAT_COMN_PTR_TYPE_MASK) << QAT_COMN_PTR_TYPE_BITPOS) | \ + (((keybuf) & QAT_COMN_PTR_TYPE_MASK) << \ + QAT_COMN_KEYBUF_USAGE_BITPOS)) + #define ICP_QAT_FW_COMN_PTR_TYPE_GET(flags) \ QAT_FIELD_GET(flags, QAT_COMN_PTR_TYPE_BITPOS, QAT_COMN_PTR_TYPE_MASK) @@ -249,6 +277,8 @@ struct icp_qat_fw_comn_resp { #define QAT_COMN_RESP_CMP_END_OF_LAST_BLK_MASK 0x1 #define QAT_COMN_RESP_UNSUPPORTED_REQUEST_BITPOS 2 #define QAT_COMN_RESP_UNSUPPORTED_REQUEST_MASK 0x1 +#define QAT_COMN_RESP_INVALID_PARAM_BITPOS 1 +#define QAT_COMN_RESP_INVALID_PARAM_MASK 0x1 #define QAT_COMN_RESP_XLT_WA_APPLIED_BITPOS 0 #define QAT_COMN_RESP_XLT_WA_APPLIED_MASK 0x1 @@ -280,6 +310,10 @@ struct icp_qat_fw_comn_resp { QAT_FIELD_GET(status, QAT_COMN_RESP_UNSUPPORTED_REQUEST_BITPOS, \ QAT_COMN_RESP_UNSUPPORTED_REQUEST_MASK) +#define ICP_QAT_FW_COMN_RESP_INVALID_PARAM_STAT_GET(status) \ + QAT_FIELD_GET(status, QAT_COMN_RESP_INVALID_PARAM_BITPOS, \ + QAT_COMN_RESP_INVALID_PARAM_MASK) + #define ICP_QAT_FW_COMN_STATUS_FLAG_OK 0 #define ICP_QAT_FW_COMN_STATUS_FLAG_ERROR 1 #define ICP_QAT_FW_COMN_STATUS_CMP_END_OF_LAST_BLK_FLAG_CLR 0 diff --git a/drivers/common/qat/qat_adf/icp_qat_fw_la.h b/drivers/common/qat/qat_adf/icp_qat_fw_la.h index 215b291b74..eba9f96685 100644 --- a/drivers/common/qat/qat_adf/icp_qat_fw_la.h +++ b/drivers/common/qat/qat_adf/icp_qat_fw_la.h @@ -22,14 +22,24 @@ enum icp_qat_fw_la_cmd_id { ICP_QAT_FW_LA_CMD_DELIMITER = 18 }; +/* In GEN_LCE Command ID 4 corresponds to AEAD */ +#define ICP_QAT_FW_LA_CMD_AEAD 4 + #define ICP_QAT_FW_LA_ICV_VER_STATUS_PASS ICP_QAT_FW_COMN_STATUS_FLAG_OK #define ICP_QAT_FW_LA_ICV_VER_STATUS_FAIL ICP_QAT_FW_COMN_STATUS_FLAG_ERROR #define ICP_QAT_FW_LA_TRNG_STATUS_PASS ICP_QAT_FW_COMN_STATUS_FLAG_OK #define ICP_QAT_FW_LA_TRNG_STATUS_FAIL ICP_QAT_FW_COMN_STATUS_FLAG_ERROR +/* GEN_LCE Hash, HMAC and GCM Verification Status */ +#define ICP_QAT_FW_LA_VER_STATUS_FAIL ICP_QAT_FW_COMN_GEN_LCE_STATUS_FLAG_ERROR + + struct icp_qat_fw_la_bulk_req { struct icp_qat_fw_comn_req_hdr comn_hdr; - struct icp_qat_fw_comn_req_hdr_cd_pars cd_pars; + union { + struct icp_qat_fw_comn_req_hdr_cd_pars cd_pars; + struct lce_key_buff_desc key_buff; + }; struct icp_qat_fw_comn_req_mid comn_mid; struct icp_qat_fw_comn_req_rqpars serv_specif_rqpars; struct icp_qat_fw_comn_req_cd_ctrl cd_ctrl; @@ -81,6 +91,21 @@ struct icp_qat_fw_la_bulk_req { #define ICP_QAT_FW_LA_PARTIAL_END 2 #define QAT_LA_PARTIAL_BITPOS 0 #define QAT_LA_PARTIAL_MASK 0x3 + +/* GEN_LCE specific Crypto Flags fields */ +#define ICP_QAT_FW_SYM_AEAD_ALGO_BITPOS 6 +#define ICP_QAT_FW_SYM_AEAD_ALGO_MASK 0x3 +#define ICP_QAT_FW_SYM_IV_SIZE_BITPOS 9 +#define ICP_QAT_FW_SYM_IV_SIZE_MASK 0x3 +#define ICP_QAT_FW_SYM_IV_IN_DESC_BITPOS 11 +#define ICP_QAT_FW_SYM_IV_IN_DESC_MASK 0x1 +#define ICP_QAT_FW_SYM_IV_IN_DESC_VALID 1 +#define ICP_QAT_FW_SYM_DIRECTION_BITPOS 15 +#define ICP_QAT_FW_SYM_DIRECTION_MASK 0x1 + +/* In GEN_LCE AEAD AES GCM Algorithm has ID 0 */ +#define QAT_LA_CRYPTO_AEAD_AES_GCM_GEN_LCE 0 + #define ICP_QAT_FW_LA_FLAGS_BUILD(zuc_proto, gcm_iv_len, auth_rslt, proto, \ cmp_auth, ret_auth, update_state, \ ciph_iv, ciphcfg, partial) \ @@ -188,6 +213,23 @@ struct icp_qat_fw_la_bulk_req { QAT_FIELD_SET(flags, val, QAT_LA_PARTIAL_BITPOS, \ QAT_LA_PARTIAL_MASK) +/* GEN_LCE specific Crypto Flags operations */ +#define ICP_QAT_FW_SYM_AEAD_ALGO_SET(flags, val) \ + QAT_FIELD_SET(flags, val, ICP_QAT_FW_SYM_AEAD_ALGO_BITPOS, \ + ICP_QAT_FW_SYM_AEAD_ALGO_MASK) + +#define ICP_QAT_FW_SYM_IV_SIZE_SET(flags, val) \ + QAT_FIELD_SET(flags, val, ICP_QAT_FW_SYM_IV_SIZE_BITPOS, \ + ICP_QAT_FW_SYM_IV_SIZE_MASK) + +#define ICP_QAT_FW_SYM_IV_IN_DESC_FLAG_SET(flags, val) \ + QAT_FIELD_SET(flags, val, ICP_QAT_FW_SYM_IV_IN_DESC_BITPOS, \ + ICP_QAT_FW_SYM_IV_IN_DESC_MASK) + +#define ICP_QAT_FW_SYM_DIR_FLAG_SET(flags, val) \ + QAT_FIELD_SET(flags, val, ICP_QAT_FW_SYM_DIRECTION_BITPOS, \ + ICP_QAT_FW_SYM_DIRECTION_MASK) + #define QAT_FW_LA_MODE2 1 #define QAT_FW_LA_NO_MODE2 0 #define QAT_FW_LA_MODE2_MASK 0x1 @@ -424,4 +466,5 @@ struct icp_qat_fw_la_cipher_30_req_params { } u; }; + #endif diff --git a/drivers/common/qat/qat_device.c b/drivers/common/qat/qat_device.c index f55dc3c6f0..6e23b9e35c 100644 --- a/drivers/common/qat/qat_device.c +++ b/drivers/common/qat/qat_device.c @@ -62,6 +62,9 @@ static const struct rte_pci_id pci_id_qat_map[] = { { RTE_PCI_DEVICE(0x8086, 0x4945), }, + { + RTE_PCI_DEVICE(0x8086, 0x1454), + }, {.device_id = 0}, }; @@ -199,6 +202,8 @@ pick_gen(const struct rte_pci_device *pci_dev) case 0x4943: case 0x4945: return QAT_GEN4; + case 0x1454: + return QAT_GEN_LCE; default: QAT_LOG(ERR, "Invalid dev_id, can't determine generation"); return QAT_N_GENS; From patchwork Wed Feb 28 14:00:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nayak, Nishikanta" X-Patchwork-Id: 137433 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7373943B7C; Wed, 28 Feb 2024 15:01:15 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 43ED643361; Wed, 28 Feb 2024 15:00:55 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by mails.dpdk.org (Postfix) with ESMTP id C2F33432D8 for ; Wed, 28 Feb 2024 15:00:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709128851; x=1740664851; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gKrFnb+x30BUmeYPb5Gy93iXjClnl5t4S4JiuY4oLSA=; b=MtsXbR+fo14/RF4YTT3rzSN+IM2vqWjWhnnXdrcKcDl32/7IQSZaQAe8 bUcoYhq9/NyQwhucsRBcjEZ+h0eZY+NoZkOPUxS1adZCoOypEtfye/sre L3nAowcmzbcSVo/DzlWuX+6aSiE0hz3yG9PdvIt3qsahD6FLCpnEWMpCb KRqqdH9Wia8SXR1Gelx3tvfKVlNpVZNQcbo0w8PBTPVPHDCEESTnXxrfV S7f0aQmk8ywaStayfbF5GOKrewwmfs1JE0mxKaEbkvJCj71bArE9p21wK UsV5IYXVc8LZzwPuNdEmr2wgX4OQY75x9tdowoWTRZG/v9jKrmwImFI8n g==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="3374032" X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="3374032" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2024 06:00:48 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="12157176" Received: from silpixa00401797.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.113]) by orviesa005.jf.intel.com with ESMTP; 28 Feb 2024 06:00:45 -0800 From: Nishikant Nayak To: dev@dpdk.org Cc: ciara.power@intel.com, kai.ji@intel.com, arkadiuszx.kusztal@intel.com, rakesh.s.joshi@intel.com, Nishikant Nayak Subject: [PATCH v6 3/4] crypto/qat: update headers for GEN LCE support Date: Wed, 28 Feb 2024 14:00:35 +0000 Message-Id: <20240228140036.1996629-4-nishikanta.nayak@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240228140036.1996629-1-nishikanta.nayak@intel.com> References: <20231220132616.318983-1-nishikanta.nayak@intel.com> <20240228140036.1996629-1-nishikanta.nayak@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch handles the changes required for updating the common header fields specific to GEN LCE, Also added/updated of the response processing APIs based on GEN LCE requirement. Signed-off-by: Nishikant Nayak Acked-by: Ciara Power --- v2: - Renamed device from GEN 5 to GEN LCE. - Removed unused code. - Updated macro names. - Added GEN LCE specific API for deque burst. - Fixed code formatting. --- --- drivers/crypto/qat/qat_sym.c | 16 ++++++- drivers/crypto/qat/qat_sym.h | 60 ++++++++++++++++++++++++++- drivers/crypto/qat/qat_sym_session.c | 62 +++++++++++++++++++++++++++- drivers/crypto/qat/qat_sym_session.h | 10 ++++- 4 files changed, 140 insertions(+), 8 deletions(-) diff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c index 6e03bde841..439a3fc00b 100644 --- a/drivers/crypto/qat/qat_sym.c +++ b/drivers/crypto/qat/qat_sym.c @@ -180,7 +180,15 @@ qat_sym_dequeue_burst(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops) { return qat_dequeue_op_burst(qp, (void **)ops, - qat_sym_process_response, nb_ops); + qat_sym_process_response, nb_ops); +} + +uint16_t +qat_sym_dequeue_burst_gen_lce(void *qp, struct rte_crypto_op **ops, + uint16_t nb_ops) +{ + return qat_dequeue_op_burst(qp, (void **)ops, + qat_sym_process_response_gen_lce, nb_ops); } int @@ -200,6 +208,7 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev, char capa_memz_name[RTE_CRYPTODEV_NAME_MAX_LEN]; struct rte_cryptodev *cryptodev; struct qat_cryptodev_private *internals; + enum qat_device_gen qat_dev_gen = qat_pci_dev->qat_dev_gen; const struct qat_crypto_gen_dev_ops *gen_dev_ops = &qat_sym_gen_dev_ops[qat_pci_dev->qat_dev_gen]; @@ -249,7 +258,10 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev, cryptodev->dev_ops = gen_dev_ops->cryptodev_ops; cryptodev->enqueue_burst = qat_sym_enqueue_burst; - cryptodev->dequeue_burst = qat_sym_dequeue_burst; + if (qat_dev_gen == QAT_GEN_LCE) + cryptodev->dequeue_burst = qat_sym_dequeue_burst_gen_lce; + else + cryptodev->dequeue_burst = qat_sym_dequeue_burst; cryptodev->feature_flags = gen_dev_ops->get_feature_flags(qat_pci_dev); diff --git a/drivers/crypto/qat/qat_sym.h b/drivers/crypto/qat/qat_sym.h index f2f197d050..3461113c13 100644 --- a/drivers/crypto/qat/qat_sym.h +++ b/drivers/crypto/qat/qat_sym.h @@ -90,7 +90,7 @@ /* * Maximum number of SGL entries */ -#define QAT_SYM_SGL_MAX_NUMBER 16 +#define QAT_SYM_SGL_MAX_NUMBER 16 /* Maximum data length for single pass GMAC: 2^14-1 */ #define QAT_AES_GMAC_SPC_MAX_SIZE 16383 @@ -142,6 +142,10 @@ uint16_t qat_sym_dequeue_burst(void *qp, struct rte_crypto_op **ops, uint16_t nb_ops); +uint16_t +qat_sym_dequeue_burst_gen_lce(void *qp, struct rte_crypto_op **ops, + uint16_t nb_ops); + #ifdef RTE_QAT_OPENSSL /** Encrypt a single partial block * Depends on openssl libcrypto @@ -390,6 +394,52 @@ qat_sym_process_response(void **op, uint8_t *resp, void *op_cookie, return 1; } +static __rte_always_inline int +qat_sym_process_response_gen_lce(void **op, uint8_t *resp, + void *op_cookie __rte_unused, + uint64_t *dequeue_err_count __rte_unused) +{ + struct icp_qat_fw_comn_resp *resp_msg = + (struct icp_qat_fw_comn_resp *)resp; + struct rte_crypto_op *rx_op = (struct rte_crypto_op *)(uintptr_t) + (resp_msg->opaque_data); + struct qat_sym_session *sess; + +#if RTE_LOG_DP_LEVEL >= RTE_LOG_DEBUG + QAT_DP_HEXDUMP_LOG(DEBUG, "qat_response:", (uint8_t *)resp_msg, + sizeof(struct icp_qat_fw_comn_resp)); +#endif + + sess = CRYPTODEV_GET_SYM_SESS_PRIV(rx_op->sym->session); + + rx_op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + + if (ICP_QAT_FW_COMN_STATUS_FLAG_OK != + ICP_QAT_FW_COMN_RESP_UNSUPPORTED_REQUEST_STAT_GET( + resp_msg->comn_hdr.comn_status)) + rx_op->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED; + + else if (ICP_QAT_FW_COMN_STATUS_FLAG_OK != + ICP_QAT_FW_COMN_RESP_INVALID_PARAM_STAT_GET( + resp_msg->comn_hdr.comn_status)) + rx_op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + + if (sess->qat_dir == ICP_QAT_HW_CIPHER_DECRYPT) { + if (ICP_QAT_FW_LA_VER_STATUS_FAIL == + ICP_QAT_FW_COMN_RESP_CRYPTO_STAT_GET( + resp_msg->comn_hdr.comn_status)) + rx_op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + } + + *op = (void *)rx_op; + + /* + * return 1 as dequeue op only move on to the next op + * if one was ready to return to API + */ + return 1; +} + int qat_sym_configure_dp_ctx(struct rte_cryptodev *dev, uint16_t qp_id, struct rte_crypto_raw_dp_ctx *raw_dp_ctx, @@ -455,7 +505,13 @@ qat_sym_preprocess_requests(void **ops __rte_unused, static inline void qat_sym_process_response(void **op __rte_unused, uint8_t *resp __rte_unused, - void *op_cookie __rte_unused) + void *op_cookie __rte_unused, uint64_t *dequeue_err_count __rte_unused) +{ +} + +static inline void +qat_sym_process_response_gen_lce(void **op __rte_unused, uint8_t *resp __rte_unused, + void *op_cookie __rte_unused, uint64_t *dequeue_err_count __rte_unused) { } diff --git a/drivers/crypto/qat/qat_sym_session.c b/drivers/crypto/qat/qat_sym_session.c index 9f4f6c3d93..8f50b61365 100644 --- a/drivers/crypto/qat/qat_sym_session.c +++ b/drivers/crypto/qat/qat_sym_session.c @@ -136,6 +136,9 @@ qat_sym_cd_auth_set(struct qat_sym_session *cdesc, static void qat_sym_session_init_common_hdr(struct qat_sym_session *session); +static void +qat_sym_session_init_gen_lce_hdr(struct qat_sym_session *session); + /* Req/cd init functions */ static void @@ -738,6 +741,12 @@ qat_sym_session_set_parameters(struct rte_cryptodev *dev, session->qat_cmd); return -ENOTSUP; } + + if (qat_dev_gen == QAT_GEN_LCE) { + qat_sym_session_init_gen_lce_hdr(session); + return 0; + } + qat_sym_session_finalize(session); return qat_sym_gen_dev_ops[qat_dev_gen].set_session((void *)dev, @@ -1016,6 +1025,12 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev, dev->data->dev_private; enum qat_device_gen qat_dev_gen = internals->qat_dev->qat_dev_gen; + if (qat_dev_gen == QAT_GEN_LCE) { + struct icp_qat_fw_la_bulk_req *req_tmpl = &session->fw_req; + struct lce_key_buff_desc *key_buff = &req_tmpl->key_buff; + + key_buff->keybuff = session->key_paddr; + } /* * Store AEAD IV parameters as cipher IV, @@ -1079,9 +1094,15 @@ qat_sym_session_configure_aead(struct rte_cryptodev *dev, } if (session->is_single_pass) { - if (qat_sym_cd_cipher_set(session, + if (qat_dev_gen != QAT_GEN_LCE) { + if (qat_sym_cd_cipher_set(session, aead_xform->key.data, aead_xform->key.length)) - return -EINVAL; + return -EINVAL; + } else { + session->auth_key_length = aead_xform->key.length; + memcpy(session->key_array, aead_xform->key.data, + aead_xform->key.length); + } } else if ((aead_xform->op == RTE_CRYPTO_AEAD_OP_ENCRYPT && aead_xform->algo == RTE_CRYPTO_AEAD_AES_GCM) || (aead_xform->op == RTE_CRYPTO_AEAD_OP_DECRYPT && @@ -1970,6 +1991,43 @@ qat_sym_session_init_common_hdr(struct qat_sym_session *session) ICP_QAT_FW_LA_NO_DIGEST_IN_BUFFER); } +static void +qat_sym_session_init_gen_lce_hdr(struct qat_sym_session *session) +{ + struct icp_qat_fw_la_bulk_req *req_tmpl = &session->fw_req; + struct icp_qat_fw_comn_req_hdr *header = &req_tmpl->comn_hdr; + + /* + * GEN_LCE specifies separate command id for AEAD operations but Cryptodev + * API processes AEAD operations as Single pass Crypto operations. + * Hence even for GEN_LCE, Session Algo Command ID is CIPHER. + * Note, however Session Algo Mode is AEAD. + */ + header->service_cmd_id = ICP_QAT_FW_LA_CMD_AEAD; + header->service_type = ICP_QAT_FW_COMN_REQ_CPM_FW_LA; + header->hdr_flags = + ICP_QAT_FW_COMN_HDR_FLAGS_BUILD_GEN_LCE(ICP_QAT_FW_COMN_REQ_FLAG_SET, + ICP_QAT_FW_COMN_GEN_LCE_DESC_LAYOUT); + header->comn_req_flags = + ICP_QAT_FW_COMN_FLAGS_BUILD_GEN_LCE(QAT_COMN_PTR_TYPE_SGL, + QAT_COMN_KEY_BUFFER_USED); + + ICP_QAT_FW_SYM_AEAD_ALGO_SET(header->serv_specif_flags, + QAT_LA_CRYPTO_AEAD_AES_GCM_GEN_LCE); + ICP_QAT_FW_SYM_IV_SIZE_SET(header->serv_specif_flags, + ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS); + ICP_QAT_FW_SYM_IV_IN_DESC_FLAG_SET(header->serv_specif_flags, + ICP_QAT_FW_SYM_IV_IN_DESC_VALID); + + if (session->qat_dir == ICP_QAT_HW_CIPHER_DECRYPT) { + ICP_QAT_FW_SYM_DIR_FLAG_SET(header->serv_specif_flags, + ICP_QAT_HW_CIPHER_DECRYPT); + } else { + ICP_QAT_FW_SYM_DIR_FLAG_SET(header->serv_specif_flags, + ICP_QAT_HW_CIPHER_ENCRYPT); + } +} + int qat_sym_cd_cipher_set(struct qat_sym_session *cdesc, const uint8_t *cipherkey, uint32_t cipherkeylen) diff --git a/drivers/crypto/qat/qat_sym_session.h b/drivers/crypto/qat/qat_sym_session.h index 9209e2e8df..958af03405 100644 --- a/drivers/crypto/qat/qat_sym_session.h +++ b/drivers/crypto/qat/qat_sym_session.h @@ -111,10 +111,16 @@ struct qat_sym_session { enum icp_qat_hw_auth_op auth_op; enum icp_qat_hw_auth_mode auth_mode; void *bpi_ctx; - struct qat_sym_cd cd; + union { + struct qat_sym_cd cd; + uint8_t key_array[32]; + }; uint8_t prefix_state[QAT_PREFIX_TBL_SIZE] __rte_cache_aligned; uint8_t *cd_cur_ptr; - phys_addr_t cd_paddr; + union { + phys_addr_t cd_paddr; + phys_addr_t key_paddr; + }; phys_addr_t prefix_paddr; struct icp_qat_fw_la_bulk_req fw_req; uint8_t aad_len; From patchwork Wed Feb 28 14:00:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nayak, Nishikanta" X-Patchwork-Id: 137434 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E2D8743B7C; Wed, 28 Feb 2024 15:01:28 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BE43A4336D; Wed, 28 Feb 2024 15:00:56 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) by mails.dpdk.org (Postfix) with ESMTP id B0AB04335E for ; Wed, 28 Feb 2024 15:00:54 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709128855; x=1740664855; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QR3ts6dmOOD29MypxvznTJevUZ82A8ba6aUPwiA6j78=; b=kmprcPAk0yydw0S8pYoRcOPCeOcDCxNOzuCcnyHw8V70zIZjqjNCm+SV tNwTkD7Suo/X4ub1LjvmztFrrDL/9vB9DIm20WKrxJhgqWh5svEa2ALSB wXXdSmcTVHx+qhjI8fhQMtWImo4d1fBFFUHnY2TULUHZ6wcldsI4LCj+f Ek/ukjaz32HVymvBs8ZdfZvmSzafN4pu5aS/dMNtGR9bcAbx26FaSBENw pAEzkZKgOn1yfYS01WhEE4VGQlFUv9kUqaSJXJ81DwHk4gVrwUtVVpWCu sud5mEBmAE3Wa7cbC9A9PQRg3EhKVFwB7BkCiNioYO93vl5sqm8dVBPnN A==; X-IronPort-AV: E=McAfee;i="6600,9927,10996"; a="3374039" X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="3374039" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Feb 2024 06:00:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,190,1705392000"; d="scan'208";a="12157194" Received: from silpixa00401797.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.222.113]) by orviesa005.jf.intel.com with ESMTP; 28 Feb 2024 06:00:48 -0800 From: Nishikant Nayak To: dev@dpdk.org Cc: ciara.power@intel.com, kai.ji@intel.com, arkadiuszx.kusztal@intel.com, rakesh.s.joshi@intel.com, Nishikant Nayak , Akhil Goyal , Fan Zhang Subject: [PATCH v6 4/4] test/cryptodev: add tests for GCM with AAD Date: Wed, 28 Feb 2024 14:00:36 +0000 Message-Id: <20240228140036.1996629-5-nishikanta.nayak@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240228140036.1996629-1-nishikanta.nayak@intel.com> References: <20231220132616.318983-1-nishikanta.nayak@intel.com> <20240228140036.1996629-1-nishikanta.nayak@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Adding one new unit test code for validating the features added as part of GCM with 64 byte AAD. The new test case adds one new test for GCM algo for both encrypt and decrypt operations. Signed-off-by: Nishikant Nayak Acked-by: Ciara Power --- v2: - Removed unused code. - Added one new unit test, AAD with GCM for GEN LCE. --- --- app/test/test_cryptodev.c | 48 +++++++++++++--- app/test/test_cryptodev_aead_test_vectors.h | 62 +++++++++++++++++++++ 2 files changed, 103 insertions(+), 7 deletions(-) diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c index 38a65aa88f..edd23731f7 100644 --- a/app/test/test_cryptodev.c +++ b/app/test/test_cryptodev.c @@ -12494,6 +12494,18 @@ test_AES_GCM_auth_decryption_test_case_256_7(void) return test_authenticated_decryption(&gcm_test_case_256_7); } +static int +test_AES_GCM_auth_decryption_test_case_256_8(void) +{ + return test_authenticated_decryption(&gcm_test_case_256_8); +} + +static int +test_AES_GCM_auth_encryption_test_case_256_8(void) +{ + return test_authenticated_encryption(&gcm_test_case_256_8); +} + static int test_AES_GCM_auth_decryption_test_case_aad_1(void) { @@ -12613,10 +12625,16 @@ test_authenticated_encryption_oop(const struct aead_test_data *tdata) /* Verify the capabilities */ struct rte_cryptodev_sym_capability_idx cap_idx; + const struct rte_cryptodev_symmetric_capability *capability; cap_idx.type = RTE_CRYPTO_SYM_XFORM_AEAD; cap_idx.algo.aead = tdata->algo; - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + capability = rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], + &cap_idx); + if (capability == NULL) + return TEST_SKIPPED; + if (rte_cryptodev_sym_capability_check_aead( + capability, tdata->key.len, tdata->auth_tag.len, + tdata->aad.len, tdata->iv.len)) return TEST_SKIPPED; rte_cryptodev_info_get(ts_params->valid_devs[0], &dev_info); @@ -12719,16 +12737,22 @@ test_authenticated_decryption_oop(const struct aead_test_data *tdata) /* Verify the capabilities */ struct rte_cryptodev_sym_capability_idx cap_idx; + const struct rte_cryptodev_symmetric_capability *capability; cap_idx.type = RTE_CRYPTO_SYM_XFORM_AEAD; cap_idx.algo.aead = tdata->algo; - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) - return TEST_SKIPPED; + capability = rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], + &cap_idx); /* not supported with CPU crypto and raw data-path APIs*/ if (gbl_action_type == RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO || global_api_test_type == CRYPTODEV_RAW_API_TEST) return TEST_SKIPPED; + if (capability == NULL) + return TEST_SKIPPED; + if (rte_cryptodev_sym_capability_check_aead( + capability, tdata->key.len, tdata->auth_tag.len, + tdata->aad.len, tdata->iv.len)) + return TEST_SKIPPED; if ((global_api_test_type == CRYPTODEV_RAW_API_TEST) && (!(feat_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP))) { @@ -15749,10 +15773,16 @@ test_authenticated_encryption_SGL(const struct aead_test_data *tdata, /* Verify the capabilities */ struct rte_cryptodev_sym_capability_idx cap_idx; + const struct rte_cryptodev_symmetric_capability *capability; cap_idx.type = RTE_CRYPTO_SYM_XFORM_AEAD; cap_idx.algo.aead = tdata->algo; - if (rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], - &cap_idx) == NULL) + capability = rte_cryptodev_sym_capability_get(ts_params->valid_devs[0], + &cap_idx); + if (capability == NULL) + return TEST_SKIPPED; + if (rte_cryptodev_sym_capability_check_aead( + capability, tdata->key.len, tdata->auth_tag.len, + tdata->aad.len, tdata->iv.len)) return TEST_SKIPPED; /* @@ -17392,6 +17422,8 @@ static struct unit_test_suite cryptodev_aes_gcm_auth_testsuite = { test_AES_GCM_auth_encryption_test_case_256_6), TEST_CASE_ST(ut_setup, ut_teardown, test_AES_GCM_auth_encryption_test_case_256_7), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encryption_test_case_256_8), /** AES GCM Authenticated Decryption 256 bits key */ TEST_CASE_ST(ut_setup, ut_teardown, @@ -17408,6 +17440,8 @@ static struct unit_test_suite cryptodev_aes_gcm_auth_testsuite = { test_AES_GCM_auth_decryption_test_case_256_6), TEST_CASE_ST(ut_setup, ut_teardown, test_AES_GCM_auth_decryption_test_case_256_7), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_decryption_test_case_256_8), /** AES GCM Authenticated Encryption big aad size */ TEST_CASE_ST(ut_setup, ut_teardown, diff --git a/app/test/test_cryptodev_aead_test_vectors.h b/app/test/test_cryptodev_aead_test_vectors.h index 07292620a4..eadf206e4d 100644 --- a/app/test/test_cryptodev_aead_test_vectors.h +++ b/app/test/test_cryptodev_aead_test_vectors.h @@ -17,6 +17,16 @@ static uint8_t gcm_aad_text[MAX_AAD_LENGTH] = { 0x00, 0xf1, 0xe2, 0xd3, 0xc4, 0xb5, 0xa6, 0x97, 0x88, 0x79, 0x6a, 0x5b, 0x4c, 0x3d, 0x2e, 0x1f }; +static uint8_t gcm_aad_64B_text[MAX_AAD_LENGTH] = { + 0xED, 0x3E, 0xA8, 0x1F, 0x74, 0xE5, 0xD1, 0x96, + 0xA4, 0xD5, 0x4B, 0x26, 0xBB, 0x20, 0x61, 0x7B, + 0x3B, 0x9C, 0x2A, 0x69, 0x90, 0xEF, 0xD7, 0x9A, + 0x94, 0xC2, 0xF5, 0x86, 0xBD, 0x00, 0xF6, 0xEA, + 0x0B, 0x14, 0x24, 0xF2, 0x08, 0x67, 0x42, 0x3A, + 0xB5, 0xB8, 0x32, 0x97, 0xB5, 0x99, 0x69, 0x75, + 0x60, 0x00, 0x8F, 0xF7, 0x6F, 0x16, 0x52, 0x66, + 0xF1, 0xA9, 0x38, 0xFD, 0xB0, 0x61, 0x60, 0xB5 }; + static uint8_t ccm_aad_test_1[8] = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 }; @@ -1736,6 +1746,58 @@ static const struct aead_test_data gcm_test_case_256_7 = { } }; +static const struct aead_test_data gcm_test_case_256_8 = { + .algo = RTE_CRYPTO_AEAD_AES_GCM, + .key = { + .data = { + 0xD8, 0xFD, 0x8F, 0x5A, 0x13, 0x7B, 0x05, 0x2C, + 0xA4, 0x64, 0x7A, 0xDD, 0x1E, 0x9A, 0x68, 0x33, + 0x04, 0x70, 0xE8, 0x1E, 0x42, 0x84, 0x64, 0xD2, + 0x23, 0xA1, 0x6A, 0x0A, 0x05, 0x7B, 0x90, 0xDE}, + .len = 32 + }, + .iv = { + .data = { + 0x8D, 0xDF, 0xB8, 0x7F, 0xD0, 0x79, 0x77, 0x55, + 0xD5, 0x48, 0x03, 0x05}, + .len = 12 + }, + .aad = { + .data = gcm_aad_64B_text, + .len = 64 + }, + .plaintext = { + .data = { + 0x4D, 0xBC, 0x2C, 0x7F, 0x25, 0x1F, 0x07, 0x25, + 0x54, 0x8C, 0x43, 0xDB, 0xD8, 0x06, 0x9F, 0xBF, + 0xCA, 0x60, 0xF4, 0xEF, 0x13, 0x87, 0xE8, 0x2F, + 0x4D, 0x9D, 0x1D, 0x87, 0x9F, 0x91, 0x79, 0x7E, + 0x3E, 0x98, 0xA3, 0x63, 0xC6, 0xFE, 0xDB, 0x35, + 0x96, 0x59, 0xB2, 0x0C, 0x80, 0x96, 0x70, 0x07, + 0x87, 0x42, 0xAB, 0x4F, 0x31, 0x73, 0xC4, 0xF9, + 0xB0, 0x1E, 0xF1, 0xBC, 0x7D, 0x45, 0xE5, 0xF3}, + .len = 64 + }, + .ciphertext = { + .data = { + 0x21, 0xFA, 0x59, 0x4F, 0x1F, 0x6B, 0x19, 0xC2, + 0x68, 0xBC, 0x05, 0x93, 0x4E, 0x48, 0x6C, 0x5B, + 0x0B, 0x7A, 0x43, 0xB7, 0x60, 0x8E, 0x00, 0xC4, + 0xAB, 0x14, 0x6B, 0xCC, 0xA1, 0x27, 0x6A, 0xDE, + 0x8E, 0xB6, 0x98, 0xBB, 0x4F, 0xD0, 0x6F, 0x30, + 0x0F, 0x04, 0xA8, 0x5B, 0xDC, 0xD8, 0xE8, 0x8A, + 0x73, 0xD9, 0xB8, 0x60, 0x7C, 0xE4, 0x32, 0x4C, + 0x3A, 0x0B, 0xC2, 0x82, 0xDA, 0x88, 0x17, 0x69}, + .len = 64 + }, + .auth_tag = { + .data = { + 0x3B, 0x80, 0x83, 0x72, 0xE5, 0x1B, 0x94, 0x15, + 0x75, 0xC8, 0x62, 0xBC, 0xA1, 0x66, 0x91, 0x45}, + .len = 16 + } +}; + /** variable AAD AES-GCM-128 Test Vectors */ static const struct aead_test_data gcm_test_case_aad_1 = { .algo = RTE_CRYPTO_AEAD_AES_GCM,