Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/107825/?format=api
{ "id": 107825, "url": "http://patchwork.dpdk.org/api/patches/107825/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/patch/20220218171527.56719-8-kai.ji@intel.com/", "project": { "id": 1, "url": "http://patchwork.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20220218171527.56719-8-kai.ji@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20220218171527.56719-8-kai.ji@intel.com", "date": "2022-02-18T17:15:25", "name": "[v9,7/9] crypto/qat: rework burst data path", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "fa5b699f4f84887b102b5d52a19151a2b31f64dc", "submitter": { "id": 2202, "url": "http://patchwork.dpdk.org/api/people/2202/?format=api", "name": "Ji, Kai", "email": "kai.ji@intel.com" }, "delegate": { "id": 6690, "url": "http://patchwork.dpdk.org/api/users/6690/?format=api", "username": "akhil", "first_name": "akhil", "last_name": "goyal", "email": "gakhil@marvell.com" }, "mbox": "http://patchwork.dpdk.org/project/dpdk/patch/20220218171527.56719-8-kai.ji@intel.com/mbox/", "series": [ { "id": 21741, "url": "http://patchwork.dpdk.org/api/series/21741/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/list/?series=21741", "date": "2022-02-18T17:15:18", "name": "drivers/qat: QAT symmetric crypto datapatch rework", "version": 9, "mbox": "http://patchwork.dpdk.org/series/21741/mbox/" } ], "comments": "http://patchwork.dpdk.org/api/patches/107825/comments/", "check": "success", "checks": "http://patchwork.dpdk.org/api/patches/107825/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 6D094A0032;\n\tFri, 18 Feb 2022 18:16:44 +0100 (CET)", "from [217.70.189.124] (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 70607411A7;\n\tFri, 18 Feb 2022 18:16:03 +0100 (CET)", "from mga02.intel.com (mga02.intel.com [134.134.136.20])\n by mails.dpdk.org (Postfix) with ESMTP id BF84441140\n for <dev@dpdk.org>; Fri, 18 Feb 2022 18:15:49 +0100 (CET)", "from orsmga005.jf.intel.com ([10.7.209.41])\n by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 18 Feb 2022 09:15:42 -0800", "from silpixa00400465.ir.intel.com ([10.55.128.22])\n by orsmga005.jf.intel.com with ESMTP; 18 Feb 2022 09:15:40 -0800" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1645204549; x=1676740549;\n h=from:to:cc:subject:date:message-id:in-reply-to: references;\n bh=b2Gv473GNnx6UTYleFXjakPZB1Lj1/zYaLyBb7dQSW4=;\n b=mezCmGg8QB/JZ/vfweoPRiBJaFStGwWEodD8Wkeak4GkxCWqElQTiCVn\n /kaZ1tQw9hdK1tLMppjrvuEPcgSy1AISKZ8giN382I7JuclWaF8nT63GV\n aZ8olzJJDQXK+guXwX8pSovLLVBPQi/N3e5RXZ8KODixcVYtta80kFcUr\n CrqxwHjpy7eki8drR7Spk9npTSE+1vhQbkZvImrP4J/pa8FY4iBR7NxFM\n jOLKmAn7GZGqxsbY306wHfdw2SGw9OpI5RX9TK8UylhqGBHeMpFPIHaTB\n evHHKfzf1FoHcdvnchR2cPWjFwI/5qqAtnVArHJEBOrjsJxZy3pFLdLn7 Q==;", "X-IronPort-AV": [ "E=McAfee;i=\"6200,9189,10262\"; a=\"238571906\"", "E=Sophos;i=\"5.88,379,1635231600\"; d=\"scan'208\";a=\"238571906\"", "E=Sophos;i=\"5.88,379,1635231600\"; d=\"scan'208\";a=\"705446288\"" ], "X-ExtLoop1": "1", "From": "Kai Ji <kai.ji@intel.com>", "To": "dev@dpdk.org", "Cc": "gakhil@marvell.com,\n\troy.fan.zhang@intel.com,\n\tKai Ji <kai.ji@intel.com>", "Subject": "[dpdk-dev v9 7/9] crypto/qat: rework burst data path", "Date": "Sat, 19 Feb 2022 01:15:25 +0800", "Message-Id": "<20220218171527.56719-8-kai.ji@intel.com>", "X-Mailer": "git-send-email 2.17.1", "In-Reply-To": "<20220218171527.56719-1-kai.ji@intel.com>", "References": "<20220217162909.22713-1-kai.ji@intel.com>\n <20220218171527.56719-1-kai.ji@intel.com>", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "This patch enable the op_build_request function in\nqat_enqueue_op_burst, and the qat_dequeue_process_response\nfunction in qat_dequeue_op_burst.\nThe op_build_request invoked in crypto build request op is based\non crypto operations setup'd during session init.\n\nSigned-off-by: Kai Ji <kai.ji@intel.com>\n---\n drivers/common/qat/qat_qp.c | 42 +-\n drivers/crypto/qat/dev/qat_sym_pmd_gen1.c | 4 -\n drivers/crypto/qat/qat_asym.c | 2 +-\n drivers/crypto/qat/qat_asym.h | 22 -\n drivers/crypto/qat/qat_sym.c | 830 +++++++---------------\n drivers/crypto/qat/qat_sym.h | 5 -\n 6 files changed, 271 insertions(+), 634 deletions(-)", "diff": "diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c\nindex b36ffa6f6d..08ac91eac4 100644\n--- a/drivers/common/qat/qat_qp.c\n+++ b/drivers/common/qat/qat_qp.c\n@@ -547,8 +547,7 @@ adf_modulo(uint32_t data, uint32_t modulo_mask)\n }\n \n uint16_t\n-qat_enqueue_op_burst(void *qp,\n-\t\t__rte_unused qat_op_build_request_t op_build_request,\n+qat_enqueue_op_burst(void *qp, qat_op_build_request_t op_build_request,\n \t\tvoid **ops, uint16_t nb_ops)\n {\n \tregister struct qat_queue *queue;\n@@ -599,29 +598,18 @@ qat_enqueue_op_burst(void *qp,\n \t\t}\n \t}\n \n-#ifdef BUILD_QAT_SYM\n+#ifdef RTE_LIB_SECURITY\n \tif (tmp_qp->service_type == QAT_SERVICE_SYMMETRIC)\n \t\tqat_sym_preprocess_requests(ops, nb_ops_possible);\n #endif\n \n+\tmemset(tmp_qp->opaque, 0xff, sizeof(tmp_qp->opaque));\n+\n \twhile (nb_ops_sent != nb_ops_possible) {\n-\t\tif (tmp_qp->service_type == QAT_SERVICE_SYMMETRIC) {\n-#ifdef BUILD_QAT_SYM\n-\t\t\tret = qat_sym_build_request(*ops, base_addr + tail,\n-\t\t\t\ttmp_qp->op_cookies[tail >> queue->trailz],\n-\t\t\t\ttmp_qp->qat_dev_gen);\n-#endif\n-\t\t} else if (tmp_qp->service_type == QAT_SERVICE_COMPRESSION) {\n-\t\t\tret = qat_comp_build_request(*ops, base_addr + tail,\n+\t\tret = op_build_request(*ops, base_addr + tail,\n \t\t\t\ttmp_qp->op_cookies[tail >> queue->trailz],\n-\t\t\t\ttmp_qp->qat_dev_gen);\n-\t\t} else if (tmp_qp->service_type == QAT_SERVICE_ASYMMETRIC) {\n-#ifdef BUILD_QAT_ASYM\n-\t\t\tret = qat_asym_build_request(*ops, base_addr + tail,\n-\t\t\t\ttmp_qp->op_cookies[tail >> queue->trailz],\n-\t\t\t\tNULL, tmp_qp->qat_dev_gen);\n-#endif\n-\t\t}\n+\t\t\t\ttmp_qp->opaque, tmp_qp->qat_dev_gen);\n+\n \t\tif (ret != 0) {\n \t\t\ttmp_qp->stats.enqueue_err_count++;\n \t\t\t/* This message cannot be enqueued */\n@@ -817,8 +805,7 @@ qat_enqueue_comp_op_burst(void *qp, void **ops, uint16_t nb_ops)\n \n uint16_t\n qat_dequeue_op_burst(void *qp, void **ops,\n-\t\t__rte_unused qat_op_dequeue_t qat_dequeue_process_response,\n-\t\tuint16_t nb_ops)\n+\t\tqat_op_dequeue_t qat_dequeue_process_response, uint16_t nb_ops)\n {\n \tstruct qat_queue *rx_queue;\n \tstruct qat_qp *tmp_qp = (struct qat_qp *)qp;\n@@ -836,21 +823,10 @@ qat_dequeue_op_burst(void *qp, void **ops,\n \n \t\tnb_fw_responses = 1;\n \n-\t\tif (tmp_qp->service_type == QAT_SERVICE_SYMMETRIC)\n-\t\t\tqat_sym_process_response(ops, resp_msg,\n-\t\t\t\ttmp_qp->op_cookies[head >> rx_queue->trailz],\n-\t\t\t\tNULL);\n-\t\telse if (tmp_qp->service_type == QAT_SERVICE_COMPRESSION)\n-\t\t\tnb_fw_responses = qat_comp_process_response(\n+\t\tnb_fw_responses = qat_dequeue_process_response(\n \t\t\t\tops, resp_msg,\n \t\t\t\ttmp_qp->op_cookies[head >> rx_queue->trailz],\n \t\t\t\t&tmp_qp->stats.dequeue_err_count);\n-#ifdef BUILD_QAT_ASYM\n-\t\telse if (tmp_qp->service_type == QAT_SERVICE_ASYMMETRIC)\n-\t\t\tqat_asym_process_response(ops, resp_msg,\n-\t\t\t\ttmp_qp->op_cookies[head >> rx_queue->trailz],\n-\t\t\t\tNULL);\n-#endif\n \n \t\thead = adf_modulo(head + rx_queue->msg_size,\n \t\t\t\t rx_queue->modulo_mask);\ndiff --git a/drivers/crypto/qat/dev/qat_sym_pmd_gen1.c b/drivers/crypto/qat/dev/qat_sym_pmd_gen1.c\nindex 501132a448..c58a628915 100644\n--- a/drivers/crypto/qat/dev/qat_sym_pmd_gen1.c\n+++ b/drivers/crypto/qat/dev/qat_sym_pmd_gen1.c\n@@ -146,10 +146,6 @@ struct rte_cryptodev_ops qat_sym_crypto_ops_gen1 = {\n \t.sym_session_get_size\t= qat_sym_session_get_private_size,\n \t.sym_session_configure\t= qat_sym_session_configure,\n \t.sym_session_clear\t= qat_sym_session_clear,\n-\n-\t/* Raw data-path API related operations */\n-\t.sym_get_raw_dp_ctx_size = qat_sym_get_dp_ctx_size,\n-\t.sym_configure_raw_dp_ctx = qat_sym_configure_dp_ctx,\n };\n \n static struct qat_capabilities_info\ndiff --git a/drivers/crypto/qat/qat_asym.c b/drivers/crypto/qat/qat_asym.c\nindex 76d50dd6f9..3c7cba6140 100644\n--- a/drivers/crypto/qat/qat_asym.c\n+++ b/drivers/crypto/qat/qat_asym.c\n@@ -507,7 +507,7 @@ qat_asym_fill_arrays(struct rte_crypto_asym_op *asym_op,\n \treturn 0;\n }\n \n-int\n+static __rte_always_inline int\n qat_asym_build_request(void *in_op, uint8_t *out_msg, void *op_cookie,\n \t\t__rte_unused uint64_t *opaque,\n \t\t__rte_unused enum qat_device_gen dev_gen)\ndiff --git a/drivers/crypto/qat/qat_asym.h b/drivers/crypto/qat/qat_asym.h\nindex 3ae95f2e7b..78caa5649c 100644\n--- a/drivers/crypto/qat/qat_asym.h\n+++ b/drivers/crypto/qat/qat_asym.h\n@@ -103,28 +103,6 @@ void\n qat_asym_session_clear(struct rte_cryptodev *dev,\n \t\tstruct rte_cryptodev_asym_session *sess);\n \n-/*\n- * Build PKE request to be sent to the fw, partially uses template\n- * request generated during session creation.\n- *\n- * @param\tin_op\t\tPointer to the crypto operation, for every\n- *\t\t\t\tservice it points to service specific struct.\n- * @param\tout_msg\t\tMessage to be returned to enqueue function\n- * @param\top_cookie\tCookie pointer that holds private metadata\n- * @param\tqat_dev_gen\tGeneration of QAT hardware\n- *\n- * @return\n- *\tThis function always returns zero,\n- *\tit is because of backward compatibility.\n- *\t- 0: Always returned\n- *\n- */\n-int\n-qat_asym_build_request(void *in_op, uint8_t *out_msg,\n-\t\tvoid *op_cookie,\n-\t\t__rte_unused uint64_t *opaque,\n-\t\tenum qat_device_gen qat_dev_gen);\n-\n /*\n * Process PKE response received from outgoing queue of QAT\n *\ndiff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c\nindex aad4b243b7..692e1f8f0a 100644\n--- a/drivers/crypto/qat/qat_sym.c\n+++ b/drivers/crypto/qat/qat_sym.c\n@@ -11,12 +11,25 @@\n #include <rte_byteorder.h>\n \n #include \"qat_sym.h\"\n-#include \"dev/qat_crypto_pmd_gens.h\"\n+#include \"qat_crypto.h\"\n+#include \"qat_qp.h\"\n \n uint8_t qat_sym_driver_id;\n \n struct qat_crypto_gen_dev_ops qat_sym_gen_dev_ops[QAT_N_GENS];\n \n+/* An rte_driver is needed in the registration of both the device and the driver\n+ * with cryptodev.\n+ * The actual qat pci's rte_driver can't be used as its name represents\n+ * the whole pci device with all services. Think of this as a holder for a name\n+ * for the crypto part of the pci device.\n+ */\n+static const char qat_sym_drv_name[] = RTE_STR(CRYPTODEV_NAME_QAT_SYM_PMD);\n+static const struct rte_driver cryptodev_qat_sym_driver = {\n+\t.name = qat_sym_drv_name,\n+\t.alias = qat_sym_drv_name\n+};\n+\n void\n qat_sym_init_op_cookie(void *op_cookie)\n {\n@@ -38,160 +51,68 @@ qat_sym_init_op_cookie(void *op_cookie)\n \t\t\topt.spc_gmac.cd_cipher);\n }\n \n-static inline void\n-set_cipher_iv(uint16_t iv_length, uint16_t iv_offset,\n-\t\tstruct icp_qat_fw_la_cipher_req_params *cipher_param,\n-\t\tstruct rte_crypto_op *op,\n-\t\tstruct icp_qat_fw_la_bulk_req *qat_req)\n+static __rte_always_inline int\n+qat_sym_build_request(void *in_op, uint8_t *out_msg,\n+\t\tvoid *op_cookie, uint64_t *opaque, enum qat_device_gen dev_gen)\n {\n-\t/* copy IV into request if it fits */\n-\tif (iv_length <= sizeof(cipher_param->u.cipher_IV_array)) {\n-\t\trte_memcpy(cipher_param->u.cipher_IV_array,\n-\t\t\t\trte_crypto_op_ctod_offset(op, uint8_t *,\n-\t\t\t\t\tiv_offset),\n-\t\t\t\tiv_length);\n-\t} else {\n-\t\tICP_QAT_FW_LA_CIPH_IV_FLD_FLAG_SET(\n-\t\t\t\tqat_req->comn_hdr.serv_specif_flags,\n-\t\t\t\tICP_QAT_FW_CIPH_IV_64BIT_PTR);\n-\t\tcipher_param->u.s.cipher_IV_ptr =\n-\t\t\t\trte_crypto_op_ctophys_offset(op,\n-\t\t\t\t\tiv_offset);\n-\t}\n-}\n+\tstruct rte_crypto_op *op = (struct rte_crypto_op *)in_op;\n+\tuintptr_t sess = (uintptr_t)opaque[0];\n+\tuintptr_t build_request_p = (uintptr_t)opaque[1];\n+\tqat_sym_build_request_t build_request = (void *)build_request_p;\n+\tstruct qat_sym_session *ctx = NULL;\n \n-/** Set IV for CCM is special case, 0th byte is set to q-1\n- * where q is padding of nonce in 16 byte block\n- */\n-static inline void\n-set_cipher_iv_ccm(uint16_t iv_length, uint16_t iv_offset,\n-\t\tstruct icp_qat_fw_la_cipher_req_params *cipher_param,\n-\t\tstruct rte_crypto_op *op, uint8_t q, uint8_t aad_len_field_sz)\n-{\n-\trte_memcpy(((uint8_t *)cipher_param->u.cipher_IV_array) +\n-\t\t\tICP_QAT_HW_CCM_NONCE_OFFSET,\n-\t\t\trte_crypto_op_ctod_offset(op, uint8_t *,\n-\t\t\t\tiv_offset) + ICP_QAT_HW_CCM_NONCE_OFFSET,\n-\t\t\tiv_length);\n-\t*(uint8_t *)&cipher_param->u.cipher_IV_array[0] =\n-\t\t\tq - ICP_QAT_HW_CCM_NONCE_OFFSET;\n-\n-\tif (aad_len_field_sz)\n-\t\trte_memcpy(&op->sym->aead.aad.data[ICP_QAT_HW_CCM_NONCE_OFFSET],\n-\t\t\trte_crypto_op_ctod_offset(op, uint8_t *,\n-\t\t\t\tiv_offset) + ICP_QAT_HW_CCM_NONCE_OFFSET,\n-\t\t\tiv_length);\n-}\n+\tif (likely(op->sess_type == RTE_CRYPTO_OP_WITH_SESSION)) {\n+\t\tctx = get_sym_session_private_data(op->sym->session,\n+\t\t\t\tqat_sym_driver_id);\n+\t\tif (unlikely(!ctx)) {\n+\t\t\tQAT_DP_LOG(ERR, \"No session for this device\");\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tif (sess != (uintptr_t)ctx) {\n+\t\t\tstruct rte_cryptodev *cdev;\n+\t\t\tstruct qat_cryptodev_private *internals;\n+\t\t\tenum rte_proc_type_t proc_type;\n+\n+\t\t\tcdev = rte_cryptodev_pmd_get_dev(ctx->dev_id);\n+\t\t\tinternals = cdev->data->dev_private;\n+\t\t\tproc_type = rte_eal_process_type();\n+\n+\t\t\tif (internals->qat_dev->qat_dev_gen != dev_gen) {\n+\t\t\t\top->status =\n+\t\t\t\t\tRTE_CRYPTO_OP_STATUS_INVALID_SESSION;\n+\t\t\t\treturn -EINVAL;\n+\t\t\t}\n \n-/** Handle Single-Pass AES-GMAC on QAT GEN3 */\n-static inline void\n-handle_spc_gmac(struct qat_sym_session *ctx, struct rte_crypto_op *op,\n-\t\tstruct qat_sym_op_cookie *cookie,\n-\t\tstruct icp_qat_fw_la_bulk_req *qat_req)\n-{\n-\tstatic const uint32_t ver_key_offset =\n-\t\t\tsizeof(struct icp_qat_hw_auth_setup) +\n-\t\t\tICP_QAT_HW_GALOIS_128_STATE1_SZ +\n-\t\t\tICP_QAT_HW_GALOIS_H_SZ + ICP_QAT_HW_GALOIS_LEN_A_SZ +\n-\t\t\tICP_QAT_HW_GALOIS_E_CTR0_SZ +\n-\t\t\tsizeof(struct icp_qat_hw_cipher_config);\n-\tstruct icp_qat_fw_cipher_cd_ctrl_hdr *cipher_cd_ctrl =\n-\t\t\t(void *) &qat_req->cd_ctrl;\n-\tstruct icp_qat_fw_la_cipher_req_params *cipher_param =\n-\t\t\t(void *) &qat_req->serv_specif_rqpars;\n-\tuint32_t data_length = op->sym->auth.data.length;\n-\n-\t/* Fill separate Content Descriptor for this op */\n-\trte_memcpy(cookie->opt.spc_gmac.cd_cipher.key,\n-\t\t\tctx->auth_op == ICP_QAT_HW_AUTH_GENERATE ?\n-\t\t\t\tctx->cd.cipher.key :\n-\t\t\t\tRTE_PTR_ADD(&ctx->cd, ver_key_offset),\n-\t\t\tctx->auth_key_length);\n-\tcookie->opt.spc_gmac.cd_cipher.cipher_config.val =\n-\t\t\tICP_QAT_HW_CIPHER_CONFIG_BUILD(\n-\t\t\t\tICP_QAT_HW_CIPHER_AEAD_MODE,\n-\t\t\t\tctx->qat_cipher_alg,\n-\t\t\t\tICP_QAT_HW_CIPHER_NO_CONVERT,\n-\t\t\t\t(ctx->auth_op == ICP_QAT_HW_AUTH_GENERATE ?\n-\t\t\t\t\tICP_QAT_HW_CIPHER_ENCRYPT :\n-\t\t\t\t\tICP_QAT_HW_CIPHER_DECRYPT));\n-\tQAT_FIELD_SET(cookie->opt.spc_gmac.cd_cipher.cipher_config.val,\n-\t\t\tctx->digest_length,\n-\t\t\tQAT_CIPHER_AEAD_HASH_CMP_LEN_BITPOS,\n-\t\t\tQAT_CIPHER_AEAD_HASH_CMP_LEN_MASK);\n-\tcookie->opt.spc_gmac.cd_cipher.cipher_config.reserved =\n-\t\t\tICP_QAT_HW_CIPHER_CONFIG_BUILD_UPPER(data_length);\n-\n-\t/* Update the request */\n-\tqat_req->cd_pars.u.s.content_desc_addr =\n-\t\t\tcookie->opt.spc_gmac.cd_phys_addr;\n-\tqat_req->cd_pars.u.s.content_desc_params_sz = RTE_ALIGN_CEIL(\n-\t\t\tsizeof(struct icp_qat_hw_cipher_config) +\n-\t\t\tctx->auth_key_length, 8) >> 3;\n-\tqat_req->comn_mid.src_length = data_length;\n-\tqat_req->comn_mid.dst_length = 0;\n-\n-\tcipher_param->spc_aad_addr = 0;\n-\tcipher_param->spc_auth_res_addr = op->sym->auth.digest.phys_addr;\n-\tcipher_param->spc_aad_sz = data_length;\n-\tcipher_param->reserved = 0;\n-\tcipher_param->spc_auth_res_sz = ctx->digest_length;\n-\n-\tqat_req->comn_hdr.service_cmd_id = ICP_QAT_FW_LA_CMD_CIPHER;\n-\tcipher_cd_ctrl->cipher_cfg_offset = 0;\n-\tICP_QAT_FW_COMN_CURR_ID_SET(cipher_cd_ctrl, ICP_QAT_FW_SLICE_CIPHER);\n-\tICP_QAT_FW_COMN_NEXT_ID_SET(cipher_cd_ctrl, ICP_QAT_FW_SLICE_DRAM_WR);\n-\tICP_QAT_FW_LA_SINGLE_PASS_PROTO_FLAG_SET(\n-\t\t\tqat_req->comn_hdr.serv_specif_flags,\n-\t\t\tICP_QAT_FW_LA_SINGLE_PASS_PROTO);\n-\tICP_QAT_FW_LA_PROTO_SET(\n-\t\t\tqat_req->comn_hdr.serv_specif_flags,\n-\t\t\tICP_QAT_FW_LA_NO_PROTO);\n-}\n+\t\t\tif (unlikely(ctx->build_request[proc_type] == NULL)) {\n+\t\t\t\tint ret =\n+\t\t\t\tqat_sym_gen_dev_ops[dev_gen].set_session(\n+\t\t\t\t\t(void *)cdev, (void *)sess);\n+\t\t\t\tif (ret < 0) {\n+\t\t\t\t\top->status =\n+\t\t\t\t\tRTE_CRYPTO_OP_STATUS_INVALID_SESSION;\n+\t\t\t\t\treturn -EINVAL;\n+\t\t\t\t}\n+\t\t\t}\n \n-int\n-qat_sym_build_request(void *in_op, uint8_t *out_msg,\n-\t\tvoid *op_cookie, __rte_unused enum qat_device_gen qat_dev_gen)\n-{\n-\tint ret = 0;\n-\tstruct qat_sym_session *ctx = NULL;\n-\tstruct icp_qat_fw_la_cipher_req_params *cipher_param;\n-\tstruct icp_qat_fw_la_cipher_20_req_params *cipher_param20;\n-\tstruct icp_qat_fw_la_auth_req_params *auth_param;\n-\tregister struct icp_qat_fw_la_bulk_req *qat_req;\n-\tuint8_t do_auth = 0, do_cipher = 0, do_aead = 0;\n-\tuint32_t cipher_len = 0, cipher_ofs = 0;\n-\tuint32_t auth_len = 0, auth_ofs = 0;\n-\tuint32_t min_ofs = 0;\n-\tuint64_t src_buf_start = 0, dst_buf_start = 0;\n-\tuint64_t auth_data_end = 0;\n-\tuint8_t do_sgl = 0;\n-\tuint8_t in_place = 1;\n-\tint alignment_adjustment = 0;\n-\tint oop_shift = 0;\n-\tstruct rte_crypto_op *op = (struct rte_crypto_op *)in_op;\n-\tstruct qat_sym_op_cookie *cookie =\n-\t\t\t\t(struct qat_sym_op_cookie *)op_cookie;\n-\n-\tif (unlikely(op->type != RTE_CRYPTO_OP_TYPE_SYMMETRIC)) {\n-\t\tQAT_DP_LOG(ERR, \"QAT PMD only supports symmetric crypto \"\n-\t\t\t\t\"operation requests, op (%p) is not a \"\n-\t\t\t\t\"symmetric operation.\", op);\n-\t\treturn -EINVAL;\n+\t\t\tbuild_request = ctx->build_request[proc_type];\n+\t\t\topaque[0] = (uintptr_t)ctx;\n+\t\t\topaque[1] = (uintptr_t)build_request;\n+\t\t}\n \t}\n \n-\tif (unlikely(op->sess_type == RTE_CRYPTO_OP_SESSIONLESS)) {\n-\t\tQAT_DP_LOG(ERR, \"QAT PMD only supports session oriented\"\n-\t\t\t\t\" requests, op (%p) is sessionless.\", op);\n-\t\treturn -EINVAL;\n-\t} else if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) {\n-\t\tctx = (struct qat_sym_session *)get_sym_session_private_data(\n-\t\t\t\top->sym->session, qat_sym_driver_id);\n #ifdef RTE_LIB_SECURITY\n-\t} else {\n-\t\tctx = (struct qat_sym_session *)get_sec_session_private_data(\n-\t\t\t\top->sym->sec_session);\n-\t\tif (likely(ctx)) {\n+\telse if (op->sess_type == RTE_CRYPTO_OP_SECURITY_SESSION) {\n+\t\tif ((void *)sess != (void *)op->sym->sec_session) {\n+\t\t\tstruct rte_cryptodev *cdev;\n+\t\t\tstruct qat_cryptodev_private *internals;\n+\t\t\tenum rte_proc_type_t proc_type;\n+\n+\t\t\tctx = get_sec_session_private_data(\n+\t\t\t\t\top->sym->sec_session);\n+\t\t\tif (unlikely(!ctx)) {\n+\t\t\t\tQAT_DP_LOG(ERR, \"No session for this device\");\n+\t\t\t\treturn -EINVAL;\n+\t\t\t}\n \t\t\tif (unlikely(ctx->bpi_ctx == NULL)) {\n \t\t\t\tQAT_DP_LOG(ERR, \"QAT PMD only supports security\"\n \t\t\t\t\t\t\" operation requests for\"\n@@ -207,463 +128,234 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg,\n \t\t\t\top->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;\n \t\t\t\treturn -EINVAL;\n \t\t\t}\n-\t\t}\n-#endif\n-\t}\n+\t\t\tcdev = rte_cryptodev_pmd_get_dev(ctx->dev_id);\n+\t\t\tinternals = cdev->data->dev_private;\n+\t\t\tproc_type = rte_eal_process_type();\n \n-\tif (unlikely(ctx == NULL)) {\n-\t\tQAT_DP_LOG(ERR, \"Session was not created for this device\");\n-\t\treturn -EINVAL;\n-\t}\n+\t\t\tif (internals->qat_dev->qat_dev_gen != dev_gen) {\n+\t\t\t\top->status =\n+\t\t\t\t\tRTE_CRYPTO_OP_STATUS_INVALID_SESSION;\n+\t\t\t\treturn -EINVAL;\n+\t\t\t}\n+\n+\t\t\tif (unlikely(ctx->build_request[proc_type] == NULL)) {\n+\t\t\t\tint ret =\n+\t\t\t\tqat_sym_gen_dev_ops[dev_gen].set_session(\n+\t\t\t\t\t(void *)cdev, (void *)sess);\n+\t\t\t\tif (ret < 0) {\n+\t\t\t\t\top->status =\n+\t\t\t\t\tRTE_CRYPTO_OP_STATUS_INVALID_SESSION;\n+\t\t\t\t\treturn -EINVAL;\n+\t\t\t\t}\n+\t\t\t}\n \n-\tqat_req = (struct icp_qat_fw_la_bulk_req *)out_msg;\n-\trte_mov128((uint8_t *)qat_req, (const uint8_t *)&(ctx->fw_req));\n-\tqat_req->comn_mid.opaque_data = (uint64_t)(uintptr_t)op;\n-\tcipher_param = (void *)&qat_req->serv_specif_rqpars;\n-\tcipher_param20 = (void *)&qat_req->serv_specif_rqpars;\n-\tauth_param = (void *)((uint8_t *)cipher_param +\n-\t\t\tICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET);\n-\n-\tif ((ctx->qat_cmd == ICP_QAT_FW_LA_CMD_HASH_CIPHER ||\n-\t\t\tctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) &&\n-\t\t\t!ctx->is_gmac) {\n-\t\t/* AES-GCM or AES-CCM */\n-\t\tif (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||\n-\t\t\tctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_64 ||\n-\t\t\t(ctx->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_AES128\n-\t\t\t&& ctx->qat_mode == ICP_QAT_HW_CIPHER_CTR_MODE\n-\t\t\t&& ctx->qat_hash_alg ==\n-\t\t\t\t\tICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC)) {\n-\t\t\tdo_aead = 1;\n-\t\t} else {\n-\t\t\tdo_auth = 1;\n-\t\t\tdo_cipher = 1;\n+\t\t\tsess = (uintptr_t)op->sym->sec_session;\n+\t\t\tbuild_request = ctx->build_request[proc_type];\n+\t\t\topaque[0] = sess;\n+\t\t\topaque[1] = (uintptr_t)build_request;\n \t\t}\n-\t} else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH || ctx->is_gmac) {\n-\t\tdo_auth = 1;\n-\t\tdo_cipher = 0;\n-\t} else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER) {\n-\t\tdo_auth = 0;\n-\t\tdo_cipher = 1;\n+\t}\n+#endif\n+\telse { /* RTE_CRYPTO_OP_SESSIONLESS */\n+\t\top->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;\n+\t\tQAT_LOG(DEBUG, \"QAT does not support sessionless operation\");\n+\t\treturn -1;\n \t}\n \n-\tif (do_cipher) {\n+\treturn build_request(op, (void *)ctx, out_msg, op_cookie);\n+}\n \n-\t\tif (ctx->qat_cipher_alg ==\n-\t\t\t\t\t ICP_QAT_HW_CIPHER_ALGO_SNOW_3G_UEA2 ||\n-\t\t\tctx->qat_cipher_alg == ICP_QAT_HW_CIPHER_ALGO_KASUMI ||\n-\t\t\tctx->qat_cipher_alg ==\n-\t\t\t\tICP_QAT_HW_CIPHER_ALGO_ZUC_3G_128_EEA3) {\n+uint16_t\n+qat_sym_enqueue_burst(void *qp, struct rte_crypto_op **ops,\n+\t\tuint16_t nb_ops)\n+{\n+\treturn qat_enqueue_op_burst(qp, qat_sym_build_request,\n+\t\t\t(void **)ops, nb_ops);\n+}\n \n-\t\t\tif (unlikely(\n-\t\t\t (op->sym->cipher.data.length % BYTE_LENGTH != 0) ||\n-\t\t\t (op->sym->cipher.data.offset % BYTE_LENGTH != 0))) {\n-\t\t\t\tQAT_DP_LOG(ERR,\n-\t\t \"SNOW3G/KASUMI/ZUC in QAT PMD only supports byte aligned values\");\n-\t\t\t\top->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;\n-\t\t\t\treturn -EINVAL;\n-\t\t\t}\n-\t\t\tcipher_len = op->sym->cipher.data.length >> 3;\n-\t\t\tcipher_ofs = op->sym->cipher.data.offset >> 3;\n-\n-\t\t} else if (ctx->bpi_ctx) {\n-\t\t\t/* DOCSIS - only send complete blocks to device.\n-\t\t\t * Process any partial block using CFB mode.\n-\t\t\t * Even if 0 complete blocks, still send this to device\n-\t\t\t * to get into rx queue for post-process and dequeuing\n-\t\t\t */\n-\t\t\tcipher_len = qat_bpicipher_preprocess(ctx, op);\n-\t\t\tcipher_ofs = op->sym->cipher.data.offset;\n-\t\t} else {\n-\t\t\tcipher_len = op->sym->cipher.data.length;\n-\t\t\tcipher_ofs = op->sym->cipher.data.offset;\n-\t\t}\n+uint16_t\n+qat_sym_dequeue_burst(void *qp, struct rte_crypto_op **ops,\n+\t\tuint16_t nb_ops)\n+{\n+\treturn qat_dequeue_op_burst(qp, (void **)ops,\n+\t\t\t\tqat_sym_process_response, nb_ops);\n+}\n \n-\t\tset_cipher_iv(ctx->cipher_iv.length, ctx->cipher_iv.offset,\n-\t\t\t\tcipher_param, op, qat_req);\n-\t\tmin_ofs = cipher_ofs;\n+int\n+qat_sym_dev_create(struct qat_pci_device *qat_pci_dev,\n+\t\tstruct qat_dev_cmd_param *qat_dev_cmd_param __rte_unused)\n+{\n+\tint i = 0, ret = 0;\n+\tstruct qat_device_info *qat_dev_instance =\n+\t\t\t&qat_pci_devs[qat_pci_dev->qat_dev_id];\n+\tstruct rte_cryptodev_pmd_init_params init_params = {\n+\t\t.name = \"\",\n+\t\t.socket_id = qat_dev_instance->pci_dev->device.numa_node,\n+\t\t.private_data_size = sizeof(struct qat_cryptodev_private)\n+\t};\n+\tchar name[RTE_CRYPTODEV_NAME_MAX_LEN];\n+\tchar capa_memz_name[RTE_CRYPTODEV_NAME_MAX_LEN];\n+\tstruct rte_cryptodev *cryptodev;\n+\tstruct qat_cryptodev_private *internals;\n+\tstruct qat_capabilities_info capa_info;\n+\tconst struct rte_cryptodev_capabilities *capabilities;\n+\tconst struct qat_crypto_gen_dev_ops *gen_dev_ops =\n+\t\t&qat_sym_gen_dev_ops[qat_pci_dev->qat_dev_gen];\n+\tuint64_t capa_size;\n+\n+\tsnprintf(name, RTE_CRYPTODEV_NAME_MAX_LEN, \"%s_%s\",\n+\t\t\tqat_pci_dev->name, \"sym\");\n+\tQAT_LOG(DEBUG, \"Creating QAT SYM device %s\", name);\n+\n+\tif (gen_dev_ops->cryptodev_ops == NULL) {\n+\t\tQAT_LOG(ERR, \"Device %s does not support symmetric crypto\",\n+\t\t\t\tname);\n+\t\treturn -(EFAULT);\n \t}\n \n-\tif (do_auth) {\n-\n-\t\tif (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2 ||\n-\t\t\tctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_KASUMI_F9 ||\n-\t\t\tctx->qat_hash_alg ==\n-\t\t\t\tICP_QAT_HW_AUTH_ALGO_ZUC_3G_128_EIA3) {\n-\t\t\tif (unlikely(\n-\t\t\t (op->sym->auth.data.offset % BYTE_LENGTH != 0) ||\n-\t\t\t (op->sym->auth.data.length % BYTE_LENGTH != 0))) {\n-\t\t\t\tQAT_DP_LOG(ERR,\n-\t\t\"For SNOW3G/KASUMI/ZUC, QAT PMD only supports byte aligned values\");\n-\t\t\t\top->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS;\n-\t\t\t\treturn -EINVAL;\n-\t\t\t}\n-\t\t\tauth_ofs = op->sym->auth.data.offset >> 3;\n-\t\t\tauth_len = op->sym->auth.data.length >> 3;\n-\n-\t\t\tauth_param->u1.aad_adr =\n-\t\t\t\t\trte_crypto_op_ctophys_offset(op,\n-\t\t\t\t\t\t\tctx->auth_iv.offset);\n-\n-\t\t} else if (ctx->qat_hash_alg ==\n-\t\t\t\t\tICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||\n-\t\t\t\tctx->qat_hash_alg ==\n-\t\t\t\t\tICP_QAT_HW_AUTH_ALGO_GALOIS_64) {\n-\t\t\t/* AES-GMAC */\n-\t\t\tset_cipher_iv(ctx->auth_iv.length,\n-\t\t\t\tctx->auth_iv.offset,\n-\t\t\t\tcipher_param, op, qat_req);\n-\t\t\tauth_ofs = op->sym->auth.data.offset;\n-\t\t\tauth_len = op->sym->auth.data.length;\n-\n-\t\t\tauth_param->u1.aad_adr = 0;\n-\t\t\tauth_param->u2.aad_sz = 0;\n-\n-\t\t} else {\n-\t\t\tauth_ofs = op->sym->auth.data.offset;\n-\t\t\tauth_len = op->sym->auth.data.length;\n-\n+\t/*\n+\t * All processes must use same driver id so they can share sessions.\n+\t * Store driver_id so we can validate that all processes have the same\n+\t * value, typically they have, but could differ if binaries built\n+\t * separately.\n+\t */\n+\tif (rte_eal_process_type() == RTE_PROC_PRIMARY) {\n+\t\tqat_pci_dev->qat_sym_driver_id =\n+\t\t\t\tqat_sym_driver_id;\n+\t} else if (rte_eal_process_type() == RTE_PROC_SECONDARY) {\n+\t\tif (qat_pci_dev->qat_sym_driver_id !=\n+\t\t\t\tqat_sym_driver_id) {\n+\t\t\tQAT_LOG(ERR,\n+\t\t\t\t\"Device %s have different driver id than corresponding device in primary process\",\n+\t\t\t\tname);\n+\t\t\treturn -(EFAULT);\n \t\t}\n-\t\tmin_ofs = auth_ofs;\n-\n-\t\tif (ctx->qat_hash_alg != ICP_QAT_HW_AUTH_ALGO_NULL ||\n-\t\t\t\tctx->auth_op == ICP_QAT_HW_AUTH_VERIFY)\n-\t\t\tauth_param->auth_res_addr =\n-\t\t\t\t\top->sym->auth.digest.phys_addr;\n-\n \t}\n \n-\tif (do_aead) {\n-\t\t/*\n-\t\t * This address may used for setting AAD physical pointer\n-\t\t * into IV offset from op\n-\t\t */\n-\t\trte_iova_t aad_phys_addr_aead = op->sym->aead.aad.phys_addr;\n-\t\tif (ctx->qat_hash_alg ==\n-\t\t\t\tICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||\n-\t\t\t\tctx->qat_hash_alg ==\n-\t\t\t\t\tICP_QAT_HW_AUTH_ALGO_GALOIS_64) {\n-\n-\t\t\tset_cipher_iv(ctx->cipher_iv.length,\n-\t\t\t\t\tctx->cipher_iv.offset,\n-\t\t\t\t\tcipher_param, op, qat_req);\n-\n-\t\t} else if (ctx->qat_hash_alg ==\n-\t\t\t\tICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC) {\n-\n-\t\t\t/* In case of AES-CCM this may point to user selected\n-\t\t\t * memory or iv offset in crypto_op\n-\t\t\t */\n-\t\t\tuint8_t *aad_data = op->sym->aead.aad.data;\n-\t\t\t/* This is true AAD length, it not includes 18 bytes of\n-\t\t\t * preceding data\n-\t\t\t */\n-\t\t\tuint8_t aad_ccm_real_len = 0;\n-\t\t\tuint8_t aad_len_field_sz = 0;\n-\t\t\tuint32_t msg_len_be =\n-\t\t\t\t\trte_bswap32(op->sym->aead.data.length);\n-\n-\t\t\tif (ctx->aad_len > ICP_QAT_HW_CCM_AAD_DATA_OFFSET) {\n-\t\t\t\taad_len_field_sz = ICP_QAT_HW_CCM_AAD_LEN_INFO;\n-\t\t\t\taad_ccm_real_len = ctx->aad_len -\n-\t\t\t\t\tICP_QAT_HW_CCM_AAD_B0_LEN -\n-\t\t\t\t\tICP_QAT_HW_CCM_AAD_LEN_INFO;\n-\t\t\t} else {\n-\t\t\t\t/*\n-\t\t\t\t * aad_len not greater than 18, so no actual aad\n-\t\t\t\t * data, then use IV after op for B0 block\n-\t\t\t\t */\n-\t\t\t\taad_data = rte_crypto_op_ctod_offset(op,\n-\t\t\t\t\t\tuint8_t *,\n-\t\t\t\t\t\tctx->cipher_iv.offset);\n-\t\t\t\taad_phys_addr_aead =\n-\t\t\t\t\t\trte_crypto_op_ctophys_offset(op,\n-\t\t\t\t\t\t\tctx->cipher_iv.offset);\n-\t\t\t}\n-\n-\t\t\tuint8_t q = ICP_QAT_HW_CCM_NQ_CONST -\n-\t\t\t\t\t\t\tctx->cipher_iv.length;\n-\n-\t\t\taad_data[0] = ICP_QAT_HW_CCM_BUILD_B0_FLAGS(\n-\t\t\t\t\t\t\taad_len_field_sz,\n-\t\t\t\t\t\t\tctx->digest_length, q);\n-\n-\t\t\tif (q > ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE) {\n-\t\t\t\tmemcpy(aad_data\t+ ctx->cipher_iv.length +\n-\t\t\t\t ICP_QAT_HW_CCM_NONCE_OFFSET +\n-\t\t\t\t (q - ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE),\n-\t\t\t\t (uint8_t *)&msg_len_be,\n-\t\t\t\t ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE);\n-\t\t\t} else {\n-\t\t\t\tmemcpy(aad_data\t+ ctx->cipher_iv.length +\n-\t\t\t\t ICP_QAT_HW_CCM_NONCE_OFFSET,\n-\t\t\t\t (uint8_t *)&msg_len_be\n-\t\t\t\t + (ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE\n-\t\t\t\t - q), q);\n-\t\t\t}\n-\n-\t\t\tif (aad_len_field_sz > 0) {\n-\t\t\t\t*(uint16_t *)&aad_data[ICP_QAT_HW_CCM_AAD_B0_LEN]\n-\t\t\t\t\t\t= rte_bswap16(aad_ccm_real_len);\n-\n-\t\t\t\tif ((aad_ccm_real_len + aad_len_field_sz)\n-\t\t\t\t\t\t% ICP_QAT_HW_CCM_AAD_B0_LEN) {\n-\t\t\t\t\tuint8_t pad_len = 0;\n-\t\t\t\t\tuint8_t pad_idx = 0;\n-\n-\t\t\t\t\tpad_len = ICP_QAT_HW_CCM_AAD_B0_LEN -\n-\t\t\t\t\t((aad_ccm_real_len + aad_len_field_sz) %\n-\t\t\t\t\t\tICP_QAT_HW_CCM_AAD_B0_LEN);\n-\t\t\t\t\tpad_idx = ICP_QAT_HW_CCM_AAD_B0_LEN +\n-\t\t\t\t\t aad_ccm_real_len + aad_len_field_sz;\n-\t\t\t\t\tmemset(&aad_data[pad_idx],\n-\t\t\t\t\t\t\t0, pad_len);\n-\t\t\t\t}\n+\t/* Populate subset device to use in cryptodev device creation */\n+\tqat_dev_instance->sym_rte_dev.driver = &cryptodev_qat_sym_driver;\n+\tqat_dev_instance->sym_rte_dev.numa_node =\n+\t\t\tqat_dev_instance->pci_dev->device.numa_node;\n+\tqat_dev_instance->sym_rte_dev.devargs = NULL;\n \n-\t\t\t}\n+\tcryptodev = rte_cryptodev_pmd_create(name,\n+\t\t\t&(qat_dev_instance->sym_rte_dev), &init_params);\n \n-\t\t\tset_cipher_iv_ccm(ctx->cipher_iv.length,\n-\t\t\t\t\tctx->cipher_iv.offset,\n-\t\t\t\t\tcipher_param, op, q,\n-\t\t\t\t\taad_len_field_sz);\n+\tif (cryptodev == NULL)\n+\t\treturn -ENODEV;\n \n-\t\t}\n+\tqat_dev_instance->sym_rte_dev.name = cryptodev->data->name;\n+\tcryptodev->driver_id = qat_sym_driver_id;\n+\tcryptodev->dev_ops = gen_dev_ops->cryptodev_ops;\n \n-\t\tcipher_len = op->sym->aead.data.length;\n-\t\tcipher_ofs = op->sym->aead.data.offset;\n-\t\tauth_len = op->sym->aead.data.length;\n-\t\tauth_ofs = op->sym->aead.data.offset;\n+\tcryptodev->enqueue_burst = qat_sym_enqueue_burst;\n+\tcryptodev->dequeue_burst = qat_sym_dequeue_burst;\n \n-\t\tauth_param->u1.aad_adr = aad_phys_addr_aead;\n-\t\tauth_param->auth_res_addr = op->sym->aead.digest.phys_addr;\n-\t\tmin_ofs = op->sym->aead.data.offset;\n-\t}\n+\tcryptodev->feature_flags = gen_dev_ops->get_feature_flags(qat_pci_dev);\n \n-\tif (op->sym->m_src->nb_segs > 1 ||\n-\t\t\t(op->sym->m_dst && op->sym->m_dst->nb_segs > 1))\n-\t\tdo_sgl = 1;\n-\n-\t/* adjust for chain case */\n-\tif (do_cipher && do_auth)\n-\t\tmin_ofs = cipher_ofs < auth_ofs ? cipher_ofs : auth_ofs;\n-\n-\tif (unlikely(min_ofs >= rte_pktmbuf_data_len(op->sym->m_src) && do_sgl))\n-\t\tmin_ofs = 0;\n-\n-\tif (unlikely((op->sym->m_dst != NULL) &&\n-\t\t\t(op->sym->m_dst != op->sym->m_src))) {\n-\t\t/* Out-of-place operation (OOP)\n-\t\t * Don't align DMA start. DMA the minimum data-set\n-\t\t * so as not to overwrite data in dest buffer\n-\t\t */\n-\t\tin_place = 0;\n-\t\tsrc_buf_start =\n-\t\t\trte_pktmbuf_iova_offset(op->sym->m_src, min_ofs);\n-\t\tdst_buf_start =\n-\t\t\trte_pktmbuf_iova_offset(op->sym->m_dst, min_ofs);\n-\t\toop_shift = min_ofs;\n+\tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n+\t\treturn 0;\n \n-\t} else {\n-\t\t/* In-place operation\n-\t\t * Start DMA at nearest aligned address below min_ofs\n-\t\t */\n-\t\tsrc_buf_start =\n-\t\t\trte_pktmbuf_iova_offset(op->sym->m_src, min_ofs)\n-\t\t\t\t\t\t& QAT_64_BTYE_ALIGN_MASK;\n-\n-\t\tif (unlikely((rte_pktmbuf_iova(op->sym->m_src) -\n-\t\t\t\t\trte_pktmbuf_headroom(op->sym->m_src))\n-\t\t\t\t\t\t\t> src_buf_start)) {\n-\t\t\t/* alignment has pushed addr ahead of start of mbuf\n-\t\t\t * so revert and take the performance hit\n-\t\t\t */\n-\t\t\tsrc_buf_start =\n-\t\t\t\trte_pktmbuf_iova_offset(op->sym->m_src,\n-\t\t\t\t\t\t\t\tmin_ofs);\n+#ifdef RTE_LIB_SECURITY\n+\tif (gen_dev_ops->create_security_ctx) {\n+\t\tcryptodev->security_ctx =\n+\t\t\tgen_dev_ops->create_security_ctx((void *)cryptodev);\n+\t\tif (cryptodev->security_ctx == NULL) {\n+\t\t\tQAT_LOG(ERR, \"rte_security_ctx memory alloc failed\");\n+\t\t\tret = -ENOMEM;\n+\t\t\tgoto error;\n \t\t}\n-\t\tdst_buf_start = src_buf_start;\n \n-\t\t/* remember any adjustment for later, note, can be +/- */\n-\t\talignment_adjustment = src_buf_start -\n-\t\t\trte_pktmbuf_iova_offset(op->sym->m_src, min_ofs);\n-\t}\n-\n-\tif (do_cipher || do_aead) {\n-\t\tcipher_param->cipher_offset =\n-\t\t\t\t(uint32_t)rte_pktmbuf_iova_offset(\n-\t\t\t\top->sym->m_src, cipher_ofs) - src_buf_start;\n-\t\tcipher_param->cipher_length = cipher_len;\n+\t\tcryptodev->feature_flags |= RTE_CRYPTODEV_FF_SECURITY;\n+\t\tQAT_LOG(INFO, \"Device %s rte_security support ensabled\", name);\n \t} else {\n-\t\tcipher_param->cipher_offset = 0;\n-\t\tcipher_param->cipher_length = 0;\n+\t\tQAT_LOG(INFO, \"Device %s rte_security support disabled\", name);\n \t}\n-\n-\tif (!ctx->is_single_pass) {\n-\t\t/* Do not let to overwrite spc_aad len */\n-\t\tif (do_auth || do_aead) {\n-\t\t\tauth_param->auth_off =\n-\t\t\t\t(uint32_t)rte_pktmbuf_iova_offset(\n-\t\t\t\top->sym->m_src, auth_ofs) - src_buf_start;\n-\t\t\tauth_param->auth_len = auth_len;\n-\t\t} else {\n-\t\t\tauth_param->auth_off = 0;\n-\t\t\tauth_param->auth_len = 0;\n+#endif\n+\tsnprintf(capa_memz_name, RTE_CRYPTODEV_NAME_MAX_LEN,\n+\t\t\t\"QAT_SYM_CAPA_GEN_%d\",\n+\t\t\tqat_pci_dev->qat_dev_gen);\n+\n+\tinternals = cryptodev->data->dev_private;\n+\tinternals->qat_dev = qat_pci_dev;\n+\n+\tinternals->dev_id = cryptodev->data->dev_id;\n+\n+\tcapa_info = gen_dev_ops->get_capabilities(qat_pci_dev);\n+\tcapabilities = capa_info.data;\n+\tcapa_size = capa_info.size;\n+\n+\tinternals->capa_mz = rte_memzone_lookup(capa_memz_name);\n+\tif (internals->capa_mz == NULL) {\n+\t\tinternals->capa_mz = rte_memzone_reserve(capa_memz_name,\n+\t\t\t\tcapa_size, rte_socket_id(), 0);\n+\t\tif (internals->capa_mz == NULL) {\n+\t\t\tQAT_LOG(DEBUG,\n+\t\t\t\t\"Error allocating memzone for capabilities, \"\n+\t\t\t\t\"destroying PMD for %s\",\n+\t\t\t\tname);\n+\t\t\tret = -EFAULT;\n+\t\t\tgoto error;\n \t\t}\n \t}\n \n-\tqat_req->comn_mid.dst_length =\n-\t\tqat_req->comn_mid.src_length =\n-\t\t(cipher_param->cipher_offset + cipher_param->cipher_length)\n-\t\t> (auth_param->auth_off + auth_param->auth_len) ?\n-\t\t(cipher_param->cipher_offset + cipher_param->cipher_length)\n-\t\t: (auth_param->auth_off + auth_param->auth_len);\n-\n-\tif (do_auth && do_cipher) {\n-\t\t/* Handle digest-encrypted cases, i.e.\n-\t\t * auth-gen-then-cipher-encrypt and\n-\t\t * cipher-decrypt-then-auth-verify\n-\t\t */\n-\t\t /* First find the end of the data */\n-\t\tif (do_sgl) {\n-\t\t\tuint32_t remaining_off = auth_param->auth_off +\n-\t\t\t\tauth_param->auth_len + alignment_adjustment + oop_shift;\n-\t\t\tstruct rte_mbuf *sgl_buf =\n-\t\t\t\t(in_place ?\n-\t\t\t\t\top->sym->m_src : op->sym->m_dst);\n-\n-\t\t\twhile (remaining_off >= rte_pktmbuf_data_len(sgl_buf)\n-\t\t\t\t\t&& sgl_buf->next != NULL) {\n-\t\t\t\tremaining_off -= rte_pktmbuf_data_len(sgl_buf);\n-\t\t\t\tsgl_buf = sgl_buf->next;\n-\t\t\t}\n+\tmemcpy(internals->capa_mz->addr, capabilities, capa_size);\n+\tinternals->qat_dev_capabilities = internals->capa_mz->addr;\n \n-\t\t\tauth_data_end = (uint64_t)rte_pktmbuf_iova_offset(\n-\t\t\t\tsgl_buf, remaining_off);\n-\t\t} else {\n-\t\t\tauth_data_end = (in_place ?\n-\t\t\t\tsrc_buf_start : dst_buf_start) +\n-\t\t\t\tauth_param->auth_off + auth_param->auth_len;\n-\t\t}\n-\t\t/* Then check if digest-encrypted conditions are met */\n-\t\tif ((auth_param->auth_off + auth_param->auth_len <\n-\t\t\t\t\tcipher_param->cipher_offset +\n-\t\t\t\t\tcipher_param->cipher_length) &&\n-\t\t\t\t(op->sym->auth.digest.phys_addr ==\n-\t\t\t\t\tauth_data_end)) {\n-\t\t\t/* Handle partial digest encryption */\n-\t\t\tif (cipher_param->cipher_offset +\n-\t\t\t\t\tcipher_param->cipher_length <\n-\t\t\t\t\tauth_param->auth_off +\n-\t\t\t\t\tauth_param->auth_len +\n-\t\t\t\t\tctx->digest_length)\n-\t\t\t\tqat_req->comn_mid.dst_length =\n-\t\t\t\t\tqat_req->comn_mid.src_length =\n-\t\t\t\t\tauth_param->auth_off +\n-\t\t\t\t\tauth_param->auth_len +\n-\t\t\t\t\tctx->digest_length;\n-\t\t\tstruct icp_qat_fw_comn_req_hdr *header =\n-\t\t\t\t&qat_req->comn_hdr;\n-\t\t\tICP_QAT_FW_LA_DIGEST_IN_BUFFER_SET(\n-\t\t\t\theader->serv_specif_flags,\n-\t\t\t\tICP_QAT_FW_LA_DIGEST_IN_BUFFER);\n-\t\t}\n+\twhile (1) {\n+\t\tif (qat_dev_cmd_param[i].name == NULL)\n+\t\t\tbreak;\n+\t\tif (!strcmp(qat_dev_cmd_param[i].name, SYM_ENQ_THRESHOLD_NAME))\n+\t\t\tinternals->min_enq_burst_threshold =\n+\t\t\t\t\tqat_dev_cmd_param[i].val;\n+\t\ti++;\n \t}\n \n-\tif (do_sgl) {\n-\n-\t\tICP_QAT_FW_COMN_PTR_TYPE_SET(qat_req->comn_hdr.comn_req_flags,\n-\t\t\t\tQAT_COMN_PTR_TYPE_SGL);\n-\t\tret = qat_sgl_fill_array(op->sym->m_src,\n-\t\t (int64_t)(src_buf_start - rte_pktmbuf_iova(op->sym->m_src)),\n-\t\t &cookie->qat_sgl_src,\n-\t\t qat_req->comn_mid.src_length,\n-\t\t QAT_SYM_SGL_MAX_NUMBER);\n-\n-\t\tif (unlikely(ret)) {\n-\t\t\tQAT_DP_LOG(ERR, \"QAT PMD Cannot fill sgl array\");\n-\t\t\treturn ret;\n-\t\t}\n+\tinternals->service_type = QAT_SERVICE_SYMMETRIC;\n+\tqat_pci_dev->sym_dev = internals;\n+\tQAT_LOG(DEBUG, \"Created QAT SYM device %s as cryptodev instance %d\",\n+\t\t\tcryptodev->data->name, internals->dev_id);\n \n-\t\tif (in_place)\n-\t\t\tqat_req->comn_mid.dest_data_addr =\n-\t\t\t\tqat_req->comn_mid.src_data_addr =\n-\t\t\t\tcookie->qat_sgl_src_phys_addr;\n-\t\telse {\n-\t\t\tret = qat_sgl_fill_array(op->sym->m_dst,\n-\t\t\t\t(int64_t)(dst_buf_start -\n-\t\t\t\t\t rte_pktmbuf_iova(op->sym->m_dst)),\n-\t\t\t\t &cookie->qat_sgl_dst,\n-\t\t\t\t qat_req->comn_mid.dst_length,\n-\t\t\t\t QAT_SYM_SGL_MAX_NUMBER);\n-\n-\t\t\tif (unlikely(ret)) {\n-\t\t\t\tQAT_DP_LOG(ERR, \"QAT PMD can't fill sgl array\");\n-\t\t\t\treturn ret;\n-\t\t\t}\n+\treturn 0;\n \n-\t\t\tqat_req->comn_mid.src_data_addr =\n-\t\t\t\tcookie->qat_sgl_src_phys_addr;\n-\t\t\tqat_req->comn_mid.dest_data_addr =\n-\t\t\t\t\tcookie->qat_sgl_dst_phys_addr;\n-\t\t}\n-\t\tqat_req->comn_mid.src_length = 0;\n-\t\tqat_req->comn_mid.dst_length = 0;\n-\t} else {\n-\t\tqat_req->comn_mid.src_data_addr = src_buf_start;\n-\t\tqat_req->comn_mid.dest_data_addr = dst_buf_start;\n-\t}\n+error:\n+#ifdef RTE_LIB_SECURITY\n+\trte_free(cryptodev->security_ctx);\n+\tcryptodev->security_ctx = NULL;\n+#endif\n+\trte_cryptodev_pmd_destroy(cryptodev);\n+\tmemset(&qat_dev_instance->sym_rte_dev, 0,\n+\t\tsizeof(qat_dev_instance->sym_rte_dev));\n \n-\tif (ctx->is_single_pass) {\n-\t\tif (ctx->is_ucs) {\n-\t\t\t/* GEN 4 */\n-\t\t\tcipher_param20->spc_aad_addr =\n-\t\t\t\top->sym->aead.aad.phys_addr;\n-\t\t\tcipher_param20->spc_auth_res_addr =\n-\t\t\t\top->sym->aead.digest.phys_addr;\n-\t\t} else {\n-\t\t\tcipher_param->spc_aad_addr =\n-\t\t\t\top->sym->aead.aad.phys_addr;\n-\t\t\tcipher_param->spc_auth_res_addr =\n-\t\t\t\t\top->sym->aead.digest.phys_addr;\n-\t\t}\n-\t} else if (ctx->is_single_pass_gmac &&\n-\t\t op->sym->auth.data.length <= QAT_AES_GMAC_SPC_MAX_SIZE) {\n-\t\t/* Handle Single-Pass AES-GMAC */\n-\t\thandle_spc_gmac(ctx, op, cookie, qat_req);\n-\t}\n+\treturn ret;\n+}\n \n-#if RTE_LOG_DP_LEVEL >= RTE_LOG_DEBUG\n-\tQAT_DP_HEXDUMP_LOG(DEBUG, \"qat_req:\", qat_req,\n-\t\t\tsizeof(struct icp_qat_fw_la_bulk_req));\n-\tQAT_DP_HEXDUMP_LOG(DEBUG, \"src_data:\",\n-\t\t\trte_pktmbuf_mtod(op->sym->m_src, uint8_t*),\n-\t\t\trte_pktmbuf_data_len(op->sym->m_src));\n-\tif (do_cipher) {\n-\t\tuint8_t *cipher_iv_ptr = rte_crypto_op_ctod_offset(op,\n-\t\t\t\t\t\tuint8_t *,\n-\t\t\t\t\t\tctx->cipher_iv.offset);\n-\t\tQAT_DP_HEXDUMP_LOG(DEBUG, \"cipher iv:\", cipher_iv_ptr,\n-\t\t\t\tctx->cipher_iv.length);\n-\t}\n+int\n+qat_sym_dev_destroy(struct qat_pci_device *qat_pci_dev)\n+{\n+\tstruct rte_cryptodev *cryptodev;\n \n-\tif (do_auth) {\n-\t\tif (ctx->auth_iv.length) {\n-\t\t\tuint8_t *auth_iv_ptr = rte_crypto_op_ctod_offset(op,\n-\t\t\t\t\t\t\tuint8_t *,\n-\t\t\t\t\t\t\tctx->auth_iv.offset);\n-\t\t\tQAT_DP_HEXDUMP_LOG(DEBUG, \"auth iv:\", auth_iv_ptr,\n-\t\t\t\t\t\tctx->auth_iv.length);\n-\t\t}\n-\t\tQAT_DP_HEXDUMP_LOG(DEBUG, \"digest:\", op->sym->auth.digest.data,\n-\t\t\t\tctx->digest_length);\n-\t}\n+\tif (qat_pci_dev == NULL)\n+\t\treturn -ENODEV;\n+\tif (qat_pci_dev->sym_dev == NULL)\n+\t\treturn 0;\n+\tif (rte_eal_process_type() == RTE_PROC_PRIMARY)\n+\t\trte_memzone_free(qat_pci_dev->sym_dev->capa_mz);\n \n-\tif (do_aead) {\n-\t\tQAT_DP_HEXDUMP_LOG(DEBUG, \"digest:\", op->sym->aead.digest.data,\n-\t\t\t\tctx->digest_length);\n-\t\tQAT_DP_HEXDUMP_LOG(DEBUG, \"aad:\", op->sym->aead.aad.data,\n-\t\t\t\tctx->aad_len);\n-\t}\n+\t/* free crypto device */\n+\tcryptodev = rte_cryptodev_pmd_get_dev(qat_pci_dev->sym_dev->dev_id);\n+#ifdef RTE_LIB_SECURITY\n+\trte_free(cryptodev->security_ctx);\n+\tcryptodev->security_ctx = NULL;\n #endif\n+\trte_cryptodev_pmd_destroy(cryptodev);\n+\tqat_pci_devs[qat_pci_dev->qat_dev_id].sym_rte_dev.name = NULL;\n+\tqat_pci_dev->sym_dev = NULL;\n+\n \treturn 0;\n }\n+\n+static struct cryptodev_driver qat_crypto_drv;\n+RTE_PMD_REGISTER_CRYPTO_DRIVER(qat_crypto_drv,\n+\t\tcryptodev_qat_sym_driver,\n+\t\tqat_sym_driver_id);\ndiff --git a/drivers/crypto/qat/qat_sym.h b/drivers/crypto/qat/qat_sym.h\nindex f4ff2ce4cd..074612c11b 100644\n--- a/drivers/crypto/qat/qat_sym.h\n+++ b/drivers/crypto/qat/qat_sym.h\n@@ -131,11 +131,6 @@ uint16_t\n qat_sym_dequeue_burst(void *qp, struct rte_crypto_op **ops,\n \t\tuint16_t nb_ops);\n \n-int\n-qat_sym_build_request(void *in_op, uint8_t *out_msg,\n-\t\tvoid *op_cookie, enum qat_device_gen qat_dev_gen);\n-\n-\n /** Encrypt a single partial block\n * Depends on openssl libcrypto\n * Uses ECB+XOR to do CFB encryption, same result, more performant\n", "prefixes": [ "v9", "7/9" ] }