From patchwork Wed Jan 10 09:42:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kumar, Ravi1" X-Patchwork-Id: 33406 X-Patchwork-Delegate: pablo.de.lara.guarch@intel.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BF1741B22E; Wed, 10 Jan 2018 10:43:34 +0100 (CET) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0075.outbound.protection.outlook.com [104.47.42.75]) by dpdk.org (Postfix) with ESMTP id 9E60C1B21D for ; Wed, 10 Jan 2018 10:43:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector1-amd-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=3zYCjwzDDbA28fN+s2mOOzYQTZ0G52iKHlnUi3Gj6Xw=; b=DUS81OofTecBHKzIr2MMWbuILkJbuQdzOlzLdSHBrqn87wVA6D+mWBc4574mTVhwmI5CuvNBQFWx5Wkp14Xrljgf0lCSsr16FiJtJxW9fNJPQHNr5a8ow68mv/MFS9DZEj+eY9w3UjMFAe77Tq7GpFu4QKAiuPDKuAfKbpuc6/0= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Ravi1.Kumar@amd.com; Received: from wallaby-smavila.amd.com (202.56.249.162) by CY4PR12MB1509.namprd12.prod.outlook.com (10.172.68.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.386.5; Wed, 10 Jan 2018 09:43:29 +0000 From: Ravi Kumar To: dev@dpdk.org Cc: pablo.de.lara.guarch@intel.com Date: Wed, 10 Jan 2018 04:42:46 -0500 Message-Id: <1515577379-18453-6-git-send-email-Ravi1.kumar@amd.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1515577379-18453-1-git-send-email-Ravi1.kumar@amd.com> References: <1515145198-97367-1-git-send-email-Ravi1.kumar@amd.com> <1515577379-18453-1-git-send-email-Ravi1.kumar@amd.com> MIME-Version: 1.0 X-Originating-IP: [202.56.249.162] X-ClientProxiedBy: MAXPR0101CA0024.INDPRD01.PROD.OUTLOOK.COM (10.174.62.34) To CY4PR12MB1509.namprd12.prod.outlook.com (10.172.68.22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: bbe37f1e-acbc-4907-9e0b-08d5580e9b3f X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020049)(4652020)(48565401081)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:CY4PR12MB1509; X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1509; 3:M+SXxLIcMVJGQeRkGAGfJULh2wBIHPJh+TsednbL2RFdoRqnsUQ9ppeJzno103xHkCKyBD2xTJwO4hvPYm1dvAW8gs0fiRzl2dEz4sx4YIbvG92FSfLA5bg/GazSXRt4vF21U21YIrYXPWoAX/+xt6DQ4qxbr+Yb0G7o0jm/Mo6LDUBtr98ny3Z86fZRduzCz6WNlzkq329dDZytMh7kriLsvLtq/0vbz/nysFjJnXkNKSRy67q4a+4bEIYWx7BS; 25:4psdcsh7HA5jE5CWnFtFSwRpk2227WnDfHkQFD3U0Qu/XdwbGxuDkNh+cbxabHrkGQjrEWh9YRSAVRsROtnhB5MnUe4nxRtdVJvZ1rYBZuRCcxPVyzAwqztdaJr+YDLIFw+3Mtf63pX8n/6nhddUNZMRezTgR3NUTc0a+Yyt+2/qPCQbP04OCe+Sm7eQn7vjwvTs5xfPIs3i+H5uXAfiiUI7CRxZrq6uKA8uifASxq93c3co203e6ac9Z3aI5amJej1OARd9kxglea74HDswcP0etP17B/JN4nD06IefAMRvyiXaLEWChSR2isBB4qmszfEEYwBiYO3J9n4SW9EC5w==; 31:Qb+4XByNAejRynRjqBvw3ljh77nVahhuOIHp6cUx0RLGZL7trbR+FEYl7TvGl5oVc92mqSHp5VXpHsxqLzUIHmsCsIzZWDONVQmo1oLfVmYujeyoE36Q/OBYF48adUYeejPayjM0j8GWemVLzNU8R1WI4JSc1i6ZOevU3L+vn+j9HOph78pOCJ3GmZoCn3XGTNFQkeW8aX1s1ZssJ7wpmQWameasXLI8jTSbx8dWaPY= X-MS-TrafficTypeDiagnostic: CY4PR12MB1509: X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1509; 20:XnBg3KjFvqMGdmXGUc5g4oU1CLLsDZ/KnBRoyjlLyJHzVG8gUHADcLdnHabexNuEhwlfTjsNe2aWZj+09rNtKGNKeHzF+HqMQRo1QDsYa7eLMVFFV9g8ubD5GkqsfrYAd9VarbLYolgjRMjskCRRSPUzbBeNaUEOf3XhXG46Sv475ICMyDh7ycRaxV7/IKQFZEoLSA1OFzhgDDBTfkSO0oRHiCUCJe3+kivCvjNoMlPBAiWKq+ZMw/KYbwITintgl27Tf7jqTFdCfuHkF9WjdH+cRnfwvSIqpDPQV/eHbCxmUaSrgtmNZaGHhsKCmM/pDPZqJ+auTa/tjUeDAAvFbGxa0udT0Rsdzu8BErw6/susaSnHxhOIfkMJmjK7LskjG422NKZ18bChb46aZBdkO6OGY6eBbfo6ZlT8Y8CBilyvbAV4R1XgqxThJDiYJgK6fjlWBSgM1Gibo7p3ddc1oE2dQJiqFHpikiigo+YZsGz4rd5hk1Ik5dkSW9JJQL5e; 4:4nrmeK14XSF10HgBifUJHF09NAtTPL5617NEkFu+xsCR1ml2X0fJBKDn3PIFOhYpHCk3OJykqvQZ6oNJa+AKDoUVYuGl2Il0+tdOKA0tQYwga4dHGLyoht7LtvV3MNgVpe4+kA5HCbrfz7iju03F+H6RU+UD4qLsS5N5w5VWSkcYA+rCoMuP6x3qp/PjiuHCOQMlzb/nef8hC00KzRQeO+KXGuagYGM5Ff7bsDyhORAsYSOukoN62t1R0Rpeo5fpBUuGw3wgxGpwfjrXkvV9sAmtPFm1TzKxxlXdZ0D5AzefNCbbhf6wJHLsTizZ/IpU X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(8121501046)(5005006)(93006095)(93001095)(3002001)(10201501046)(3231023)(944501119)(6055026)(6041268)(20161123562045)(20161123558120)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:CY4PR12MB1509; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:CY4PR12MB1509; X-Forefront-PRVS: 0548586081 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(366004)(39860400002)(396003)(346002)(376002)(39380400002)(189003)(199004)(25786009)(8676002)(81166006)(47776003)(48376002)(6666003)(2950100002)(81156014)(6916009)(66066001)(50226002)(68736007)(36756003)(8936002)(16586007)(72206003)(50466002)(16526018)(316002)(53936002)(478600001)(5660300001)(86362001)(575784001)(4326008)(386003)(2361001)(2351001)(6486002)(53416004)(97736004)(51416003)(76176011)(106356001)(105586002)(7736002)(6116002)(3846002)(2906002)(59450400001)(305945005)(7696005)(52116002); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR12MB1509; H:wallaby-smavila.amd.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR12MB1509; 23:aYZVqX6AKnhIu34m13LvhZT7Gd4nUdEk/0qw27amv?= 4IL9UFwbrtrR1AWKmtBLehQThE71f6tKAi3Vph56XcKARTk0KIB4G6ZWfnyZ5QqA5oU4a81PsZzKah755kktLiNk5m29Jw0Y/35F6qtoubguY9VKjJVS9TvPsHTnjVNhC7WFX+eDYe2JlFUbKzjQO85DsvJyl2PAOx0+0dpGciTBYw/BXmZd+dMO+t+ogYuroS3qM4gxxloxxPF5KkK60f6AgFeoHqTLEPKdDfLDB9HNaA8adpTXLKvjj8sD7g5KHrOMrVviMuPRVKScuhj9rsu1VQE6+B+Y4XNU+yT7sQIJfZKk9EFpHNlh9tLtcWubJzyw6RAEeJL5C+t8uAvYgQiqAulLas24OngpZVu6qHAo/JlaQK77B9dDgoPKLvAs5XyAou5vmBLWFVh77Fufsg1FgsXtyJyOPvrTC/1UvmoOxGRiic4raXw7n3WPuch3QUk/GgYpTGx2qf0YgWi+uXK0p5l1WxFE4gwf/VY6+w6/2gLCP2oGhA//ZZgCvotAdPtXjRcQNhv+oQXGHl/DTCXRcRoYCP+lqHcWsRvjmwXNDph5Jin7bDtD9yoHBHmEQ5yQiur8f2CpPZ3IFHlG5k3NokLU6IGh8OpkC1LFWUFWripKcWQJy/Y6yMtOGIYAimsl/iIm3aWqPsQNnrs1hkMsfpE1gYBT9q7SfKzubIousqLrI/dqGecODudH+c+h7SDLsbsrnkevc3lOr6JOC1wm1aHb7PicFqbGN9MQmgSvrCefviBeMv3rgGfKi9B03zR+sCvw7iDHv+Q4B+fCrgMVMJHwiuOZ3p/dUbtw3WIPBfbDP9dMxxphQwJR5+ZlsTcMWNDJ9jiNOK6vxHNKAvubCT6Vc5NKCeGqdfIlcAod0yEsANJf6QFdqpnTY/Tiq6AdSdg8gnCddN/5lq3zZA/FyRZ68I5Lxo5SJ2W0aA3FVzZMxitCiAERCwklP72CUcQzsRKH2I+Rh0yrltbph00xX/lw1AWGX9QGvhUdoD0ng0AYGjqDvZuquNHoYzolJZyAArWFQ4uNpDx5jlIH/MuG/SDq9iZjTekViwxMjBUjhhR4h7hINF6nlce8Fs7fY5IOXNC2h1sM7SiKDGkOx/xoev6iFJUCdeEBCPTlWK4wgMPThcftEbKGD7esV7O45g= X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1509; 6:A/4F8DPSPW8ZlB3/WA2IypgbSneoRur/DYnh6plW/ydtbyOa6Mu54yMgMub/eYwZpQZNZW0tlYAtyKp2DsfuKUkN5sNkpYrYLju8oQT781EP0DrieNn/BfryBnyv2JJDXtDRWwe8YqMxW8enPcGhUOso7bGU3/sCWFuHQHfyVDwO0u5epgEuQIROMlPzFuOc4jHlKAy8DTHQRW2yqez3F8oUF6wCMmyIqnzIP8j9tZmHR4yKK63U8LF2PgAZJYn0a6C4wqVi+qsf4rmHDJ0JMrqvohu3l+EmPHQNHtyiaZsv4OV+omkzPZObgrzl5qPvRw8e2RB3LRilvSB5tRrtKbi3dJYnJ9dDDg1RKjvzgDw=; 5:jYyprBf2F2XCLhAzQgfc8eY7B3RvDPND9Z8tanDOodo2GvuclnROHGm/TLydSRpH2UgA6gy9v3s48+UXCS8pmSn0fHwXcY4jGgiDKvtW3J1ym+WLOaZQf3fJAt3GjWe9upsqZI3filC3dxnhBkX/rnBKSpzpPkr2erpX7dIMeZ0=; 24:YEB9nimiqINybIzl1LYmxjieltMjXFVnMV7FwUnL6tAhaC5UDeN7VaPYWtCuOfMH8JjWGpMGP7l+kOcP05TinSLkqzqUtLHiE0XfgQrqQX8=; 7:59aNQtNUJTzF5pTFKkJoxrWGCj1QXTmqaijFsQVwno8e/oH6MmeMGMVaDHybEzw8CU62ykm82gNLv7GQJ+BbKzsHLA4K3784qHhnXQ5psjM4BTqKF0h9FgRI6JcRRFps27osjWizUYQRCVSaFfgFWVre8tMycoqfgv7eT4paTyYJhRe2xBgIS1fmuQQM8Lh9ZPkia1NTsDsVLKv5eIiax+rODB2CGxmdbH/MT/8bkdjwczhPFMnZ5ZYESlaF0ipn SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR12MB1509; 20:uDnkxvux8LC9MQlDZEOk2WoQkLF+x4TC0zC/aX1v0kW/UGvf9RXd80fQxeu5l249ps0I+mQw4iLwpT4GD1oTI4HeugLXloWM6IXs+o0y1me0N9b6Bo6oWhLWlmuI+FCI82VSrwPuZG/eJio3yln1W5WvlD0xHXtE+I/EqOwqlTtySUTRrZFRp3IC+i5Q/vLbfoEAVbVJQ7lkY45N78muCePGrBaVZKH4WOmkzZksM5+vHq1CLr5J1a41akYVypn0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2018 09:43:29.0998 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bbe37f1e-acbc-4907-9e0b-08d5580e9b3f X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1509 Subject: [dpdk-dev] [PATCH v3 06/19] crypto/ccp: support crypto enqueue and dequeue burst api X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Signed-off-by: Ravi Kumar --- drivers/crypto/ccp/ccp_crypto.c | 360 +++++++++++++++++++++++++++++++++++++++ drivers/crypto/ccp/ccp_crypto.h | 35 ++++ drivers/crypto/ccp/ccp_dev.c | 27 +++ drivers/crypto/ccp/ccp_dev.h | 9 + drivers/crypto/ccp/rte_ccp_pmd.c | 64 ++++++- 5 files changed, 488 insertions(+), 7 deletions(-) diff --git a/drivers/crypto/ccp/ccp_crypto.c b/drivers/crypto/ccp/ccp_crypto.c index c365c0f..c17e84f 100644 --- a/drivers/crypto/ccp/ccp_crypto.c +++ b/drivers/crypto/ccp/ccp_crypto.c @@ -227,3 +227,363 @@ ccp_set_session_parameters(struct ccp_session *sess, } return ret; } + +/* calculate CCP descriptors requirement */ +static inline int +ccp_cipher_slot(struct ccp_session *session) +{ + int count = 0; + + switch (session->cipher.algo) { + default: + CCP_LOG_ERR("Unsupported cipher algo %d", + session->cipher.algo); + } + return count; +} + +static inline int +ccp_auth_slot(struct ccp_session *session) +{ + int count = 0; + + switch (session->auth.algo) { + default: + CCP_LOG_ERR("Unsupported auth algo %d", + session->auth.algo); + } + + return count; +} + +static int +ccp_aead_slot(struct ccp_session *session) +{ + int count = 0; + + switch (session->aead_algo) { + default: + CCP_LOG_ERR("Unsupported aead algo %d", + session->aead_algo); + } + return count; +} + +int +ccp_compute_slot_count(struct ccp_session *session) +{ + int count = 0; + + switch (session->cmd_id) { + case CCP_CMD_CIPHER: + count = ccp_cipher_slot(session); + break; + case CCP_CMD_AUTH: + count = ccp_auth_slot(session); + break; + case CCP_CMD_CIPHER_HASH: + case CCP_CMD_HASH_CIPHER: + count = ccp_cipher_slot(session); + count += ccp_auth_slot(session); + break; + case CCP_CMD_COMBINED: + count = ccp_aead_slot(session); + break; + default: + CCP_LOG_ERR("Unsupported cmd_id"); + + } + + return count; +} + +static inline int +ccp_crypto_cipher(struct rte_crypto_op *op, + struct ccp_queue *cmd_q __rte_unused, + struct ccp_batch_info *b_info __rte_unused) +{ + int result = 0; + struct ccp_session *session; + + session = (struct ccp_session *)get_session_private_data( + op->sym->session, + ccp_cryptodev_driver_id); + + switch (session->cipher.algo) { + default: + CCP_LOG_ERR("Unsupported cipher algo %d", + session->cipher.algo); + return -ENOTSUP; + } + return result; +} + +static inline int +ccp_crypto_auth(struct rte_crypto_op *op, + struct ccp_queue *cmd_q __rte_unused, + struct ccp_batch_info *b_info __rte_unused) +{ + + int result = 0; + struct ccp_session *session; + + session = (struct ccp_session *)get_session_private_data( + op->sym->session, + ccp_cryptodev_driver_id); + + switch (session->auth.algo) { + default: + CCP_LOG_ERR("Unsupported auth algo %d", + session->auth.algo); + return -ENOTSUP; + } + + return result; +} + +static inline int +ccp_crypto_aead(struct rte_crypto_op *op, + struct ccp_queue *cmd_q __rte_unused, + struct ccp_batch_info *b_info __rte_unused) +{ + int result = 0; + struct ccp_session *session; + + session = (struct ccp_session *)get_session_private_data( + op->sym->session, + ccp_cryptodev_driver_id); + + switch (session->aead_algo) { + default: + CCP_LOG_ERR("Unsupported aead algo %d", + session->aead_algo); + return -ENOTSUP; + } + return result; +} + +int +process_ops_to_enqueue(const struct ccp_qp *qp, + struct rte_crypto_op **op, + struct ccp_queue *cmd_q, + uint16_t nb_ops, + int slots_req) +{ + int i, result = 0; + struct ccp_batch_info *b_info; + struct ccp_session *session; + + if (rte_mempool_get(qp->batch_mp, (void **)&b_info)) { + CCP_LOG_ERR("batch info allocation failed"); + return 0; + } + /* populate batch info necessary for dequeue */ + b_info->op_idx = 0; + b_info->lsb_buf_idx = 0; + b_info->desccnt = 0; + b_info->cmd_q = cmd_q; + b_info->lsb_buf_phys = + (phys_addr_t)rte_mem_virt2phy((void *)b_info->lsb_buf); + rte_atomic64_sub(&b_info->cmd_q->free_slots, slots_req); + + b_info->head_offset = (uint32_t)(cmd_q->qbase_phys_addr + cmd_q->qidx * + Q_DESC_SIZE); + for (i = 0; i < nb_ops; i++) { + session = (struct ccp_session *)get_session_private_data( + op[i]->sym->session, + ccp_cryptodev_driver_id); + switch (session->cmd_id) { + case CCP_CMD_CIPHER: + result = ccp_crypto_cipher(op[i], cmd_q, b_info); + break; + case CCP_CMD_AUTH: + result = ccp_crypto_auth(op[i], cmd_q, b_info); + break; + case CCP_CMD_CIPHER_HASH: + result = ccp_crypto_cipher(op[i], cmd_q, b_info); + if (result) + break; + result = ccp_crypto_auth(op[i], cmd_q, b_info); + break; + case CCP_CMD_HASH_CIPHER: + result = ccp_crypto_auth(op[i], cmd_q, b_info); + if (result) + break; + result = ccp_crypto_cipher(op[i], cmd_q, b_info); + break; + case CCP_CMD_COMBINED: + result = ccp_crypto_aead(op[i], cmd_q, b_info); + break; + default: + CCP_LOG_ERR("Unsupported cmd_id"); + result = -1; + } + if (unlikely(result < 0)) { + rte_atomic64_add(&b_info->cmd_q->free_slots, + (slots_req - b_info->desccnt)); + break; + } + b_info->op[i] = op[i]; + } + + b_info->opcnt = i; + b_info->tail_offset = (uint32_t)(cmd_q->qbase_phys_addr + cmd_q->qidx * + Q_DESC_SIZE); + + rte_wmb(); + /* Write the new tail address back to the queue register */ + CCP_WRITE_REG(cmd_q->reg_base, CMD_Q_TAIL_LO_BASE, + b_info->tail_offset); + /* Turn the queue back on using our cached control register */ + CCP_WRITE_REG(cmd_q->reg_base, CMD_Q_CONTROL_BASE, + cmd_q->qcontrol | CMD_Q_RUN); + + rte_ring_enqueue(qp->processed_pkts, (void *)b_info); + + return i; +} + +static inline void ccp_auth_dq_prepare(struct rte_crypto_op *op) +{ + struct ccp_session *session; + uint8_t *digest_data, *addr; + struct rte_mbuf *m_last; + int offset, digest_offset; + uint8_t digest_le[64]; + + session = (struct ccp_session *)get_session_private_data( + op->sym->session, + ccp_cryptodev_driver_id); + + if (session->cmd_id == CCP_CMD_COMBINED) { + digest_data = op->sym->aead.digest.data; + digest_offset = op->sym->aead.data.offset + + op->sym->aead.data.length; + } else { + digest_data = op->sym->auth.digest.data; + digest_offset = op->sym->auth.data.offset + + op->sym->auth.data.length; + } + m_last = rte_pktmbuf_lastseg(op->sym->m_src); + addr = (uint8_t *)((char *)m_last->buf_addr + m_last->data_off + + m_last->data_len - session->auth.ctx_len); + + rte_mb(); + offset = session->auth.offset; + + if (session->auth.engine == CCP_ENGINE_SHA) + if ((session->auth.ut.sha_type != CCP_SHA_TYPE_1) && + (session->auth.ut.sha_type != CCP_SHA_TYPE_224) && + (session->auth.ut.sha_type != CCP_SHA_TYPE_256)) { + /* All other algorithms require byte + * swap done by host + */ + unsigned int i; + + offset = session->auth.ctx_len - + session->auth.offset - 1; + for (i = 0; i < session->auth.digest_length; i++) + digest_le[i] = addr[offset - i]; + offset = 0; + addr = digest_le; + } + + op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + if (session->auth.op == CCP_AUTH_OP_VERIFY) { + if (memcmp(addr + offset, digest_data, + session->auth.digest_length) != 0) + op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; + + } else { + if (unlikely(digest_data == 0)) + digest_data = rte_pktmbuf_mtod_offset( + op->sym->m_dst, uint8_t *, + digest_offset); + rte_memcpy(digest_data, addr + offset, + session->auth.digest_length); + } + /* Trim area used for digest from mbuf. */ + rte_pktmbuf_trim(op->sym->m_src, + session->auth.ctx_len); +} + +static int +ccp_prepare_ops(struct rte_crypto_op **op_d, + struct ccp_batch_info *b_info, + uint16_t nb_ops) +{ + int i, min_ops; + struct ccp_session *session; + + min_ops = RTE_MIN(nb_ops, b_info->opcnt); + + for (i = 0; i < min_ops; i++) { + op_d[i] = b_info->op[b_info->op_idx++]; + session = (struct ccp_session *)get_session_private_data( + op_d[i]->sym->session, + ccp_cryptodev_driver_id); + switch (session->cmd_id) { + case CCP_CMD_CIPHER: + op_d[i]->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + break; + case CCP_CMD_AUTH: + case CCP_CMD_CIPHER_HASH: + case CCP_CMD_HASH_CIPHER: + case CCP_CMD_COMBINED: + ccp_auth_dq_prepare(op_d[i]); + break; + default: + CCP_LOG_ERR("Unsupported cmd_id"); + } + } + + b_info->opcnt -= min_ops; + return min_ops; +} + +int +process_ops_to_dequeue(struct ccp_qp *qp, + struct rte_crypto_op **op, + uint16_t nb_ops) +{ + struct ccp_batch_info *b_info; + uint32_t cur_head_offset; + + if (qp->b_info != NULL) { + b_info = qp->b_info; + if (unlikely(b_info->op_idx > 0)) + goto success; + } else if (rte_ring_dequeue(qp->processed_pkts, + (void **)&b_info)) + return 0; + cur_head_offset = CCP_READ_REG(b_info->cmd_q->reg_base, + CMD_Q_HEAD_LO_BASE); + + if (b_info->head_offset < b_info->tail_offset) { + if ((cur_head_offset >= b_info->head_offset) && + (cur_head_offset < b_info->tail_offset)) { + qp->b_info = b_info; + return 0; + } + } else { + if ((cur_head_offset >= b_info->head_offset) || + (cur_head_offset < b_info->tail_offset)) { + qp->b_info = b_info; + return 0; + } + } + + +success: + nb_ops = ccp_prepare_ops(op, b_info, nb_ops); + rte_atomic64_add(&b_info->cmd_q->free_slots, b_info->desccnt); + b_info->desccnt = 0; + if (b_info->opcnt > 0) { + qp->b_info = b_info; + } else { + rte_mempool_put(qp->batch_mp, (void *)b_info); + qp->b_info = NULL; + } + + return nb_ops; +} diff --git a/drivers/crypto/ccp/ccp_crypto.h b/drivers/crypto/ccp/ccp_crypto.h index 346d5ee..4455497 100644 --- a/drivers/crypto/ccp/ccp_crypto.h +++ b/drivers/crypto/ccp/ccp_crypto.h @@ -264,4 +264,39 @@ struct ccp_qp; int ccp_set_session_parameters(struct ccp_session *sess, const struct rte_crypto_sym_xform *xform); +/** + * Find count of slots + * + * @param session CCP private session + * @return count of free slots available + */ +int ccp_compute_slot_count(struct ccp_session *session); + +/** + * process crypto ops to be enqueued + * + * @param qp CCP crypto queue-pair + * @param op crypto ops table + * @param cmd_q CCP cmd queue + * @param nb_ops No. of ops to be submitted + * @return 0 on success otherwise -1 + */ +int process_ops_to_enqueue(const struct ccp_qp *qp, + struct rte_crypto_op **op, + struct ccp_queue *cmd_q, + uint16_t nb_ops, + int slots_req); + +/** + * process crypto ops to be dequeued + * + * @param qp CCP crypto queue-pair + * @param op crypto ops table + * @param nb_ops requested no. of ops + * @return 0 on success otherwise -1 + */ +int process_ops_to_dequeue(struct ccp_qp *qp, + struct rte_crypto_op **op, + uint16_t nb_ops); + #endif /* _CCP_CRYPTO_H_ */ diff --git a/drivers/crypto/ccp/ccp_dev.c b/drivers/crypto/ccp/ccp_dev.c index 57bccf4..fee90e3 100644 --- a/drivers/crypto/ccp/ccp_dev.c +++ b/drivers/crypto/ccp/ccp_dev.c @@ -61,6 +61,33 @@ ccp_dev_start(struct rte_cryptodev *dev) return 0; } +struct ccp_queue * +ccp_allot_queue(struct rte_cryptodev *cdev, int slot_req) +{ + int i, ret = 0; + struct ccp_device *dev; + struct ccp_private *priv = cdev->data->dev_private; + + dev = TAILQ_NEXT(priv->last_dev, next); + if (unlikely(dev == NULL)) + dev = TAILQ_FIRST(&ccp_list); + priv->last_dev = dev; + if (dev->qidx >= dev->cmd_q_count) + dev->qidx = 0; + ret = rte_atomic64_read(&dev->cmd_q[dev->qidx].free_slots); + if (ret >= slot_req) + return &dev->cmd_q[dev->qidx]; + for (i = 0; i < dev->cmd_q_count; i++) { + dev->qidx++; + if (dev->qidx >= dev->cmd_q_count) + dev->qidx = 0; + ret = rte_atomic64_read(&dev->cmd_q[dev->qidx].free_slots); + if (ret >= slot_req) + return &dev->cmd_q[dev->qidx]; + } + return NULL; +} + static const struct rte_memzone * ccp_queue_dma_zone_reserve(const char *queue_name, uint32_t queue_size, diff --git a/drivers/crypto/ccp/ccp_dev.h b/drivers/crypto/ccp/ccp_dev.h index a16ba81..cfb3b03 100644 --- a/drivers/crypto/ccp/ccp_dev.h +++ b/drivers/crypto/ccp/ccp_dev.h @@ -445,4 +445,13 @@ int ccp_dev_start(struct rte_cryptodev *dev); */ int ccp_probe_devices(const struct rte_pci_id *ccp_id); +/** + * allocate a ccp command queue + * + * @dev rte crypto device + * @param slot_req number of required + * @return allotted CCP queue on success otherwise NULL + */ +struct ccp_queue *ccp_allot_queue(struct rte_cryptodev *dev, int slot_req); + #endif /* _CCP_DEV_H_ */ diff --git a/drivers/crypto/ccp/rte_ccp_pmd.c b/drivers/crypto/ccp/rte_ccp_pmd.c index cc35a97..ed6ca5d 100644 --- a/drivers/crypto/ccp/rte_ccp_pmd.c +++ b/drivers/crypto/ccp/rte_ccp_pmd.c @@ -38,6 +38,7 @@ #include #include +#include "ccp_crypto.h" #include "ccp_dev.h" #include "ccp_pmd_private.h" @@ -47,23 +48,72 @@ static unsigned int ccp_pmd_init_done; uint8_t ccp_cryptodev_driver_id; +static struct ccp_session * +get_ccp_session(struct ccp_qp *qp __rte_unused, struct rte_crypto_op *op) +{ + struct ccp_session *sess = NULL; + + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + if (unlikely(op->sym->session == NULL)) + return NULL; + + sess = (struct ccp_session *) + get_session_private_data( + op->sym->session, + ccp_cryptodev_driver_id); + } + + return sess; +} + static uint16_t -ccp_pmd_enqueue_burst(void *queue_pair __rte_unused, - struct rte_crypto_op **ops __rte_unused, - uint16_t nb_ops __rte_unused) +ccp_pmd_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, + uint16_t nb_ops) { - uint16_t enq_cnt = 0; + struct ccp_session *sess = NULL; + struct ccp_qp *qp = queue_pair; + struct ccp_queue *cmd_q; + struct rte_cryptodev *dev = qp->dev; + uint16_t i, enq_cnt = 0, slots_req = 0; + + if (nb_ops == 0) + return 0; + + if (unlikely(rte_ring_full(qp->processed_pkts) != 0)) + return 0; + + for (i = 0; i < nb_ops; i++) { + sess = get_ccp_session(qp, ops[i]); + if (unlikely(sess == NULL) && (i == 0)) { + qp->qp_stats.enqueue_err_count++; + return 0; + } else if (sess == NULL) { + nb_ops = i; + break; + } + slots_req += ccp_compute_slot_count(sess); + } + + cmd_q = ccp_allot_queue(dev, slots_req); + if (unlikely(cmd_q == NULL)) + return 0; + enq_cnt = process_ops_to_enqueue(qp, ops, cmd_q, nb_ops, slots_req); + qp->qp_stats.enqueued_count += enq_cnt; return enq_cnt; } static uint16_t -ccp_pmd_dequeue_burst(void *queue_pair __rte_unused, - struct rte_crypto_op **ops __rte_unused, - uint16_t nb_ops __rte_unused) +ccp_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, + uint16_t nb_ops) { + struct ccp_qp *qp = queue_pair; uint16_t nb_dequeued = 0; + nb_dequeued = process_ops_to_dequeue(qp, ops, nb_ops); + + qp->qp_stats.dequeued_count += nb_dequeued; + return nb_dequeued; }