Patch Detail

HTTP 200 OK
Allow: GET, PUT, PATCH, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "id": 48148,
    "url": "http://patchwork.dpdk.org/api/patches/48148/?format=api",
    "web_url": "http://patchwork.dpdk.org/project/dpdk/patch/1542326031-5263-7-git-send-email-konstantin.ananyev@intel.com/",
    "project": {
        "id": 1,
        "url": "http://patchwork.dpdk.org/api/projects/1/?format=api",
        "name": "DPDK",
        "link_name": "dpdk",
        "list_id": "dev.dpdk.org",
        "list_email": "dev@dpdk.org",
        "web_url": "http://core.dpdk.org",
        "scm_url": "git://dpdk.org/dpdk",
        "webscm_url": "http://git.dpdk.org/dpdk",
        "list_archive_url": "https://inbox.dpdk.org/dev",
        "list_archive_url_format": "https://inbox.dpdk.org/dev/{}",
        "commit_url_format": ""
    },
    "msgid": "<1542326031-5263-7-git-send-email-konstantin.ananyev@intel.com>",
    "list_archive_url": "https://inbox.dpdk.org/dev/1542326031-5263-7-git-send-email-konstantin.ananyev@intel.com",
    "date": "2018-11-15T23:53:48",
    "name": "[6/9] ipsec: implement SA data-path API",
    "commit_ref": null,
    "pull_url": null,
    "state": "superseded",
    "archived": true,
    "hash": "fb6b66179a70f18ee522d06649e4b8146733434f",
    "submitter": {
        "id": 33,
        "url": "http://patchwork.dpdk.org/api/people/33/?format=api",
        "name": "Ananyev, Konstantin",
        "email": "konstantin.ananyev@intel.com"
    },
    "delegate": {
        "id": 1,
        "url": "http://patchwork.dpdk.org/api/users/1/?format=api",
        "username": "tmonjalo",
        "first_name": "Thomas",
        "last_name": "Monjalon",
        "email": "thomas@monjalon.net"
    },
    "mbox": "http://patchwork.dpdk.org/project/dpdk/patch/1542326031-5263-7-git-send-email-konstantin.ananyev@intel.com/mbox/",
    "series": [
        {
            "id": 2456,
            "url": "http://patchwork.dpdk.org/api/series/2456/?format=api",
            "web_url": "http://patchwork.dpdk.org/project/dpdk/list/?series=2456",
            "date": "2018-11-15T23:53:48",
            "name": null,
            "version": 1,
            "mbox": "http://patchwork.dpdk.org/series/2456/mbox/"
        }
    ],
    "comments": "http://patchwork.dpdk.org/api/patches/48148/comments/",
    "check": "fail",
    "checks": "http://patchwork.dpdk.org/api/patches/48148/checks/",
    "tags": {},
    "related": [],
    "headers": {
        "Return-Path": "<dev-bounces@dpdk.org>",
        "X-Original-To": "patchwork@dpdk.org",
        "Delivered-To": "patchwork@dpdk.org",
        "Received": [
            "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 497865681;\n\tFri, 16 Nov 2018 00:54:17 +0100 (CET)",
            "from mga09.intel.com (mga09.intel.com [134.134.136.24])\n\tby dpdk.org (Postfix) with ESMTP id C11884CB5\n\tfor <dev@dpdk.org>; Fri, 16 Nov 2018 00:54:08 +0100 (CET)",
            "from orsmga007.jf.intel.com ([10.7.209.58])\n\tby orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t15 Nov 2018 15:54:08 -0800",
            "from sivswdev08.ir.intel.com (HELO localhost.localdomain)\n\t([10.237.217.47])\n\tby orsmga007.jf.intel.com with ESMTP; 15 Nov 2018 15:54:07 -0800"
        ],
        "X-Amp-Result": "SKIPPED(no attachment in message)",
        "X-Amp-File-Uploaded": "False",
        "X-ExtLoop1": "1",
        "X-IronPort-AV": "E=Sophos;i=\"5.56,238,1539673200\"; d=\"scan'208\";a=\"89697381\"",
        "From": "Konstantin Ananyev <konstantin.ananyev@intel.com>",
        "To": "dev@dpdk.org",
        "Cc": "Konstantin Ananyev <konstantin.ananyev@intel.com>,\n\tMohammad Abdul Awal <mohammad.abdul.awal@intel.com>",
        "Date": "Thu, 15 Nov 2018 23:53:48 +0000",
        "Message-Id": "<1542326031-5263-7-git-send-email-konstantin.ananyev@intel.com>",
        "X-Mailer": "git-send-email 1.7.0.7",
        "In-Reply-To": "<1535129598-27301-1-git-send-email-konstantin.ananyev@intel.com>",
        "References": "<1535129598-27301-1-git-send-email-konstantin.ananyev@intel.com>",
        "Subject": "[dpdk-dev] [PATCH 6/9] ipsec: implement SA data-path API",
        "X-BeenThere": "dev@dpdk.org",
        "X-Mailman-Version": "2.1.15",
        "Precedence": "list",
        "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
        "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>",
        "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
        "List-Post": "<mailto:dev@dpdk.org>",
        "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
        "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>",
        "Errors-To": "dev-bounces@dpdk.org",
        "Sender": "\"dev\" <dev-bounces@dpdk.org>"
    },
    "content": "Provide implementation for rte_ipsec_pkt_crypto_prepare() and\nrte_ipsec_pkt_process().\nCurrent implementation:\n - supports ESP protocol tunnel mode.\n - supports ESP protocol transport mode.\n - supports ESN and replay window.\n - supports algorithms: AES-CBC, AES-GCM, HMAC-SHA1, NULL.\n - covers all currently defined security session types:\n        - RTE_SECURITY_ACTION_TYPE_NONE\n        - RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO\n        - RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL\n        - RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL\n\nFor first two types SQN check/update is done by SW (inside the library).\nFor last two type it is HW/PMD responsibility.\n\nSigned-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>\nSigned-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>\n---\n lib/librte_ipsec/crypto.h    |  119 ++++\n lib/librte_ipsec/iph.h       |   63 ++\n lib/librte_ipsec/ipsec_sqn.h |  186 ++++++\n lib/librte_ipsec/pad.h       |   45 ++\n lib/librte_ipsec/sa.c        | 1050 +++++++++++++++++++++++++++++++++-\n 5 files changed, 1461 insertions(+), 2 deletions(-)\n create mode 100644 lib/librte_ipsec/crypto.h\n create mode 100644 lib/librte_ipsec/iph.h\n create mode 100644 lib/librte_ipsec/pad.h",
    "diff": "diff --git a/lib/librte_ipsec/crypto.h b/lib/librte_ipsec/crypto.h\nnew file mode 100644\nindex 000000000..98f9989af\n--- /dev/null\n+++ b/lib/librte_ipsec/crypto.h\n@@ -0,0 +1,119 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#ifndef _CRYPTO_H_\n+#define _CRYPTO_H_\n+\n+/**\n+ * @file crypto.h\n+ * Contains crypto specific functions/structures/macros used internally\n+ * by ipsec library.\n+ */\n+\n+ /*\n+  * AES-GCM devices have some specific requirements for IV and AAD formats.\n+  * Ideally that to be done by the driver itself.\n+  */\n+\n+struct aead_gcm_iv {\n+\tuint32_t salt;\n+\tuint64_t iv;\n+\tuint32_t cnt;\n+} __attribute__((packed));\n+\n+struct aead_gcm_aad {\n+\tuint32_t spi;\n+\t/*\n+\t * RFC 4106, section 5:\n+\t * Two formats of the AAD are defined:\n+\t * one for 32-bit sequence numbers, and one for 64-bit ESN.\n+\t */\n+\tunion {\n+\t\tuint32_t u32;\n+\t\tuint64_t u64;\n+\t} sqn;\n+\tuint32_t align0; /* align to 16B boundary */\n+} __attribute__((packed));\n+\n+struct gcm_esph_iv {\n+\tstruct esp_hdr esph;\n+\tuint64_t iv;\n+} __attribute__((packed));\n+\n+\n+static inline void\n+aead_gcm_iv_fill(struct aead_gcm_iv *gcm, uint64_t iv, uint32_t salt)\n+{\n+\tgcm->salt = salt;\n+\tgcm->iv = iv;\n+\tgcm->cnt = rte_cpu_to_be_32(1);\n+}\n+\n+/*\n+ * RFC 4106, 5 AAD Construction\n+ * spi and sqn should already be converted into network byte order.\n+ */\n+static inline void\n+aead_gcm_aad_fill(struct aead_gcm_aad *aad, rte_be32_t spi, rte_be64_t sqn,\n+\tint esn)\n+{\n+\taad->spi = spi;\n+\tif (esn)\n+\t\taad->sqn.u64 = sqn;\n+\telse\n+\t\taad->sqn.u32 = sqn_low32(sqn);\n+}\n+\n+static inline void\n+gen_iv(uint64_t iv[IPSEC_MAX_IV_QWORD], rte_be64_t sqn)\n+{\n+\tiv[0] = sqn;\n+\tiv[1] = 0;\n+}\n+\n+/*\n+ * from RFC 4303 3.3.2.1.4:\n+ * If the ESN option is enabled for the SA, the high-order 32\n+ * bits of the sequence number are appended after the Next Header field\n+ * for purposes of this computation, but are not transmitted.\n+ */\n+\n+/*\n+ * Helper function that moves ICV by 4B below, and inserts SQN.hibits.\n+ * icv parameter points to the new start of ICV.\n+ */\n+static inline void\n+insert_sqh(uint32_t sqh, void *picv, uint32_t icv_len)\n+{\n+\tuint32_t *icv;\n+\tint32_t i;\n+\n+\tRTE_ASSERT(icv_len % sizeof(uint32_t) == 0);\n+\n+\ticv = picv;\n+\ticv_len = icv_len / sizeof(uint32_t);\n+\tfor (i = icv_len; i-- != 0; icv[i] = icv[i - 1])\n+\t\t;\n+\n+\ticv[i] = sqh;\n+}\n+\n+/*\n+ * Helper function that moves ICV by 4B up, and removes SQN.hibits.\n+ * icv parameter points to the new start of ICV.\n+ */\n+static inline void\n+remove_sqh(void *picv, uint32_t icv_len)\n+{\n+\tuint32_t i, *icv;\n+\n+\tRTE_ASSERT(icv_len % sizeof(uint32_t) == 0);\n+\n+\ticv = picv;\n+\ticv_len = icv_len / sizeof(uint32_t);\n+\tfor (i = 0; i != icv_len; i++)\n+\t\ticv[i] = icv[i + 1];\n+}\n+\n+#endif /* _CRYPTO_H_ */\ndiff --git a/lib/librte_ipsec/iph.h b/lib/librte_ipsec/iph.h\nnew file mode 100644\nindex 000000000..c85bd2866\n--- /dev/null\n+++ b/lib/librte_ipsec/iph.h\n@@ -0,0 +1,63 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#ifndef _IPH_H_\n+#define _IPH_H_\n+\n+/**\n+ * @file iph.h\n+ * Contains functions/structures/macros to manipulate IPv/IPv6 headers\n+ * used internally by ipsec library.\n+ */\n+\n+/*\n+ * Move preceding (L3) headers down to remove ESP header and IV.\n+ */\n+static inline void\n+remove_esph(char *np, char *op, uint32_t hlen)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = hlen; i-- != 0; np[i] = op[i])\n+\t\t;\n+}\n+\n+/*\n+ * Move preceding (L3) headers up to free space for ESP header and IV.\n+ */\n+static inline void\n+insert_esph(char *np, char *op, uint32_t hlen)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i != hlen; i++)\n+\t\tnp[i] = op[i];\n+}\n+\n+static inline int\n+update_trs_l3hdr(const struct rte_ipsec_sa *sa, void *p, uint32_t plen,\n+\t\tuint32_t l2len, uint32_t l3len, uint8_t proto)\n+{\n+\tstruct ipv4_hdr *v4h;\n+\tstruct ipv6_hdr *v6h;\n+\tint32_t rc;\n+\n+\tif ((sa->type & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {\n+\t\tv4h = p;\n+\t\trc = v4h->next_proto_id;\n+\t\tv4h->next_proto_id = proto;\n+\t\tv4h->total_length = rte_cpu_to_be_16(plen - l2len);\n+\t} else if (l3len == sizeof(*v6h)) {\n+\t\tv6h = p;\n+\t\trc = v6h->proto;\n+\t\tv6h->proto = proto;\n+\t\tv6h->payload_len = rte_cpu_to_be_16(plen - l2len - l3len);\n+\t/* need to add support for IPv6 with options */\n+\t} else\n+\t\trc = -ENOTSUP;\n+\n+\treturn rc;\n+}\n+\n+#endif /* _IPH_H_ */\ndiff --git a/lib/librte_ipsec/ipsec_sqn.h b/lib/librte_ipsec/ipsec_sqn.h\nindex 4471814f9..a33ff9cca 100644\n--- a/lib/librte_ipsec/ipsec_sqn.h\n+++ b/lib/librte_ipsec/ipsec_sqn.h\n@@ -15,6 +15,45 @@\n \n #define IS_ESN(sa)\t((sa)->sqn_mask == UINT64_MAX)\n \n+/*\n+ * gets SQN.hi32 bits, SQN supposed to be in network byte order.\n+ */\n+static inline rte_be32_t\n+sqn_hi32(rte_be64_t sqn)\n+{\n+#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN\n+\treturn (sqn >> 32);\n+#else\n+\treturn sqn;\n+#endif\n+}\n+\n+/*\n+ * gets SQN.low32 bits, SQN supposed to be in network byte order.\n+ */\n+static inline rte_be32_t\n+sqn_low32(rte_be64_t sqn)\n+{\n+#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN\n+\treturn sqn;\n+#else\n+\treturn (sqn >> 32);\n+#endif\n+}\n+\n+/*\n+ * gets SQN.low16 bits, SQN supposed to be in network byte order.\n+ */\n+static inline rte_be16_t\n+sqn_low16(rte_be64_t sqn)\n+{\n+#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN\n+\treturn sqn;\n+#else\n+\treturn (sqn >> 48);\n+#endif\n+}\n+\n /*\n  * for given size, calculate required number of buckets.\n  */\n@@ -30,6 +69,153 @@ replay_num_bucket(uint32_t wsz)\n \treturn nb;\n }\n \n+/*\n+ * According to RFC4303 A2.1, determine the high-order bit of sequence number.\n+ * use 32bit arithmetic inside, return uint64_t.\n+ */\n+static inline uint64_t\n+reconstruct_esn(uint64_t t, uint32_t sqn, uint32_t w)\n+{\n+\tuint32_t th, tl, bl;\n+\n+\ttl = t;\n+\tth = t >> 32;\n+\tbl = tl - w + 1;\n+\n+\t/* case A: window is within one sequence number subspace */\n+\tif (tl >= (w - 1))\n+\t\tth += (sqn < bl);\n+\t/* case B: window spans two sequence number subspaces */\n+\telse if (th != 0)\n+\t\tth -= (sqn >= bl);\n+\n+\t/* return constructed sequence with proper high-order bits */\n+\treturn (uint64_t)th << 32 | sqn;\n+}\n+\n+/**\n+ * Perform the replay checking.\n+ *\n+ * struct rte_ipsec_sa contains the window and window related parameters,\n+ * such as the window size, bitmask, and the last acknowledged sequence number.\n+ *\n+ * Based on RFC 6479.\n+ * Blocks are 64 bits unsigned integers\n+ */\n+static inline int32_t\n+esn_inb_check_sqn(const struct replay_sqn *rsn, const struct rte_ipsec_sa *sa,\n+\tuint64_t sqn)\n+{\n+\tuint32_t bit, bucket;\n+\n+\t/* replay not enabled */\n+\tif (sa->replay.win_sz == 0)\n+\t\treturn 0;\n+\n+\t/* seq is larger than lastseq */\n+\tif (sqn > rsn->sqn)\n+\t\treturn 0;\n+\n+\t/* seq is outside window */\n+\tif (sqn == 0 || sqn + sa->replay.win_sz < rsn->sqn)\n+\t\treturn -EINVAL;\n+\n+\t/* seq is inside the window */\n+\tbit = sqn & WINDOW_BIT_LOC_MASK;\n+\tbucket = (sqn >> WINDOW_BUCKET_BITS) & sa->replay.bucket_index_mask;\n+\n+\t/* already seen packet */\n+\tif (rsn->window[bucket] & ((uint64_t)1 << bit))\n+\t\treturn -EINVAL;\n+\n+\treturn 0;\n+}\n+\n+/**\n+ * For outbound SA perform the sequence number update.\n+ */\n+static inline uint64_t\n+esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num)\n+{\n+\tuint64_t n, s, sqn;\n+\n+\tn = *num;\n+\tsqn = sa->sqn.outb + n;\n+\tsa->sqn.outb = sqn;\n+\n+\t/* overflow */\n+\tif (sqn > sa->sqn_mask) {\n+\t\ts = sqn - sa->sqn_mask;\n+\t\t*num = (s < n) ?  n - s : 0;\n+\t}\n+\n+\treturn sqn - n;\n+}\n+\n+/**\n+ * For inbound SA perform the sequence number and replay window update.\n+ */\n+static inline int32_t\n+esn_inb_update_sqn(struct replay_sqn *rsn, const struct rte_ipsec_sa *sa,\n+\tuint64_t sqn)\n+{\n+\tuint32_t bit, bucket, last_bucket, new_bucket, diff, i;\n+\n+\t/* replay not enabled */\n+\tif (sa->replay.win_sz == 0)\n+\t\treturn 0;\n+\n+\t/* handle ESN */\n+\tif (IS_ESN(sa))\n+\t\tsqn = reconstruct_esn(rsn->sqn, sqn, sa->replay.win_sz);\n+\n+\t/* seq is outside window*/\n+\tif (sqn == 0 || sqn + sa->replay.win_sz < rsn->sqn)\n+\t\treturn -EINVAL;\n+\n+\t/* update the bit */\n+\tbucket = (sqn >> WINDOW_BUCKET_BITS);\n+\n+\t/* check if the seq is within the range */\n+\tif (sqn > rsn->sqn) {\n+\t\tlast_bucket = rsn->sqn >> WINDOW_BUCKET_BITS;\n+\t\tdiff = bucket - last_bucket;\n+\t\t/* seq is way after the range of WINDOW_SIZE */\n+\t\tif (diff > sa->replay.nb_bucket)\n+\t\t\tdiff = sa->replay.nb_bucket;\n+\n+\t\tfor (i = 0; i != diff; i++) {\n+\t\t\tnew_bucket = (i + last_bucket + 1) &\n+\t\t\t\tsa->replay.bucket_index_mask;\n+\t\t\trsn->window[new_bucket] = 0;\n+\t\t}\n+\t\trsn->sqn = sqn;\n+\t}\n+\n+\tbucket &= sa->replay.bucket_index_mask;\n+\tbit = (uint64_t)1 << (sqn & WINDOW_BIT_LOC_MASK);\n+\n+\t/* already seen packet */\n+\tif (rsn->window[bucket] & bit)\n+\t\treturn -EINVAL;\n+\n+\trsn->window[bucket] |= bit;\n+\treturn 0;\n+}\n+\n+/**\n+ * To achieve ability to do multiple readers single writer for\n+ * SA replay window information and sequence number (RSN)\n+ * basic RCU schema is used:\n+ * SA have 2 copies of RSN (one for readers, another for writers).\n+ * Each RSN contains a rwlock that has to be grabbed (for read/write)\n+ * to avoid races between readers and writer.\n+ * Writer is responsible to make a copy or reader RSN, update it\n+ * and mark newly updated RSN as readers one.\n+ * That approach is intended to minimize contention and cache sharing\n+ * between writer and readers.\n+ */\n+\n /**\n  * Based on number of buckets calculated required size for the\n  * structure that holds replay window and sequnce number (RSN) information.\ndiff --git a/lib/librte_ipsec/pad.h b/lib/librte_ipsec/pad.h\nnew file mode 100644\nindex 000000000..2f5ccd00e\n--- /dev/null\n+++ b/lib/librte_ipsec/pad.h\n@@ -0,0 +1,45 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#ifndef _PAD_H_\n+#define _PAD_H_\n+\n+#define IPSEC_MAX_PAD_SIZE\tUINT8_MAX\n+\n+static const uint8_t esp_pad_bytes[IPSEC_MAX_PAD_SIZE] = {\n+\t1, 2, 3, 4, 5, 6, 7, 8,\n+\t9, 10, 11, 12, 13, 14, 15, 16,\n+\t17, 18, 19, 20, 21, 22, 23, 24,\n+\t25, 26, 27, 28, 29, 30, 31, 32,\n+\t33, 34, 35, 36, 37, 38, 39, 40,\n+\t41, 42, 43, 44, 45, 46, 47, 48,\n+\t49, 50, 51, 52, 53, 54, 55, 56,\n+\t57, 58, 59, 60, 61, 62, 63, 64,\n+\t65, 66, 67, 68, 69, 70, 71, 72,\n+\t73, 74, 75, 76, 77, 78, 79, 80,\n+\t81, 82, 83, 84, 85, 86, 87, 88,\n+\t89, 90, 91, 92, 93, 94, 95, 96,\n+\t97, 98, 99, 100, 101, 102, 103, 104,\n+\t105, 106, 107, 108, 109, 110, 111, 112,\n+\t113, 114, 115, 116, 117, 118, 119, 120,\n+\t121, 122, 123, 124, 125, 126, 127, 128,\n+\t129, 130, 131, 132, 133, 134, 135, 136,\n+\t137, 138, 139, 140, 141, 142, 143, 144,\n+\t145, 146, 147, 148, 149, 150, 151, 152,\n+\t153, 154, 155, 156, 157, 158, 159, 160,\n+\t161, 162, 163, 164, 165, 166, 167, 168,\n+\t169, 170, 171, 172, 173, 174, 175, 176,\n+\t177, 178, 179, 180, 181, 182, 183, 184,\n+\t185, 186, 187, 188, 189, 190, 191, 192,\n+\t193, 194, 195, 196, 197, 198, 199, 200,\n+\t201, 202, 203, 204, 205, 206, 207, 208,\n+\t209, 210, 211, 212, 213, 214, 215, 216,\n+\t217, 218, 219, 220, 221, 222, 223, 224,\n+\t225, 226, 227, 228, 229, 230, 231, 232,\n+\t233, 234, 235, 236, 237, 238, 239, 240,\n+\t241, 242, 243, 244, 245, 246, 247, 248,\n+\t249, 250, 251, 252, 253, 254, 255,\n+};\n+\n+#endif /* _PAD_H_ */\ndiff --git a/lib/librte_ipsec/sa.c b/lib/librte_ipsec/sa.c\nindex 7f9baa602..00b3c8044 100644\n--- a/lib/librte_ipsec/sa.c\n+++ b/lib/librte_ipsec/sa.c\n@@ -6,9 +6,13 @@\n #include <rte_esp.h>\n #include <rte_ip.h>\n #include <rte_errno.h>\n+#include <rte_cryptodev.h>\n \n #include \"sa.h\"\n #include \"ipsec_sqn.h\"\n+#include \"crypto.h\"\n+#include \"iph.h\"\n+#include \"pad.h\"\n \n /* some helper structures */\n struct crypto_xform {\n@@ -192,6 +196,7 @@ esp_sa_init(struct rte_ipsec_sa *sa, const struct rte_ipsec_sa_prm *prm,\n \t\t/* RFC 4106 */\n \t\tif (cxf->aead->algo != RTE_CRYPTO_AEAD_AES_GCM)\n \t\t\treturn -EINVAL;\n+\t\tsa->aad_len = sizeof(struct aead_gcm_aad);\n \t\tsa->icv_len = cxf->aead->digest_length;\n \t\tsa->iv_ofs = cxf->aead->iv.offset;\n \t\tsa->iv_len = sizeof(uint64_t);\n@@ -306,18 +311,1059 @@ rte_ipsec_sa_init(struct rte_ipsec_sa *sa, const struct rte_ipsec_sa_prm *prm,\n \treturn sz;\n }\n \n+static inline void\n+mbuf_bulk_copy(struct rte_mbuf *dst[], struct rte_mbuf * const src[],\n+\tuint32_t num)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i != num; i++)\n+\t\tdst[i] = src[i];\n+}\n+\n+static inline void\n+lksd_none_cop_prepare(const struct rte_ipsec_session *ss,\n+\tstruct rte_mbuf *mb[], struct rte_crypto_op *cop[], uint16_t num)\n+{\n+\tuint32_t i;\n+\tstruct rte_crypto_sym_op *sop;\n+\n+\tfor (i = 0; i != num; i++) {\n+\t\tsop = cop[i]->sym;\n+\t\tcop[i]->type = RTE_CRYPTO_OP_TYPE_SYMMETRIC;\n+\t\tcop[i]->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;\n+\t\tcop[i]->sess_type = RTE_CRYPTO_OP_WITH_SESSION;\n+\t\tsop->m_src = mb[i];\n+\t\t__rte_crypto_sym_op_attach_sym_session(sop, ss->crypto.ses);\n+\t}\n+}\n+\n+static inline void\n+esp_outb_cop_prepare(struct rte_crypto_op *cop,\n+\tconst struct rte_ipsec_sa *sa, const uint64_t ivp[IPSEC_MAX_IV_QWORD],\n+\tconst union sym_op_data *icv, uint32_t hlen, uint32_t plen)\n+{\n+\tstruct rte_crypto_sym_op *sop;\n+\tstruct aead_gcm_iv *gcm;\n+\n+\t/* fill sym op fields */\n+\tsop = cop->sym;\n+\n+\t/* AEAD (AES_GCM) case */\n+\tif (sa->aad_len != 0) {\n+\t\tsop->aead.data.offset = sa->ctp.cipher.offset + hlen;\n+\t\tsop->aead.data.length = sa->ctp.cipher.length + plen;\n+\t\tsop->aead.digest.data = icv->va;\n+\t\tsop->aead.digest.phys_addr = icv->pa;\n+\t\tsop->aead.aad.data = icv->va + sa->icv_len;\n+\t\tsop->aead.aad.phys_addr = icv->pa + sa->icv_len;\n+\n+\t\t/* fill AAD IV (located inside crypto op) */\n+\t\tgcm = rte_crypto_op_ctod_offset(cop, struct aead_gcm_iv *,\n+\t\t\tsa->iv_ofs);\n+\t\taead_gcm_iv_fill(gcm, ivp[0], sa->salt);\n+\t/* CRYPT+AUTH case */\n+\t} else {\n+\t\tsop->cipher.data.offset = sa->ctp.cipher.offset + hlen;\n+\t\tsop->cipher.data.length = sa->ctp.cipher.length + plen;\n+\t\tsop->auth.data.offset = sa->ctp.auth.offset + hlen;\n+\t\tsop->auth.data.length = sa->ctp.auth.length + plen;\n+\t\tsop->auth.digest.data = icv->va;\n+\t\tsop->auth.digest.phys_addr = icv->pa;\n+\t}\n+}\n+\n+static inline int32_t\n+esp_outb_tun_pkt_prepare(struct rte_ipsec_sa *sa, rte_be64_t sqc,\n+\tconst uint64_t ivp[IPSEC_MAX_IV_QWORD], struct rte_mbuf *mb,\n+\tunion sym_op_data *icv)\n+{\n+\tuint32_t clen, hlen, pdlen, pdofs, tlen;\n+\tstruct rte_mbuf *ml;\n+\tstruct esp_hdr *esph;\n+\tstruct esp_tail *espt;\n+\tchar *ph, *pt;\n+\tuint64_t *iv;\n+\n+\t/* calculate extra header space required */\n+\thlen = sa->hdr_len + sa->iv_len + sizeof(*esph);\n+\n+\t/* number of bytes to encrypt */\n+\tclen = mb->pkt_len + sizeof(*espt);\n+\tclen = RTE_ALIGN_CEIL(clen, sa->pad_align);\n+\n+\t/* pad length + esp tail */\n+\tpdlen = clen - mb->pkt_len;\n+\ttlen = pdlen + sa->icv_len;\n+\n+\t/* do append and prepend */\n+\tml = rte_pktmbuf_lastseg(mb);\n+\tif (tlen + sa->sqh_len + sa->aad_len > rte_pktmbuf_tailroom(ml))\n+\t\treturn -ENOSPC;\n+\n+\t/* prepend header */\n+\tph = rte_pktmbuf_prepend(mb, hlen);\n+\tif (ph == NULL)\n+\t\treturn -ENOSPC;\n+\n+\t/* append tail */\n+\tpdofs = ml->data_len;\n+\tml->data_len += tlen;\n+\tmb->pkt_len += tlen;\n+\tpt = rte_pktmbuf_mtod_offset(ml, typeof(pt), pdofs);\n+\n+\t/* update pkt l2/l3 len */\n+\tmb->l2_len = sa->hdr_l3_off;\n+\tmb->l3_len = sa->hdr_len - sa->hdr_l3_off;\n+\n+\t/* copy tunnel pkt header */\n+\trte_memcpy(ph, sa->hdr, sa->hdr_len);\n+\n+\t/* update original and new ip header fields */\n+\tif (sa->type & RTE_IPSEC_SATP_MODE_TUNLV4) {\n+\t\tstruct ipv4_hdr *l3h;\n+\t\tl3h = (struct ipv4_hdr *)(ph + sa->hdr_l3_off);\n+\t\tl3h->packet_id = sqn_low16(sqc);\n+\t\tl3h->total_length = rte_cpu_to_be_16(mb->pkt_len -\n+\t\t\tsa->hdr_l3_off);\n+\t} else {\n+\t\tstruct ipv6_hdr *l3h;\n+\t\tl3h = (struct ipv6_hdr *)(ph + sa->hdr_l3_off);\n+\t\tl3h->payload_len = rte_cpu_to_be_16(mb->pkt_len -\n+\t\t\tsa->hdr_l3_off - sizeof(*l3h));\n+\t}\n+\n+\t/* update spi, seqn and iv */\n+\tesph = (struct esp_hdr *)(ph + sa->hdr_len);\n+\tiv = (uint64_t *)(esph + 1);\n+\trte_memcpy(iv, ivp, sa->iv_len);\n+\n+\tesph->spi = sa->spi;\n+\tesph->seq = sqn_low32(sqc);\n+\n+\t/* offset for ICV */\n+\tpdofs += pdlen + sa->sqh_len;\n+\n+\t/* pad length */\n+\tpdlen -= sizeof(*espt);\n+\n+\t/* copy padding data */\n+\trte_memcpy(pt, esp_pad_bytes, pdlen);\n+\n+\t/* update esp trailer */\n+\tespt = (struct esp_tail *)(pt + pdlen);\n+\tespt->pad_len = pdlen;\n+\tespt->next_proto = sa->proto;\n+\n+\ticv->va = rte_pktmbuf_mtod_offset(ml, void *, pdofs);\n+\ticv->pa = rte_pktmbuf_iova_offset(ml, pdofs);\n+\n+\treturn clen;\n+}\n+\n+/*\n+ * for pure cryptodev (lookaside none) depending on SA settings,\n+ * we might have to write some extra data to the packet.\n+ */\n+static inline void\n+outb_pkt_xprepare(const struct rte_ipsec_sa *sa, rte_be64_t sqc,\n+\tconst union sym_op_data *icv)\n+{\n+\tuint32_t *psqh;\n+\tstruct aead_gcm_aad *aad;\n+\n+\t/* insert SQN.hi between ESP trailer and ICV */\n+\tif (sa->sqh_len != 0) {\n+\t\tpsqh = (uint32_t *)(icv->va - sa->sqh_len);\n+\t\tpsqh[0] = sqn_hi32(sqc);\n+\t}\n+\n+\t/*\n+\t * fill IV and AAD fields, if any (aad fields are placed after icv),\n+\t * right now we support only one AEAD algorithm: AES-GCM .\n+\t */\n+\tif (sa->aad_len != 0) {\n+\t\taad = (struct aead_gcm_aad *)(icv->va + sa->icv_len);\n+\t\taead_gcm_aad_fill(aad, sa->spi, sqc, IS_ESN(sa));\n+\t}\n+}\n+\n+static uint16_t\n+outb_tun_prepare(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tstruct rte_crypto_op *cop[], uint16_t num)\n+{\n+\tint32_t rc;\n+\tuint32_t i, k, n;\n+\tuint64_t sqn;\n+\trte_be64_t sqc;\n+\tstruct rte_ipsec_sa *sa;\n+\tunion sym_op_data icv;\n+\tuint64_t iv[IPSEC_MAX_IV_QWORD];\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\tn = num;\n+\tsqn = esn_outb_update_sqn(sa, &n);\n+\tif (n != num)\n+\t\trte_errno = EOVERFLOW;\n+\n+\tk = 0;\n+\tfor (i = 0; i != n; i++) {\n+\n+\t\tsqc = rte_cpu_to_be_64(sqn + i);\n+\t\tgen_iv(iv, sqc);\n+\n+\t\t/* try to update the packet itself */\n+\t\trc = esp_outb_tun_pkt_prepare(sa, sqc, iv, mb[i], &icv);\n+\n+\t\t/* success, setup crypto op */\n+\t\tif (rc >= 0) {\n+\t\t\tmb[k] = mb[i];\n+\t\t\toutb_pkt_xprepare(sa, sqc, &icv);\n+\t\t\tesp_outb_cop_prepare(cop[k], sa, iv, &icv, 0, rc);\n+\t\t\tk++;\n+\t\t/* failure, put packet into the death-row */\n+\t\t} else {\n+\t\t\tdr[i - k] = mb[i];\n+\t\t\trte_errno = -rc;\n+\t\t}\n+\t}\n+\n+\t/* update cops */\n+\tlksd_none_cop_prepare(ss, mb, cop, k);\n+\n+\t /* copy not prepared mbufs beyond good ones */\n+\tif (k != num && k != 0)\n+\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\n+\treturn k;\n+}\n+\n+static inline int32_t\n+esp_outb_trs_pkt_prepare(struct rte_ipsec_sa *sa, rte_be64_t sqc,\n+\tconst uint64_t ivp[IPSEC_MAX_IV_QWORD], struct rte_mbuf *mb,\n+\tuint32_t l2len, uint32_t l3len, union sym_op_data *icv)\n+{\n+\tuint8_t np;\n+\tuint32_t clen, hlen, pdlen, pdofs, plen, tlen, uhlen;\n+\tstruct rte_mbuf *ml;\n+\tstruct esp_hdr *esph;\n+\tstruct esp_tail *espt;\n+\tchar *ph, *pt;\n+\tuint64_t *iv;\n+\n+\tuhlen = l2len + l3len;\n+\tplen = mb->pkt_len - uhlen;\n+\n+\t/* calculate extra header space required */\n+\thlen = sa->iv_len + sizeof(*esph);\n+\n+\t/* number of bytes to encrypt */\n+\tclen = plen + sizeof(*espt);\n+\tclen = RTE_ALIGN_CEIL(clen, sa->pad_align);\n+\n+\t/* pad length + esp tail */\n+\tpdlen = clen - plen;\n+\ttlen = pdlen + sa->icv_len;\n+\n+\t/* do append and insert */\n+\tml = rte_pktmbuf_lastseg(mb);\n+\tif (tlen + sa->sqh_len + sa->aad_len > rte_pktmbuf_tailroom(ml))\n+\t\treturn -ENOSPC;\n+\n+\t/* prepend space for ESP header */\n+\tph = rte_pktmbuf_prepend(mb, hlen);\n+\tif (ph == NULL)\n+\t\treturn -ENOSPC;\n+\n+\t/* append tail */\n+\tpdofs = ml->data_len;\n+\tml->data_len += tlen;\n+\tmb->pkt_len += tlen;\n+\tpt = rte_pktmbuf_mtod_offset(ml, typeof(pt), pdofs);\n+\n+\t/* shift L2/L3 headers */\n+\tinsert_esph(ph, ph + hlen, uhlen);\n+\n+\t/* update ip  header fields */\n+\tnp = update_trs_l3hdr(sa, ph + l2len, mb->pkt_len, l2len, l3len,\n+\t\t\tIPPROTO_ESP);\n+\n+\t/* update spi, seqn and iv */\n+\tesph = (struct esp_hdr *)(ph + uhlen);\n+\tiv = (uint64_t *)(esph + 1);\n+\trte_memcpy(iv, ivp, sa->iv_len);\n+\n+\tesph->spi = sa->spi;\n+\tesph->seq = sqn_low32(sqc);\n+\n+\t/* offset for ICV */\n+\tpdofs += pdlen + sa->sqh_len;\n+\n+\t/* pad length */\n+\tpdlen -= sizeof(*espt);\n+\n+\t/* copy padding data */\n+\trte_memcpy(pt, esp_pad_bytes, pdlen);\n+\n+\t/* update esp trailer */\n+\tespt = (struct esp_tail *)(pt + pdlen);\n+\tespt->pad_len = pdlen;\n+\tespt->next_proto = np;\n+\n+\ticv->va = rte_pktmbuf_mtod_offset(ml, void *, pdofs);\n+\ticv->pa = rte_pktmbuf_iova_offset(ml, pdofs);\n+\n+\treturn clen;\n+}\n+\n+static uint16_t\n+outb_trs_prepare(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tstruct rte_crypto_op *cop[], uint16_t num)\n+{\n+\tint32_t rc;\n+\tuint32_t i, k, n, l2, l3;\n+\tuint64_t sqn;\n+\trte_be64_t sqc;\n+\tstruct rte_ipsec_sa *sa;\n+\tunion sym_op_data icv;\n+\tuint64_t iv[IPSEC_MAX_IV_QWORD];\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\tn = num;\n+\tsqn = esn_outb_update_sqn(sa, &n);\n+\tif (n != num)\n+\t\trte_errno = EOVERFLOW;\n+\n+\tk = 0;\n+\tfor (i = 0; i != n; i++) {\n+\n+\t\tl2 = mb[i]->l2_len;\n+\t\tl3 = mb[i]->l3_len;\n+\n+\t\tsqc = rte_cpu_to_be_64(sqn + i);\n+\t\tgen_iv(iv, sqc);\n+\n+\t\t/* try to update the packet itself */\n+\t\trc = esp_outb_trs_pkt_prepare(sa, sqc, iv, mb[i],\n+\t\t\t\tl2, l3, &icv);\n+\n+\t\t/* success, setup crypto op */\n+\t\tif (rc >= 0) {\n+\t\t\tmb[k] = mb[i];\n+\t\t\toutb_pkt_xprepare(sa, sqc, &icv);\n+\t\t\tesp_outb_cop_prepare(cop[k], sa, iv, &icv, l2 + l3, rc);\n+\t\t\tk++;\n+\t\t/* failure, put packet into the death-row */\n+\t\t} else {\n+\t\t\tdr[i - k] = mb[i];\n+\t\t\trte_errno = -rc;\n+\t\t}\n+\t}\n+\n+\t/* update cops */\n+\tlksd_none_cop_prepare(ss, mb, cop, k);\n+\n+\t/* copy not prepared mbufs beyond good ones */\n+\tif (k != num && k != 0)\n+\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\n+\treturn k;\n+}\n+\n+static inline int32_t\n+esp_inb_tun_cop_prepare(struct rte_crypto_op *cop,\n+\tconst struct rte_ipsec_sa *sa, struct rte_mbuf *mb,\n+\tconst union sym_op_data *icv, uint32_t pofs, uint32_t plen)\n+{\n+\tstruct rte_crypto_sym_op *sop;\n+\tstruct aead_gcm_iv *gcm;\n+\tuint64_t *ivc, *ivp;\n+\tuint32_t clen;\n+\n+\tclen = plen - sa->ctp.cipher.length;\n+\tif ((int32_t)clen < 0 || (clen & (sa->pad_align - 1)) != 0)\n+\t\treturn -EINVAL;\n+\n+\t/* fill sym op fields */\n+\tsop = cop->sym;\n+\n+\t/* AEAD (AES_GCM) case */\n+\tif (sa->aad_len != 0) {\n+\t\tsop->aead.data.offset = pofs + sa->ctp.cipher.offset;\n+\t\tsop->aead.data.length = clen;\n+\t\tsop->aead.digest.data = icv->va;\n+\t\tsop->aead.digest.phys_addr = icv->pa;\n+\t\tsop->aead.aad.data = icv->va + sa->icv_len;\n+\t\tsop->aead.aad.phys_addr = icv->pa + sa->icv_len;\n+\n+\t\t/* fill AAD IV (located inside crypto op) */\n+\t\tgcm = rte_crypto_op_ctod_offset(cop, struct aead_gcm_iv *,\n+\t\t\tsa->iv_ofs);\n+\t\tivp = rte_pktmbuf_mtod_offset(mb, uint64_t *,\n+\t\t\tpofs + sizeof(struct esp_hdr));\n+\t\taead_gcm_iv_fill(gcm, ivp[0], sa->salt);\n+\t/* CRYPT+AUTH case */\n+\t} else {\n+\t\tsop->cipher.data.offset = pofs + sa->ctp.cipher.offset;\n+\t\tsop->cipher.data.length = clen;\n+\t\tsop->auth.data.offset = pofs + sa->ctp.auth.offset;\n+\t\tsop->auth.data.length = plen - sa->ctp.auth.length;\n+\t\tsop->auth.digest.data = icv->va;\n+\t\tsop->auth.digest.phys_addr = icv->pa;\n+\n+\t\t/* copy iv from the input packet to the cop */\n+\t\tivc = rte_crypto_op_ctod_offset(cop, uint64_t *, sa->iv_ofs);\n+\t\tivp = rte_pktmbuf_mtod_offset(mb, uint64_t *,\n+\t\t\tpofs + sizeof(struct esp_hdr));\n+\t\trte_memcpy(ivc, ivp, sa->iv_len);\n+\t}\n+\treturn 0;\n+}\n+\n+/*\n+ * for pure cryptodev (lookaside none) depending on SA settings,\n+ * we might have to write some extra data to the packet.\n+ */\n+static inline void\n+inb_pkt_xprepare(const struct rte_ipsec_sa *sa, rte_be64_t sqc,\n+\tconst union sym_op_data *icv)\n+{\n+\tstruct aead_gcm_aad *aad;\n+\n+\t/* insert SQN.hi between ESP trailer and ICV */\n+\tif (sa->sqh_len != 0)\n+\t\tinsert_sqh(sqn_hi32(sqc), icv->va, sa->icv_len);\n+\n+\t/*\n+\t * fill AAD fields, if any (aad fields are placed after icv),\n+\t * right now we support only one AEAD algorithm: AES-GCM.\n+\t */\n+\tif (sa->aad_len != 0) {\n+\t\taad = (struct aead_gcm_aad *)(icv->va + sa->icv_len);\n+\t\taead_gcm_aad_fill(aad, sa->spi, sqc, IS_ESN(sa));\n+\t}\n+}\n+\n+static inline int32_t\n+esp_inb_tun_pkt_prepare(const struct rte_ipsec_sa *sa,\n+\tconst struct replay_sqn *rsn, struct rte_mbuf *mb,\n+\tuint32_t hlen, union sym_op_data *icv)\n+{\n+\tint32_t rc;\n+\tuint64_t sqn;\n+\tuint32_t icv_ofs, plen;\n+\tstruct rte_mbuf *ml;\n+\tstruct esp_hdr *esph;\n+\n+\tesph = rte_pktmbuf_mtod_offset(mb, struct esp_hdr *, hlen);\n+\n+\t/*\n+\t * retrieve and reconstruct SQN, then check it, then\n+\t * convert it back into network byte order.\n+\t */\n+\tsqn = rte_be_to_cpu_32(esph->seq);\n+\tif (IS_ESN(sa))\n+\t\tsqn = reconstruct_esn(rsn->sqn, sqn, sa->replay.win_sz);\n+\n+\trc = esn_inb_check_sqn(rsn, sa, sqn);\n+\tif (rc != 0)\n+\t\treturn rc;\n+\n+\tsqn = rte_cpu_to_be_64(sqn);\n+\n+\t/* start packet manipulation */\n+\tplen = mb->pkt_len;\n+\tplen = plen - hlen;\n+\n+\tml = rte_pktmbuf_lastseg(mb);\n+\ticv_ofs = ml->data_len - sa->icv_len + sa->sqh_len;\n+\n+\t/* we have to allocate space for AAD somewhere,\n+\t * right now - just use free trailing space at the last segment.\n+\t * Would probably be more convenient to reserve space for AAD\n+\t * inside rte_crypto_op itself\n+\t * (again for IV space is already reserved inside cop).\n+\t */\n+\tif (sa->aad_len + sa->sqh_len > rte_pktmbuf_tailroom(ml))\n+\t\treturn -ENOSPC;\n+\n+\ticv->va = rte_pktmbuf_mtod_offset(ml, void *, icv_ofs);\n+\ticv->pa = rte_pktmbuf_iova_offset(ml, icv_ofs);\n+\n+\tinb_pkt_xprepare(sa, sqn, icv);\n+\treturn plen;\n+}\n+\n+static uint16_t\n+inb_pkt_prepare(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tstruct rte_crypto_op *cop[], uint16_t num)\n+{\n+\tint32_t rc;\n+\tuint32_t i, k, hl;\n+\tstruct rte_ipsec_sa *sa;\n+\tstruct replay_sqn *rsn;\n+\tunion sym_op_data icv;\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\trsn = sa->sqn.inb;\n+\n+\tk = 0;\n+\tfor (i = 0; i != num; i++) {\n+\n+\t\thl = mb[i]->l2_len + mb[i]->l3_len;\n+\t\trc = esp_inb_tun_pkt_prepare(sa, rsn, mb[i], hl, &icv);\n+\t\tif (rc >= 0)\n+\t\t\trc = esp_inb_tun_cop_prepare(cop[k], sa, mb[i], &icv,\n+\t\t\t\thl, rc);\n+\n+\t\tif (rc == 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\telse {\n+\t\t\tdr[i - k] = mb[i];\n+\t\t\trte_errno = -rc;\n+\t\t}\n+\t}\n+\n+\t/* update cops */\n+\tlksd_none_cop_prepare(ss, mb, cop, k);\n+\n+\t/* copy not prepared mbufs beyond good ones */\n+\tif (k != num && k != 0)\n+\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\n+\treturn k;\n+}\n+\n+static inline void\n+lksd_proto_cop_prepare(const struct rte_ipsec_session *ss,\n+\tstruct rte_mbuf *mb[], struct rte_crypto_op *cop[], uint16_t num)\n+{\n+\tuint32_t i;\n+\tstruct rte_crypto_sym_op *sop;\n+\n+\tfor (i = 0; i != num; i++) {\n+\t\tsop = cop[i]->sym;\n+\t\tcop[i]->type = RTE_CRYPTO_OP_TYPE_SYMMETRIC;\n+\t\tcop[i]->status = RTE_CRYPTO_OP_STATUS_NOT_PROCESSED;\n+\t\tcop[i]->sess_type = RTE_CRYPTO_OP_SECURITY_SESSION;\n+\t\tsop->m_src = mb[i];\n+\t\t__rte_security_attach_session(sop, ss->security.ses);\n+\t}\n+}\n+\n+static uint16_t\n+lksd_proto_prepare(const struct rte_ipsec_session *ss,\n+\tstruct rte_mbuf *mb[], struct rte_crypto_op *cop[], uint16_t num)\n+{\n+\tlksd_proto_cop_prepare(ss, mb, cop, num);\n+\treturn num;\n+}\n+\n+static inline int\n+esp_inb_tun_single_pkt_process(struct rte_ipsec_sa *sa, struct rte_mbuf *mb,\n+\tuint32_t *sqn)\n+{\n+\tuint32_t hlen, icv_len, tlen;\n+\tstruct esp_hdr *esph;\n+\tstruct esp_tail *espt;\n+\tstruct rte_mbuf *ml;\n+\tchar *pd;\n+\n+\tif (mb->ol_flags & PKT_RX_SEC_OFFLOAD_FAILED)\n+\t\treturn -EBADMSG;\n+\n+\ticv_len = sa->icv_len;\n+\n+\tml = rte_pktmbuf_lastseg(mb);\n+\tespt = rte_pktmbuf_mtod_offset(ml, struct esp_tail *,\n+\t\tml->data_len - icv_len - sizeof(*espt));\n+\n+\t/*\n+\t * check padding and next proto.\n+\t * return an error if something is wrong.\n+\t */\n+\tpd = (char *)espt - espt->pad_len;\n+\tif (espt->next_proto != sa->proto ||\n+\t\t\tmemcmp(pd, esp_pad_bytes, espt->pad_len))\n+\t\treturn -EINVAL;\n+\n+\t/* cut of ICV, ESP tail and padding bytes */\n+\ttlen = icv_len + sizeof(*espt) + espt->pad_len;\n+\tml->data_len -= tlen;\n+\tmb->pkt_len -= tlen;\n+\n+\t/* cut of L2/L3 headers, ESP header and IV */\n+\thlen = mb->l2_len + mb->l3_len;\n+\tesph = rte_pktmbuf_mtod_offset(mb, struct esp_hdr *, hlen);\n+\trte_pktmbuf_adj(mb, hlen + sa->ctp.cipher.offset);\n+\n+\t/* retrieve SQN for later check */\n+\t*sqn = rte_be_to_cpu_32(esph->seq);\n+\n+\t/* reset mbuf metatdata: L2/L3 len, packet type */\n+\tmb->packet_type = RTE_PTYPE_UNKNOWN;\n+\tmb->l2_len = 0;\n+\tmb->l3_len = 0;\n+\n+\t/* clear the PKT_RX_SEC_OFFLOAD flag if set */\n+\tmb->ol_flags &= ~(mb->ol_flags & PKT_RX_SEC_OFFLOAD);\n+\treturn 0;\n+}\n+\n+static inline int\n+esp_inb_trs_single_pkt_process(struct rte_ipsec_sa *sa, struct rte_mbuf *mb,\n+\tuint32_t *sqn)\n+{\n+\tuint32_t hlen, icv_len, l2len, l3len, tlen;\n+\tstruct esp_hdr *esph;\n+\tstruct esp_tail *espt;\n+\tstruct rte_mbuf *ml;\n+\tchar *np, *op, *pd;\n+\n+\tif (mb->ol_flags & PKT_RX_SEC_OFFLOAD_FAILED)\n+\t\treturn -EBADMSG;\n+\n+\ticv_len = sa->icv_len;\n+\n+\tml = rte_pktmbuf_lastseg(mb);\n+\tespt = rte_pktmbuf_mtod_offset(ml, struct esp_tail *,\n+\t\tml->data_len - icv_len - sizeof(*espt));\n+\n+\t/* check padding, return an error if something is wrong. */\n+\tpd = (char *)espt - espt->pad_len;\n+\tif (memcmp(pd, esp_pad_bytes, espt->pad_len))\n+\t\treturn -EINVAL;\n+\n+\t/* cut of ICV, ESP tail and padding bytes */\n+\ttlen = icv_len + sizeof(*espt) + espt->pad_len;\n+\tml->data_len -= tlen;\n+\tmb->pkt_len -= tlen;\n+\n+\t/* retrieve SQN for later check */\n+\tl2len = mb->l2_len;\n+\tl3len = mb->l3_len;\n+\thlen = l2len + l3len;\n+\top = rte_pktmbuf_mtod(mb, char *);\n+\tesph = (struct esp_hdr *)(op + hlen);\n+\t*sqn = rte_be_to_cpu_32(esph->seq);\n+\n+\t/* cut off ESP header and IV, update L3 header */\n+\tnp = rte_pktmbuf_adj(mb, sa->ctp.cipher.offset);\n+\tremove_esph(np, op, hlen);\n+\tupdate_trs_l3hdr(sa, np + l2len, mb->pkt_len, l2len, l3len,\n+\t\t\tespt->next_proto);\n+\n+\t/* reset mbuf packet type */\n+\tmb->packet_type &= (RTE_PTYPE_L2_MASK | RTE_PTYPE_L3_MASK);\n+\n+\t/* clear the PKT_RX_SEC_OFFLOAD flag if set */\n+\tmb->ol_flags &= ~(mb->ol_flags & PKT_RX_SEC_OFFLOAD);\n+\treturn 0;\n+}\n+\n+static inline uint16_t\n+esp_inb_rsn_update(struct rte_ipsec_sa *sa, const uint32_t sqn[],\n+\tstruct rte_mbuf *mb[], struct rte_mbuf *dr[], uint16_t num)\n+{\n+\tuint32_t i, k;\n+\tstruct replay_sqn *rsn;\n+\n+\trsn = sa->sqn.inb;\n+\n+\tk = 0;\n+\tfor (i = 0; i != num; i++) {\n+\t\tif (esn_inb_update_sqn(rsn, sa, sqn[i]) == 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\telse\n+\t\t\tdr[i - k] = mb[i];\n+\t}\n+\n+\treturn k;\n+}\n+\n+static uint16_t\n+inb_tun_pkt_process(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tuint16_t num)\n+{\n+\tuint32_t i, k;\n+\tstruct rte_ipsec_sa *sa;\n+\tuint32_t sqn[num];\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\t/* process packets, extract seq numbers */\n+\n+\tk = 0;\n+\tfor (i = 0; i != num; i++) {\n+\t\t/* good packet */\n+\t\tif (esp_inb_tun_single_pkt_process(sa, mb[i], sqn + k) == 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\t/* bad packet, will drop from furhter processing */\n+\t\telse\n+\t\t\tdr[i - k] = mb[i];\n+\t}\n+\n+\t/* update seq # and replay winow */\n+\tk = esp_inb_rsn_update(sa, sqn, mb, dr + i - k, k);\n+\n+\t/* handle unprocessed mbufs */\n+\tif (k != num) {\n+\t\trte_errno = EBADMSG;\n+\t\tif (k != 0)\n+\t\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\t}\n+\n+\treturn k;\n+}\n+\n+static uint16_t\n+inb_trs_pkt_process(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tuint16_t num)\n+{\n+\tuint32_t i, k;\n+\tuint32_t sqn[num];\n+\tstruct rte_ipsec_sa *sa;\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\t/* process packets, extract seq numbers */\n+\n+\tk = 0;\n+\tfor (i = 0; i != num; i++) {\n+\t\t/* good packet */\n+\t\tif (esp_inb_trs_single_pkt_process(sa, mb[i], sqn + k) == 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\t/* bad packet, will drop from furhter processing */\n+\t\telse\n+\t\t\tdr[i - k] = mb[i];\n+\t}\n+\n+\t/* update seq # and replay winow */\n+\tk = esp_inb_rsn_update(sa, sqn, mb, dr + i - k, k);\n+\n+\t/* handle unprocessed mbufs */\n+\tif (k != num) {\n+\t\trte_errno = EBADMSG;\n+\t\tif (k != 0)\n+\t\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\t}\n+\n+\treturn k;\n+}\n+\n+/*\n+ * process outbound packets for SA with ESN support,\n+ * for algorithms that require SQN.hibits to be implictly included\n+ * into digest computation.\n+ * In that case we have to move ICV bytes back to their proper place.\n+ */\n+static uint16_t\n+outb_sqh_process(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tuint16_t num)\n+{\n+\tuint32_t i, k, icv_len, *icv;\n+\tstruct rte_mbuf *ml;\n+\tstruct rte_ipsec_sa *sa;\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\tk = 0;\n+\ticv_len = sa->icv_len;\n+\n+\tfor (i = 0; i != num; i++) {\n+\t\tif ((mb[i]->ol_flags & PKT_RX_SEC_OFFLOAD_FAILED) == 0) {\n+\t\t\tml = rte_pktmbuf_lastseg(mb[i]);\n+\t\t\ticv = rte_pktmbuf_mtod_offset(ml, void *,\n+\t\t\t\tml->data_len - icv_len);\n+\t\t\tremove_sqh(icv, icv_len);\n+\t\t\tmb[k++] = mb[i];\n+\t\t} else\n+\t\t\tdr[i - k] = mb[i];\n+\t}\n+\n+\t/* handle unprocessed mbufs */\n+\tif (k != num) {\n+\t\trte_errno = EBADMSG;\n+\t\tif (k != 0)\n+\t\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\t}\n+\n+\treturn k;\n+}\n+\n+/*\n+ * simplest pkt process routine:\n+ * all actual processing is done already doneby HW/PMD,\n+ * just check mbuf ol_flags.\n+ * used for:\n+ * - inbound for RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL\n+ * - inbound/outbound for RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL\n+ * - outbound for RTE_SECURITY_ACTION_TYPE_NONE when ESN is disabled\n+ */\n+static uint16_t\n+pkt_flag_process(const struct rte_ipsec_session *ss, struct rte_mbuf *mb[],\n+\tuint16_t num)\n+{\n+\tuint32_t i, k;\n+\tstruct rte_mbuf *dr[num];\n+\n+\tRTE_SET_USED(ss);\n+\n+\tk = 0;\n+\tfor (i = 0; i != num; i++) {\n+\t\tif ((mb[i]->ol_flags & PKT_RX_SEC_OFFLOAD_FAILED) == 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\telse\n+\t\t\tdr[i - k] = mb[i];\n+\t}\n+\n+\t/* handle unprocessed mbufs */\n+\tif (k != num) {\n+\t\trte_errno = EBADMSG;\n+\t\tif (k != 0)\n+\t\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\t}\n+\n+\treturn k;\n+}\n+\n+/*\n+ * prepare packets for inline ipsec processing:\n+ * set ol_flags and attach metadata.\n+ */\n+static inline void\n+inline_outb_mbuf_prepare(const struct rte_ipsec_session *ss,\n+\tstruct rte_mbuf *mb[], uint16_t num)\n+{\n+\tuint32_t i, ol_flags;\n+\n+\tol_flags = ss->security.ol_flags & RTE_SECURITY_TX_OLOAD_NEED_MDATA;\n+\tfor (i = 0; i != num; i++) {\n+\n+\t\tmb[i]->ol_flags |= PKT_TX_SEC_OFFLOAD;\n+\t\tif (ol_flags != 0)\n+\t\t\trte_security_set_pkt_metadata(ss->security.ctx,\n+\t\t\t\tss->security.ses, mb[i], NULL);\n+\t}\n+}\n+\n+static uint16_t\n+inline_outb_tun_pkt_process(const struct rte_ipsec_session *ss,\n+\tstruct rte_mbuf *mb[], uint16_t num)\n+{\n+\tint32_t rc;\n+\tuint32_t i, k, n;\n+\tuint64_t sqn;\n+\trte_be64_t sqc;\n+\tstruct rte_ipsec_sa *sa;\n+\tunion sym_op_data icv;\n+\tuint64_t iv[IPSEC_MAX_IV_QWORD];\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\tn = num;\n+\tsqn = esn_outb_update_sqn(sa, &n);\n+\tif (n != num)\n+\t\trte_errno = EOVERFLOW;\n+\n+\tk = 0;\n+\tfor (i = 0; i != n; i++) {\n+\n+\t\tsqc = rte_cpu_to_be_64(sqn + i);\n+\t\tgen_iv(iv, sqc);\n+\n+\t\t/* try to update the packet itself */\n+\t\trc = esp_outb_tun_pkt_prepare(sa, sqc, iv, mb[i], &icv);\n+\n+\t\t/* success, update mbuf fields */\n+\t\tif (rc >= 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\t/* failure, put packet into the death-row */\n+\t\telse {\n+\t\t\tdr[i - k] = mb[i];\n+\t\t\trte_errno = -rc;\n+\t\t}\n+\t}\n+\n+\tinline_outb_mbuf_prepare(ss, mb, k);\n+\n+\t/* copy not processed mbufs beyond good ones */\n+\tif (k != num && k != 0)\n+\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\n+\treturn k;\n+}\n+\n+static uint16_t\n+inline_outb_trs_pkt_process(const struct rte_ipsec_session *ss,\n+\tstruct rte_mbuf *mb[], uint16_t num)\n+{\n+\tint32_t rc;\n+\tuint32_t i, k, n, l2, l3;\n+\tuint64_t sqn;\n+\trte_be64_t sqc;\n+\tstruct rte_ipsec_sa *sa;\n+\tunion sym_op_data icv;\n+\tuint64_t iv[IPSEC_MAX_IV_QWORD];\n+\tstruct rte_mbuf *dr[num];\n+\n+\tsa = ss->sa;\n+\n+\tn = num;\n+\tsqn = esn_outb_update_sqn(sa, &n);\n+\tif (n != num)\n+\t\trte_errno = EOVERFLOW;\n+\n+\tk = 0;\n+\tfor (i = 0; i != n; i++) {\n+\n+\t\tl2 = mb[i]->l2_len;\n+\t\tl3 = mb[i]->l3_len;\n+\n+\t\tsqc = rte_cpu_to_be_64(sqn + i);\n+\t\tgen_iv(iv, sqc);\n+\n+\t\t/* try to update the packet itself */\n+\t\trc = esp_outb_trs_pkt_prepare(sa, sqc, iv, mb[i],\n+\t\t\t\tl2, l3, &icv);\n+\n+\t\t/* success, update mbuf fields */\n+\t\tif (rc >= 0)\n+\t\t\tmb[k++] = mb[i];\n+\t\t/* failure, put packet into the death-row */\n+\t\telse {\n+\t\t\tdr[i - k] = mb[i];\n+\t\t\trte_errno = -rc;\n+\t\t}\n+\t}\n+\n+\tinline_outb_mbuf_prepare(ss, mb, k);\n+\n+\t/* copy not processed mbufs beyond good ones */\n+\tif (k != num && k != 0)\n+\t\tmbuf_bulk_copy(mb + k, dr, num - k);\n+\n+\treturn k;\n+}\n+\n+/*\n+ * outbound for RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL:\n+ * actual processing is done by HW/PMD, just set flags and metadata.\n+ */\n+static uint16_t\n+outb_inline_proto_process(const struct rte_ipsec_session *ss,\n+\t\tstruct rte_mbuf *mb[], uint16_t num)\n+{\n+\tinline_outb_mbuf_prepare(ss, mb, num);\n+\treturn num;\n+}\n+\n+static int\n+lksd_none_pkt_func_select(const struct rte_ipsec_sa *sa,\n+\t\tstruct rte_ipsec_sa_pkt_func *pf)\n+{\n+\tint32_t rc;\n+\n+\tstatic const uint64_t msk = RTE_IPSEC_SATP_DIR_MASK |\n+\t\t\tRTE_IPSEC_SATP_MODE_MASK;\n+\n+\trc = 0;\n+\tswitch (sa->type & msk) {\n+\tcase (RTE_IPSEC_SATP_DIR_IB | RTE_IPSEC_SATP_MODE_TUNLV4):\n+\tcase (RTE_IPSEC_SATP_DIR_IB | RTE_IPSEC_SATP_MODE_TUNLV6):\n+\t\tpf->prepare = inb_pkt_prepare;\n+\t\tpf->process = inb_tun_pkt_process;\n+\t\tbreak;\n+\tcase (RTE_IPSEC_SATP_DIR_IB | RTE_IPSEC_SATP_MODE_TRANS):\n+\t\tpf->prepare = inb_pkt_prepare;\n+\t\tpf->process = inb_trs_pkt_process;\n+\t\tbreak;\n+\tcase (RTE_IPSEC_SATP_DIR_OB | RTE_IPSEC_SATP_MODE_TUNLV4):\n+\tcase (RTE_IPSEC_SATP_DIR_OB | RTE_IPSEC_SATP_MODE_TUNLV6):\n+\t\tpf->prepare = outb_tun_prepare;\n+\t\tpf->process = (sa->sqh_len != 0) ?\n+\t\t\toutb_sqh_process : pkt_flag_process;\n+\t\tbreak;\n+\tcase (RTE_IPSEC_SATP_DIR_OB | RTE_IPSEC_SATP_MODE_TRANS):\n+\t\tpf->prepare = outb_trs_prepare;\n+\t\tpf->process = (sa->sqh_len != 0) ?\n+\t\t\toutb_sqh_process : pkt_flag_process;\n+\t\tbreak;\n+\tdefault:\n+\t\trc = -ENOTSUP;\n+\t}\n+\n+\treturn rc;\n+}\n+\n+static int\n+inline_crypto_pkt_func_select(const struct rte_ipsec_sa *sa,\n+\t\tstruct rte_ipsec_sa_pkt_func *pf)\n+{\n+\tint32_t rc;\n+\n+\tstatic const uint64_t msk = RTE_IPSEC_SATP_DIR_MASK |\n+\t\t\tRTE_IPSEC_SATP_MODE_MASK;\n+\n+\trc = 0;\n+\tswitch (sa->type & msk) {\n+\tcase (RTE_IPSEC_SATP_DIR_IB | RTE_IPSEC_SATP_MODE_TUNLV4):\n+\tcase (RTE_IPSEC_SATP_DIR_IB | RTE_IPSEC_SATP_MODE_TUNLV6):\n+\t\tpf->process = inb_tun_pkt_process;\n+\t\tbreak;\n+\tcase (RTE_IPSEC_SATP_DIR_IB | RTE_IPSEC_SATP_MODE_TRANS):\n+\t\tpf->process = inb_trs_pkt_process;\n+\t\tbreak;\n+\tcase (RTE_IPSEC_SATP_DIR_OB | RTE_IPSEC_SATP_MODE_TUNLV4):\n+\tcase (RTE_IPSEC_SATP_DIR_OB | RTE_IPSEC_SATP_MODE_TUNLV6):\n+\t\tpf->process = inline_outb_tun_pkt_process;\n+\t\tbreak;\n+\tcase (RTE_IPSEC_SATP_DIR_OB | RTE_IPSEC_SATP_MODE_TRANS):\n+\t\tpf->process = inline_outb_trs_pkt_process;\n+\t\tbreak;\n+\tdefault:\n+\t\trc = -ENOTSUP;\n+\t}\n+\n+\treturn rc;\n+}\n+\n int\n ipsec_sa_pkt_func_select(const struct rte_ipsec_session *ss,\n \tconst struct rte_ipsec_sa *sa, struct rte_ipsec_sa_pkt_func *pf)\n {\n \tint32_t rc;\n \n-\tRTE_SET_USED(sa);\n-\n \trc = 0;\n \tpf[0] = (struct rte_ipsec_sa_pkt_func) { 0 };\n \n \tswitch (ss->type) {\n+\tcase RTE_SECURITY_ACTION_TYPE_NONE:\n+\t\trc = lksd_none_pkt_func_select(sa, pf);\n+\t\tbreak;\n+\tcase RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO:\n+\t\trc = inline_crypto_pkt_func_select(sa, pf);\n+\t\tbreak;\n+\tcase RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL:\n+\t\tif ((sa->type & RTE_IPSEC_SATP_DIR_MASK) ==\n+\t\t\t\tRTE_IPSEC_SATP_DIR_IB)\n+\t\t\tpf->process = pkt_flag_process;\n+\t\telse\n+\t\t\tpf->process = outb_inline_proto_process;\n+\t\tbreak;\n+\tcase RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL:\n+\t\tpf->prepare = lksd_proto_prepare;\n+\t\tpf->process = pkt_flag_process;\n+\t\tbreak;\n \tdefault:\n \t\trc = -ENOTSUP;\n \t}\n",
    "prefixes": [
        "6/9"
    ]
}