From patchwork Wed Jun 21 07:47:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 25599 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id CBF99910D; Wed, 21 Jun 2017 17:48:15 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 4001B7CFA for ; Wed, 21 Jun 2017 17:47:40 +0200 (CEST) Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Jun 2017 08:47:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,369,1493708400"; d="scan'208";a="100153564" Received: from silpixa00399464.ir.intel.com (HELO silpixa00399464.ger.corp.intel.com) ([10.237.222.157]) by orsmga004.jf.intel.com with ESMTP; 21 Jun 2017 08:47:38 -0700 From: Pablo de Lara To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org Date: Wed, 21 Jun 2017 08:47:30 +0100 Message-Id: <20170621074731.45013-21-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.9.4 In-Reply-To: <20170621074731.45013-1-pablo.de.lara.guarch@intel.com> References: <20170621074731.45013-1-pablo.de.lara.guarch@intel.com> Subject: [dpdk-dev] [PATCH 21/22] cryptodev: use AES-GCM/CCM as AEAD algorithms X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Signed-off-by: Pablo de Lara --- app/test-crypto-perf/cperf_ops.c | 134 +++---- app/test-crypto-perf/cperf_options.h | 21 +- app/test-crypto-perf/cperf_options_parsing.c | 137 +++++-- app/test-crypto-perf/cperf_test_latency.c | 8 +- app/test-crypto-perf/cperf_test_throughput.c | 8 +- app/test-crypto-perf/cperf_test_vector_parsing.c | 12 +- app/test-crypto-perf/cperf_test_vectors.c | 106 +++--- app/test-crypto-perf/cperf_test_vectors.h | 12 + app/test-crypto-perf/cperf_test_verify.c | 10 +- app/test-crypto-perf/main.c | 40 +- doc/guides/sample_app_ug/ipsec_secgw.rst | 43 ++- doc/guides/sample_app_ug/l2_forward_crypto.rst | 24 +- doc/guides/tools/cryptoperf.rst | 36 +- drivers/crypto/aesni_gcm/aesni_gcm_pmd.c | 76 ++-- drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c | 24 +- drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c | 8 - drivers/crypto/openssl/rte_openssl_pmd.c | 140 +++++-- drivers/crypto/openssl/rte_openssl_pmd_ops.c | 26 +- drivers/crypto/openssl/rte_openssl_pmd_private.h | 4 + drivers/crypto/qat/qat_crypto.c | 187 +++++++--- drivers/crypto/qat/qat_crypto.h | 4 + drivers/crypto/qat/qat_crypto_capabilities.h | 34 +- examples/ipsec-secgw/esp.c | 223 +++++++----- examples/ipsec-secgw/ipsec.h | 1 + examples/ipsec-secgw/sa.c | 298 ++++++++++----- examples/l2fwd-crypto/main.c | 386 +++++++++++++++++--- lib/librte_cryptodev/rte_crypto_sym.h | 105 +----- lib/librte_cryptodev/rte_cryptodev.c | 6 +- lib/librte_cryptodev/rte_cryptodev.h | 2 + test/test/test_cryptodev.c | 218 +++++------ test/test/test_cryptodev_perf.c | 446 ++++++++++++----------- 31 files changed, 1713 insertions(+), 1066 deletions(-) diff --git a/app/test-crypto-perf/cperf_ops.c b/app/test-crypto-perf/cperf_ops.c index 9548584..a76ad8c 100644 --- a/app/test-crypto-perf/cperf_ops.c +++ b/app/test-crypto-perf/cperf_ops.c @@ -176,8 +176,6 @@ cperf_set_ops_auth(struct rte_crypto_op **ops, uint8_t *, offset); sym_op->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(buf, offset); - sym_op->auth.aad.phys_addr = test_vector->aad.phys_addr; - sym_op->auth.aad.data = test_vector->aad.data; } @@ -262,8 +260,6 @@ cperf_set_ops_cipher_auth(struct rte_crypto_op **ops, uint8_t *, offset); sym_op->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(buf, offset); - sym_op->auth.aad.phys_addr = test_vector->aad.phys_addr; - sym_op->auth.aad.data = test_vector->aad.data; } if (options->auth_algo == RTE_CRYPTO_AUTH_SNOW3G_UIA2 || @@ -301,23 +297,22 @@ cperf_set_ops_aead(struct rte_crypto_op **ops, test_vector->cipher_iv.data, test_vector->cipher_iv.length); - /* cipher parameters */ - sym_op->cipher.data.length = options->test_buffer_size; - sym_op->cipher.data.offset = - RTE_ALIGN_CEIL(options->auth_aad_sz, 16); + /* AEAD parameters */ + sym_op->aead.data.length = options->test_buffer_size; + sym_op->aead.data.offset = + RTE_ALIGN_CEIL(options->aead_aad_sz, 16); - sym_op->auth.aad.data = rte_pktmbuf_mtod(bufs_in[i], uint8_t *); - sym_op->auth.aad.phys_addr = rte_pktmbuf_mtophys(bufs_in[i]); + sym_op->aead.aad.data = rte_pktmbuf_mtod(bufs_in[i], uint8_t *); + sym_op->aead.aad.phys_addr = rte_pktmbuf_mtophys(bufs_in[i]); - /* authentication parameters */ - if (options->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { - sym_op->auth.digest.data = test_vector->digest.data; - sym_op->auth.digest.phys_addr = + if (options->aead_op == RTE_CRYPTO_AEAD_OP_DECRYPT) { + sym_op->aead.digest.data = test_vector->digest.data; + sym_op->aead.digest.phys_addr = test_vector->digest.phys_addr; } else { - uint32_t offset = sym_op->cipher.data.length + - sym_op->cipher.data.offset; + uint32_t offset = sym_op->aead.data.length + + sym_op->aead.data.offset; struct rte_mbuf *buf, *tbuf; if (options->out_of_place) { @@ -333,14 +328,11 @@ cperf_set_ops_aead(struct rte_crypto_op **ops, } } - sym_op->auth.digest.data = rte_pktmbuf_mtod_offset(buf, + sym_op->aead.digest.data = rte_pktmbuf_mtod_offset(buf, uint8_t *, offset); - sym_op->auth.digest.phys_addr = + sym_op->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset(buf, offset); } - - sym_op->auth.data.length = options->test_buffer_size; - sym_op->auth.data.offset = options->auth_aad_sz; } return 0; @@ -354,6 +346,7 @@ cperf_create_session(uint8_t dev_id, { struct rte_crypto_sym_xform cipher_xform; struct rte_crypto_sym_xform auth_xform; + struct rte_crypto_sym_xform aead_xform; struct rte_cryptodev_sym_session *sess = NULL; /* @@ -393,9 +386,7 @@ cperf_create_session(uint8_t dev_id, /* auth different than null */ if (options->auth_algo != RTE_CRYPTO_AUTH_NULL) { auth_xform.auth.digest_length = - options->auth_digest_sz; - auth_xform.auth.add_auth_data_length = - options->auth_aad_sz; + options->digest_sz; auth_xform.auth.key.length = test_vector->auth_key.length; auth_xform.auth.key.data = test_vector->auth_key.data; @@ -414,9 +405,7 @@ cperf_create_session(uint8_t dev_id, * cipher and auth */ } else if (options->op_type == CPERF_CIPHER_THEN_AUTH - || options->op_type == CPERF_AUTH_THEN_CIPHER - || options->op_type == CPERF_AEAD) { - + || options->op_type == CPERF_AUTH_THEN_CIPHER) { /* * cipher */ @@ -450,23 +439,12 @@ cperf_create_session(uint8_t dev_id, /* auth different than null */ if (options->auth_algo != RTE_CRYPTO_AUTH_NULL) { - auth_xform.auth.digest_length = options->auth_digest_sz; - auth_xform.auth.add_auth_data_length = - options->auth_aad_sz; - /* auth options for aes gcm */ - if (options->cipher_algo == RTE_CRYPTO_CIPHER_AES_GCM && - options->auth_algo == RTE_CRYPTO_AUTH_AES_GCM) { - auth_xform.auth.key.length = 0; - auth_xform.auth.key.data = NULL; - auth_xform.auth.iv.length = 0; - } else { /* auth options for others */ - auth_xform.auth.key.length = + auth_xform.auth.digest_length = options->digest_sz; + auth_xform.auth.iv.length = test_vector->auth_iv.length; + auth_xform.auth.key.length = test_vector->auth_key.length; - auth_xform.auth.key.data = - test_vector->auth_key.data; - auth_xform.auth.iv.length = - test_vector->auth_iv.length; - } + auth_xform.auth.key.data = + test_vector->auth_key.data; } else { auth_xform.auth.digest_length = 0; auth_xform.auth.add_auth_data_length = 0; @@ -475,35 +453,39 @@ cperf_create_session(uint8_t dev_id, auth_xform.auth.iv.length = 0; } - /* create crypto session for aes gcm */ - if (options->cipher_algo == RTE_CRYPTO_CIPHER_AES_GCM) { - if (options->cipher_op == - RTE_CRYPTO_CIPHER_OP_ENCRYPT) { - cipher_xform.next = &auth_xform; - /* create crypto session */ - sess = rte_cryptodev_sym_session_create(dev_id, - &cipher_xform); - } else { /* decrypt */ - auth_xform.next = &cipher_xform; - /* create crypto session */ - sess = rte_cryptodev_sym_session_create(dev_id, - &auth_xform); - } - } else { /* create crypto session for other */ - /* cipher then auth */ - if (options->op_type == CPERF_CIPHER_THEN_AUTH) { - cipher_xform.next = &auth_xform; - /* create crypto session */ - sess = rte_cryptodev_sym_session_create(dev_id, + /* cipher then auth */ + if (options->op_type == CPERF_CIPHER_THEN_AUTH) { + cipher_xform.next = &auth_xform; + /* create crypto session */ + sess = rte_cryptodev_sym_session_create(dev_id, &cipher_xform); - } else { /* auth then cipher */ - auth_xform.next = &cipher_xform; - /* create crypto session */ - sess = rte_cryptodev_sym_session_create(dev_id, - &auth_xform); - } + } else { /* auth then cipher */ + auth_xform.next = &cipher_xform; + /* create crypto session */ + sess = rte_cryptodev_sym_session_create(dev_id, + &auth_xform); } + } else { /* options->op_type == CPERF_AEAD */ + aead_xform.type = RTE_CRYPTO_SYM_XFORM_AEAD; + aead_xform.next = NULL; + aead_xform.aead.algo = options->aead_algo; + aead_xform.aead.op = options->aead_op; + aead_xform.aead.iv.offset = iv_offset; + + aead_xform.aead.key.data = + test_vector->aead_key.data; + aead_xform.aead.key.length = + test_vector->aead_key.length; + aead_xform.aead.iv.length = test_vector->aead_iv.length; + + aead_xform.aead.digest_length = options->digest_sz; + aead_xform.aead.add_auth_data_length = + options->aead_aad_sz; + + /* Create crypto session */ + sess = rte_cryptodev_sym_session_create(dev_id, &aead_xform); } + return sess; } @@ -515,14 +497,14 @@ cperf_get_op_functions(const struct cperf_options *options, op_fns->sess_create = cperf_create_session; - if (options->op_type == CPERF_AEAD - || options->op_type == CPERF_AUTH_THEN_CIPHER + if (options->op_type == CPERF_AEAD) { + op_fns->populate_ops = cperf_set_ops_aead; + return 0; + } + + if (options->op_type == CPERF_AUTH_THEN_CIPHER || options->op_type == CPERF_CIPHER_THEN_AUTH) { - if (options->cipher_algo == RTE_CRYPTO_CIPHER_AES_GCM && - options->auth_algo == RTE_CRYPTO_AUTH_AES_GCM) - op_fns->populate_ops = cperf_set_ops_aead; - else - op_fns->populate_ops = cperf_set_ops_cipher_auth; + op_fns->populate_ops = cperf_set_ops_cipher_auth; return 0; } if (options->op_type == CPERF_AUTH_ONLY) { diff --git a/app/test-crypto-perf/cperf_options.h b/app/test-crypto-perf/cperf_options.h index 0e53c03..2f1e282 100644 --- a/app/test-crypto-perf/cperf_options.h +++ b/app/test-crypto-perf/cperf_options.h @@ -29,8 +29,15 @@ #define CPERF_AUTH_OP ("auth-op") #define CPERF_AUTH_KEY_SZ ("auth-key-sz") #define CPERF_AUTH_IV_SZ ("auth-iv-sz") -#define CPERF_AUTH_DIGEST_SZ ("auth-digest-sz") -#define CPERF_AUTH_AAD_SZ ("auth-aad-sz") + +#define CPERF_AEAD_ALGO ("aead-algo") +#define CPERF_AEAD_OP ("aead-op") +#define CPERF_AEAD_KEY_SZ ("aead-key-sz") +#define CPERF_AEAD_IV_SZ ("aead-iv-sz") +#define CPERF_AEAD_AAD_SZ ("aead-aad-sz") + +#define CPERF_DIGEST_SZ ("digest-sz") + #define CPERF_CSV ("csv-friendly") #define MAX_LIST 32 @@ -79,7 +86,15 @@ struct cperf_options { uint16_t auth_key_sz; uint16_t auth_iv_sz; uint16_t auth_digest_sz; - uint16_t auth_aad_sz; + + enum rte_crypto_aead_algorithm aead_algo; + enum rte_crypto_aead_operation aead_op; + + uint16_t aead_key_sz; + uint16_t aead_iv_sz; + uint16_t aead_aad_sz; + + uint16_t digest_sz; char device_type[RTE_CRYPTODEV_NAME_LEN]; enum cperf_op_type op_type; diff --git a/app/test-crypto-perf/cperf_options_parsing.c b/app/test-crypto-perf/cperf_options_parsing.c index 5c2dcff..3ec2c50 100644 --- a/app/test-crypto-perf/cperf_options_parsing.c +++ b/app/test-crypto-perf/cperf_options_parsing.c @@ -543,9 +543,9 @@ parse_auth_key_sz(struct cperf_options *opts, const char *arg) } static int -parse_auth_digest_sz(struct cperf_options *opts, const char *arg) +parse_digest_sz(struct cperf_options *opts, const char *arg) { - return parse_uint16_t(&opts->auth_digest_sz, arg); + return parse_uint16_t(&opts->digest_sz, arg); } static int @@ -555,9 +555,64 @@ parse_auth_iv_sz(struct cperf_options *opts, const char *arg) } static int -parse_auth_aad_sz(struct cperf_options *opts, const char *arg) +parse_aead_algo(struct cperf_options *opts, const char *arg) { - return parse_uint16_t(&opts->auth_aad_sz, arg); + enum rte_crypto_aead_algorithm aead_algo; + + if (rte_cryptodev_get_aead_algo_enum(&aead_algo, arg) < 0) { + RTE_LOG(ERR, USER1, "Invalid AEAD algorithm specified\n"); + return -1; + } + + opts->aead_algo = aead_algo; + + return 0; +} + +static int +parse_aead_op(struct cperf_options *opts, const char *arg) +{ + struct name_id_map aead_op_namemap[] = { + { + rte_crypto_aead_operation_strings + [RTE_CRYPTO_AEAD_OP_ENCRYPT], + RTE_CRYPTO_AEAD_OP_ENCRYPT }, + { + rte_crypto_aead_operation_strings + [RTE_CRYPTO_AEAD_OP_DECRYPT], + RTE_CRYPTO_AEAD_OP_DECRYPT + } + }; + + int id = get_str_key_id_mapping(aead_op_namemap, + RTE_DIM(aead_op_namemap), arg); + if (id < 0) { + RTE_LOG(ERR, USER1, "invalid AEAD operation specified" + "\n"); + return -1; + } + + opts->aead_op = (enum rte_crypto_aead_operation)id; + + return 0; +} + +static int +parse_aead_key_sz(struct cperf_options *opts, const char *arg) +{ + return parse_uint16_t(&opts->aead_key_sz, arg); +} + +static int +parse_aead_iv_sz(struct cperf_options *opts, const char *arg) +{ + return parse_uint16_t(&opts->aead_iv_sz, arg); +} + +static int +parse_aead_aad_sz(struct cperf_options *opts, const char *arg) +{ + return parse_uint16_t(&opts->aead_aad_sz, arg); } static int @@ -606,8 +661,17 @@ static struct option lgopts[] = { { CPERF_AUTH_OP, required_argument, 0, 0 }, { CPERF_AUTH_KEY_SZ, required_argument, 0, 0 }, - { CPERF_AUTH_DIGEST_SZ, required_argument, 0, 0 }, - { CPERF_AUTH_AAD_SZ, required_argument, 0, 0 }, + { CPERF_AUTH_IV_SZ, required_argument, 0, 0 }, + + { CPERF_AEAD_ALGO, required_argument, 0, 0 }, + { CPERF_AEAD_OP, required_argument, 0, 0 }, + + { CPERF_AEAD_KEY_SZ, required_argument, 0, 0 }, + { CPERF_AEAD_AAD_SZ, required_argument, 0, 0 }, + { CPERF_AEAD_IV_SZ, required_argument, 0, 0 }, + + { CPERF_DIGEST_SZ, required_argument, 0, 0 }, + { CPERF_CSV, no_argument, 0, 0}, { NULL, 0, 0, 0 } @@ -658,7 +722,12 @@ cperf_options_default(struct cperf_options *opts) opts->auth_key_sz = 64; opts->auth_digest_sz = 12; opts->auth_iv_sz = 0; - opts->auth_aad_sz = 0; + + opts->aead_key_sz = 0; + opts->aead_iv_sz = 0; + opts->aead_aad_sz = 0; + + opts->digest_sz = 12; } static int @@ -686,9 +755,13 @@ cperf_opts_parse_long(int opt_idx, struct cperf_options *opts) { CPERF_AUTH_OP, parse_auth_op }, { CPERF_AUTH_KEY_SZ, parse_auth_key_sz }, { CPERF_AUTH_IV_SZ, parse_auth_iv_sz }, - { CPERF_AUTH_DIGEST_SZ, parse_auth_digest_sz }, - { CPERF_AUTH_AAD_SZ, parse_auth_aad_sz }, - { CPERF_CSV, parse_csv_friendly}, + { CPERF_AEAD_ALGO, parse_aead_algo }, + { CPERF_AEAD_OP, parse_aead_op }, + { CPERF_AEAD_KEY_SZ, parse_aead_key_sz }, + { CPERF_AEAD_IV_SZ, parse_aead_iv_sz }, + { CPERF_AEAD_AAD_SZ, parse_aead_aad_sz }, + { CPERF_DIGEST_SZ, parse_digest_sz }, + { CPERF_CSV, parse_csv_friendly}, }; unsigned int i; @@ -803,30 +876,7 @@ cperf_options_check(struct cperf_options *options) " options: decrypt and verify.\n"); return -EINVAL; } - } else if (options->op_type == CPERF_AEAD) { - if (!(options->cipher_op == RTE_CRYPTO_CIPHER_OP_ENCRYPT && - options->auth_op == - RTE_CRYPTO_AUTH_OP_GENERATE) && - !(options->cipher_op == - RTE_CRYPTO_CIPHER_OP_DECRYPT && - options->auth_op == - RTE_CRYPTO_AUTH_OP_VERIFY)) { - RTE_LOG(ERR, USER1, "Use together options: encrypt and" - " generate or decrypt and verify.\n"); - return -EINVAL; - } - } - - if (options->cipher_algo == RTE_CRYPTO_CIPHER_AES_GCM || - options->cipher_algo == RTE_CRYPTO_CIPHER_AES_CCM || - options->auth_algo == RTE_CRYPTO_AUTH_AES_GCM || - options->auth_algo == RTE_CRYPTO_AUTH_AES_CCM) { - if (options->op_type != CPERF_AEAD) { - RTE_LOG(ERR, USER1, "Use --optype aead\n"); - return -EINVAL; - } } - if (options->cipher_algo == RTE_CRYPTO_CIPHER_AES_CBC || options->cipher_algo == RTE_CRYPTO_CIPHER_AES_ECB) { if (options->inc_buffer_size != 0) @@ -914,23 +964,20 @@ cperf_options_dump(struct cperf_options *opts) if (opts->op_type == CPERF_AUTH_ONLY || opts->op_type == CPERF_CIPHER_THEN_AUTH || - opts->op_type == CPERF_AUTH_THEN_CIPHER || - opts->op_type == CPERF_AEAD) { + opts->op_type == CPERF_AUTH_THEN_CIPHER) { printf("# auth algorithm: %s\n", rte_crypto_auth_algorithm_strings[opts->auth_algo]); printf("# auth operation: %s\n", rte_crypto_auth_operation_strings[opts->auth_op]); printf("# auth key size: %u\n", opts->auth_key_sz); printf("# auth iv size: %u\n", opts->auth_iv_sz); - printf("# auth digest size: %u\n", opts->auth_digest_sz); - printf("# auth aad size: %u\n", opts->auth_aad_sz); + printf("# auth digest size: %u\n", opts->digest_sz); printf("#\n"); } if (opts->op_type == CPERF_CIPHER_ONLY || opts->op_type == CPERF_CIPHER_THEN_AUTH || - opts->op_type == CPERF_AUTH_THEN_CIPHER || - opts->op_type == CPERF_AEAD) { + opts->op_type == CPERF_AUTH_THEN_CIPHER) { printf("# cipher algorithm: %s\n", rte_crypto_cipher_algorithm_strings[opts->cipher_algo]); printf("# cipher operation: %s\n", @@ -939,4 +986,16 @@ cperf_options_dump(struct cperf_options *opts) printf("# cipher iv size: %u\n", opts->cipher_iv_sz); printf("#\n"); } + + if (opts->op_type == CPERF_AEAD) { + printf("# aead algorithm: %s\n", + rte_crypto_aead_algorithm_strings[opts->aead_algo]); + printf("# aead operation: %s\n", + rte_crypto_aead_operation_strings[opts->aead_op]); + printf("# aead key size: %u\n", opts->aead_key_sz); + printf("# aead iv size: %u\n", opts->aead_iv_sz); + printf("# aead digest size: %u\n", opts->digest_sz); + printf("# aead aad size: %u\n", opts->aead_aad_sz); + printf("#\n"); + } } diff --git a/app/test-crypto-perf/cperf_test_latency.c b/app/test-crypto-perf/cperf_test_latency.c index dc0ba84..7a89b97 100644 --- a/app/test-crypto-perf/cperf_test_latency.c +++ b/app/test-crypto-perf/cperf_test_latency.c @@ -168,14 +168,14 @@ cperf_mbuf_create(struct rte_mempool *mempool, if (options->op_type != CPERF_CIPHER_ONLY) { mbuf_data = (uint8_t *)rte_pktmbuf_append(mbuf, - options->auth_digest_sz); + options->digest_sz); if (mbuf_data == NULL) goto error; } if (options->op_type == CPERF_AEAD) { uint8_t *aead = (uint8_t *)rte_pktmbuf_prepend(mbuf, - RTE_ALIGN_CEIL(options->auth_aad_sz, 16)); + RTE_ALIGN_CEIL(options->aead_aad_sz, 16)); if (aead == NULL) goto error; @@ -230,7 +230,7 @@ cperf_latency_test_constructor(uint8_t dev_id, uint16_t qp_id, RTE_CACHE_LINE_ROUNDUP( (options->max_buffer_size / options->segments_nb) + (options->max_buffer_size % options->segments_nb) + - options->auth_digest_sz), + options->digest_sz), rte_socket_id()); if (ctx->pkt_mbuf_pool_in == NULL) @@ -260,7 +260,7 @@ cperf_latency_test_constructor(uint8_t dev_id, uint16_t qp_id, RTE_PKTMBUF_HEADROOM + RTE_CACHE_LINE_ROUNDUP( options->max_buffer_size + - options->auth_digest_sz), + options->digest_sz), rte_socket_id()); if (ctx->pkt_mbuf_pool_out == NULL) diff --git a/app/test-crypto-perf/cperf_test_throughput.c b/app/test-crypto-perf/cperf_test_throughput.c index 1e3f3b3..1ff1560 100644 --- a/app/test-crypto-perf/cperf_test_throughput.c +++ b/app/test-crypto-perf/cperf_test_throughput.c @@ -151,14 +151,14 @@ cperf_mbuf_create(struct rte_mempool *mempool, if (options->op_type != CPERF_CIPHER_ONLY) { mbuf_data = (uint8_t *)rte_pktmbuf_append(mbuf, - options->auth_digest_sz); + options->digest_sz); if (mbuf_data == NULL) goto error; } if (options->op_type == CPERF_AEAD) { uint8_t *aead = (uint8_t *)rte_pktmbuf_prepend(mbuf, - RTE_ALIGN_CEIL(options->auth_aad_sz, 16)); + RTE_ALIGN_CEIL(options->aead_aad_sz, 16)); if (aead == NULL) goto error; @@ -212,7 +212,7 @@ cperf_throughput_test_constructor(uint8_t dev_id, uint16_t qp_id, RTE_CACHE_LINE_ROUNDUP( (options->max_buffer_size / options->segments_nb) + (options->max_buffer_size % options->segments_nb) + - options->auth_digest_sz), + options->digest_sz), rte_socket_id()); if (ctx->pkt_mbuf_pool_in == NULL) @@ -240,7 +240,7 @@ cperf_throughput_test_constructor(uint8_t dev_id, uint16_t qp_id, RTE_PKTMBUF_HEADROOM + RTE_CACHE_LINE_ROUNDUP( options->max_buffer_size + - options->auth_digest_sz), + options->digest_sz), rte_socket_id()); if (ctx->pkt_mbuf_pool_out == NULL) diff --git a/app/test-crypto-perf/cperf_test_vector_parsing.c b/app/test-crypto-perf/cperf_test_vector_parsing.c index 277ff1e..e462d2c 100644 --- a/app/test-crypto-perf/cperf_test_vector_parsing.c +++ b/app/test-crypto-perf/cperf_test_vector_parsing.c @@ -363,12 +363,12 @@ parse_entry(char *entry, struct cperf_test_vector *vector, if (tc_found) vector->aad.length = data_length; else { - if (opts->auth_aad_sz > data_length) { + if (opts->aead_aad_sz > data_length) { printf("Global aad shorter than " - "auth_aad_sz\n"); + "aead_aad_sz\n"); return -1; } - vector->aad.length = opts->auth_aad_sz; + vector->aad.length = opts->aead_aad_sz; } } else if (strstr(key_token, "digest")) { @@ -379,12 +379,12 @@ parse_entry(char *entry, struct cperf_test_vector *vector, if (tc_found) vector->digest.length = data_length; else { - if (opts->auth_digest_sz > data_length) { + if (opts->digest_sz > data_length) { printf("Global digest shorter than " - "auth_digest_sz\n"); + "digest_sz\n"); return -1; } - vector->digest.length = opts->auth_digest_sz; + vector->digest.length = opts->digest_sz; } } else { printf("Not valid key: '%s'\n", trim_space(key_token)); diff --git a/app/test-crypto-perf/cperf_test_vectors.c b/app/test-crypto-perf/cperf_test_vectors.c index 2e5339c..03bc995 100644 --- a/app/test-crypto-perf/cperf_test_vectors.c +++ b/app/test-crypto-perf/cperf_test_vectors.c @@ -385,6 +385,13 @@ uint8_t auth_key[] = { 0x78, 0x79, 0x7A, 0x7B, 0x7C, 0x7D, 0x7E, 0x7F }; +/* AEAD key */ +uint8_t aead_key[] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B, + 0x0C, 0x0D, 0x0E, 0x0F, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F +}; + /* Digests */ uint8_t digest[2048] = { 0x00 }; @@ -403,8 +410,7 @@ cperf_test_vector_get_dummy(struct cperf_options *options) if (options->op_type == CPERF_CIPHER_ONLY || options->op_type == CPERF_CIPHER_THEN_AUTH || - options->op_type == CPERF_AUTH_THEN_CIPHER || - options->op_type == CPERF_AEAD) { + options->op_type == CPERF_AUTH_THEN_CIPHER) { if (options->cipher_algo == RTE_CRYPTO_CIPHER_NULL) { t_vec->cipher_key.length = 0; t_vec->ciphertext.data = plaintext; @@ -441,40 +447,32 @@ cperf_test_vector_get_dummy(struct cperf_options *options) if (options->op_type == CPERF_AUTH_ONLY || options->op_type == CPERF_CIPHER_THEN_AUTH || - options->op_type == CPERF_AUTH_THEN_CIPHER || - options->op_type == CPERF_AEAD) { - uint8_t aad_alloc = 0; - - t_vec->auth_key.length = options->auth_key_sz; - - switch (options->auth_algo) { - case RTE_CRYPTO_AUTH_NULL: - t_vec->auth_key.data = NULL; - aad_alloc = 0; - break; - case RTE_CRYPTO_AUTH_AES_GCM: + options->op_type == CPERF_AUTH_THEN_CIPHER) { + if (options->auth_algo == RTE_CRYPTO_AUTH_NULL) { + t_vec->auth_key.length = 0; t_vec->auth_key.data = NULL; - aad_alloc = 1; - break; - default: + t_vec->digest.data = NULL; + t_vec->digest.length = 0; + } else { + t_vec->auth_key.length = options->auth_key_sz; t_vec->auth_key.data = auth_key; - aad_alloc = 0; - break; - } - if (aad_alloc && options->auth_aad_sz) { - t_vec->aad.data = rte_malloc(NULL, - options->auth_aad_sz, 16); - if (t_vec->aad.data == NULL) { - if (options->op_type != CPERF_AUTH_ONLY) - rte_free(t_vec->cipher_iv.data); + t_vec->digest.data = rte_malloc(NULL, + options->digest_sz, + 16); + if (t_vec->digest.data == NULL) { + rte_free(t_vec->cipher_iv.data); rte_free(t_vec); return NULL; } - memcpy(t_vec->aad.data, aad, options->auth_aad_sz); - } else { - t_vec->aad.data = NULL; + t_vec->digest.phys_addr = + rte_malloc_virt2phy(t_vec->digest.data); + t_vec->digest.length = options->digest_sz; + memcpy(t_vec->digest.data, digest, + options->digest_sz); } + t_vec->data.auth_offset = 0; + t_vec->data.auth_length = options->max_buffer_size; /* Set IV parameters */ t_vec->auth_iv.data = rte_malloc(NULL, options->auth_iv_sz, @@ -487,26 +485,52 @@ cperf_test_vector_get_dummy(struct cperf_options *options) } memcpy(t_vec->auth_iv.data, iv, options->auth_iv_sz); t_vec->auth_iv.length = options->auth_iv_sz; + } - t_vec->aad.phys_addr = rte_malloc_virt2phy(t_vec->aad.data); - t_vec->aad.length = options->auth_aad_sz; - t_vec->digest.data = rte_malloc(NULL, options->auth_digest_sz, - 16); + if (options->op_type == CPERF_AEAD) { + t_vec->aead_key.length = options->aead_key_sz; + t_vec->aead_key.data = aead_key; + + if (options->aead_aad_sz) { + t_vec->aad.data = rte_malloc(NULL, + options->aead_aad_sz, 16); + if (t_vec->aad.data == NULL) { + rte_free(t_vec); + return NULL; + } + memcpy(t_vec->aad.data, aad, options->aead_aad_sz); + t_vec->aad.phys_addr = rte_malloc_virt2phy(t_vec->aad.data); + t_vec->aad.length = options->aead_aad_sz; + } else { + t_vec->aad.data = NULL; + t_vec->aad.length = 0; + } + + t_vec->digest.data = rte_malloc(NULL, options->digest_sz, + 16); if (t_vec->digest.data == NULL) { - if (options->op_type != CPERF_AUTH_ONLY) - rte_free(t_vec->cipher_iv.data); - rte_free(t_vec->auth_iv.data); rte_free(t_vec->aad.data); rte_free(t_vec); return NULL; } t_vec->digest.phys_addr = rte_malloc_virt2phy(t_vec->digest.data); - t_vec->digest.length = options->auth_digest_sz; - memcpy(t_vec->digest.data, digest, options->auth_digest_sz); - t_vec->data.auth_offset = 0; - t_vec->data.auth_length = options->max_buffer_size; - } + t_vec->digest.length = options->digest_sz; + memcpy(t_vec->digest.data, digest, options->digest_sz); + t_vec->data.aead_offset = 0; + t_vec->data.aead_length = options->max_buffer_size; + /* Set IV parameters */ + t_vec->aead_iv.data = rte_malloc(NULL, options->aead_iv_sz, + 16); + if (options->aead_iv_sz && t_vec->aead_iv.data == NULL) { + rte_free(t_vec->aad.data); + rte_free(t_vec->digest.data); + rte_free(t_vec); + return NULL; + } + memcpy(t_vec->aead_iv.data, iv, options->aead_iv_sz); + t_vec->aead_iv.length = options->aead_iv_sz; + } return t_vec; } diff --git a/app/test-crypto-perf/cperf_test_vectors.h b/app/test-crypto-perf/cperf_test_vectors.h index 7f9c4fa..8595570 100644 --- a/app/test-crypto-perf/cperf_test_vectors.h +++ b/app/test-crypto-perf/cperf_test_vectors.h @@ -54,6 +54,11 @@ struct cperf_test_vector { struct { uint8_t *data; uint16_t length; + } aead_key; + + struct { + uint8_t *data; + uint16_t length; } cipher_iv; struct { @@ -63,6 +68,11 @@ struct cperf_test_vector { struct { uint8_t *data; + uint16_t length; + } aead_iv; + + struct { + uint8_t *data; uint32_t length; } ciphertext; @@ -83,6 +93,8 @@ struct cperf_test_vector { uint32_t auth_length; uint32_t cipher_offset; uint32_t cipher_length; + uint32_t aead_offset; + uint32_t aead_length; } data; }; diff --git a/app/test-crypto-perf/cperf_test_verify.c b/app/test-crypto-perf/cperf_test_verify.c index 81057ff..bba8019 100644 --- a/app/test-crypto-perf/cperf_test_verify.c +++ b/app/test-crypto-perf/cperf_test_verify.c @@ -155,14 +155,14 @@ cperf_mbuf_create(struct rte_mempool *mempool, if (options->op_type != CPERF_CIPHER_ONLY) { mbuf_data = (uint8_t *)rte_pktmbuf_append(mbuf, - options->auth_digest_sz); + options->digest_sz); if (mbuf_data == NULL) goto error; } if (options->op_type == CPERF_AEAD) { uint8_t *aead = (uint8_t *)rte_pktmbuf_prepend(mbuf, - RTE_ALIGN_CEIL(options->auth_aad_sz, 16)); + RTE_ALIGN_CEIL(options->aead_aad_sz, 16)); if (aead == NULL) goto error; @@ -216,7 +216,7 @@ cperf_verify_test_constructor(uint8_t dev_id, uint16_t qp_id, RTE_CACHE_LINE_ROUNDUP( (options->max_buffer_size / options->segments_nb) + (options->max_buffer_size % options->segments_nb) + - options->auth_digest_sz), + options->digest_sz), rte_socket_id()); if (ctx->pkt_mbuf_pool_in == NULL) @@ -244,7 +244,7 @@ cperf_verify_test_constructor(uint8_t dev_id, uint16_t qp_id, RTE_PKTMBUF_HEADROOM + RTE_CACHE_LINE_ROUNDUP( options->max_buffer_size + - options->auth_digest_sz), + options->digest_sz), rte_socket_id()); if (ctx->pkt_mbuf_pool_out == NULL) @@ -379,7 +379,7 @@ cperf_verify_op(struct rte_crypto_op *op, if (options->auth_op == RTE_CRYPTO_AUTH_OP_GENERATE) res += memcmp(data + auth_offset, vector->digest.data, - options->auth_digest_sz); + options->digest_sz); } return !!res; diff --git a/app/test-crypto-perf/main.c b/app/test-crypto-perf/main.c index cf4fa4f..b39b1cb 100644 --- a/app/test-crypto-perf/main.c +++ b/app/test-crypto-perf/main.c @@ -123,8 +123,7 @@ cperf_verify_devices_capabilities(struct cperf_options *opts, if (opts->op_type == CPERF_AUTH_ONLY || opts->op_type == CPERF_CIPHER_THEN_AUTH || - opts->op_type == CPERF_AUTH_THEN_CIPHER || - opts->op_type == CPERF_AEAD) { + opts->op_type == CPERF_AUTH_THEN_CIPHER) { cap_idx.type = RTE_CRYPTO_SYM_XFORM_AUTH; cap_idx.algo.auth = opts->auth_algo; @@ -138,7 +137,7 @@ cperf_verify_devices_capabilities(struct cperf_options *opts, capability, opts->auth_key_sz, opts->auth_digest_sz, - opts->auth_aad_sz, + 0, opts->auth_iv_sz); if (ret != 0) return ret; @@ -146,8 +145,7 @@ cperf_verify_devices_capabilities(struct cperf_options *opts, if (opts->op_type == CPERF_CIPHER_ONLY || opts->op_type == CPERF_CIPHER_THEN_AUTH || - opts->op_type == CPERF_AUTH_THEN_CIPHER || - opts->op_type == CPERF_AEAD) { + opts->op_type == CPERF_AUTH_THEN_CIPHER) { cap_idx.type = RTE_CRYPTO_SYM_XFORM_CIPHER; cap_idx.algo.cipher = opts->cipher_algo; @@ -164,6 +162,26 @@ cperf_verify_devices_capabilities(struct cperf_options *opts, if (ret != 0) return ret; } + + if (opts->op_type == CPERF_AEAD) { + + cap_idx.type = RTE_CRYPTO_SYM_XFORM_AEAD; + cap_idx.algo.aead = opts->aead_algo; + + capability = rte_cryptodev_sym_capability_get(cdev_id, + &cap_idx); + if (capability == NULL) + return -1; + + ret = rte_cryptodev_sym_capability_check_aead( + capability, + opts->aead_key_sz, + opts->digest_sz, + opts->aead_aad_sz, + opts->aead_iv_sz); + if (ret != 0) + return ret; + } } return 0; @@ -212,7 +230,7 @@ cperf_check_test_vector(struct cperf_options *opts, return -1; if (test_vec->digest.data == NULL) return -1; - if (test_vec->digest.length < opts->auth_digest_sz) + if (test_vec->digest.length < opts->digest_sz) return -1; } @@ -253,7 +271,7 @@ cperf_check_test_vector(struct cperf_options *opts, return -1; if (test_vec->digest.data == NULL) return -1; - if (test_vec->digest.length < opts->auth_digest_sz) + if (test_vec->digest.length < opts->digest_sz) return -1; } } else if (opts->op_type == CPERF_AEAD) { @@ -265,17 +283,17 @@ cperf_check_test_vector(struct cperf_options *opts, return -1; if (test_vec->ciphertext.length < opts->max_buffer_size) return -1; - if (test_vec->cipher_iv.data == NULL) + if (test_vec->aead_iv.data == NULL) return -1; - if (test_vec->cipher_iv.length != opts->cipher_iv_sz) + if (test_vec->aead_iv.length != opts->aead_iv_sz) return -1; if (test_vec->aad.data == NULL) return -1; - if (test_vec->aad.length != opts->auth_aad_sz) + if (test_vec->aad.length != opts->aead_aad_sz) return -1; if (test_vec->digest.data == NULL) return -1; - if (test_vec->digest.length < opts->auth_digest_sz) + if (test_vec->digest.length < opts->digest_sz) return -1; } return 0; diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst index 885c77e..75b960f 100644 --- a/doc/guides/sample_app_ug/ipsec_secgw.rst +++ b/doc/guides/sample_app_ug/ipsec_secgw.rst @@ -412,14 +412,13 @@ where each options means: * Cipher algorithm - * Optional: No + * Optional: Yes, unless is not used * Available options: * *null*: NULL algorithm * *aes-128-cbc*: AES-CBC 128-bit algorithm * *aes-128-ctr*: AES-CTR 128-bit algorithm - * *aes-128-gcm*: AES-GCM 128-bit algorithm * Syntax: *cipher_algo * @@ -427,7 +426,8 @@ where each options means: * Cipher key, NOT available when 'null' algorithm is used - * Optional: No, must followed by option + * Optional: Yes, unless is not used. + Must be followed by option * Syntax: Hexadecimal bytes (0x0-0xFF) concatenate by colon symbol ':'. The number of bytes should be as same as the specified cipher algorithm @@ -440,20 +440,20 @@ where each options means: * Authentication algorithm - * Optional: No + * Optional: Yes, unless is not used * Available options: * *null*: NULL algorithm * *sha1-hmac*: HMAC SHA1 algorithm - * *aes-128-gcm*: AES-GCM 128-bit algorithm ```` * Authentication key, NOT available when 'null' or 'aes-128-gcm' algorithm is used. - * Optional: No, must followed by option + * Optional: Yes, unless is not used. + Must be followed by option * Syntax: Hexadecimal bytes (0x0-0xFF) concatenate by colon symbol ':'. The number of bytes should be as same as the specified authentication @@ -462,6 +462,32 @@ where each options means: For example: *auth_key A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4: A1:B2:C3:D4* +```` + + * AEAD algorithm + + * Optional: Yes, unless and are not used + + * Available options: + + * *aes-128-gcm*: AES-GCM 128-bit algorithm + + * Syntax: *cipher_algo * + +```` + + * Cipher key, NOT available when 'null' algorithm is used + + * Optional: Yes, unless and are not used. + Must be followed by option + + * Syntax: Hexadecimal bytes (0x0-0xFF) concatenate by colon symbol ':'. + The number of bytes should be as same as the specified AEAD algorithm + key size. + + For example: *aead_key A1:B2:C3:D4:A1:B2:C3:D4:A1:B2:C3:D4: + A1:B2:C3:D4* + ```` * The operation mode @@ -515,9 +541,8 @@ Example SA rules: src 1111:1111:1111:1111:1111:1111:1111:5555 \ dst 2222:2222:2222:2222:2222:2222:2222:5555 - sa in 105 cipher_algo aes-128-gcm \ - cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \ - auth_algo aes-128-gcm \ + sa in 105 aead_algo aes-128-gcm \ + aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \ mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 Routing rule syntax diff --git a/doc/guides/sample_app_ug/l2_forward_crypto.rst b/doc/guides/sample_app_ug/l2_forward_crypto.rst index b9aa573..2a61af7 100644 --- a/doc/guides/sample_app_ug/l2_forward_crypto.rst +++ b/doc/guides/sample_app_ug/l2_forward_crypto.rst @@ -110,7 +110,9 @@ where, * chain: select the operation chaining to perform: Cipher->Hash (CIPHER_HASH), - Hash->Cipher (HASH_CIPHER), Cipher (CIPHER_ONLY), Hash(HASH_ONLY) + Hash->Cipher (HASH_CIPHER), Cipher (CIPHER_ONLY), Hash (HASH_ONLY) + + or AEAD (AEAD) (default is Cipher->Hash) @@ -154,6 +156,26 @@ where, Note that if --auth_iv is used, this will be ignored. +* aead_algo: select the AEAD algorithm (default is aes-gcm) + +* aead_op: select the AEAD operation to perform: ENCRYPT or DECRYPT + + (default is ENCRYPT) + +* aead_key: set the AEAD key to be used. Bytes has to be separated with ":" + +* aead_key_random_size: set the size of the AEAD key, + + which will be generated randomly. + + Note that if --aead_key is used, this will be ignored. + +* aead_iv: set the AEAD IV to be used. Bytes has to be separated with ":" + +* aead_iv_random_size: set the size of the AEAD IV, which will be generated randomly. + + Note that if --aead_iv is used, this will be ignored. + * aad: set the AAD to be used. Bytes has to be separated with ":" * aad_random_size: set the size of the AAD, which will be generated randomly. diff --git a/doc/guides/tools/cryptoperf.rst b/doc/guides/tools/cryptoperf.rst index 8834df7..f1e99a1 100644 --- a/doc/guides/tools/cryptoperf.rst +++ b/doc/guides/tools/cryptoperf.rst @@ -223,10 +223,8 @@ The following are the appication command-line options: 3des-ecb 3des-ctr aes-cbc - aes-ccm aes-ctr aes-ecb - aes-gcm aes-f8 aes-xts arc4 @@ -257,9 +255,7 @@ The following are the appication command-line options: 3des-cbc aes-cbc-mac - aes-ccm aes-cmac - aes-gcm aes-gmac aes-xcbc-mac md5 @@ -294,13 +290,37 @@ The following are the appication command-line options: Set the size of auth iv. -* ``--auth-digest-sz `` +* ``--aead-algo `` + + Set AEAD algorithm name, where ``name`` is one + of the following:: + + aes-ccm + aes-gcm + +* ``--aead-op `` + + Set AEAD operation mode, where ``mode`` is one of + the following:: + + encrypt + decrypt + +* ``--aead-key-sz `` + + Set the size of AEAD key. + +* ``--aead-iv-sz `` + + Set the size of AEAD iv. + +* ``--aead-aad-sz `` - Set the size of authentication digest. + Set the size of AEAD aad. -* ``--auth-aad-sz `` +* ``--digest-sz `` - Set the size of authentication aad. + Set the size of digest. * ``--csv-friendly`` diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c index 4c55355..f5ca150 100644 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c +++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c @@ -76,13 +76,13 @@ aesni_gcm_set_session_parameters(struct aesni_gcm_session *sess, const struct rte_crypto_sym_xform *xform) { const struct rte_crypto_sym_xform *auth_xform; - const struct rte_crypto_sym_xform *cipher_xform; + const struct rte_crypto_sym_xform *aead_xform; uint16_t digest_length; uint8_t key_length; uint8_t *key; /* AES-GMAC */ - if (xform->next == NULL) { + if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { auth_xform = xform; if (auth_xform->auth.algo != RTE_CRYPTO_AUTH_AES_GMAC) { GCM_LOG_ERR("Only AES GMAC is supported as an " @@ -101,52 +101,39 @@ aesni_gcm_set_session_parameters(struct aesni_gcm_session *sess, key_length = auth_xform->auth.key.length; key = auth_xform->auth.key.data; + digest_length = auth_xform->auth.digest_length; + /* AES-GCM */ - } else { - if (xform->type == RTE_CRYPTO_SYM_XFORM_CIPHER && - xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) { - auth_xform = xform->next; - cipher_xform = xform; - } else if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH && - xform->next->type == RTE_CRYPTO_SYM_XFORM_CIPHER) { - auth_xform = xform; - cipher_xform = xform->next; - } else { - GCM_LOG_ERR("Cipher and auth xform required " - "when using AES GCM"); - return -EINVAL; - } + } else if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { + aead_xform = xform; - if (!(cipher_xform->cipher.algo == RTE_CRYPTO_CIPHER_AES_GCM && - (auth_xform->auth.algo == RTE_CRYPTO_AUTH_AES_GCM))) { + if (aead_xform->aead.algo != RTE_CRYPTO_AEAD_AES_GCM) { GCM_LOG_ERR("The only combined operation " "supported is AES GCM"); return -EINVAL; } /* Set IV parameters */ - sess->iv.offset = cipher_xform->cipher.iv.offset; - sess->iv.length = cipher_xform->cipher.iv.length; + sess->iv.offset = aead_xform->aead.iv.offset; + sess->iv.length = aead_xform->aead.iv.length; /* Select Crypto operation */ - if (cipher_xform->cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT && - auth_xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) + if (aead_xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) sess->op = AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION; - else if (cipher_xform->cipher.op == RTE_CRYPTO_CIPHER_OP_DECRYPT && - auth_xform->auth.op == RTE_CRYPTO_AUTH_OP_VERIFY) + else sess->op = AESNI_GCM_OP_AUTHENTICATED_DECRYPTION; - else { - GCM_LOG_ERR("Cipher/Auth operations: Encrypt/Generate or" - " Decrypt/Verify are valid only"); - return -EINVAL; - } - key_length = cipher_xform->auth.key.length; - key = cipher_xform->auth.key.data; + key_length = aead_xform->aead.key.length; + key = aead_xform->aead.key.data; - sess->aad_length = auth_xform->auth.add_auth_data_length; + sess->aad_length = aead_xform->aead.add_auth_data_length; + digest_length = aead_xform->aead.digest_length; + } else { + GCM_LOG_ERR("Wrong xform type, has to be AEAD or authentication"); + return -EINVAL; } + /* IV check */ if (sess->iv.length != 16 && sess->iv.length != 12 && sess->iv.length != 0) { @@ -154,8 +141,6 @@ aesni_gcm_set_session_parameters(struct aesni_gcm_session *sess, return -EINVAL; } - digest_length = auth_xform->auth.digest_length; - /* Check key length and calculate GCM pre-compute. */ switch (key_length) { case 16: @@ -169,7 +154,7 @@ aesni_gcm_set_session_parameters(struct aesni_gcm_session *sess, break; default: - GCM_LOG_ERR("Unsupported cipher/auth key length"); + GCM_LOG_ERR("Unsupported key length"); return -EINVAL; } @@ -240,9 +225,9 @@ process_gcm_crypto_op(struct rte_crypto_op *op, if (session->op == AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION || session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION) { - offset = sym_op->cipher.data.offset; + offset = sym_op->aead.data.offset; data_offset = offset; - data_length = sym_op->cipher.data.length; + data_length = sym_op->aead.data.length; } else { offset = sym_op->auth.data.offset; data_offset = offset; @@ -295,7 +280,7 @@ process_gcm_crypto_op(struct rte_crypto_op *op, aesni_gcm_enc[session->key].init(&session->gdata, IV_ptr, - sym_op->auth.aad.data, + sym_op->aead.aad.data, (uint64_t)session->aad_length); aesni_gcm_enc[session->key].update(&session->gdata, dst, src, @@ -319,7 +304,7 @@ process_gcm_crypto_op(struct rte_crypto_op *op, } aesni_gcm_enc[session->key].finalize(&session->gdata, - sym_op->auth.digest.data, + sym_op->aead.digest.data, (uint64_t)session->digest_length); } else if (session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION) { uint8_t *auth_tag = (uint8_t *)rte_pktmbuf_append(sym_op->m_dst ? @@ -333,7 +318,7 @@ process_gcm_crypto_op(struct rte_crypto_op *op, aesni_gcm_dec[session->key].init(&session->gdata, IV_ptr, - sym_op->auth.aad.data, + sym_op->aead.aad.data, (uint64_t)session->aad_length); aesni_gcm_dec[session->key].update(&session->gdata, dst, src, @@ -413,19 +398,24 @@ post_process_gcm_crypto_op(struct rte_crypto_op *op) /* Verify digest if required */ if (session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION || session->op == AESNI_GMAC_OP_VERIFY) { + uint8_t *digest; uint8_t *tag = rte_pktmbuf_mtod_offset(m, uint8_t *, m->data_len - session->digest_length); + if (session->op == AESNI_GMAC_OP_VERIFY) + digest = op->sym->auth.digest.data; + else + digest = op->sym->aead.digest.data; + #ifdef RTE_LIBRTE_PMD_AESNI_GCM_DEBUG rte_hexdump(stdout, "auth tag (orig):", - op->sym->auth.digest.data, session->digest_length); + digest, session->digest_length); rte_hexdump(stdout, "auth tag (calc):", tag, session->digest_length); #endif - if (memcmp(tag, op->sym->auth.digest.data, - session->digest_length) != 0) + if (memcmp(tag, digest, session->digest_length) != 0) op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; /* trim area used for digest from mbuf */ diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c index 39285d0..5317657 100644 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c +++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c @@ -65,12 +65,12 @@ static const struct rte_cryptodev_capabilities aesni_gcm_pmd_capabilities[] = { }, } }, } }, - { /* AES GCM (AUTH) */ + { /* AES GCM */ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_AES_GCM, + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, + {.aead = { + .algo = RTE_CRYPTO_AEAD_AES_GCM, .block_size = 16, .key_size = { .min = 16, @@ -87,22 +87,6 @@ static const struct rte_cryptodev_capabilities aesni_gcm_pmd_capabilities[] = { .max = 65535, .increment = 1 }, - .iv_size = { 0 } - }, } - }, } - }, - { /* AES GCM (CIPHER) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, - {.cipher = { - .algo = RTE_CRYPTO_CIPHER_AES_GCM, - .block_size = 16, - .key_size = { - .min = 16, - .max = 32, - .increment = 16 - }, .iv_size = { .min = 12, .max = 12, diff --git a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c index e3666f6..8ec6d4a 100644 --- a/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c +++ b/drivers/crypto/dpaa2_sec/dpaa2_sec_dpseci.c @@ -817,8 +817,6 @@ dpaa2_sec_cipher_init(struct rte_cryptodev *dev, break; case RTE_CRYPTO_CIPHER_AES_CTR: case RTE_CRYPTO_CIPHER_3DES_CTR: - case RTE_CRYPTO_CIPHER_AES_GCM: - case RTE_CRYPTO_CIPHER_AES_CCM: case RTE_CRYPTO_CIPHER_AES_ECB: case RTE_CRYPTO_CIPHER_3DES_ECB: case RTE_CRYPTO_CIPHER_AES_XTS: @@ -946,7 +944,6 @@ dpaa2_sec_auth_init(struct rte_cryptodev *dev, session->auth_alg = RTE_CRYPTO_AUTH_SHA224_HMAC; break; case RTE_CRYPTO_AUTH_AES_XCBC_MAC: - case RTE_CRYPTO_AUTH_AES_GCM: case RTE_CRYPTO_AUTH_SNOW3G_UIA2: case RTE_CRYPTO_AUTH_NULL: case RTE_CRYPTO_AUTH_SHA1: @@ -955,7 +952,6 @@ dpaa2_sec_auth_init(struct rte_cryptodev *dev, case RTE_CRYPTO_AUTH_SHA224: case RTE_CRYPTO_AUTH_SHA384: case RTE_CRYPTO_AUTH_MD5: - case RTE_CRYPTO_AUTH_AES_CCM: case RTE_CRYPTO_AUTH_AES_GMAC: case RTE_CRYPTO_AUTH_KASUMI_F9: case RTE_CRYPTO_AUTH_AES_CMAC: @@ -1100,7 +1096,6 @@ dpaa2_sec_aead_init(struct rte_cryptodev *dev, session->auth_alg = RTE_CRYPTO_AUTH_SHA512_HMAC; break; case RTE_CRYPTO_AUTH_AES_XCBC_MAC: - case RTE_CRYPTO_AUTH_AES_GCM: case RTE_CRYPTO_AUTH_SNOW3G_UIA2: case RTE_CRYPTO_AUTH_NULL: case RTE_CRYPTO_AUTH_SHA1: @@ -1109,7 +1104,6 @@ dpaa2_sec_aead_init(struct rte_cryptodev *dev, case RTE_CRYPTO_AUTH_SHA224: case RTE_CRYPTO_AUTH_SHA384: case RTE_CRYPTO_AUTH_MD5: - case RTE_CRYPTO_AUTH_AES_CCM: case RTE_CRYPTO_AUTH_AES_GMAC: case RTE_CRYPTO_AUTH_KASUMI_F9: case RTE_CRYPTO_AUTH_AES_CMAC: @@ -1141,13 +1135,11 @@ dpaa2_sec_aead_init(struct rte_cryptodev *dev, session->cipher_alg = RTE_CRYPTO_CIPHER_3DES_CBC; ctxt->iv.length = TDES_CBC_IV_LEN; break; - case RTE_CRYPTO_CIPHER_AES_GCM: case RTE_CRYPTO_CIPHER_SNOW3G_UEA2: case RTE_CRYPTO_CIPHER_NULL: case RTE_CRYPTO_CIPHER_3DES_ECB: case RTE_CRYPTO_CIPHER_AES_ECB: case RTE_CRYPTO_CIPHER_AES_CTR: - case RTE_CRYPTO_CIPHER_AES_CCM: case RTE_CRYPTO_CIPHER_KASUMI_F8: RTE_LOG(ERR, PMD, "Crypto: Unsupported Cipher alg %u", cipher_xform->algo); diff --git a/drivers/crypto/openssl/rte_openssl_pmd.c b/drivers/crypto/openssl/rte_openssl_pmd.c index 89fd11b..a33c36e 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd.c +++ b/drivers/crypto/openssl/rte_openssl_pmd.c @@ -88,6 +88,8 @@ openssl_get_chain_order(const struct rte_crypto_sym_xform *xform) else if (xform->next->type == RTE_CRYPTO_SYM_XFORM_AUTH) res = OPENSSL_CHAIN_CIPHER_AUTH; } + if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) + res = OPENSSL_CHAIN_COMBINED; } return res; @@ -183,21 +185,6 @@ get_cipher_algo(enum rte_crypto_cipher_algorithm sess_algo, size_t keylen, res = -EINVAL; } break; - case RTE_CRYPTO_CIPHER_AES_GCM: - switch (keylen) { - case 16: - *algo = EVP_aes_128_gcm(); - break; - case 24: - *algo = EVP_aes_192_gcm(); - break; - case 32: - *algo = EVP_aes_256_gcm(); - break; - default: - res = -EINVAL; - } - break; default: res = -EINVAL; break; @@ -253,6 +240,41 @@ get_auth_algo(enum rte_crypto_auth_algorithm sessalgo, return res; } +/** Get adequate openssl function for input cipher algorithm */ +static uint8_t +get_aead_algo(enum rte_crypto_aead_algorithm sess_algo, size_t keylen, + const EVP_CIPHER **algo) +{ + int res = 0; + + if (algo != NULL) { + switch (sess_algo) { + case RTE_CRYPTO_AEAD_AES_GCM: + switch (keylen) { + case 16: + *algo = EVP_aes_128_gcm(); + break; + case 24: + *algo = EVP_aes_192_gcm(); + break; + case 32: + *algo = EVP_aes_256_gcm(); + break; + default: + res = -EINVAL; + } + break; + default: + res = -EINVAL; + break; + } + } else { + res = -EINVAL; + } + + return res; +} + /** Set session cipher parameters */ static int openssl_set_session_cipher_parameters(struct openssl_session *sess, @@ -272,7 +294,6 @@ openssl_set_session_cipher_parameters(struct openssl_session *sess, case RTE_CRYPTO_CIPHER_3DES_CBC: case RTE_CRYPTO_CIPHER_AES_CBC: case RTE_CRYPTO_CIPHER_AES_CTR: - case RTE_CRYPTO_CIPHER_AES_GCM: sess->cipher.mode = OPENSSL_CIPHER_LIB; sess->cipher.algo = xform->cipher.algo; sess->cipher.ctx = EVP_CIPHER_CTX_new(); @@ -329,12 +350,6 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, /* Select auth algo */ switch (xform->auth.algo) { - case RTE_CRYPTO_AUTH_AES_GCM: - /* Check additional condition for AES_GCM */ - if (sess->cipher.algo != RTE_CRYPTO_CIPHER_AES_GCM) - return -EINVAL; - sess->chain_order = OPENSSL_CHAIN_COMBINED; - break; case RTE_CRYPTO_AUTH_AES_GMAC: sess->chain_order = OPENSSL_CHAIN_COMBINED; @@ -355,7 +370,7 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, sess->cipher.key.length = xform->auth.key.length; sess->cipher.ctx = EVP_CIPHER_CTX_new(); - if (get_cipher_algo(RTE_CRYPTO_CIPHER_AES_GCM, + if (get_aead_algo(RTE_CRYPTO_AEAD_AES_GCM, sess->cipher.key.length, &sess->cipher.evp_algo) != 0) return -EINVAL; @@ -403,6 +418,50 @@ openssl_set_session_auth_parameters(struct openssl_session *sess, return 0; } +/* Set session AEAD parameters */ +static int +openssl_set_session_aead_parameters(struct openssl_session *sess, + const struct rte_crypto_sym_xform *xform) +{ + /* Select cipher direction */ + sess->cipher.direction = xform->cipher.op; + /* Select cipher key */ + sess->cipher.key.length = xform->aead.key.length; + + /* Set IV parameters */ + sess->iv.offset = xform->aead.iv.offset; + sess->iv.length = xform->aead.iv.length; + + /* Select auth generate/verify */ + sess->auth.operation = xform->auth.op; + sess->auth.algo = xform->auth.algo; + + /* Select auth algo */ + switch (xform->aead.algo) { + case RTE_CRYPTO_AEAD_AES_GCM: + sess->cipher.mode = OPENSSL_CIPHER_LIB; + sess->aead_algo = xform->aead.algo; + sess->cipher.ctx = EVP_CIPHER_CTX_new(); + + if (get_aead_algo(sess->aead_algo, sess->cipher.key.length, + &sess->cipher.evp_algo) != 0) + return -EINVAL; + + get_cipher_key(xform->cipher.key.data, sess->cipher.key.length, + sess->cipher.key.data); + + sess->chain_order = OPENSSL_CHAIN_COMBINED; + break; + default: + return -EINVAL; + } + + sess->auth.aad_length = xform->aead.add_auth_data_length; + sess->auth.digest_length = xform->aead.digest_length; + + return 0; +} + /** Parse crypto xform chain and set private session parameters */ int openssl_set_session_parameters(struct openssl_session *sess, @@ -410,6 +469,7 @@ openssl_set_session_parameters(struct openssl_session *sess, { const struct rte_crypto_sym_xform *cipher_xform = NULL; const struct rte_crypto_sym_xform *auth_xform = NULL; + const struct rte_crypto_sym_xform *aead_xform = NULL; sess->chain_order = openssl_get_chain_order(xform); switch (sess->chain_order) { @@ -427,6 +487,9 @@ openssl_set_session_parameters(struct openssl_session *sess, auth_xform = xform; cipher_xform = xform->next; break; + case OPENSSL_CHAIN_COMBINED: + aead_xform = xform; + break; default: return -EINVAL; } @@ -452,6 +515,14 @@ openssl_set_session_parameters(struct openssl_session *sess, } } + if (aead_xform) { + if (openssl_set_session_aead_parameters(sess, aead_xform)) { + OPENSSL_LOG_ERR( + "Invalid/unsupported auth parameters"); + return -EINVAL; + } + } + return 0; } @@ -964,26 +1035,27 @@ process_openssl_combined_op iv = rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv.offset); ivlen = sess->iv.length; - tag = op->sym->auth.digest.data; - if (tag == NULL) - tag = rte_pktmbuf_mtod_offset(mbuf_dst, uint8_t *, - op->sym->cipher.data.offset + - op->sym->cipher.data.length); - if (sess->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { srclen = 0; offset = op->sym->auth.data.offset; aadlen = op->sym->auth.data.length; aad = rte_pktmbuf_mtod_offset(mbuf_src, uint8_t *, op->sym->auth.data.offset); - + tag = op->sym->auth.digest.data; + if (tag == NULL) + tag = rte_pktmbuf_mtod_offset(mbuf_dst, uint8_t *, + offset + aadlen); } else { - srclen = op->sym->cipher.data.length; + srclen = op->sym->aead.data.length; dst = rte_pktmbuf_mtod_offset(mbuf_dst, uint8_t *, - op->sym->cipher.data.offset); - offset = op->sym->cipher.data.offset; - aad = op->sym->auth.aad.data; + op->sym->aead.data.offset); + offset = op->sym->aead.data.offset; + aad = op->sym->aead.aad.data; aadlen = sess->auth.aad_length; + tag = op->sym->aead.digest.data; + if (tag == NULL) + tag = rte_pktmbuf_mtod_offset(mbuf_dst, uint8_t *, + offset + srclen); } if (sess->cipher.direction == RTE_CRYPTO_CIPHER_OP_ENCRYPT) diff --git a/drivers/crypto/openssl/rte_openssl_pmd_ops.c b/drivers/crypto/openssl/rte_openssl_pmd_ops.c index fc525d9..26265b8 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd_ops.c +++ b/drivers/crypto/openssl/rte_openssl_pmd_ops.c @@ -344,12 +344,12 @@ static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = { }, } }, } }, - { /* AES GCM (AUTH) */ + { /* AES GCM */ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, - {.auth = { - .algo = RTE_CRYPTO_AUTH_AES_GCM, + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, + {.aead = { + .algo = RTE_CRYPTO_AEAD_AES_GCM, .block_size = 16, .key_size = { .min = 16, @@ -366,27 +366,11 @@ static const struct rte_cryptodev_capabilities openssl_pmd_capabilities[] = { .max = 65535, .increment = 1 }, - .iv_size = { 0 } - }, } - }, } - }, - { /* AES GCM (CIPHER) */ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, - {.sym = { - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, - {.cipher = { - .algo = RTE_CRYPTO_CIPHER_AES_GCM, - .block_size = 16, - .key_size = { - .min = 16, - .max = 32, - .increment = 8 - }, .iv_size = { .min = 12, .max = 16, .increment = 4 - } + }, }, } }, } }, diff --git a/drivers/crypto/openssl/rte_openssl_pmd_private.h b/drivers/crypto/openssl/rte_openssl_pmd_private.h index 4c9be05..7799407 100644 --- a/drivers/crypto/openssl/rte_openssl_pmd_private.h +++ b/drivers/crypto/openssl/rte_openssl_pmd_private.h @@ -113,6 +113,10 @@ struct openssl_session { uint16_t offset; } iv; /**< IV parameters */ + + enum rte_crypto_aead_algorithm aead_algo; + /**< AEAD algorithm */ + /** Cipher Parameters */ struct { enum rte_crypto_cipher_operation direction; diff --git a/drivers/crypto/qat/qat_crypto.c b/drivers/crypto/qat/qat_crypto.c index 963c311..f29c016 100644 --- a/drivers/crypto/qat/qat_crypto.c +++ b/drivers/crypto/qat/qat_crypto.c @@ -245,6 +245,14 @@ qat_get_cmd_id(const struct rte_crypto_sym_xform *xform) if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH && xform->next == NULL) return ICP_QAT_FW_LA_CMD_AUTH; + /* AEAD */ + if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { + if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) + return ICP_QAT_FW_LA_CMD_CIPHER_HASH; + else + return ICP_QAT_FW_LA_CMD_HASH_CIPHER; + } + if (xform->next == NULL) return -1; @@ -309,14 +317,6 @@ qat_crypto_sym_configure_session_cipher(struct rte_cryptodev *dev, } session->qat_mode = ICP_QAT_HW_CIPHER_CBC_MODE; break; - case RTE_CRYPTO_CIPHER_AES_GCM: - if (qat_alg_validate_aes_key(cipher_xform->key.length, - &session->qat_cipher_alg) != 0) { - PMD_DRV_LOG(ERR, "Invalid AES cipher key size"); - goto error_out; - } - session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE; - break; case RTE_CRYPTO_CIPHER_AES_CTR: if (qat_alg_validate_aes_key(cipher_xform->key.length, &session->qat_cipher_alg) != 0) { @@ -417,7 +417,6 @@ qat_crypto_sym_configure_session_cipher(struct rte_cryptodev *dev, break; case RTE_CRYPTO_CIPHER_3DES_ECB: case RTE_CRYPTO_CIPHER_AES_ECB: - case RTE_CRYPTO_CIPHER_AES_CCM: case RTE_CRYPTO_CIPHER_AES_F8: case RTE_CRYPTO_CIPHER_AES_XTS: case RTE_CRYPTO_CIPHER_ARC4: @@ -479,12 +478,26 @@ qat_crypto_sym_configure_session(struct rte_cryptodev *dev, session = qat_crypto_sym_configure_session_auth(dev, xform, session); break; case ICP_QAT_FW_LA_CMD_CIPHER_HASH: - session = qat_crypto_sym_configure_session_cipher(dev, xform, session); - session = qat_crypto_sym_configure_session_auth(dev, xform, session); + if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) + session = qat_crypto_sym_configure_session_aead(xform, + session); + else { + session = qat_crypto_sym_configure_session_cipher(dev, + xform, session); + session = qat_crypto_sym_configure_session_auth(dev, + xform, session); + } break; case ICP_QAT_FW_LA_CMD_HASH_CIPHER: - session = qat_crypto_sym_configure_session_auth(dev, xform, session); - session = qat_crypto_sym_configure_session_cipher(dev, xform, session); + if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) + session = qat_crypto_sym_configure_session_aead(xform, + session); + else { + session = qat_crypto_sym_configure_session_auth(dev, + xform, session); + session = qat_crypto_sym_configure_session_cipher(dev, + xform, session); + } break; case ICP_QAT_FW_LA_CMD_TRNG_GET_RANDOM: case ICP_QAT_FW_LA_CMD_TRNG_TEST: @@ -518,7 +531,6 @@ qat_crypto_sym_configure_session_auth(struct rte_cryptodev *dev, struct qat_session *session = session_private; struct rte_crypto_auth_xform *auth_xform = NULL; - struct rte_crypto_cipher_xform *cipher_xform = NULL; struct qat_pmd_private *internals = dev->data->dev_private; auth_xform = qat_get_auth_xform(xform); uint8_t *key_data = auth_xform->key.data; @@ -543,14 +555,6 @@ qat_crypto_sym_configure_session_auth(struct rte_cryptodev *dev, case RTE_CRYPTO_AUTH_AES_XCBC_MAC: session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_AES_XCBC_MAC; break; - case RTE_CRYPTO_AUTH_AES_GCM: - cipher_xform = qat_get_cipher_xform(xform); - - session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_GALOIS_128; - - key_data = cipher_xform->key.data; - key_length = cipher_xform->key.length; - break; case RTE_CRYPTO_AUTH_AES_GMAC: if (qat_alg_validate_aes_key(auth_xform->key.length, &session->qat_cipher_alg) != 0) { @@ -588,8 +592,6 @@ qat_crypto_sym_configure_session_auth(struct rte_cryptodev *dev, case RTE_CRYPTO_AUTH_SHA224: case RTE_CRYPTO_AUTH_SHA384: case RTE_CRYPTO_AUTH_MD5: - case RTE_CRYPTO_AUTH_AES_CCM: - case RTE_CRYPTO_AUTH_AES_CMAC: case RTE_CRYPTO_AUTH_AES_CBC_MAC: PMD_DRV_LOG(ERR, "Crypto: Unsupported hash alg %u", auth_xform->algo); @@ -649,7 +651,7 @@ qat_crypto_sym_configure_session_auth(struct rte_cryptodev *dev, if (qat_alg_aead_session_create_content_desc_auth(session, key_data, key_length, - auth_xform->add_auth_data_length, + 0, auth_xform->digest_length, auth_xform->op)) goto error_out; @@ -662,6 +664,85 @@ qat_crypto_sym_configure_session_auth(struct rte_cryptodev *dev, return NULL; } +struct qat_session * +qat_crypto_sym_configure_session_aead(struct rte_crypto_sym_xform *xform, + struct qat_session *session_private) +{ + struct qat_session *session = session_private; + struct rte_crypto_aead_xform *aead_xform = &xform->aead; + + /* + * Store AEAD IV parameters as cipher IV, + * to avoid unnecessary memory usage + */ + session->cipher_iv.offset = xform->aead.iv.offset; + session->cipher_iv.length = xform->aead.iv.length; + + switch (aead_xform->algo) { + case RTE_CRYPTO_AEAD_AES_GCM: + if (qat_alg_validate_aes_key(aead_xform->key.length, + &session->qat_cipher_alg) != 0) { + PMD_DRV_LOG(ERR, "Invalid AES key size"); + goto error_out; + } + session->qat_mode = ICP_QAT_HW_CIPHER_CTR_MODE; + session->qat_hash_alg = ICP_QAT_HW_AUTH_ALGO_GALOIS_128; + break; + case RTE_CRYPTO_AEAD_AES_CCM: + PMD_DRV_LOG(ERR, "Crypto QAT PMD: Unsupported AEAD alg %u", + aead_xform->algo); + goto error_out; + default: + PMD_DRV_LOG(ERR, "Crypto: Undefined AEAD specified %u\n", + aead_xform->algo); + goto error_out; + } + + if (aead_xform->op == RTE_CRYPTO_AEAD_OP_ENCRYPT) { + session->qat_dir = ICP_QAT_HW_CIPHER_ENCRYPT; + /* + * It needs to create cipher desc content first, + * then authentication + */ + if (qat_alg_aead_session_create_content_desc_cipher(session, + aead_xform->key.data, + aead_xform->key.length)) + goto error_out; + + if (qat_alg_aead_session_create_content_desc_auth(session, + aead_xform->key.data, + aead_xform->key.length, + aead_xform->add_auth_data_length, + aead_xform->digest_length, + RTE_CRYPTO_AUTH_OP_GENERATE)) + goto error_out; + } else { + session->qat_dir = ICP_QAT_HW_CIPHER_DECRYPT; + /* + * It needs to create authentication desc content first, + * then cipher + */ + if (qat_alg_aead_session_create_content_desc_auth(session, + aead_xform->key.data, + aead_xform->key.length, + aead_xform->add_auth_data_length, + aead_xform->digest_length, + RTE_CRYPTO_AUTH_OP_VERIFY)) + goto error_out; + + if (qat_alg_aead_session_create_content_desc_cipher(session, + aead_xform->key.data, + aead_xform->key.length)) + goto error_out; + } + + session->digest_length = aead_xform->digest_length; + return session; + +error_out: + return NULL; +} + unsigned qat_crypto_sym_get_session_private_size( struct rte_cryptodev *dev __rte_unused) { @@ -984,7 +1065,7 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, struct icp_qat_fw_la_cipher_req_params *cipher_param; struct icp_qat_fw_la_auth_req_params *auth_param; register struct icp_qat_fw_la_bulk_req *qat_req; - uint8_t do_auth = 0, do_cipher = 0; + uint8_t do_auth = 0, do_cipher = 0, do_aead = 0; uint32_t cipher_len = 0, cipher_ofs = 0; uint32_t auth_len = 0, auth_ofs = 0; uint32_t min_ofs = 0; @@ -1018,9 +1099,15 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, auth_param = (void *)((uint8_t *)cipher_param + sizeof(*cipher_param)); if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_HASH_CIPHER || - ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) { - do_auth = 1; - do_cipher = 1; + ctx->qat_cmd == ICP_QAT_FW_LA_CMD_CIPHER_HASH) { + /* AES-GCM */ + if (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_128 || + ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_64) { + do_aead = 1; + } else { + do_auth = 1; + do_cipher = 1; + } } else if (ctx->qat_cmd == ICP_QAT_FW_LA_CMD_AUTH) { do_auth = 1; do_cipher = 0; @@ -1112,18 +1199,10 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, ICP_QAT_HW_AUTH_ALGO_GALOIS_128 || ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_64) { - /* AES-GCM */ - if (do_cipher) { - auth_ofs = op->sym->cipher.data.offset; - auth_len = op->sym->cipher.data.length; - - auth_param->u1.aad_adr = op->sym->auth.aad.phys_addr; /* AES-GMAC */ - } else { - set_cipher_IV(ctx->auth_iv.length, - ctx->auth_iv.offset, - cipher_param, op, qat_req); - } + set_cipher_IV(ctx->auth_iv.length, + ctx->auth_iv.offset, + cipher_param, op, qat_req); } else { auth_ofs = op->sym->auth.data.offset; auth_len = op->sym->auth.data.length; @@ -1134,6 +1213,19 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, } + if (do_aead) { + cipher_len = op->sym->aead.data.length; + cipher_ofs = op->sym->aead.data.offset; + auth_len = op->sym->aead.data.length; + auth_ofs = op->sym->aead.data.offset; + + auth_param->u1.aad_adr = op->sym->aead.aad.phys_addr; + auth_param->auth_res_addr = op->sym->aead.digest.phys_addr; + set_cipher_IV(ctx->cipher_iv.length, ctx->cipher_iv.offset, + cipher_param, op, qat_req); + min_ofs = op->sym->aead.data.offset; + } + if (op->sym->m_src->next || (op->sym->m_dst && op->sym->m_dst->next)) do_sgl = 1; @@ -1175,7 +1267,7 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, dst_buf_start = src_buf_start; } - if (do_cipher) { + if (do_cipher || do_aead) { cipher_param->cipher_offset = (uint32_t)rte_pktmbuf_mtophys_offset( op->sym->m_src, cipher_ofs) - src_buf_start; @@ -1184,7 +1276,8 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, cipher_param->cipher_offset = 0; cipher_param->cipher_length = 0; } - if (do_auth) { + + if (do_auth || do_aead) { auth_param->auth_off = (uint32_t)rte_pktmbuf_mtophys_offset( op->sym->m_src, auth_ofs) - src_buf_start; auth_param->auth_len = auth_len; @@ -1192,6 +1285,7 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, auth_param->auth_off = 0; auth_param->auth_len = 0; } + qat_req->comn_mid.dst_length = qat_req->comn_mid.src_length = (cipher_param->cipher_offset + cipher_param->cipher_length) @@ -1250,7 +1344,7 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS); } /* GMAC */ - if (!do_cipher) { + if (!do_aead) { qat_req->comn_mid.dst_length = qat_req->comn_mid.src_length = rte_pktmbuf_data_len(op->sym->m_src); @@ -1285,7 +1379,12 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg, } rte_hexdump(stdout, "digest:", op->sym->auth.digest.data, ctx->digest_lengthh); - rte_hexdump(stdout, "aad:", op->sym->auth.aad.data, + } + + if (do_aead) { + rte_hexdump(stdout, "digest:", op->sym->aead.digest.data, + ctx->digest_length); + rte_hexdump(stdout, "aad:", op->sym->aead.aad.data, ctx->aad_len); } #endif diff --git a/drivers/crypto/qat/qat_crypto.h b/drivers/crypto/qat/qat_crypto.h index b740d6b..f76f3ca 100644 --- a/drivers/crypto/qat/qat_crypto.h +++ b/drivers/crypto/qat/qat_crypto.h @@ -117,6 +117,10 @@ qat_crypto_sym_configure_session(struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform, void *session_private); struct qat_session * +qat_crypto_sym_configure_session_aead(struct rte_crypto_sym_xform *xform, + struct qat_session *session_private); + +struct qat_session * qat_crypto_sym_configure_session_auth(struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform, struct qat_session *session_private); diff --git a/drivers/crypto/qat/qat_crypto_capabilities.h b/drivers/crypto/qat/qat_crypto_capabilities.h index d863ccd..fee8ee1 100644 --- a/drivers/crypto/qat/qat_crypto_capabilities.h +++ b/drivers/crypto/qat/qat_crypto_capabilities.h @@ -189,12 +189,12 @@ }, } \ }, } \ }, \ - { /* AES GCM (AUTH) */ \ + { /* AES GCM */ \ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \ {.sym = { \ - .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, \ - {.auth = { \ - .algo = RTE_CRYPTO_AUTH_AES_GCM, \ + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, \ + {.aead = { \ + .algo = RTE_CRYPTO_AEAD_AES_GCM, \ .block_size = 16, \ .key_size = { \ .min = 16, \ @@ -211,7 +211,11 @@ .max = 240, \ .increment = 1 \ }, \ - .iv_size = { 0 } \ + .iv_size = { \ + .min = 12, \ + .max = 12, \ + .increment = 0 \ + }, \ }, } \ }, } \ }, \ @@ -266,26 +270,6 @@ }, } \ }, } \ }, \ - { /* AES GCM (CIPHER) */ \ - .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \ - {.sym = { \ - .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, \ - {.cipher = { \ - .algo = RTE_CRYPTO_CIPHER_AES_GCM, \ - .block_size = 16, \ - .key_size = { \ - .min = 16, \ - .max = 32, \ - .increment = 8 \ - }, \ - .iv_size = { \ - .min = 12, \ - .max = 12, \ - .increment = 0 \ - } \ - }, } \ - }, } \ - }, \ { /* AES CBC */ \ .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, \ {.sym = { \ diff --git a/examples/ipsec-secgw/esp.c b/examples/ipsec-secgw/esp.c index 1f3efae..e1cfca0 100644 --- a/examples/ipsec-secgw/esp.c +++ b/examples/ipsec-secgw/esp.c @@ -84,62 +84,79 @@ esp_inbound(struct rte_mbuf *m, struct ipsec_sa *sa, } sym_cop = get_sym_cop(cop); - sym_cop->m_src = m; - sym_cop->cipher.data.offset = ip_hdr_len + sizeof(struct esp_hdr) + - sa->iv_len; - sym_cop->cipher.data.length = payload_len; - - struct cnt_blk *icb; - uint8_t *aad; - uint8_t *iv = RTE_PTR_ADD(ip4, ip_hdr_len + sizeof(struct esp_hdr)); - uint8_t *IV_ptr = rte_crypto_op_ctod_offset(cop, - uint8_t *, IV_OFFSET); - - switch (sa->cipher_algo) { - case RTE_CRYPTO_CIPHER_NULL: - case RTE_CRYPTO_CIPHER_AES_CBC: - /* Copy IV at the end of crypto operation */ - rte_memcpy(IV_ptr, iv, sa->iv_len); - break; - case RTE_CRYPTO_CIPHER_AES_CTR: - case RTE_CRYPTO_CIPHER_AES_GCM: + + if (sa->aead_algo == RTE_CRYPTO_AEAD_AES_GCM) { + sym_cop->aead.data.offset = ip_hdr_len + sizeof(struct esp_hdr) + + sa->iv_len; + sym_cop->aead.data.length = payload_len; + + struct cnt_blk *icb; + uint8_t *aad; + uint8_t *iv = RTE_PTR_ADD(ip4, ip_hdr_len + sizeof(struct esp_hdr)); + icb = get_cnt_blk(m); icb->salt = sa->salt; memcpy(&icb->iv, iv, 8); icb->cnt = rte_cpu_to_be_32(1); - break; - default: - RTE_LOG(ERR, IPSEC_ESP, "unsupported cipher algorithm %u\n", - sa->cipher_algo); - return -EINVAL; - } - switch (sa->auth_algo) { - case RTE_CRYPTO_AUTH_NULL: - case RTE_CRYPTO_AUTH_SHA1_HMAC: - case RTE_CRYPTO_AUTH_SHA256_HMAC: - sym_cop->auth.data.offset = ip_hdr_len; - sym_cop->auth.data.length = sizeof(struct esp_hdr) + - sa->iv_len + payload_len; - break; - case RTE_CRYPTO_AUTH_AES_GCM: aad = get_aad(m); memcpy(aad, iv - sizeof(struct esp_hdr), 8); sym_cop->auth.aad.data = aad; sym_cop->auth.aad.phys_addr = rte_pktmbuf_mtophys_offset(m, aad - rte_pktmbuf_mtod(m, uint8_t *)); - break; - default: - RTE_LOG(ERR, IPSEC_ESP, "unsupported auth algorithm %u\n", - sa->auth_algo); - return -EINVAL; - } - sym_cop->auth.digest.data = rte_pktmbuf_mtod_offset(m, void*, - rte_pktmbuf_pkt_len(m) - sa->digest_len); - sym_cop->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, - rte_pktmbuf_pkt_len(m) - sa->digest_len); + sym_cop->aead.digest.data = rte_pktmbuf_mtod_offset(m, void*, + rte_pktmbuf_pkt_len(m) - sa->digest_len); + sym_cop->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, + rte_pktmbuf_pkt_len(m) - sa->digest_len); + } else { + sym_cop->cipher.data.offset = ip_hdr_len + sizeof(struct esp_hdr) + + sa->iv_len; + sym_cop->cipher.data.length = payload_len; + + struct cnt_blk *icb; + uint8_t *iv = RTE_PTR_ADD(ip4, ip_hdr_len + sizeof(struct esp_hdr)); + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(cop, + uint8_t *, IV_OFFSET); + + switch (sa->cipher_algo) { + case RTE_CRYPTO_CIPHER_NULL: + case RTE_CRYPTO_CIPHER_AES_CBC: + /* Copy IV at the end of crypto operation */ + rte_memcpy(IV_ptr, iv, sa->iv_len); + break; + case RTE_CRYPTO_CIPHER_AES_CTR: + icb = get_cnt_blk(m); + icb->salt = sa->salt; + memcpy(&icb->iv, iv, 8); + icb->cnt = rte_cpu_to_be_32(1); + break; + default: + RTE_LOG(ERR, IPSEC_ESP, "unsupported cipher algorithm %u\n", + sa->cipher_algo); + return -EINVAL; + } + + switch (sa->auth_algo) { + case RTE_CRYPTO_AUTH_NULL: + case RTE_CRYPTO_AUTH_SHA1_HMAC: + case RTE_CRYPTO_AUTH_SHA256_HMAC: + sym_cop->auth.data.offset = ip_hdr_len; + sym_cop->auth.data.length = sizeof(struct esp_hdr) + + sa->iv_len + payload_len; + break; + default: + RTE_LOG(ERR, IPSEC_ESP, "unsupported auth algorithm %u\n", + sa->auth_algo); + return -EINVAL; + } + + sym_cop->auth.digest.data = rte_pktmbuf_mtod_offset(m, void*, + rte_pktmbuf_pkt_len(m) - sa->digest_len); + sym_cop->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, + rte_pktmbuf_pkt_len(m) - sa->digest_len); + } return 0; } @@ -308,65 +325,87 @@ esp_outbound(struct rte_mbuf *m, struct ipsec_sa *sa, sym_cop = get_sym_cop(cop); sym_cop->m_src = m; - switch (sa->cipher_algo) { - case RTE_CRYPTO_CIPHER_NULL: - case RTE_CRYPTO_CIPHER_AES_CBC: - memset(iv, 0, sa->iv_len); - sym_cop->cipher.data.offset = ip_hdr_len + - sizeof(struct esp_hdr); - sym_cop->cipher.data.length = pad_payload_len + sa->iv_len; - break; - case RTE_CRYPTO_CIPHER_AES_CTR: - case RTE_CRYPTO_CIPHER_AES_GCM: + + if (sa->aead_algo == RTE_CRYPTO_AEAD_AES_GCM) { + uint8_t *aad; + *iv = sa->seq; - sym_cop->cipher.data.offset = ip_hdr_len + + sym_cop->aead.data.offset = ip_hdr_len + sizeof(struct esp_hdr) + sa->iv_len; - sym_cop->cipher.data.length = pad_payload_len; - break; - default: - RTE_LOG(ERR, IPSEC_ESP, "unsupported cipher algorithm %u\n", - sa->cipher_algo); - return -EINVAL; - } + sym_cop->aead.data.length = pad_payload_len; + + /* Fill pad_len using default sequential scheme */ + for (i = 0; i < pad_len - 2; i++) + padding[i] = i + 1; + padding[pad_len - 2] = pad_len - 2; + padding[pad_len - 1] = nlp; + + struct cnt_blk *icb = get_cnt_blk(m); + icb->salt = sa->salt; + icb->iv = sa->seq; + icb->cnt = rte_cpu_to_be_32(1); - /* Fill pad_len using default sequential scheme */ - for (i = 0; i < pad_len - 2; i++) - padding[i] = i + 1; - padding[pad_len - 2] = pad_len - 2; - padding[pad_len - 1] = nlp; - - struct cnt_blk *icb = get_cnt_blk(m); - icb->salt = sa->salt; - icb->iv = sa->seq; - icb->cnt = rte_cpu_to_be_32(1); - - uint8_t *aad; - - switch (sa->auth_algo) { - case RTE_CRYPTO_AUTH_NULL: - case RTE_CRYPTO_AUTH_SHA1_HMAC: - case RTE_CRYPTO_AUTH_SHA256_HMAC: - sym_cop->auth.data.offset = ip_hdr_len; - sym_cop->auth.data.length = sizeof(struct esp_hdr) + - sa->iv_len + pad_payload_len; - break; - case RTE_CRYPTO_AUTH_AES_GCM: aad = get_aad(m); memcpy(aad, esp, 8); sym_cop->auth.aad.data = aad; sym_cop->auth.aad.phys_addr = rte_pktmbuf_mtophys_offset(m, aad - rte_pktmbuf_mtod(m, uint8_t *)); - break; - default: - RTE_LOG(ERR, IPSEC_ESP, "unsupported auth algorithm %u\n", - sa->auth_algo); - return -EINVAL; - } - sym_cop->auth.digest.data = rte_pktmbuf_mtod_offset(m, uint8_t *, + sym_cop->aead.digest.data = rte_pktmbuf_mtod_offset(m, uint8_t *, rte_pktmbuf_pkt_len(m) - sa->digest_len); - sym_cop->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, + sym_cop->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, rte_pktmbuf_pkt_len(m) - sa->digest_len); + } else { + switch (sa->cipher_algo) { + case RTE_CRYPTO_CIPHER_NULL: + case RTE_CRYPTO_CIPHER_AES_CBC: + memset(iv, 0, sa->iv_len); + sym_cop->cipher.data.offset = ip_hdr_len + + sizeof(struct esp_hdr); + sym_cop->cipher.data.length = pad_payload_len + sa->iv_len; + break; + case RTE_CRYPTO_CIPHER_AES_CTR: + *iv = sa->seq; + sym_cop->cipher.data.offset = ip_hdr_len + + sizeof(struct esp_hdr) + sa->iv_len; + sym_cop->cipher.data.length = pad_payload_len; + break; + default: + RTE_LOG(ERR, IPSEC_ESP, "unsupported cipher algorithm %u\n", + sa->cipher_algo); + return -EINVAL; + } + + /* Fill pad_len using default sequential scheme */ + for (i = 0; i < pad_len - 2; i++) + padding[i] = i + 1; + padding[pad_len - 2] = pad_len - 2; + padding[pad_len - 1] = nlp; + + struct cnt_blk *icb = get_cnt_blk(m); + icb->salt = sa->salt; + icb->iv = sa->seq; + icb->cnt = rte_cpu_to_be_32(1); + + switch (sa->auth_algo) { + case RTE_CRYPTO_AUTH_NULL: + case RTE_CRYPTO_AUTH_SHA1_HMAC: + case RTE_CRYPTO_AUTH_SHA256_HMAC: + sym_cop->auth.data.offset = ip_hdr_len; + sym_cop->auth.data.length = sizeof(struct esp_hdr) + + sa->iv_len + pad_payload_len; + break; + default: + RTE_LOG(ERR, IPSEC_ESP, "unsupported auth algorithm %u\n", + sa->auth_algo); + return -EINVAL; + } + + sym_cop->auth.digest.data = rte_pktmbuf_mtod_offset(m, uint8_t *, + rte_pktmbuf_pkt_len(m) - sa->digest_len); + sym_cop->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, + rte_pktmbuf_pkt_len(m) - sa->digest_len); + } return 0; } diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h index 405cf3d..f8569ca 100644 --- a/examples/ipsec-secgw/ipsec.h +++ b/examples/ipsec-secgw/ipsec.h @@ -103,6 +103,7 @@ struct ipsec_sa { struct rte_cryptodev_sym_session *crypto_session; enum rte_crypto_cipher_algorithm cipher_algo; enum rte_crypto_auth_algorithm auth_algo; + enum rte_crypto_aead_algorithm aead_algo; uint16_t digest_len; uint16_t iv_len; uint16_t block_size; diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c index 85e4d4e..f0a3af6 100644 --- a/examples/ipsec-secgw/sa.c +++ b/examples/ipsec-secgw/sa.c @@ -64,10 +64,20 @@ struct supported_auth_algo { enum rte_crypto_auth_algorithm algo; uint16_t digest_len; uint16_t key_len; - uint8_t aad_len; uint8_t key_not_req; }; +struct supported_aead_algo { + const char *keyword; + enum rte_crypto_aead_algorithm algo; + uint16_t iv_len; + uint16_t block_size; + uint16_t digest_len; + uint16_t key_len; + uint8_t aad_len; +}; + + const struct supported_cipher_algo cipher_algos[] = { { .keyword = "null", @@ -84,13 +94,6 @@ const struct supported_cipher_algo cipher_algos[] = { .key_len = 16 }, { - .keyword = "aes-128-gcm", - .algo = RTE_CRYPTO_CIPHER_AES_GCM, - .iv_len = 8, - .block_size = 4, - .key_len = 20 - }, - { .keyword = "aes-128-ctr", .algo = RTE_CRYPTO_CIPHER_AES_CTR, .iv_len = 8, @@ -118,13 +121,18 @@ const struct supported_auth_algo auth_algos[] = { .algo = RTE_CRYPTO_AUTH_SHA256_HMAC, .digest_len = 12, .key_len = 32 - }, + } +}; + +const struct supported_aead_algo aead_algos[] = { { .keyword = "aes-128-gcm", - .algo = RTE_CRYPTO_AUTH_AES_GCM, + .algo = RTE_CRYPTO_AEAD_AES_GCM, + .iv_len = 8, + .block_size = 4, + .key_len = 20, .digest_len = 16, .aad_len = 8, - .key_not_req = 1 } }; @@ -166,6 +174,22 @@ find_match_auth_algo(const char *auth_keyword) return NULL; } +static const struct supported_aead_algo * +find_match_aead_algo(const char *aead_keyword) +{ + size_t i; + + for (i = 0; i < RTE_DIM(aead_algos); i++) { + const struct supported_aead_algo *algo = + &aead_algos[i]; + + if (strcmp(aead_keyword, algo->keyword) == 0) + return algo; + } + + return NULL; +} + /** parse_key_string * parse x:x:x:x.... hex number key string into uint8_t *key * return: @@ -210,6 +234,7 @@ parse_sa_tokens(char **tokens, uint32_t n_tokens, uint32_t *ri /*rule index*/; uint32_t cipher_algo_p = 0; uint32_t auth_algo_p = 0; + uint32_t aead_algo_p = 0; uint32_t src_p = 0; uint32_t dst_p = 0; uint32_t mode_p = 0; @@ -319,8 +344,7 @@ parse_sa_tokens(char **tokens, uint32_t n_tokens, if (algo->algo == RTE_CRYPTO_CIPHER_AES_CBC) rule->salt = (uint32_t)rte_rand(); - if ((algo->algo == RTE_CRYPTO_CIPHER_AES_CTR) || - (algo->algo == RTE_CRYPTO_CIPHER_AES_GCM)) { + if (algo->algo == RTE_CRYPTO_CIPHER_AES_CTR) { key_len -= 4; rule->cipher_key_len = key_len; memcpy(&rule->salt, @@ -386,6 +410,61 @@ parse_sa_tokens(char **tokens, uint32_t n_tokens, continue; } + if (strcmp(tokens[ti], "aead_algo") == 0) { + const struct supported_aead_algo *algo; + uint32_t key_len; + + APP_CHECK_PRESENCE(aead_algo_p, tokens[ti], + status); + if (status->status < 0) + return; + + INCREMENT_TOKEN_INDEX(ti, n_tokens, status); + if (status->status < 0) + return; + + algo = find_match_aead_algo(tokens[ti]); + + APP_CHECK(algo != NULL, status, "unrecognized " + "input \"%s\"", tokens[ti]); + + rule->aead_algo = algo->algo; + rule->cipher_key_len = algo->key_len; + rule->digest_len = algo->digest_len; + rule->aad_len = algo->key_len; + rule->block_size = algo->block_size; + rule->iv_len = algo->iv_len; + + INCREMENT_TOKEN_INDEX(ti, n_tokens, status); + if (status->status < 0) + return; + + APP_CHECK(strcmp(tokens[ti], "aead_key") == 0, + status, "unrecognized input \"%s\", " + "expect \"aead_key\"", tokens[ti]); + if (status->status < 0) + return; + + INCREMENT_TOKEN_INDEX(ti, n_tokens, status); + if (status->status < 0) + return; + + key_len = parse_key_string(tokens[ti], + rule->cipher_key); + APP_CHECK(key_len == rule->cipher_key_len, status, + "unrecognized input \"%s\"", tokens[ti]); + if (status->status < 0) + return; + + key_len -= 4; + rule->cipher_key_len = key_len; + memcpy(&rule->salt, + &rule->cipher_key[key_len], 4); + + aead_algo_p = 1; + continue; + } + if (strcmp(tokens[ti], "src") == 0) { APP_CHECK_PRESENCE(src_p, tokens[ti], status); if (status->status < 0) @@ -477,13 +556,25 @@ parse_sa_tokens(char **tokens, uint32_t n_tokens, return; } - APP_CHECK(cipher_algo_p == 1, status, "missing cipher options"); - if (status->status < 0) - return; + if (aead_algo_p) { + APP_CHECK(cipher_algo_p == 0, status, + "AEAD used, no need for cipher options"); + if (status->status < 0) + return; - APP_CHECK(auth_algo_p == 1, status, "missing auth options"); - if (status->status < 0) - return; + APP_CHECK(auth_algo_p == 0, status, + "AEAD used, no need for auth options"); + if (status->status < 0) + return; + } else { + APP_CHECK(cipher_algo_p == 1, status, "missing cipher or AEAD options"); + if (status->status < 0) + return; + + APP_CHECK(auth_algo_p == 1, status, "missing auth or AEAD options"); + if (status->status < 0) + return; + } APP_CHECK(mode_p == 1, status, "missing mode option"); if (status->status < 0) @@ -514,6 +605,13 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound) } } + for (i = 0; i < RTE_DIM(aead_algos); i++) { + if (aead_algos[i].algo == sa->aead_algo) { + printf("%s ", aead_algos[i].keyword); + break; + } + } + printf("mode:"); switch (sa->flags) { @@ -608,75 +706,109 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[], sa->dst.ip.ip4 = rte_cpu_to_be_32(sa->dst.ip.ip4); } - switch (sa->cipher_algo) { - case RTE_CRYPTO_CIPHER_NULL: - case RTE_CRYPTO_CIPHER_AES_CBC: - iv_length = sa->iv_len; - break; - case RTE_CRYPTO_CIPHER_AES_CTR: - case RTE_CRYPTO_CIPHER_AES_GCM: + if (sa->aead_algo == RTE_CRYPTO_AEAD_AES_GCM) { iv_length = 16; - break; - default: - RTE_LOG(ERR, IPSEC_ESP, "unsupported cipher algorithm %u\n", - sa->cipher_algo); - return -EINVAL; - } - if (inbound) { - sa_ctx->xf[idx].b.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - sa_ctx->xf[idx].b.cipher.algo = sa->cipher_algo; - sa_ctx->xf[idx].b.cipher.key.data = sa->cipher_key; - sa_ctx->xf[idx].b.cipher.key.length = - sa->cipher_key_len; - sa_ctx->xf[idx].b.cipher.op = - RTE_CRYPTO_CIPHER_OP_DECRYPT; - sa_ctx->xf[idx].b.cipher.iv.offset = IV_OFFSET; - sa_ctx->xf[idx].b.cipher.iv.length = iv_length; - sa_ctx->xf[idx].b.next = NULL; + if (inbound) { + sa_ctx->xf[idx].a.type = RTE_CRYPTO_SYM_XFORM_AEAD; + sa_ctx->xf[idx].a.aead.algo = sa->aead_algo; + sa_ctx->xf[idx].a.aead.key.data = sa->cipher_key; + sa_ctx->xf[idx].a.aead.key.length = + sa->cipher_key_len; + sa_ctx->xf[idx].a.aead.op = + RTE_CRYPTO_AEAD_OP_DECRYPT; + sa_ctx->xf[idx].a.next = NULL; + sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET; + sa_ctx->xf[idx].a.aead.iv.length = iv_length; + sa_ctx->xf[idx].a.aead.add_auth_data_length = + sa->aad_len; + sa_ctx->xf[idx].a.aead.digest_length = + sa->digest_len; + } else { /* outbound */ + sa_ctx->xf[idx].a.type = RTE_CRYPTO_SYM_XFORM_AEAD; + sa_ctx->xf[idx].a.aead.algo = sa->aead_algo; + sa_ctx->xf[idx].a.aead.key.data = sa->cipher_key; + sa_ctx->xf[idx].a.aead.key.length = + sa->cipher_key_len; + sa_ctx->xf[idx].a.aead.op = + RTE_CRYPTO_AEAD_OP_ENCRYPT; + sa_ctx->xf[idx].a.next = NULL; + sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET; + sa_ctx->xf[idx].a.aead.iv.length = iv_length; + sa_ctx->xf[idx].a.aead.add_auth_data_length = + sa->aad_len; + sa_ctx->xf[idx].a.aead.digest_length = + sa->digest_len; + } - sa_ctx->xf[idx].a.type = RTE_CRYPTO_SYM_XFORM_AUTH; - sa_ctx->xf[idx].a.auth.algo = sa->auth_algo; - sa_ctx->xf[idx].a.auth.add_auth_data_length = - sa->aad_len; - sa_ctx->xf[idx].a.auth.key.data = sa->auth_key; - sa_ctx->xf[idx].a.auth.key.length = - sa->auth_key_len; - sa_ctx->xf[idx].a.auth.digest_length = - sa->digest_len; - sa_ctx->xf[idx].a.auth.op = - RTE_CRYPTO_AUTH_OP_VERIFY; - - } else { /* outbound */ - sa_ctx->xf[idx].a.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - sa_ctx->xf[idx].a.cipher.algo = sa->cipher_algo; - sa_ctx->xf[idx].a.cipher.key.data = sa->cipher_key; - sa_ctx->xf[idx].a.cipher.key.length = - sa->cipher_key_len; - sa_ctx->xf[idx].a.cipher.op = - RTE_CRYPTO_CIPHER_OP_ENCRYPT; - sa_ctx->xf[idx].a.cipher.iv.offset = IV_OFFSET; - sa_ctx->xf[idx].a.cipher.iv.length = iv_length; - sa_ctx->xf[idx].a.next = NULL; - - sa_ctx->xf[idx].b.type = RTE_CRYPTO_SYM_XFORM_AUTH; - sa_ctx->xf[idx].b.auth.algo = sa->auth_algo; - sa_ctx->xf[idx].b.auth.add_auth_data_length = - sa->aad_len; - sa_ctx->xf[idx].b.auth.key.data = sa->auth_key; - sa_ctx->xf[idx].b.auth.key.length = - sa->auth_key_len; - sa_ctx->xf[idx].b.auth.digest_length = - sa->digest_len; - sa_ctx->xf[idx].b.auth.op = - RTE_CRYPTO_AUTH_OP_GENERATE; - } + sa->xforms = &sa_ctx->xf[idx].a; - sa_ctx->xf[idx].a.next = &sa_ctx->xf[idx].b; - sa_ctx->xf[idx].b.next = NULL; - sa->xforms = &sa_ctx->xf[idx].a; + print_one_sa_rule(sa, inbound); + } else { + switch (sa->cipher_algo) { + case RTE_CRYPTO_CIPHER_NULL: + case RTE_CRYPTO_CIPHER_AES_CBC: + iv_length = sa->iv_len; + break; + case RTE_CRYPTO_CIPHER_AES_CTR: + iv_length = 16; + break; + default: + RTE_LOG(ERR, IPSEC_ESP, "unsupported cipher algorithm %u\n", + sa->cipher_algo); + return -EINVAL; + } + + if (inbound) { + sa_ctx->xf[idx].b.type = RTE_CRYPTO_SYM_XFORM_CIPHER; + sa_ctx->xf[idx].b.cipher.algo = sa->cipher_algo; + sa_ctx->xf[idx].b.cipher.key.data = sa->cipher_key; + sa_ctx->xf[idx].b.cipher.key.length = + sa->cipher_key_len; + sa_ctx->xf[idx].b.cipher.op = + RTE_CRYPTO_CIPHER_OP_DECRYPT; + sa_ctx->xf[idx].b.next = NULL; + sa_ctx->xf[idx].b.cipher.iv.offset = IV_OFFSET; + sa_ctx->xf[idx].b.cipher.iv.length = iv_length; + + sa_ctx->xf[idx].a.type = RTE_CRYPTO_SYM_XFORM_AUTH; + sa_ctx->xf[idx].a.auth.algo = sa->auth_algo; + sa_ctx->xf[idx].a.auth.key.data = sa->auth_key; + sa_ctx->xf[idx].a.auth.key.length = + sa->auth_key_len; + sa_ctx->xf[idx].a.auth.digest_length = + sa->digest_len; + sa_ctx->xf[idx].a.auth.op = + RTE_CRYPTO_AUTH_OP_VERIFY; + } else { /* outbound */ + sa_ctx->xf[idx].a.type = RTE_CRYPTO_SYM_XFORM_CIPHER; + sa_ctx->xf[idx].a.cipher.algo = sa->cipher_algo; + sa_ctx->xf[idx].a.cipher.key.data = sa->cipher_key; + sa_ctx->xf[idx].a.cipher.key.length = + sa->cipher_key_len; + sa_ctx->xf[idx].a.cipher.op = + RTE_CRYPTO_CIPHER_OP_ENCRYPT; + sa_ctx->xf[idx].a.next = NULL; + sa_ctx->xf[idx].a.cipher.iv.offset = IV_OFFSET; + sa_ctx->xf[idx].a.cipher.iv.length = iv_length; + + sa_ctx->xf[idx].b.type = RTE_CRYPTO_SYM_XFORM_AUTH; + sa_ctx->xf[idx].b.auth.algo = sa->auth_algo; + sa_ctx->xf[idx].b.auth.key.data = sa->auth_key; + sa_ctx->xf[idx].b.auth.key.length = + sa->auth_key_len; + sa_ctx->xf[idx].b.auth.digest_length = + sa->digest_len; + sa_ctx->xf[idx].b.auth.op = + RTE_CRYPTO_AUTH_OP_GENERATE; + } - print_one_sa_rule(sa, inbound); + sa_ctx->xf[idx].a.next = &sa_ctx->xf[idx].b; + sa_ctx->xf[idx].b.next = NULL; + sa->xforms = &sa_ctx->xf[idx].a; + + print_one_sa_rule(sa, inbound); + } } return 0; diff --git a/examples/l2fwd-crypto/main.c b/examples/l2fwd-crypto/main.c index edf4db7..9778537 100644 --- a/examples/l2fwd-crypto/main.c +++ b/examples/l2fwd-crypto/main.c @@ -130,7 +130,8 @@ enum l2fwd_crypto_xform_chain { L2FWD_CRYPTO_CIPHER_HASH, L2FWD_CRYPTO_HASH_CIPHER, L2FWD_CRYPTO_CIPHER_ONLY, - L2FWD_CRYPTO_HASH_ONLY + L2FWD_CRYPTO_HASH_ONLY, + L2FWD_CRYPTO_AEAD }; struct l2fwd_key { @@ -172,6 +173,14 @@ struct l2fwd_crypto_options { unsigned int auth_iv_param; int auth_iv_random_size; + struct rte_crypto_sym_xform aead_xform; + unsigned aead_key_param; + int aead_key_random_size; + + struct l2fwd_iv aead_iv; + unsigned int aead_iv_param; + int aead_iv_random_size; + struct l2fwd_key aad; unsigned aad_param; int aad_random_size; @@ -199,10 +208,12 @@ struct l2fwd_crypto_params { uint8_t do_cipher; uint8_t do_hash; + uint8_t do_aead; uint8_t hash_verify; enum rte_crypto_cipher_algorithm cipher_algo; enum rte_crypto_auth_algorithm auth_algo; + enum rte_crypto_aead_algorithm aead_algo; }; /** lcore configuration */ @@ -490,14 +501,6 @@ l2fwd_simple_crypto_enqueue(struct rte_mbuf *m, op->sym->auth.data.offset = ipdata_offset; op->sym->auth.data.length = data_len; } - - if (cparams->aad.length) { - op->sym->auth.aad.data = cparams->aad.data; - op->sym->auth.aad.phys_addr = cparams->aad.phys_addr; - } else { - op->sym->auth.aad.data = NULL; - op->sym->auth.aad.phys_addr = 0; - } } if (cparams->do_cipher) { @@ -519,6 +522,33 @@ l2fwd_simple_crypto_enqueue(struct rte_mbuf *m, } } + if (cparams->do_aead) { + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + IV_OFFSET); + /* Copy IV at the end of the crypto operation */ + rte_memcpy(IV_ptr, cparams->aead.iv.data, cparams->aead.iv.length); + + op->sym->aead.data.offset = ipdata_offset; + op->sym->aead.data.length = data_len; + + if (!cparams->hash_verify) { + /* Append space for digest to end of packet */ + op->sym->aead.digest.data = (uint8_t *)rte_pktmbuf_append(m, + cparams->digest_length); + } else { + op->sym->aead.digest.data = rte_pktmbuf_mtod(m, + uint8_t *) + ipdata_offset + data_len; + } + + op->sym->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, + rte_pktmbuf_pkt_len(m) - cparams->digest_length); + + if (cparams->aad.length) { + op->sym->aead.aad.data = cparams->aad.data; + op->sym->aead.aad.phys_addr = cparams->aad.phys_addr; + } + } + op->sym->m_src = m; return l2fwd_crypto_enqueue(op, cparams); @@ -615,7 +645,9 @@ initialize_crypto_session(struct l2fwd_crypto_options *options, { struct rte_crypto_sym_xform *first_xform; - if (options->xform_chain == L2FWD_CRYPTO_CIPHER_HASH) { + if (options->xform_chain == L2FWD_CRYPTO_AEAD) { + first_xform = &options->aead_xform; + } else if (options->xform_chain == L2FWD_CRYPTO_CIPHER_HASH) { first_xform = &options->cipher_xform; first_xform->next = &options->auth_xform; } else if (options->xform_chain == L2FWD_CRYPTO_HASH_CIPHER) { @@ -667,8 +699,12 @@ l2fwd_main_loop(struct l2fwd_crypto_options *options) for (i = 0; i < qconf->nb_crypto_devs; i++) { port_cparams[i].do_cipher = 0; port_cparams[i].do_hash = 0; + port_cparams[i].do_aead = 0; switch (options->xform_chain) { + case L2FWD_CRYPTO_AEAD: + port_cparams[i].do_aead = 1; + break; case L2FWD_CRYPTO_CIPHER_HASH: case L2FWD_CRYPTO_HASH_CIPHER: port_cparams[i].do_cipher = 1; @@ -693,6 +729,12 @@ l2fwd_main_loop(struct l2fwd_crypto_options *options) if (!options->auth_iv_param) generate_random_key(port_cparams[i].auth_iv.data, port_cparams[i].auth_iv.length); + if (options->auth_xform.auth.op == RTE_CRYPTO_AUTH_OP_VERIFY) + port_cparams[i].hash_verify = 1; + else + port_cparams[i].hash_verify = 0; + + port_cparams[i].auth_algo = options->auth_xform.auth.algo; /* Set IV parameters */ if (options->auth_iv.length) { options->auth_xform.auth.iv.offset = @@ -700,11 +742,16 @@ l2fwd_main_loop(struct l2fwd_crypto_options *options) options->auth_xform.auth.iv.length = options->auth_iv.length; } + } + + if (port_cparams[i].do_aead) { + port_cparams[i].aead_algo = options->aead_xform.aead.algo; port_cparams[i].digest_length = - options->auth_xform.auth.digest_length; - if (options->auth_xform.auth.add_auth_data_length) { + options->aead_xform.aead.digest_length; + if (options->aead_xform.aead.add_auth_data_length) { port_cparams[i].aad.data = options->aad.data; port_cparams[i].aad.phys_addr = options->aad.phys_addr; + port_cparams[i].aad.length = options->aad.length; if (!options->aad_param) generate_random_key(port_cparams[i].aad.data, port_cparams[i].aad.length); @@ -712,12 +759,14 @@ l2fwd_main_loop(struct l2fwd_crypto_options *options) } else port_cparams[i].aad.length = 0; - if (options->auth_xform.auth.op == RTE_CRYPTO_AUTH_OP_VERIFY) + if (options->aead_xform.aead.op == RTE_CRYPTO_AEAD_OP_DECRYPT) port_cparams[i].hash_verify = 1; else port_cparams[i].hash_verify = 0; - port_cparams[i].auth_algo = options->auth_xform.auth.algo; + /* Set IV parameters */ + options->aead_xform.aead.iv.offset = IV_OFFSET; + options->aead_xform.aead.iv.length = options->aead_iv.length; } if (port_cparams[i].do_cipher) { @@ -878,7 +927,7 @@ l2fwd_crypto_usage(const char *prgname) " --cdev_type HW / SW / ANY\n" " --chain HASH_CIPHER / CIPHER_HASH / CIPHER_ONLY /" - " HASH_ONLY\n" + " HASH_ONLY / AEAD\n" " --cipher_algo ALGO\n" " --cipher_op ENCRYPT / DECRYPT\n" @@ -893,8 +942,16 @@ l2fwd_crypto_usage(const char *prgname) " --auth_key_random_size SIZE: size of auth key when generated randomly\n" " --auth_iv IV (bytes separated with \":\")\n" " --auth_iv_random_size SIZE: size of auth IV when generated randomly\n" + + " --aead_algo ALGO\n" + " --aead_op ENCRYPT / DECRYPT\n" + " --aead_key KEY (bytes separated with \":\")\n" + " --aead_key_random_size SIZE: size of AEAD key when generated randomly\n" + " --aead_iv IV (bytes separated with \":\")\n" + " --aead_iv_random_size SIZE: size of AEAD IV when generated randomly\n" " --aad AAD (bytes separated with \":\")\n" " --aad_random_size SIZE: size of AAD when generated randomly\n" + " --digest_size SIZE: size of digest to be generated/verified\n" " --sessionless\n" @@ -936,6 +993,9 @@ parse_crypto_opt_chain(struct l2fwd_crypto_options *options, char *optarg) } else if (strcmp("HASH_ONLY", optarg) == 0) { options->xform_chain = L2FWD_CRYPTO_HASH_ONLY; return 0; + } else if (strcmp("AEAD", optarg) == 0) { + options->xform_chain = L2FWD_CRYPTO_AEAD; + return 0; } return -1; @@ -1043,6 +1103,32 @@ parse_auth_op(enum rte_crypto_auth_operation *op, char *optarg) } static int +parse_aead_algo(enum rte_crypto_aead_algorithm *algo, char *optarg) +{ + if (rte_cryptodev_get_aead_algo_enum(algo, optarg) < 0) { + RTE_LOG(ERR, USER1, "AEAD algorithm specified " + "not supported!\n"); + return -1; + } + + return 0; +} + +static int +parse_aead_op(enum rte_crypto_aead_operation *op, char *optarg) +{ + if (strcmp("ENCRYPT", optarg) == 0) { + *op = RTE_CRYPTO_AEAD_OP_ENCRYPT; + return 0; + } else if (strcmp("DECRYPT", optarg) == 0) { + *op = RTE_CRYPTO_AEAD_OP_DECRYPT; + return 0; + } + + printf("AEAD operation specified not supported!\n"); + return -1; +} +static int parse_cryptodev_mask(struct l2fwd_crypto_options *options, const char *q_arg) { @@ -1140,7 +1226,6 @@ l2fwd_crypto_parse_args_long_options(struct l2fwd_crypto_options *options, return parse_size(&options->akey_random_size, optarg); } - else if (strcmp(lgopts[option_index].name, "auth_iv") == 0) { options->auth_iv_param = 1; options->auth_iv.length = @@ -1154,6 +1239,43 @@ l2fwd_crypto_parse_args_long_options(struct l2fwd_crypto_options *options, else if (strcmp(lgopts[option_index].name, "auth_iv_random_size") == 0) return parse_size(&options->auth_iv_random_size, optarg); + /* AEAD options */ + else if (strcmp(lgopts[option_index].name, "aead_algo") == 0) { + return parse_aead_algo(&options->aead_xform.aead.algo, + optarg); + } + + else if (strcmp(lgopts[option_index].name, "aead_op") == 0) + return parse_aead_op(&options->aead_xform.aead.op, + optarg); + + else if (strcmp(lgopts[option_index].name, "aead_key") == 0) { + options->aead_key_param = 1; + options->aead_xform.aead.key.length = + parse_key(options->aead_xform.aead.key.data, optarg); + if (options->aead_xform.aead.key.length > 0) + return 0; + else + return -1; + } + + else if (strcmp(lgopts[option_index].name, "aead_key_random_size") == 0) { + return parse_size(&options->aead_key_random_size, optarg); + + + else if (strcmp(lgopts[option_index].name, "aead_iv") == 0) { + options->aead_iv_param = 1; + options->aead_iv.length = + parse_key(options->aead_iv.data, optarg); + if (options->aead_iv.length > 0) + return 0; + else + return -1; + } + + else if (strcmp(lgopts[option_index].name, "aead_iv_random_size") == 0) + return parse_size(&options->aead_iv_random_size, optarg); + else if (strcmp(lgopts[option_index].name, "aad") == 0) { options->aad_param = 1; options->aad.length = @@ -1285,13 +1407,31 @@ l2fwd_crypto_default_options(struct l2fwd_crypto_options *options) options->akey_param = 0; options->akey_random_size = -1; options->auth_xform.auth.key.length = 0; + options->cipher_iv_param = 0; + options->cipher_iv_random_size = -1; + options->cipher_iv.length = 0; + + options->auth_xform.auth.algo = RTE_CRYPTO_AUTH_SHA1_HMAC; + options->auth_xform.auth.op = RTE_CRYPTO_AUTH_OP_GENERATE; + + /* AEAD Data */ + options->aead_xform.type = RTE_CRYPTO_SYM_XFORM_AEAD; + options->aead_xform.next = NULL; + options->aead_key_param = 0; + options->aead_key_random_size = -1; + options->aead_xform.aead.key.length = 0; + options->aead_iv_param = 0; + options->aead_iv_random_size = -1; + options->aead_iv.length = 0; + + options->auth_xform.auth.algo = RTE_CRYPTO_AEAD_AES_GCM; + options->auth_xform.auth.op = RTE_CRYPTO_AEAD_OP_ENCRYPT; + options->aad_param = 0; options->aad_random_size = -1; options->aad.length = 0; - options->digest_size = -1; - options->auth_xform.auth.algo = RTE_CRYPTO_AUTH_SHA1_HMAC; - options->auth_xform.auth.op = RTE_CRYPTO_AUTH_OP_GENERATE; + options->digest_size = -1; options->type = CDEV_TYPE_ANY; options->cryptodev_mask = UINT64_MAX; @@ -1318,6 +1458,17 @@ display_auth_info(struct l2fwd_crypto_options *options) rte_hexdump(stdout, "Auth key:", options->auth_xform.auth.key.data, options->auth_xform.auth.key.length); +} + +static void +display_aead_info(struct l2fwd_crypto_options *options) +{ + printf("\n---- AEAD information ---\n"); + printf("Algorithm: %s\n", + rte_crypto_aead_algorithm_strings[options->aead_xform.aead.algo]); + rte_hexdump(stdout, "AEAD key:", + options->aead_xform.aead.key.data, + options->aead_xform.aead.key.length); rte_hexdump(stdout, "AAD:", options->aad.data, options->aad.length); } @@ -1326,6 +1477,7 @@ l2fwd_crypto_options_print(struct l2fwd_crypto_options *options) { char string_cipher_op[MAX_STR_LEN]; char string_auth_op[MAX_STR_LEN]; + char string_aead_op[MAX_STR_LEN]; if (options->cipher_xform.cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) strcpy(string_cipher_op, "Encrypt"); @@ -1337,6 +1489,12 @@ l2fwd_crypto_options_print(struct l2fwd_crypto_options *options) else strcpy(string_auth_op, "Auth verify"); + if (options->aead_xform.aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) + strcpy(string_aead_op, "Authenticated encryption"); + else + strcpy(string_aead_op, "Authenticated decryption"); + + printf("Options:-\nn"); printf("portmask: %x\n", options->portmask); printf("ports per lcore: %u\n", options->nb_ports_per_lcore); @@ -1363,6 +1521,10 @@ l2fwd_crypto_options_print(struct l2fwd_crypto_options *options) printf("\nCrypto chain: "); switch (options->xform_chain) { + case L2FWD_CRYPTO_AEAD: + printf("Input --> %s --> Output\n", string_aead_op); + display_aead_info(options); + break; case L2FWD_CRYPTO_CIPHER_HASH: printf("Input --> %s --> %s --> Output\n", string_cipher_op, string_auth_op); @@ -1400,6 +1562,9 @@ l2fwd_crypto_parse_args(struct l2fwd_crypto_options *options, { "cdev_type", required_argument, 0, 0 }, { "chain", required_argument, 0, 0 }, + { "iv", required_argument, 0, 0 }, + { "iv_random_size", required_argument, 0, 0 }, + { "cipher_algo", required_argument, 0, 0 }, { "cipher_op", required_argument, 0, 0 }, { "cipher_key", required_argument, 0, 0 }, @@ -1410,10 +1575,13 @@ l2fwd_crypto_parse_args(struct l2fwd_crypto_options *options, { "auth_key", required_argument, 0, 0 }, { "auth_key_random_size", required_argument, 0, 0 }, - { "iv", required_argument, 0, 0 }, - { "iv_random_size", required_argument, 0, 0 }, + { "aead_algo", required_argument, 0, 0 }, + { "aead_op", required_argument, 0, 0 }, + { "aead_key", required_argument, 0, 0 }, + { "aead_key_random_size", required_argument, 0, 0 }, { "aad", required_argument, 0, 0 }, { "aad_random_size", required_argument, 0, 0 }, + { "digest_size", required_argument, 0, 0 }, { "sessionless", no_argument, 0, 0 }, @@ -1641,6 +1809,8 @@ initialize_cryptodevs(struct l2fwd_crypto_options *options, unsigned nb_ports, enum rte_crypto_auth_algorithm opt_auth_algo; enum rte_crypto_cipher_algorithm cap_cipher_algo; enum rte_crypto_cipher_algorithm opt_cipher_algo; + enum rte_crypto_aead_algorithm cap_aead_algo; + enum rte_crypto_aead_algorithm opt_aead_algo; int retval; cdev_count = rte_cryptodev_count(); @@ -1668,6 +1838,128 @@ initialize_cryptodevs(struct l2fwd_crypto_options *options, unsigned nb_ports, rte_cryptodev_info_get(cdev_id, &dev_info); + /* Set AEAD parameters */ + if (options->xform_chain == L2FWD_CRYPTO_AEAD) { + /* Check if device supports AEAD algo */ + i = 0; + opt_aead_algo = options->aead_xform.aead.algo; + cap = &dev_info.capabilities[i]; + while (cap->op != RTE_CRYPTO_OP_TYPE_UNDEFINED) { + cap_aead_algo = cap->sym.aead.algo; + if (cap->sym.xform_type == + RTE_CRYPTO_SYM_XFORM_AEAD) { + if (cap_aead_algo == opt_aead_algo) { + if (check_type(options, &dev_info) == 0) + break; + } + } + cap = &dev_info.capabilities[++i]; + } + + if (cap->op == RTE_CRYPTO_OP_TYPE_UNDEFINED) { + printf("Algorithm %s not supported by cryptodev %u" + " or device not of preferred type (%s)\n", + rte_crypto_aead_algorithm_strings[opt_aead_algo], + cdev_id, + options->string_type); + continue; + } + + options->block_size = cap->sym.aead.block_size; + + check_iv_param(cap, options); + + /* + * Check if length of provided AEAD key is supported + * by the algorithm chosen. + */ + if (options->aead_key_param) { + if (check_supported_size( + options->aead_xform.aead.key.length, + cap->sym.aead.key_size.min, + cap->sym.aead.key_size.max, + cap->sym.aead.key_size.increment) + != 0) { + printf("Unsupported aead key length\n"); + return -1; + } + /* + * Check if length of the aead key to be randomly generated + * is supported by the algorithm chosen. + */ + } else if (options->aead_key_random_size != -1) { + if (check_supported_size(options->ckey_random_size, + cap->sym.aead.key_size.min, + cap->sym.aead.key_size.max, + cap->sym.aead.key_size.increment) + != 0) { + printf("Unsupported aead key length\n"); + return -1; + } + options->aead_xform.aead.key.length = + options->ckey_random_size; + /* No size provided, use minimum size. */ + } else + options->aead_xform.aead.key.length = + cap->sym.aead.key_size.min; + + if (!options->aead_key_param) + generate_random_key( + options->aead_xform.aead.key.data, + options->aead_xform.aead.key.length); + + /* + * Check if length of provided AAD is supported + * by the algorithm chosen. + */ + if (options->aad_param) { + if (check_supported_size(options->aad.length, + cap->sym.aead.aad_size.min, + cap->sym.aead.aad_size.max, + cap->sym.aead.aad_size.increment) + != 0) { + printf("Unsupported AAD length\n"); + return -1; + } + /* + * Check if length of AAD to be randomly generated + * is supported by the algorithm chosen. + */ + } else if (options->aad_random_size != -1) { + if (check_supported_size(options->aad_random_size, + cap->sym.aead.aad_size.min, + cap->sym.aead.aad_size.max, + cap->sym.aead.aad_size.increment) + != 0) { + printf("Unsupported AAD length\n"); + return -1; + } + options->aad.length = options->aad_random_size; + /* No size provided, use minimum size. */ + } else + options->aad.length = cap->sym.auth.aad_size.min; + + options->aead_xform.aead.add_auth_data_length = + options->aad.length; + + /* Check if digest size is supported by the algorithm. */ + if (options->digest_size != -1) { + if (check_supported_size(options->digest_size, + cap->sym.aead.digest_size.min, + cap->sym.aead.digest_size.max, + cap->sym.aead.digest_size.increment) + != 0) { + printf("Unsupported digest length\n"); + return -1; + } + options->aead_xform.aead.digest_length = + options->digest_size; + /* No size provided, use minimum size. */ + } else + options->aead_xform.aead.digest_length = + cap->sym.aead.digest_size.min; + } + /* Set cipher parameters */ if (options->xform_chain == L2FWD_CRYPTO_CIPHER_HASH || options->xform_chain == L2FWD_CRYPTO_HASH_CIPHER || @@ -1777,40 +2069,6 @@ initialize_cryptodevs(struct l2fwd_crypto_options *options, unsigned nb_ports, options->auth_iv_random_size, &options->auth_iv.length); /* - * Check if length of provided AAD is supported - * by the algorithm chosen. - */ - if (options->aad_param) { - if (check_supported_size(options->aad.length, - cap->sym.auth.aad_size.min, - cap->sym.auth.aad_size.max, - cap->sym.auth.aad_size.increment) - != 0) { - printf("Unsupported AAD length\n"); - return -1; - } - /* - * Check if length of AAD to be randomly generated - * is supported by the algorithm chosen. - */ - } else if (options->aad_random_size != -1) { - if (check_supported_size(options->aad_random_size, - cap->sym.auth.aad_size.min, - cap->sym.auth.aad_size.max, - cap->sym.auth.aad_size.increment) - != 0) { - printf("Unsupported AAD length\n"); - return -1; - } - options->aad.length = options->aad_random_size; - /* No size provided, use minimum size. */ - } else - options->aad.length = cap->sym.auth.aad_size.min; - - options->auth_xform.auth.add_auth_data_length = - options->aad.length; - - /* * Check if length of provided auth key is supported * by the algorithm chosen. */ @@ -2011,15 +2269,27 @@ reserve_key_memory(struct l2fwd_crypto_options *options) if (options->cipher_xform.cipher.key.data == NULL) rte_exit(EXIT_FAILURE, "Failed to allocate memory for cipher key"); - options->auth_xform.auth.key.data = rte_malloc("auth key", MAX_KEY_SIZE, 0); if (options->auth_xform.auth.key.data == NULL) rte_exit(EXIT_FAILURE, "Failed to allocate memory for auth key"); - options->cipher_iv.data = rte_malloc("iv", MAX_KEY_SIZE, 0); + options->aead_xform.aead.key.data = rte_malloc("aead key", + MAX_KEY_SIZE, 0); + if (options->aead_xform.aead.key.data == NULL) + rte_exit(EXIT_FAILURE, "Failed to allocate memory for AEAD key"); + + options->cipher_iv.data = rte_malloc("cipher iv", MAX_KEY_SIZE, 0); if (options->cipher_iv.data == NULL) - rte_exit(EXIT_FAILURE, "Failed to allocate memory for IV"); + rte_exit(EXIT_FAILURE, "Failed to allocate memory for cipher IV"); + + options->auth_iv.data = rte_malloc("auth_iv", MAX_KEY_SIZE, 0); + if (options->auth_iv.data == NULL) + rte_exit(EXIT_FAILURE, "Failed to allocate memory for auth iv"); + + options->aead_iv.data = rte_malloc("aead_iv", MAX_KEY_SIZE, 0); + if (options->aead_iv.data == NULL) + rte_exit(EXIT_FAILURE, "Failed to allocate memory for AEAD iv"); options->aad.data = rte_malloc("aad", MAX_KEY_SIZE, 0); if (options->aad.data == NULL) diff --git a/lib/librte_cryptodev/rte_crypto_sym.h b/lib/librte_cryptodev/rte_crypto_sym.h index fc0d06b..9d9592e 100644 --- a/lib/librte_cryptodev/rte_crypto_sym.h +++ b/lib/librte_cryptodev/rte_crypto_sym.h @@ -68,27 +68,12 @@ enum rte_crypto_cipher_algorithm { RTE_CRYPTO_CIPHER_AES_CBC, /**< AES algorithm in CBC mode */ - RTE_CRYPTO_CIPHER_AES_CCM, - /**< AES algorithm in CCM mode. When this cipher algorithm is used the - * *RTE_CRYPTO_AUTH_AES_CCM* element of the - * *rte_crypto_hash_algorithm* enum MUST be used to set up the related - * *rte_crypto_auth_xform* structure in the session context or in - * the op_params of the crypto operation structure in the case of a - * session-less crypto operation - */ RTE_CRYPTO_CIPHER_AES_CTR, /**< AES algorithm in Counter mode */ RTE_CRYPTO_CIPHER_AES_ECB, /**< AES algorithm in ECB mode */ RTE_CRYPTO_CIPHER_AES_F8, /**< AES algorithm in F8 mode */ - RTE_CRYPTO_CIPHER_AES_GCM, - /**< AES algorithm in GCM mode. When this cipher algorithm is used the - * *RTE_CRYPTO_AUTH_AES_GCM* element of the *rte_crypto_auth_algorithm* - * enum MUST be used to set up the related *rte_crypto_auth_setup_data* - * structure in the session context or in the op_params of the crypto - * operation structure in the case of a session-less crypto operation. - */ RTE_CRYPTO_CIPHER_AES_XTS, /**< AES algorithm in XTS mode */ @@ -247,25 +232,8 @@ enum rte_crypto_auth_algorithm { RTE_CRYPTO_AUTH_AES_CBC_MAC, /**< AES-CBC-MAC algorithm. Only 128-bit keys are supported. */ - RTE_CRYPTO_AUTH_AES_CCM, - /**< AES algorithm in CCM mode. This is an authenticated cipher. When - * this hash algorithm is used, the *RTE_CRYPTO_CIPHER_AES_CCM* - * element of the *rte_crypto_cipher_algorithm* enum MUST be used to - * set up the related rte_crypto_cipher_setup_data structure in the - * session context or the corresponding parameter in the crypto - * operation data structures op_params parameter MUST be set for a - * session-less crypto operation. - */ RTE_CRYPTO_AUTH_AES_CMAC, /**< AES CMAC algorithm. */ - RTE_CRYPTO_AUTH_AES_GCM, - /**< AES algorithm in GCM mode. When this hash algorithm - * is used, the RTE_CRYPTO_CIPHER_AES_GCM element of the - * rte_crypto_cipher_algorithm enum MUST be used to set up the related - * rte_crypto_cipher_setup_data structure in the session context, or - * the corresponding parameter in the crypto operation data structures - * op_params parameter MUST be set for a session-less crypto operation. - */ RTE_CRYPTO_AUTH_AES_GMAC, /**< AES GMAC algorithm. */ RTE_CRYPTO_AUTH_AES_XCBC_MAC, @@ -363,20 +331,6 @@ struct rte_crypto_auth_xform { * The maximum permitted value is 65535 (2^16 - 1) bytes, unless * otherwise specified below. * - * This field must be specified when the hash algorithm is one of the - * following: - * - * - For GCM (@ref RTE_CRYPTO_AUTH_AES_GCM). In this case, this is - * the length of the Additional Authenticated Data (called A, in NIST - * SP800-38D). - * - * - For CCM (@ref RTE_CRYPTO_AUTH_AES_CCM). In this case, this is - * the length of the associated data (called A, in NIST SP800-38C). - * Note that this does NOT include the length of any padding, or the - * 18 bytes reserved at the start of the above field to store the - * block B0 and the encoded length. The maximum permitted value in - * this case is 222 bytes. - * */ struct { @@ -443,6 +397,18 @@ struct rte_crypto_aead_xform { uint32_t digest_length; + struct { + uint16_t offset; + /**< Starting point for Initialisation Vector or Counter, + * specified as number of bytes from start of crypto + * operation. + * + * For optimum performance, the data pointed to SHOULD + * be 8-byte aligned. + */ + uint16_t length; + /**< Length of valid IV data.*/ + } iv; /**< Initialisation vector parameters */ uint16_t add_auth_data_length; /**< The length of the additional authenticated data (AAD) in bytes. */ }; @@ -624,14 +590,6 @@ struct rte_crypto_sym_op { * same as the result length. * * @note - * In the case of CCM @ref RTE_CRYPTO_AUTH_AES_CCM, - * this value should not include the length of the - * padding or the length of the MAC; the driver will - * compute the actual number of bytes over which the - * encryption will occur, which will include these - * values. - * - * @note * For SNOW 3G @ RTE_CRYPTO_AUTH_SNOW3G_UEA2, * KASUMI @ RTE_CRYPTO_CIPHER_KASUMI_F8 * and ZUC @ RTE_CRYPTO_CIPHER_ZUC_EEA3, @@ -648,11 +606,6 @@ struct rte_crypto_sym_op { * buffer. * * @note - * For CCM and GCM modes of operation, this field is - * ignored. The field @ref aad field - * should be set instead. - * - * @note * For SNOW 3G @ RTE_CRYPTO_AUTH_SNOW3G_UIA2, * KASUMI @ RTE_CRYPTO_AUTH_KASUMI_F9 * and ZUC @ RTE_CRYPTO_AUTH_ZUC_EIA3, @@ -663,11 +616,6 @@ struct rte_crypto_sym_op { * buffer that the hash will be computed on. * * @note - * For CCM and GCM modes of operation, this field is - * ignored. The field @ref aad field should be set - * instead. - * - * @note * For SNOW 3G @ RTE_CRYPTO_AUTH_SNOW3G_UIA2, * KASUMI @ RTE_CRYPTO_AUTH_KASUMI_F9 * and ZUC @ RTE_CRYPTO_AUTH_ZUC_EIA3, @@ -692,9 +640,6 @@ struct rte_crypto_sym_op { * For digest generation, the digest result will * overwrite any data at this location. * - * @note - * For GCM (@ref RTE_CRYPTO_AUTH_AES_GCM), for - * "digest result" read "authentication tag T". */ phys_addr_t phys_addr; /**< Physical address of digest */ @@ -712,32 +657,6 @@ struct rte_crypto_sym_op { * rte_cryptodev_sym_session_create function call. * This length must not exceed 65535 (2^16-1) bytes. * - * Specifically for CCM (@ref RTE_CRYPTO_AUTH_AES_CCM), - * the caller should setup this field as follows: - * - * - the nonce should be written starting at an offset - * of one byte into the array, leaving room for the - * implementation to write in the flags to the first - * byte. - * - * - the additional authentication data itself should - * be written starting at an offset of 18 bytes into - * the array, leaving room for the length encoding in - * the first two bytes of the second block. - * - * - the array should be big enough to hold the above - * fields, plus any padding to round this up to the - * nearest multiple of the block size (16 bytes). - * Padding will be added by the implementation. - * - * Finally, for GCM (@ref RTE_CRYPTO_AUTH_AES_GCM), the - * caller should setup this field as follows: - * - * - the AAD is written in starting at byte 0 - * - the array must be big enough to hold the AAD, plus - * any space to round this up to the nearest multiple - * of the block size (16 bytes). - * */ phys_addr_t phys_addr; /**< physical address */ } aad; diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c index acd74a1..3198fb5 100644 --- a/lib/librte_cryptodev/rte_cryptodev.c +++ b/lib/librte_cryptodev/rte_cryptodev.c @@ -124,11 +124,9 @@ rte_crypto_cipher_algorithm_strings[] = { [RTE_CRYPTO_CIPHER_3DES_CTR] = "3des-ctr", [RTE_CRYPTO_CIPHER_AES_CBC] = "aes-cbc", - [RTE_CRYPTO_CIPHER_AES_CCM] = "aes-ccm", [RTE_CRYPTO_CIPHER_AES_CTR] = "aes-ctr", [RTE_CRYPTO_CIPHER_AES_DOCSISBPI] = "aes-docsisbpi", [RTE_CRYPTO_CIPHER_AES_ECB] = "aes-ecb", - [RTE_CRYPTO_CIPHER_AES_GCM] = "aes-gcm", [RTE_CRYPTO_CIPHER_AES_F8] = "aes-f8", [RTE_CRYPTO_CIPHER_AES_XTS] = "aes-xts", @@ -161,9 +159,7 @@ rte_crypto_cipher_operation_strings[] = { const char * rte_crypto_auth_algorithm_strings[] = { [RTE_CRYPTO_AUTH_AES_CBC_MAC] = "aes-cbc-mac", - [RTE_CRYPTO_AUTH_AES_CCM] = "aes-ccm", [RTE_CRYPTO_AUTH_AES_CMAC] = "aes-cmac", - [RTE_CRYPTO_AUTH_AES_GCM] = "aes-gcm", [RTE_CRYPTO_AUTH_AES_GMAC] = "aes-gmac", [RTE_CRYPTO_AUTH_AES_XCBC_MAC] = "aes-xcbc-mac", @@ -464,7 +460,7 @@ rte_cryptodev_sym_capability_check_aead( if (param_range_check(aad_size, capability->aead.aad_size)) return -1; - if (param_range_check(iv_size, capability->iv_size)) + if (param_range_check(iv_size, capability->aead.iv_size)) return -1; return 0; diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h index 4bd58fe..afbbb14 100644 --- a/lib/librte_cryptodev/rte_cryptodev.h +++ b/lib/librte_cryptodev/rte_cryptodev.h @@ -204,6 +204,8 @@ struct rte_cryptodev_symmetric_capability { /**< AEAD key size range */ struct rte_crypto_param_range digest_size; /**< digest size range */ + struct rte_crypto_param_range iv_size; + /**< Initialisation vector data size range */ struct rte_crypto_param_range aad_size; /**< Additional authentication data size range */ } aead; diff --git a/test/test/test_cryptodev.c b/test/test/test_cryptodev.c index b4b1289..77407f7 100644 --- a/test/test/test_cryptodev.c +++ b/test/test/test_cryptodev.c @@ -76,6 +76,7 @@ struct crypto_testsuite_params { struct crypto_unittest_params { struct rte_crypto_sym_xform cipher_xform; struct rte_crypto_sym_xform auth_xform; + struct rte_crypto_sym_xform aead_xform; struct rte_cryptodev_sym_session *sess; @@ -4630,54 +4631,34 @@ test_3DES_cipheronly_openssl_all(void) /* ***** AES-GCM Tests ***** */ static int -create_gcm_session(uint8_t dev_id, enum rte_crypto_cipher_operation op, +create_gcm_session(uint8_t dev_id, enum rte_crypto_aead_operation op, const uint8_t *key, const uint8_t key_len, const uint16_t aad_len, const uint8_t auth_len, - uint8_t iv_len, - enum rte_crypto_auth_operation auth_op) + uint8_t iv_len) { - uint8_t cipher_key[key_len]; + uint8_t aead_key[key_len]; struct crypto_unittest_params *ut_params = &unittest_params; - memcpy(cipher_key, key, key_len); + memcpy(aead_key, key, key_len); - /* Setup Cipher Parameters */ - ut_params->cipher_xform.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - ut_params->cipher_xform.next = NULL; - - ut_params->cipher_xform.cipher.algo = RTE_CRYPTO_CIPHER_AES_GCM; - ut_params->auth_xform.auth.op = auth_op; - ut_params->cipher_xform.cipher.op = op; - ut_params->cipher_xform.cipher.key.data = cipher_key; - ut_params->cipher_xform.cipher.key.length = key_len; - ut_params->cipher_xform.cipher.iv.offset = IV_OFFSET; - ut_params->cipher_xform.cipher.iv.length = iv_len; + /* Setup AEAD Parameters */ + ut_params->aead_xform.type = RTE_CRYPTO_SYM_XFORM_AEAD; + ut_params->aead_xform.next = NULL; + ut_params->aead_xform.aead.algo = RTE_CRYPTO_AEAD_AES_GCM; + ut_params->aead_xform.aead.op = op; + ut_params->aead_xform.aead.key.data = aead_key; + ut_params->aead_xform.aead.key.length = key_len; + ut_params->aead_xform.aead.iv.offset = IV_OFFSET; + ut_params->aead_xform.aead.iv.length = iv_len; + ut_params->aead_xform.aead.digest_length = auth_len; + ut_params->aead_xform.aead.add_auth_data_length = aad_len; TEST_HEXDUMP(stdout, "key:", key, key_len); - /* Setup Authentication Parameters */ - ut_params->auth_xform.type = RTE_CRYPTO_SYM_XFORM_AUTH; - ut_params->auth_xform.next = NULL; - - ut_params->auth_xform.auth.algo = RTE_CRYPTO_AUTH_AES_GCM; - - ut_params->auth_xform.auth.digest_length = auth_len; - ut_params->auth_xform.auth.add_auth_data_length = aad_len; - ut_params->auth_xform.auth.key.length = 0; - ut_params->auth_xform.auth.key.data = NULL; - - if (op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) { - ut_params->cipher_xform.next = &ut_params->auth_xform; - - /* Create Crypto session*/ - ut_params->sess = rte_cryptodev_sym_session_create(dev_id, - &ut_params->cipher_xform); - } else {/* Create Crypto session*/ - ut_params->auth_xform.next = &ut_params->cipher_xform; - ut_params->sess = rte_cryptodev_sym_session_create(dev_id, - &ut_params->auth_xform); - } + /* Create Crypto session*/ + ut_params->sess = rte_cryptodev_sym_session_create(dev_id, + &ut_params->aead_xform); TEST_ASSERT_NOT_NULL(ut_params->sess, "Session creation failed"); @@ -4686,43 +4667,35 @@ create_gcm_session(uint8_t dev_id, enum rte_crypto_cipher_operation op, static int create_gcm_xforms(struct rte_crypto_op *op, - enum rte_crypto_cipher_operation cipher_op, + enum rte_crypto_aead_operation aead_op, uint8_t *key, const uint8_t key_len, const uint8_t aad_len, const uint8_t auth_len, - uint8_t iv_len, - enum rte_crypto_auth_operation auth_op) + uint8_t iv_len) { - TEST_ASSERT_NOT_NULL(rte_crypto_op_sym_xforms_alloc(op, 2), - "failed to allocate space for crypto transforms"); + TEST_ASSERT_NOT_NULL(rte_crypto_op_sym_xforms_alloc(op, 1), + "failed to allocate space for crypto transform"); struct rte_crypto_sym_op *sym_op = op->sym; - /* Setup Cipher Parameters */ - sym_op->xform->type = RTE_CRYPTO_SYM_XFORM_CIPHER; - sym_op->xform->cipher.algo = RTE_CRYPTO_CIPHER_AES_GCM; - sym_op->xform->cipher.op = cipher_op; - sym_op->xform->cipher.key.data = key; - sym_op->xform->cipher.key.length = key_len; - sym_op->xform->cipher.iv.offset = IV_OFFSET; - sym_op->xform->cipher.iv.length = iv_len; + /* Setup AEAD Parameters */ + sym_op->xform->type = RTE_CRYPTO_SYM_XFORM_AEAD; + sym_op->xform->next = NULL; + sym_op->xform->aead.algo = RTE_CRYPTO_AEAD_AES_GCM; + sym_op->xform->aead.op = aead_op; + sym_op->xform->aead.key.data = key; + sym_op->xform->aead.key.length = key_len; + sym_op->xform->aead.iv.offset = IV_OFFSET; + sym_op->xform->aead.iv.length = iv_len; + sym_op->xform->aead.digest_length = auth_len; + sym_op->xform->aead.add_auth_data_length = aad_len; TEST_HEXDUMP(stdout, "key:", key, key_len); - /* Setup Authentication Parameters */ - sym_op->xform->next->type = RTE_CRYPTO_SYM_XFORM_AUTH; - sym_op->xform->next->auth.algo = RTE_CRYPTO_AUTH_AES_GCM; - sym_op->xform->next->auth.op = auth_op; - sym_op->xform->next->auth.digest_length = auth_len; - sym_op->xform->next->auth.add_auth_data_length = aad_len; - sym_op->xform->next->auth.key.length = 0; - sym_op->xform->next->auth.key.data = NULL; - sym_op->xform->next->next = NULL; - return 0; } static int -create_gcm_operation(enum rte_crypto_cipher_operation op, +create_gcm_operation(enum rte_crypto_aead_operation op, const struct gcm_test_data *tdata) { struct crypto_testsuite_params *ts_params = &testsuite_params; @@ -4741,15 +4714,15 @@ create_gcm_operation(enum rte_crypto_cipher_operation op, /* Append aad data */ aad_pad_len = RTE_ALIGN_CEIL(tdata->aad.len, 16); - sym_op->auth.aad.data = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf, + sym_op->aead.aad.data = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf, aad_pad_len); - TEST_ASSERT_NOT_NULL(sym_op->auth.aad.data, + TEST_ASSERT_NOT_NULL(sym_op->aead.aad.data, "no room to append aad"); - sym_op->auth.aad.phys_addr = + sym_op->aead.aad.phys_addr = rte_pktmbuf_mtophys(ut_params->ibuf); - memcpy(sym_op->auth.aad.data, tdata->aad.data, tdata->aad.len); - TEST_HEXDUMP(stdout, "aad:", sym_op->auth.aad.data, + memcpy(sym_op->aead.aad.data, tdata->aad.data, tdata->aad.len); + TEST_HEXDUMP(stdout, "aad:", sym_op->aead.aad.data, tdata->aad.len); /* Append IV at the end of the crypto operation*/ @@ -4761,7 +4734,7 @@ create_gcm_operation(enum rte_crypto_cipher_operation op, tdata->iv.len); /* Append plaintext/ciphertext */ - if (op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) { + if (op == RTE_CRYPTO_AEAD_OP_ENCRYPT) { plaintext_pad_len = RTE_ALIGN_CEIL(tdata->plaintext.len, 16); plaintext = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf, plaintext_pad_len); @@ -4806,40 +4779,37 @@ create_gcm_operation(enum rte_crypto_cipher_operation op, } /* Append digest data */ - if (op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) { - sym_op->auth.digest.data = (uint8_t *)rte_pktmbuf_append( + if (op == RTE_CRYPTO_AEAD_OP_ENCRYPT) { + sym_op->aead.digest.data = (uint8_t *)rte_pktmbuf_append( ut_params->obuf ? ut_params->obuf : ut_params->ibuf, tdata->auth_tag.len); - TEST_ASSERT_NOT_NULL(sym_op->auth.digest.data, + TEST_ASSERT_NOT_NULL(sym_op->aead.digest.data, "no room to append digest"); - memset(sym_op->auth.digest.data, 0, tdata->auth_tag.len); - sym_op->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset( + memset(sym_op->aead.digest.data, 0, tdata->auth_tag.len); + sym_op->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset( ut_params->obuf ? ut_params->obuf : ut_params->ibuf, plaintext_pad_len + aad_pad_len); } else { - sym_op->auth.digest.data = (uint8_t *)rte_pktmbuf_append( + sym_op->aead.digest.data = (uint8_t *)rte_pktmbuf_append( ut_params->ibuf, tdata->auth_tag.len); - TEST_ASSERT_NOT_NULL(sym_op->auth.digest.data, + TEST_ASSERT_NOT_NULL(sym_op->aead.digest.data, "no room to append digest"); - sym_op->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset( + sym_op->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset( ut_params->ibuf, plaintext_pad_len + aad_pad_len); - rte_memcpy(sym_op->auth.digest.data, tdata->auth_tag.data, + rte_memcpy(sym_op->aead.digest.data, tdata->auth_tag.data, tdata->auth_tag.len); TEST_HEXDUMP(stdout, "digest:", - sym_op->auth.digest.data, + sym_op->aead.digest.data, tdata->auth_tag.len); } - sym_op->cipher.data.length = tdata->plaintext.len; - sym_op->cipher.data.offset = aad_pad_len; - - sym_op->auth.data.length = tdata->plaintext.len; - sym_op->auth.data.offset = aad_pad_len; + sym_op->aead.data.length = tdata->plaintext.len; + sym_op->aead.data.offset = aad_pad_len; return 0; } @@ -4857,11 +4827,10 @@ test_mb_AES_GCM_authenticated_encryption(const struct gcm_test_data *tdata) /* Create GCM session */ retval = create_gcm_session(ts_params->valid_devs[0], - RTE_CRYPTO_CIPHER_OP_ENCRYPT, + RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata->key.data, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_GENERATE); + tdata->iv.len); if (retval < 0) return retval; @@ -4878,7 +4847,7 @@ test_mb_AES_GCM_authenticated_encryption(const struct gcm_test_data *tdata) rte_pktmbuf_tailroom(ut_params->ibuf)); /* Create GCM operation */ - retval = create_gcm_operation(RTE_CRYPTO_CIPHER_OP_ENCRYPT, tdata); + retval = create_gcm_operation(RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata); if (retval < 0) return retval; @@ -5035,11 +5004,10 @@ test_mb_AES_GCM_authenticated_decryption(const struct gcm_test_data *tdata) /* Create GCM session */ retval = create_gcm_session(ts_params->valid_devs[0], - RTE_CRYPTO_CIPHER_OP_DECRYPT, + RTE_CRYPTO_AEAD_OP_DECRYPT, tdata->key.data, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_VERIFY); + tdata->iv.len); if (retval < 0) return retval; @@ -5056,7 +5024,7 @@ test_mb_AES_GCM_authenticated_decryption(const struct gcm_test_data *tdata) rte_pktmbuf_tailroom(ut_params->ibuf)); /* Create GCM operation */ - retval = create_gcm_operation(RTE_CRYPTO_CIPHER_OP_DECRYPT, tdata); + retval = create_gcm_operation(RTE_CRYPTO_AEAD_OP_DECRYPT, tdata); if (retval < 0) return retval; @@ -5202,11 +5170,10 @@ test_AES_GCM_authenticated_encryption_oop(const struct gcm_test_data *tdata) /* Create GCM session */ retval = create_gcm_session(ts_params->valid_devs[0], - RTE_CRYPTO_CIPHER_OP_ENCRYPT, + RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata->key.data, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_GENERATE); + tdata->iv.len); if (retval < 0) return retval; @@ -5220,7 +5187,7 @@ test_AES_GCM_authenticated_encryption_oop(const struct gcm_test_data *tdata) rte_pktmbuf_tailroom(ut_params->obuf)); /* Create GCM operation */ - retval = create_gcm_operation(RTE_CRYPTO_CIPHER_OP_ENCRYPT, tdata); + retval = create_gcm_operation(RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata); if (retval < 0) return retval; @@ -5279,11 +5246,10 @@ test_AES_GCM_authenticated_decryption_oop(const struct gcm_test_data *tdata) /* Create GCM session */ retval = create_gcm_session(ts_params->valid_devs[0], - RTE_CRYPTO_CIPHER_OP_DECRYPT, + RTE_CRYPTO_AEAD_OP_DECRYPT, tdata->key.data, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_VERIFY); + tdata->iv.len); if (retval < 0) return retval; @@ -5297,7 +5263,7 @@ test_AES_GCM_authenticated_decryption_oop(const struct gcm_test_data *tdata) rte_pktmbuf_tailroom(ut_params->obuf)); /* Create GCM operation */ - retval = create_gcm_operation(RTE_CRYPTO_CIPHER_OP_DECRYPT, tdata); + retval = create_gcm_operation(RTE_CRYPTO_AEAD_OP_DECRYPT, tdata); if (retval < 0) return retval; @@ -5356,18 +5322,17 @@ test_AES_GCM_authenticated_encryption_sessionless( rte_pktmbuf_tailroom(ut_params->ibuf)); /* Create GCM operation */ - retval = create_gcm_operation(RTE_CRYPTO_CIPHER_OP_ENCRYPT, tdata); + retval = create_gcm_operation(RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata); if (retval < 0) return retval; /* Create GCM xforms */ memcpy(key, tdata->key.data, tdata->key.len); retval = create_gcm_xforms(ut_params->op, - RTE_CRYPTO_CIPHER_OP_ENCRYPT, + RTE_CRYPTO_AEAD_OP_ENCRYPT, key, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_GENERATE); + tdata->iv.len); if (retval < 0) return retval; @@ -5437,18 +5402,17 @@ test_AES_GCM_authenticated_decryption_sessionless( rte_pktmbuf_tailroom(ut_params->ibuf)); /* Create GCM operation */ - retval = create_gcm_operation(RTE_CRYPTO_CIPHER_OP_DECRYPT, tdata); + retval = create_gcm_operation(RTE_CRYPTO_AEAD_OP_DECRYPT, tdata); if (retval < 0) return retval; /* Create GCM xforms */ memcpy(key, tdata->key.data, tdata->key.len); retval = create_gcm_xforms(ut_params->op, - RTE_CRYPTO_CIPHER_OP_DECRYPT, + RTE_CRYPTO_AEAD_OP_DECRYPT, key, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_VERIFY); + tdata->iv.len); if (retval < 0) return retval; @@ -7138,7 +7102,7 @@ test_authenticated_decryption_fail_when_corruption( } static int -create_gcm_operation_SGL(enum rte_crypto_cipher_operation op, +create_gcm_operation_SGL(enum rte_crypto_aead_operation op, const struct gcm_test_data *tdata, void *digest_mem, uint64_t digest_phys) { @@ -7157,18 +7121,18 @@ create_gcm_operation_SGL(enum rte_crypto_cipher_operation op, struct rte_crypto_sym_op *sym_op = ut_params->op->sym; - sym_op->auth.digest.data = digest_mem; + sym_op->aead.digest.data = digest_mem; - TEST_ASSERT_NOT_NULL(sym_op->auth.digest.data, + TEST_ASSERT_NOT_NULL(sym_op->aead.digest.data, "no room to append digest"); - sym_op->auth.digest.phys_addr = digest_phys; + sym_op->aead.digest.phys_addr = digest_phys; - if (op == RTE_CRYPTO_CIPHER_OP_DECRYPT) { - rte_memcpy(sym_op->auth.digest.data, tdata->auth_tag.data, + if (op == RTE_CRYPTO_AEAD_OP_DECRYPT) { + rte_memcpy(sym_op->aead.digest.data, tdata->auth_tag.data, auth_tag_len); TEST_HEXDUMP(stdout, "digest:", - sym_op->auth.digest.data, + sym_op->aead.digest.data, auth_tag_len); } @@ -7177,25 +7141,22 @@ create_gcm_operation_SGL(enum rte_crypto_cipher_operation op, rte_memcpy(IV_ptr, tdata->iv.data, iv_len); - sym_op->auth.aad.data = (uint8_t *)rte_pktmbuf_prepend( + sym_op->aead.aad.data = (uint8_t *)rte_pktmbuf_prepend( ut_params->ibuf, aad_len); - TEST_ASSERT_NOT_NULL(sym_op->auth.aad.data, + TEST_ASSERT_NOT_NULL(sym_op->aead.aad.data, "no room to prepend aad"); - sym_op->auth.aad.phys_addr = rte_pktmbuf_mtophys( + sym_op->aead.aad.phys_addr = rte_pktmbuf_mtophys( ut_params->ibuf); - memset(sym_op->auth.aad.data, 0, aad_len); - rte_memcpy(sym_op->auth.aad.data, tdata->aad.data, aad_len); + memset(sym_op->aead.aad.data, 0, aad_len); + rte_memcpy(sym_op->aead.aad.data, tdata->aad.data, aad_len); TEST_HEXDUMP(stdout, "iv:", IV_ptr, iv_len); TEST_HEXDUMP(stdout, "aad:", - sym_op->auth.aad.data, aad_len); + sym_op->aead.aad.data, aad_len); - sym_op->cipher.data.length = tdata->plaintext.len; - sym_op->cipher.data.offset = aad_len; - - sym_op->auth.data.offset = aad_len; - sym_op->auth.data.length = tdata->plaintext.len; + sym_op->aead.data.length = tdata->plaintext.len; + sym_op->aead.data.offset = aad_len; return 0; } @@ -7250,11 +7211,10 @@ test_AES_GCM_authenticated_encryption_SGL(const struct gcm_test_data *tdata, /* Create GCM session */ retval = create_gcm_session(ts_params->valid_devs[0], - RTE_CRYPTO_CIPHER_OP_ENCRYPT, + RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata->key.data, tdata->key.len, tdata->aad.len, tdata->auth_tag.len, - tdata->iv.len, - RTE_CRYPTO_AUTH_OP_GENERATE); + tdata->iv.len); if (retval < 0) return retval; @@ -7380,7 +7340,7 @@ test_AES_GCM_authenticated_encryption_SGL(const struct gcm_test_data *tdata, } /* Create GCM opertaion */ - retval = create_gcm_operation_SGL(RTE_CRYPTO_CIPHER_OP_ENCRYPT, + retval = create_gcm_operation_SGL(RTE_CRYPTO_AEAD_OP_ENCRYPT, tdata, digest_mem, digest_phys); if (retval < 0) diff --git a/test/test/test_cryptodev_perf.c b/test/test/test_cryptodev_perf.c index 71b03cd..8e9c33a 100644 --- a/test/test/test_cryptodev_perf.c +++ b/test/test/test_cryptodev_perf.c @@ -65,7 +65,8 @@ enum chain_mode { CIPHER_HASH, HASH_CIPHER, CIPHER_ONLY, - HASH_ONLY + HASH_ONLY, + AEAD }; @@ -86,6 +87,7 @@ struct symmetric_op { struct symmetric_session_attrs { enum rte_crypto_cipher_operation cipher; enum rte_crypto_auth_operation auth; + enum rte_crypto_aead_operation aead; enum rte_crypto_cipher_algorithm cipher_algorithm; const uint8_t *key_cipher_data; @@ -95,6 +97,10 @@ struct symmetric_session_attrs { const uint8_t *key_auth_data; uint32_t key_auth_len; + enum rte_crypto_aead_algorithm aead_algorithm; + const uint8_t *key_aead_data; + uint32_t key_aead_len; + const uint8_t *iv_data; uint16_t iv_len; uint16_t aad_len; @@ -124,8 +130,9 @@ struct perf_test_params { enum chain_mode chain; enum rte_crypto_cipher_algorithm cipher_algo; - unsigned cipher_key_length; + unsigned key_length; enum rte_crypto_auth_algorithm auth_algo; + enum rte_crypto_aead_algorithm aead_algo; struct symmetric_session_attrs *session_attrs; @@ -157,7 +164,8 @@ static struct rte_cryptodev_sym_session * test_perf_create_openssl_session(uint8_t dev_id, enum chain_mode chain, enum rte_crypto_cipher_algorithm cipher_algo, unsigned int cipher_key_len, - enum rte_crypto_auth_algorithm auth_algo); + enum rte_crypto_auth_algorithm auth_algo, + enum rte_crypto_aead_algorithm aead_algo); static struct rte_cryptodev_sym_session * test_perf_create_armv8_session(uint8_t dev_id, enum chain_mode chain, enum rte_crypto_cipher_algorithm cipher_algo, @@ -191,6 +199,7 @@ static const char *chain_mode_name(enum chain_mode mode) case HASH_CIPHER: return "hash_cipher"; break; case CIPHER_ONLY: return "cipher_only"; break; case HASH_ONLY: return "hash_only"; break; + case AEAD: return "aead"; break; default: return ""; break; } } @@ -2085,7 +2094,7 @@ test_perf_snow3G_optimise_cyclecount(struct perf_test_params *pparams) /* Create Crypto session*/ sess = test_perf_create_snow3g_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate Crypto op data structure(s)*/ @@ -2106,7 +2115,16 @@ test_perf_snow3G_optimise_cyclecount(struct perf_test_params *pparams) c_ops[i] = op; } - printf("\nOn %s dev%u qp%u, %s, cipher algo:%s, auth_algo:%s, " + if (pparams->chain == AEAD) + printf("\nOn %s dev%u qp%u, %s, aead algo:%s, " + "Packet Size %u bytes", + pmd_name(gbl_cryptodev_perftest_devtype), + ts_params->dev_id, 0, + chain_mode_name(pparams->chain), + rte_crypto_aead_algorithm_strings[pparams->aead_algo], + pparams->buf_size); + else + printf("\nOn %s dev%u qp%u, %s, cipher algo:%s, auth_algo:%s, " "Packet Size %u bytes", pmd_name(gbl_cryptodev_perftest_devtype), ts_params->dev_id, 0, @@ -2196,14 +2214,14 @@ test_perf_snow3G_vary_burst_size(void) { .chain = CIPHER_ONLY, .cipher_algo = RTE_CRYPTO_CIPHER_SNOW3G_UEA2, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_NULL, }, { .chain = HASH_ONLY, .cipher_algo = RTE_CRYPTO_CIPHER_NULL, .auth_algo = RTE_CRYPTO_AUTH_SNOW3G_UIA2, - .cipher_key_length = 16 + .key_length = 16 }, }; @@ -2260,7 +2278,8 @@ test_perf_openssl_optimise_cyclecount(struct perf_test_params *pparams) /* Create Crypto session*/ sess = test_perf_create_openssl_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo, + pparams->aead_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate Crypto op data structure(s)*/ @@ -2275,21 +2294,22 @@ test_perf_openssl_optimise_cyclecount(struct perf_test_params *pparams) RTE_CRYPTO_OP_TYPE_SYMMETRIC); TEST_ASSERT_NOT_NULL(op, "Failed to allocate op"); - switch (pparams->cipher_algo) { - case RTE_CRYPTO_CIPHER_3DES_CBC: - case RTE_CRYPTO_CIPHER_3DES_CTR: - test_perf_set_crypto_op = test_perf_set_crypto_op_3des; - break; - case RTE_CRYPTO_CIPHER_AES_CBC: - case RTE_CRYPTO_CIPHER_AES_CTR: - test_perf_set_crypto_op = test_perf_set_crypto_op_aes; - break; - case RTE_CRYPTO_CIPHER_AES_GCM: + if (pparams->chain == AEAD) test_perf_set_crypto_op = test_perf_set_crypto_op_aes_gcm; - break; - default: - return TEST_FAILED; + else { + switch (pparams->cipher_algo) { + case RTE_CRYPTO_CIPHER_3DES_CBC: + case RTE_CRYPTO_CIPHER_3DES_CTR: + test_perf_set_crypto_op = test_perf_set_crypto_op_3des; + break; + case RTE_CRYPTO_CIPHER_AES_CBC: + case RTE_CRYPTO_CIPHER_AES_CTR: + test_perf_set_crypto_op = test_perf_set_crypto_op_aes; + break; + default: + return TEST_FAILED; + } } op = test_perf_set_crypto_op(op, m, sess, pparams->buf_size, @@ -2299,14 +2319,24 @@ test_perf_openssl_optimise_cyclecount(struct perf_test_params *pparams) c_ops[i] = op; } - printf("\nOn %s dev%u qp%u, %s, cipher algo:%s, cipher key length:%u, " - "auth_algo:%s, Packet Size %u bytes", + if (pparams->chain == AEAD) + printf("\nOn %s dev%u qp%u, %s, aead_algo:%s, " + "key length:%u, Packet Size %u bytes", + pmd_name(gbl_cryptodev_perftest_devtype), + ts_params->dev_id, 0, + chain_mode_name(pparams->chain), + rte_crypto_aead_algorithm_strings[pparams->aead_algo], + pparams->key_length, + pparams->buf_size); + else + printf("\nOn %s dev%u qp%u, %s, cipher algo:%s, auth_algo:%s, " + "key length:%u, Packet Size %u bytes", pmd_name(gbl_cryptodev_perftest_devtype), ts_params->dev_id, 0, chain_mode_name(pparams->chain), rte_crypto_cipher_algorithm_strings[pparams->cipher_algo], - pparams->cipher_key_length, rte_crypto_auth_algorithm_strings[pparams->auth_algo], + pparams->key_length, pparams->buf_size); printf("\nOps Tx\tOps Rx\tOps/burst "); printf("Retries EmptyPolls\tIACycles/CyOp\tIACycles/Burst\t" @@ -2410,7 +2440,7 @@ test_perf_armv8_optimise_cyclecount(struct perf_test_params *pparams) /* Create Crypto session*/ sess = test_perf_create_armv8_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate Crypto op data structure(s)*/ @@ -2438,7 +2468,7 @@ test_perf_armv8_optimise_cyclecount(struct perf_test_params *pparams) ts_params->dev_id, 0, chain_mode_name(pparams->chain), rte_crypto_cipher_algorithm_strings[pparams->cipher_algo], - pparams->cipher_key_length, + pparams->key_length, rte_crypto_auth_algorithm_strings[pparams->auth_algo], pparams->buf_size); printf("\nOps Tx\tOps Rx\tOps/burst "); @@ -2532,8 +2562,6 @@ static uint32_t get_auth_key_max_length(enum rte_crypto_auth_algorithm algo) return 128; case RTE_CRYPTO_AUTH_SHA512_HMAC: return 128; - case RTE_CRYPTO_AUTH_AES_GCM: - return 0; default: return 0; } @@ -2554,7 +2582,15 @@ static uint32_t get_auth_digest_length(enum rte_crypto_auth_algorithm algo) return TRUNCATED_DIGEST_BYTE_LENGTH_SHA384; case RTE_CRYPTO_AUTH_SHA512_HMAC: return TRUNCATED_DIGEST_BYTE_LENGTH_SHA512; - case RTE_CRYPTO_AUTH_AES_GCM: + default: + return 0; + } +} + +static uint32_t get_aead_digest_length(enum rte_crypto_aead_algorithm algo) +{ + switch (algo) { + case RTE_CRYPTO_AEAD_AES_GCM: return DIGEST_BYTE_LENGTH_AES_GCM; default: return 0; @@ -2732,55 +2768,73 @@ test_perf_create_snow3g_session(uint8_t dev_id, enum chain_mode chain, static struct rte_cryptodev_sym_session * test_perf_create_openssl_session(uint8_t dev_id, enum chain_mode chain, enum rte_crypto_cipher_algorithm cipher_algo, - unsigned int cipher_key_len, - enum rte_crypto_auth_algorithm auth_algo) + unsigned int key_len, + enum rte_crypto_auth_algorithm auth_algo, + enum rte_crypto_aead_algorithm aead_algo) { struct rte_crypto_sym_xform cipher_xform = { 0 }; struct rte_crypto_sym_xform auth_xform = { 0 }; + struct rte_crypto_sym_xform aead_xform = { 0 }; - /* Setup Cipher Parameters */ - cipher_xform.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - cipher_xform.cipher.algo = cipher_algo; - cipher_xform.cipher.iv.offset = IV_OFFSET; - cipher_xform.cipher.op = RTE_CRYPTO_CIPHER_OP_ENCRYPT; + if (chain == CIPHER_HASH || chain == HASH_CIPHER) { + /* Setup Cipher Parameters */ + cipher_xform.type = RTE_CRYPTO_SYM_XFORM_CIPHER; + cipher_xform.cipher.algo = cipher_algo; + cipher_xform.cipher.iv.offset = IV_OFFSET; + cipher_xform.cipher.op = RTE_CRYPTO_CIPHER_OP_ENCRYPT; - switch (cipher_algo) { - case RTE_CRYPTO_CIPHER_3DES_CBC: - case RTE_CRYPTO_CIPHER_3DES_CTR: - cipher_xform.cipher.key.data = triple_des_key; - cipher_xform.cipher.iv.length = TRIPLE_DES_CIPHER_IV_LENGTH; - break; - case RTE_CRYPTO_CIPHER_AES_CBC: - case RTE_CRYPTO_CIPHER_AES_CTR: - case RTE_CRYPTO_CIPHER_AES_GCM: - cipher_xform.cipher.key.data = aes_key; - cipher_xform.cipher.iv.length = AES_CIPHER_IV_LENGTH; - break; - default: - return NULL; - } + switch (cipher_algo) { + case RTE_CRYPTO_CIPHER_3DES_CBC: + case RTE_CRYPTO_CIPHER_3DES_CTR: + cipher_xform.cipher.key.data = triple_des_key; + cipher_xform.cipher.iv.length = TRIPLE_DES_CIPHER_IV_LENGTH; + break; + case RTE_CRYPTO_CIPHER_AES_CBC: + case RTE_CRYPTO_CIPHER_AES_CTR: + cipher_xform.cipher.key.data = aes_key; + cipher_xform.cipher.iv.length = AES_CIPHER_IV_LENGTH; + break; + default: + return NULL; + } - cipher_xform.cipher.key.length = cipher_key_len; + cipher_xform.cipher.key.length = key_len; - /* Setup Auth Parameters */ - auth_xform.type = RTE_CRYPTO_SYM_XFORM_AUTH; - auth_xform.auth.op = RTE_CRYPTO_AUTH_OP_GENERATE; - auth_xform.auth.algo = auth_algo; + /* Setup Auth Parameters */ + auth_xform.type = RTE_CRYPTO_SYM_XFORM_AUTH; + auth_xform.auth.op = RTE_CRYPTO_AUTH_OP_GENERATE; + auth_xform.auth.algo = auth_algo; - switch (auth_algo) { - case RTE_CRYPTO_AUTH_SHA1_HMAC: - auth_xform.auth.key.data = hmac_sha_key; - break; - case RTE_CRYPTO_AUTH_AES_GCM: - auth_xform.auth.key.data = NULL; - auth_xform.auth.add_auth_data_length = AES_GCM_AAD_LENGTH; - break; - default: - return NULL; - } + switch (auth_algo) { + case RTE_CRYPTO_AUTH_SHA1_HMAC: + auth_xform.auth.key.data = hmac_sha_key; + break; + default: + return NULL; + } - auth_xform.auth.key.length = get_auth_key_max_length(auth_algo); - auth_xform.auth.digest_length = get_auth_digest_length(auth_algo); + auth_xform.auth.key.length = get_auth_key_max_length(auth_algo); + auth_xform.auth.digest_length = get_auth_digest_length(auth_algo); + } else if (chain == AEAD) { + /* Setup AEAD Parameters */ + aead_xform.type = RTE_CRYPTO_SYM_XFORM_AEAD; + aead_xform.aead.op = RTE_CRYPTO_AEAD_OP_ENCRYPT; + aead_xform.aead.algo = aead_algo; + aead_xform.aead.iv.offset = IV_OFFSET; + + switch (aead_algo) { + case RTE_CRYPTO_AEAD_AES_GCM: + aead_xform.aead.key.data = aes_key; + aead_xform.aead.iv.length = AES_CIPHER_IV_LENGTH; + aead_xform.aead.add_auth_data_length = AES_GCM_AAD_LENGTH; + aead_xform.aead.digest_length = get_aead_digest_length(auth_algo); + break; + default: + return NULL; + } + + aead_xform.aead.key.length = key_len; + } switch (chain) { case CIPHER_HASH: @@ -2793,6 +2847,9 @@ test_perf_create_openssl_session(uint8_t dev_id, enum chain_mode chain, cipher_xform.next = NULL; /* Create Crypto session*/ return rte_cryptodev_sym_session_create(dev_id, &auth_xform); + case AEAD: + /* Create Crypto session*/ + return rte_cryptodev_sym_session_create(dev_id, &aead_xform); default: return NULL; } @@ -2916,22 +2973,19 @@ test_perf_set_crypto_op_aes_gcm(struct rte_crypto_op *op, struct rte_mbuf *m, } /* Authentication Parameters */ - op->sym->auth.digest.data = (uint8_t *)m->buf_addr + + op->sym->aead.digest.data = (uint8_t *)m->buf_addr + (m->data_off + data_len); - op->sym->auth.digest.phys_addr = + op->sym->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset(m, data_len); - op->sym->auth.aad.data = aes_gcm_aad; + op->sym->aead.aad.data = aes_gcm_aad; /* Copy IV at the end of the crypto operation */ rte_memcpy(rte_crypto_op_ctod_offset(op, uint8_t *, IV_OFFSET), aes_iv, AES_CIPHER_IV_LENGTH); /* Data lengths/offsets Parameters */ - op->sym->auth.data.offset = 0; - op->sym->auth.data.length = data_len; - - op->sym->cipher.data.offset = 0; - op->sym->cipher.data.length = data_len; + op->sym->aead.data.offset = 0; + op->sym->aead.data.length = data_len; op->sym->m_src = m; @@ -3102,7 +3156,7 @@ test_perf_aes_sha(uint8_t dev_id, uint16_t queue_id, /* Create Crypto session*/ sess = test_perf_create_aes_sha_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate a burst of crypto operations */ @@ -3235,7 +3289,7 @@ test_perf_snow3g(uint8_t dev_id, uint16_t queue_id, /* Create Crypto session*/ sess = test_perf_create_snow3g_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate a burst of crypto operations */ @@ -3394,20 +3448,22 @@ test_perf_openssl(uint8_t dev_id, uint16_t queue_id, unsigned int, enum chain_mode); - switch (pparams->cipher_algo) { - case RTE_CRYPTO_CIPHER_3DES_CBC: - case RTE_CRYPTO_CIPHER_3DES_CTR: - test_perf_set_crypto_op = test_perf_set_crypto_op_3des; - break; - case RTE_CRYPTO_CIPHER_AES_CBC: - case RTE_CRYPTO_CIPHER_AES_CTR: - test_perf_set_crypto_op = test_perf_set_crypto_op_aes; - break; - case RTE_CRYPTO_CIPHER_AES_GCM: - test_perf_set_crypto_op = test_perf_set_crypto_op_aes_gcm; - break; - default: - return TEST_FAILED; + if (pparams->chain == AEAD) + test_perf_set_crypto_op = + test_perf_set_crypto_op_aes_gcm; + else { + switch (pparams->cipher_algo) { + case RTE_CRYPTO_CIPHER_3DES_CBC: + case RTE_CRYPTO_CIPHER_3DES_CTR: + test_perf_set_crypto_op = test_perf_set_crypto_op_3des; + break; + case RTE_CRYPTO_CIPHER_AES_CBC: + case RTE_CRYPTO_CIPHER_AES_CTR: + test_perf_set_crypto_op = test_perf_set_crypto_op_aes; + break; + default: + return TEST_FAILED; + } } if (rte_cryptodev_count() == 0) { @@ -3418,7 +3474,8 @@ test_perf_openssl(uint8_t dev_id, uint16_t queue_id, /* Create Crypto session*/ sess = test_perf_create_openssl_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo, + pparams->aead_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate a burst of crypto operations */ @@ -3548,7 +3605,7 @@ test_perf_armv8(uint8_t dev_id, uint16_t queue_id, /* Create Crypto session*/ sess = test_perf_create_armv8_session(ts_params->dev_id, pparams->chain, pparams->cipher_algo, - pparams->cipher_key_length, pparams->auth_algo); + pparams->key_length, pparams->auth_algo); TEST_ASSERT_NOT_NULL(sess, "Session creation failed"); /* Generate a burst of crypto operations */ @@ -3674,48 +3731,48 @@ test_perf_aes_cbc_encrypt_digest_vary_pkt_size(void) { .chain = CIPHER_ONLY, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_NULL }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA256_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA512_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 32, + .key_length = 32, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 32, + .key_length = 32, .auth_algo = RTE_CRYPTO_AUTH_SHA256_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 32, + .key_length = 32, .auth_algo = RTE_CRYPTO_AUTH_SHA512_HMAC }, }; @@ -3729,7 +3786,7 @@ test_perf_aes_cbc_encrypt_digest_vary_pkt_size(void) chain_mode_name(params_set[i].chain), rte_crypto_cipher_algorithm_strings[params_set[i].cipher_algo], rte_crypto_auth_algorithm_strings[params_set[i].auth_algo], - params_set[i].cipher_key_length, + params_set[i].key_length, burst_size); printf("\nBuffer Size(B)\tOPS(M)\tThroughput(Gbps)\t" "Retries\tEmptyPolls\n"); @@ -3755,14 +3812,14 @@ test_perf_snow3G_vary_pkt_size(void) { .chain = CIPHER_ONLY, .cipher_algo = RTE_CRYPTO_CIPHER_SNOW3G_UEA2, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_NULL, }, { .chain = HASH_ONLY, .cipher_algo = RTE_CRYPTO_CIPHER_NULL, .auth_algo = RTE_CRYPTO_AUTH_SNOW3G_UIA2, - .cipher_key_length = 16 + .key_length = 16 }, }; @@ -3813,63 +3870,77 @@ test_perf_openssl_vary_pkt_size(void) .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CBC, - .cipher_key_length = 24, + .key_length = 24, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CTR, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CTR, - .cipher_key_length = 32, + .key_length = 32, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CTR, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CTR, - .cipher_key_length = 24, + .key_length = 24, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { - .chain = CIPHER_HASH, + .chain = AEAD, - .cipher_algo = RTE_CRYPTO_CIPHER_AES_GCM, - .cipher_key_length = 16, - .auth_algo = RTE_CRYPTO_AUTH_AES_GCM + .aead_algo = RTE_CRYPTO_AEAD_AES_GCM, + .key_length = 16, }, }; for (i = 0; i < RTE_DIM(params_set); i++) { params_set[i].total_operations = total_operations; params_set[i].burst_size = burst_size; - printf("\n%s. cipher algo: %s auth algo: %s cipher key size=%u." - " burst_size: %d ops\n", - chain_mode_name(params_set[i].chain), - rte_crypto_cipher_algorithm_strings[params_set[i].cipher_algo], - rte_crypto_auth_algorithm_strings[params_set[i].auth_algo], - params_set[i].cipher_key_length, - burst_size); + if (params_set[i].chain == AEAD) { + enum rte_crypto_aead_algorithm aead_algo = + params_set[i].aead_algo; + printf("\n%s. aead algo: %s key size=%u." + " burst_size: %d ops\n", + chain_mode_name(params_set[i].chain), + rte_crypto_aead_algorithm_strings[aead_algo], + params_set[i].key_length, + burst_size); + } else { + enum rte_crypto_cipher_algorithm cipher_algo = + params_set[i].cipher_algo; + enum rte_crypto_auth_algorithm auth_algo = + params_set[i].auth_algo; + printf("\n%s. cipher algo: %s auth algo: %s key size=%u." + " burst_size: %d ops\n", + chain_mode_name(params_set[i].chain), + rte_crypto_cipher_algorithm_strings[cipher_algo], + rte_crypto_auth_algorithm_strings[auth_algo], + params_set[i].key_length, + burst_size); + } printf("\nBuffer Size(B)\tOPS(M)\tThroughput(Gbps)\tRetries\t" "EmptyPolls\n"); for (j = 0; j < RTE_DIM(buf_lengths); j++) { @@ -3894,50 +3965,49 @@ test_perf_openssl_vary_burst_size(void) .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CBC, - .cipher_key_length = 24, + .key_length = 24, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CTR, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CTR, - .cipher_key_length = 32, + .key_length = 32, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CTR, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_3DES_CTR, - .cipher_key_length = 24, + .key_length = 24, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { - .chain = CIPHER_HASH, + .chain = AEAD, - .cipher_algo = RTE_CRYPTO_CIPHER_AES_GCM, - .cipher_key_length = 16, - .auth_algo = RTE_CRYPTO_AUTH_AES_GCM + .aead_algo = RTE_CRYPTO_AEAD_AES_GCM, + .key_length = 16, }, }; @@ -3974,28 +4044,28 @@ test_perf_armv8_vary_pkt_size(void) .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = HASH_CIPHER, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA256_HMAC }, { .chain = HASH_CIPHER, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA256_HMAC }, }; @@ -4008,7 +4078,7 @@ test_perf_armv8_vary_pkt_size(void) chain_mode_name(params_set[i].chain), rte_crypto_cipher_algorithm_strings[params_set[i].cipher_algo], rte_crypto_auth_algorithm_strings[params_set[i].auth_algo], - params_set[i].cipher_key_length, + params_set[i].key_length, burst_size); printf("\nBuffer Size(B)\tOPS(M)\tThroughput(Gbps)\tRetries\t" "EmptyPolls\n"); @@ -4034,28 +4104,28 @@ test_perf_armv8_vary_burst_size(void) .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = HASH_CIPHER, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }, { .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA256_HMAC }, { .chain = HASH_CIPHER, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA256_HMAC }, }; @@ -4090,48 +4160,26 @@ static struct rte_cryptodev_sym_session * test_perf_create_session(uint8_t dev_id, struct perf_test_params *pparams) { static struct rte_cryptodev_sym_session *sess; - struct rte_crypto_sym_xform cipher_xform = { 0 }; - struct rte_crypto_sym_xform auth_xform = { 0 }; + struct rte_crypto_sym_xform aead_xform = { 0 }; - uint8_t cipher_key[pparams->session_attrs->key_cipher_len]; - uint8_t auth_key[pparams->session_attrs->key_auth_len]; + uint8_t aead_key[pparams->session_attrs->key_aead_len]; - memcpy(cipher_key, pparams->session_attrs->key_cipher_data, - pparams->session_attrs->key_cipher_len); - memcpy(auth_key, pparams->session_attrs->key_auth_data, - pparams->session_attrs->key_auth_len); - - cipher_xform.type = RTE_CRYPTO_SYM_XFORM_CIPHER; - cipher_xform.next = NULL; - - cipher_xform.cipher.algo = pparams->session_attrs->cipher_algorithm; - cipher_xform.cipher.op = pparams->session_attrs->cipher; - cipher_xform.cipher.key.data = cipher_key; - cipher_xform.cipher.key.length = pparams->session_attrs->key_cipher_len; - cipher_xform.cipher.iv.length = pparams->session_attrs->iv_len; - cipher_xform.cipher.iv.offset = IV_OFFSET; - - auth_xform.type = RTE_CRYPTO_SYM_XFORM_AUTH; - auth_xform.next = NULL; + memcpy(aead_key, pparams->session_attrs->key_aead_data, + pparams->session_attrs->key_aead_len); - auth_xform.auth.op = pparams->session_attrs->auth; - auth_xform.auth.algo = pparams->session_attrs->auth_algorithm; + aead_xform.type = RTE_CRYPTO_SYM_XFORM_AEAD; + aead_xform.next = NULL; - auth_xform.auth.add_auth_data_length = pparams->session_attrs->aad_len; - auth_xform.auth.digest_length = pparams->session_attrs->digest_len; - auth_xform.auth.key.length = pparams->session_attrs->key_auth_len; + aead_xform.aead.algo = pparams->session_attrs->aead_algorithm; + aead_xform.aead.op = pparams->session_attrs->aead; + aead_xform.aead.key.data = aead_key; + aead_xform.aead.key.length = pparams->session_attrs->key_aead_len; + aead_xform.aead.iv.length = pparams->session_attrs->iv_len; + aead_xform.aead.iv.offset = IV_OFFSET; + aead_xform.aead.add_auth_data_length = pparams->session_attrs->aad_len; + aead_xform.aead.digest_length = pparams->session_attrs->digest_len; - - cipher_xform.cipher.op = RTE_CRYPTO_CIPHER_OP_ENCRYPT; - if (cipher_xform.cipher.op == RTE_CRYPTO_CIPHER_OP_ENCRYPT) { - cipher_xform.next = &auth_xform; - sess = rte_cryptodev_sym_session_create(dev_id, - &cipher_xform); - } else { - auth_xform.next = &cipher_xform; - sess = rte_cryptodev_sym_session_create(dev_id, - &auth_xform); - } + sess = rte_cryptodev_sym_session_create(dev_id, &aead_xform); return sess; } @@ -4150,17 +4198,17 @@ perf_gcm_set_crypto_op(struct rte_crypto_op *op, struct rte_mbuf *m, return NULL; } - op->sym->auth.digest.data = m_hlp->digest; - op->sym->auth.digest.phys_addr = rte_pktmbuf_mtophys_offset( + op->sym->aead.digest.data = m_hlp->digest; + op->sym->aead.digest.phys_addr = rte_pktmbuf_mtophys_offset( m, params->session_attrs->aad_len + params->symmetric_op->p_len); - op->sym->auth.aad.data = m_hlp->aad; - op->sym->auth.aad.phys_addr = rte_pktmbuf_mtophys(m); + op->sym->aead.aad.data = m_hlp->aad; + op->sym->aead.aad.phys_addr = rte_pktmbuf_mtophys(m); - rte_memcpy(op->sym->auth.aad.data, params->symmetric_op->aad_data, + rte_memcpy(op->sym->aead.aad.data, params->symmetric_op->aad_data, params->session_attrs->aad_len); rte_memcpy(IV_ptr, params->session_attrs->iv_data, @@ -4168,13 +4216,9 @@ perf_gcm_set_crypto_op(struct rte_crypto_op *op, struct rte_mbuf *m, if (params->session_attrs->iv_len == 12) IV_ptr[15] = 1; - op->sym->auth.data.offset = - params->session_attrs->aad_len; - op->sym->auth.data.length = params->symmetric_op->p_len; - - op->sym->cipher.data.offset = + op->sym->aead.data.offset = params->session_attrs->aad_len; - op->sym->cipher.data.length = params->symmetric_op->p_len; + op->sym->aead.data.length = params->symmetric_op->p_len; op->sym->m_src = m; @@ -4388,20 +4432,14 @@ test_perf_AES_GCM(int continual_buf_len, int continual_size) gcm_test = gcm_tests[i]; - session_attrs[i].cipher = - RTE_CRYPTO_CIPHER_OP_ENCRYPT; - session_attrs[i].cipher_algorithm = - RTE_CRYPTO_CIPHER_AES_GCM; - session_attrs[i].key_cipher_data = + session_attrs[i].aead = + RTE_CRYPTO_AEAD_OP_ENCRYPT; + session_attrs[i].aead_algorithm = + RTE_CRYPTO_AEAD_AES_GCM; + session_attrs[i].key_aead_data = gcm_test->key.data; - session_attrs[i].key_cipher_len = + session_attrs[i].key_aead_len = gcm_test->key.len; - session_attrs[i].auth_algorithm = - RTE_CRYPTO_AUTH_AES_GCM; - session_attrs[i].auth = - RTE_CRYPTO_AUTH_OP_GENERATE; - session_attrs[i].key_auth_data = NULL; - session_attrs[i].key_auth_len = 0; session_attrs[i].aad_len = gcm_test->aad.len; session_attrs[i].digest_len = gcm_test->auth_tag.len; @@ -4416,7 +4454,7 @@ test_perf_AES_GCM(int continual_buf_len, int continual_size) ops_set[i].t_data = gcm_test->auth_tags[i].data; ops_set[i].t_len = gcm_test->auth_tags[i].len; - params_set[i].chain = CIPHER_HASH; + params_set[i].chain = AEAD; params_set[i].session_attrs = &session_attrs[i]; params_set[i].symmetric_op = &ops_set[i]; if (continual_buf_len) @@ -4518,7 +4556,7 @@ test_perf_continual_performance_test(void) .chain = CIPHER_HASH, .cipher_algo = RTE_CRYPTO_CIPHER_AES_CBC, - .cipher_key_length = 16, + .key_length = 16, .auth_algo = RTE_CRYPTO_AUTH_SHA1_HMAC }; @@ -4528,7 +4566,7 @@ test_perf_continual_performance_test(void) chain_mode_name(params_set.chain), rte_crypto_cipher_algorithm_strings[params_set.cipher_algo], rte_crypto_auth_algorithm_strings[params_set.auth_algo], - params_set.cipher_key_length, + params_set.key_length, burst_size); printf("\nBuffer Size(B)\tOPS(M)\tThroughput(Gbps)\t" "Retries\tEmptyPolls\n");