From patchwork Fri Aug 25 14:57:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 27989 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id C19857D83; Fri, 25 Aug 2017 17:01:24 +0200 (CEST) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 0480C7D22 for ; Fri, 25 Aug 2017 17:01:20 +0200 (CEST) Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga105.jf.intel.com with ESMTP; 25 Aug 2017 08:01:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.41,426,1498546800"; d="scan'208";a="143899022" Received: from silpixa00383879.ir.intel.com (HELO silpixa00383879.ger.corp.intel.com) ([10.237.223.127]) by fmsmga005.fm.intel.com with ESMTP; 25 Aug 2017 08:01:18 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: Radu Nicolau Date: Fri, 25 Aug 2017 15:57:25 +0100 Message-Id: <1503673046-30651-5-git-send-email-radu.nicolau@intel.com> X-Mailer: git-send-email 2.7.5 In-Reply-To: <1503673046-30651-1-git-send-email-radu.nicolau@intel.com> References: <1503673046-30651-1-git-send-email-radu.nicolau@intel.com> Subject: [dpdk-dev] [RFC PATCH 4/5] ixgbe: enable inline ipsec X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Signed-off-by: Radu Nicolau --- config/common_base | 1 + drivers/net/ixgbe/Makefile | 4 +- drivers/net/ixgbe/ixgbe_ethdev.c | 3 + drivers/net/ixgbe/ixgbe_ethdev.h | 10 +- drivers/net/ixgbe/ixgbe_ipsec.c | 617 +++++++++++++++++++++++++++++++++ drivers/net/ixgbe/ixgbe_ipsec.h | 142 ++++++++ drivers/net/ixgbe/ixgbe_rxtx.c | 33 +- drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c | 44 +++ 8 files changed, 850 insertions(+), 4 deletions(-) create mode 100644 drivers/net/ixgbe/ixgbe_ipsec.c create mode 100644 drivers/net/ixgbe/ixgbe_ipsec.h diff --git a/config/common_base b/config/common_base index 5e97a08..2084609 100644 --- a/config/common_base +++ b/config/common_base @@ -179,6 +179,7 @@ CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n CONFIG_RTE_IXGBE_INC_VECTOR=y CONFIG_RTE_LIBRTE_IXGBE_BYPASS=n +CONFIG_RTE_LIBRTE_IXGBE_IPSEC=y # # Compile burst-oriented I40E PMD driver diff --git a/drivers/net/ixgbe/Makefile b/drivers/net/ixgbe/Makefile index 5e57cb3..1180900 100644 --- a/drivers/net/ixgbe/Makefile +++ b/drivers/net/ixgbe/Makefile @@ -118,11 +118,13 @@ SRCS-$(CONFIG_RTE_IXGBE_INC_VECTOR) += ixgbe_rxtx_vec_neon.c else SRCS-$(CONFIG_RTE_IXGBE_INC_VECTOR) += ixgbe_rxtx_vec_sse.c endif - ifeq ($(CONFIG_RTE_LIBRTE_IXGBE_BYPASS),y) SRCS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += ixgbe_bypass.c SRCS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += ixgbe_82599_bypass.c endif +ifeq ($(CONFIG_RTE_LIBRTE_IXGBE_IPSEC),y) +SRCS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += ixgbe_ipsec.c +endif SRCS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += rte_pmd_ixgbe.c SRCS-$(CONFIG_RTE_LIBRTE_IXGBE_PMD) += ixgbe_tm.c diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c index 22171d8..73de5e6 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/ixgbe/ixgbe_ethdev.c @@ -1135,6 +1135,9 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev) PMD_INIT_FUNC_TRACE(); eth_dev->dev_ops = &ixgbe_eth_dev_ops; +#ifdef RTE_LIBRTE_IXGBE_IPSEC + eth_dev->sec_ops = &ixgbe_security_ops; +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ eth_dev->rx_pkt_burst = &ixgbe_recv_pkts; eth_dev->tx_pkt_burst = &ixgbe_xmit_pkts; eth_dev->tx_pkt_prepare = &ixgbe_prep_pkts; diff --git a/drivers/net/ixgbe/ixgbe_ethdev.h b/drivers/net/ixgbe/ixgbe_ethdev.h index caa50c8..d1a84e2 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.h +++ b/drivers/net/ixgbe/ixgbe_ethdev.h @@ -38,6 +38,9 @@ #include "base/ixgbe_dcb_82599.h" #include "base/ixgbe_dcb_82598.h" #include "ixgbe_bypass.h" +#ifdef RTE_LIBRTE_IXGBE_IPSEC +#include "ixgbe_ipsec.h" +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ #include #include #include @@ -529,7 +532,9 @@ struct ixgbe_adapter { struct ixgbe_filter_info filter; struct ixgbe_l2_tn_info l2_tn; struct ixgbe_bw_conf bw_conf; - +#ifdef RTE_LIBRTE_IXGBE_IPSEC + struct ixgbe_ipsec ipsec; +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ bool rx_bulk_alloc_allowed; bool rx_vec_allowed; struct rte_timecounter systime_tc; @@ -586,6 +591,9 @@ struct ixgbe_adapter { #define IXGBE_DEV_PRIVATE_TO_TM_CONF(adapter) \ (&((struct ixgbe_adapter *)adapter)->tm_conf) +#define IXGBE_DEV_PRIVATE_TO_IPSEC(adapter)\ + (&((struct ixgbe_adapter *)adapter)->ipsec) + /* * RX/TX function prototypes */ diff --git a/drivers/net/ixgbe/ixgbe_ipsec.c b/drivers/net/ixgbe/ixgbe_ipsec.c new file mode 100644 index 0000000..d866cd8 --- /dev/null +++ b/drivers/net/ixgbe/ixgbe_ipsec.c @@ -0,0 +1,617 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include +#include +#include +#include +#include + +#include "base/ixgbe_type.h" +#include "base/ixgbe_api.h" +#include "ixgbe_ethdev.h" +#include "ixgbe_ipsec.h" + + +#define IXGBE_WAIT_RW(__reg, __rw) \ +{ \ + IXGBE_WRITE_REG(hw, (__reg), reg); \ + while ((IXGBE_READ_REG(hw, (__reg))) & (__rw)) \ + ; \ +} +#define IXGBE_WAIT_RREAD IXGBE_WAIT_RW(IXGBE_IPSRXIDX, IPSRXIDX_READ) +#define IXGBE_WAIT_RWRITE IXGBE_WAIT_RW(IXGBE_IPSRXIDX, IPSRXIDX_WRITE) +#define IXGBE_WAIT_TREAD IXGBE_WAIT_RW(IXGBE_IPSTXIDX, IPSRXIDX_READ) +#define IXGBE_WAIT_TWRITE IXGBE_WAIT_RW(IXGBE_IPSTXIDX, IPSRXIDX_WRITE) + +#define CMP_IP(a, b) \ + ((a).ipv6[0] == (b).ipv6[0] && (a).ipv6[1] == (b).ipv6[1] && \ + (a).ipv6[2] == (b).ipv6[2] && (a).ipv6[3] == (b).ipv6[3]) + + +static void +ixgbe_crypto_clear_ipsec_tables(struct rte_eth_dev *dev) +{ + struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private); + int i = 0; + + /* clear Rx IP table*/ + for (i = 0; i < IPSEC_MAX_RX_IP_COUNT; i++) { + uint16_t index = i << 3; + uint32_t reg = IPSRXIDX_WRITE | IPSRXIDX_TABLE_IP | index; + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(3), 0); + IXGBE_WAIT_RWRITE; + } + + /* clear Rx SPI and Rx/Tx SA tables*/ + for (i = 0; i < IPSEC_MAX_SA_COUNT; i++) { + uint32_t index = i << 3; + uint32_t reg = IPSRXIDX_WRITE | IPSRXIDX_TABLE_SPI | index; + IXGBE_WRITE_REG(hw, IXGBE_IPSRXSPI, 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPIDX, 0); + IXGBE_WAIT_RWRITE; + reg = IPSRXIDX_WRITE | IPSRXIDX_TABLE_KEY | index; + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(3), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXSALT, 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXMOD, 0); + IXGBE_WAIT_RWRITE; + reg = IPSRXIDX_WRITE | index; + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(3), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXSALT, 0); + IXGBE_WAIT_TWRITE; + } +} + +static int +ixgbe_crypto_enable_ipsec(struct rte_eth_dev *dev) +{ + struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct rte_eth_link link; + uint32_t reg; + + /* Halt the data paths */ + reg = IXGBE_SECTXCTRL_TX_DIS; + IXGBE_WRITE_REG(hw, IXGBE_SECTXCTRL, reg); + reg = IXGBE_SECRXCTRL_RX_DIS; + IXGBE_WRITE_REG(hw, IXGBE_SECRXCTRL, reg); + + /* Wait for Tx path to empty */ + do { + rte_eth_link_get_nowait(dev->data->port_id, &link); + if (link.link_status != ETH_LINK_UP) { + /* Fix for HSD:4426139 + * If the Tx FIFO has data but no link, + * we can't clear the Tx Sec block. So set MAC + * loopback before block clear + */ + reg = IXGBE_READ_REG(hw, IXGBE_MACC); + reg |= IXGBE_MACC_FLU; + IXGBE_WRITE_REG(hw, IXGBE_MACC, reg); + + reg = IXGBE_READ_REG(hw, IXGBE_HLREG0); + reg |= IXGBE_HLREG0_LPBK; + IXGBE_WRITE_REG(hw, IXGBE_HLREG0, reg); + struct timespec time; + time.tv_sec = 0; + time.tv_nsec = 1000000 * 3; + nanosleep(&time, NULL); + } + + reg = IXGBE_READ_REG(hw, IXGBE_SECTXSTAT); + + rte_eth_link_get_nowait(dev->data->port_id, &link); + if (link.link_status != ETH_LINK_UP) { + reg = IXGBE_READ_REG(hw, IXGBE_MACC); + reg &= ~(IXGBE_MACC_FLU); + IXGBE_WRITE_REG(hw, IXGBE_MACC, reg); + + reg = IXGBE_READ_REG(hw, IXGBE_HLREG0); + reg &= ~(IXGBE_HLREG0_LPBK); + IXGBE_WRITE_REG(hw, IXGBE_HLREG0, reg); + } + } while (!(reg & IXGBE_SECTXSTAT_SECTX_RDY)); + + /* Wait for Rx path to empty*/ + do { + reg = IXGBE_READ_REG(hw, IXGBE_SECRXSTAT); + } while (!(reg & IXGBE_SECRXSTAT_SECRX_RDY)); + + /* Set IXGBE_SECTXBUFFAF to 0x15 as required in the datasheet*/ + IXGBE_WRITE_REG(hw, IXGBE_SECTXBUFFAF, 0x15); + + /* IFG needs to be set to 3 when we are using security. Otherwise a Tx + * hang will occur with heavy traffic. + */ + reg = IXGBE_READ_REG(hw, IXGBE_SECTXMINIFG); + reg = (reg & 0xFFFFFFF0) | 0x3; + IXGBE_WRITE_REG(hw, IXGBE_SECTXMINIFG, reg); + + reg = IXGBE_READ_REG(hw, IXGBE_HLREG0); + reg |= IXGBE_HLREG0_TXCRCEN | IXGBE_HLREG0_RXCRCSTRP; + IXGBE_WRITE_REG(hw, IXGBE_HLREG0, reg); + + /* Enable the Tx crypto engine and restart the Tx data path; + * set the STORE_FORWARD bit for IPSec. + */ + IXGBE_WRITE_REG(hw, IXGBE_SECTXCTRL, IXGBE_SECTXCTRL_STORE_FORWARD); + + /* Enable the Rx crypto engine and restart the Rx data path*/ + IXGBE_WRITE_REG(hw, IXGBE_SECRXCTRL, 0); + + /* Test if crypto was enabled */ + reg = IXGBE_READ_REG(hw, IXGBE_SECTXCTRL); + if (reg != IXGBE_SECTXCTRL_STORE_FORWARD) { + PMD_DRV_LOG(ERR, "Error enabling Tx Crypto"); + return -1; + } + reg = IXGBE_READ_REG(hw, IXGBE_SECRXCTRL); + if (reg != 0) { + PMD_DRV_LOG(ERR, "Error enabling Rx Crypto"); + return -1; + } + + ixgbe_crypto_clear_ipsec_tables(dev); + + /* create hash table*/ + { + struct ixgbe_ipsec *internals = IXGBE_DEV_PRIVATE_TO_IPSEC( + dev->data->dev_private); + struct rte_hash_parameters params = { 0 }; + params.entries = IPSEC_MAX_SA_COUNT; + params.key_len = sizeof(uint32_t); + params.hash_func = rte_jhash; + params.hash_func_init_val = 0; + params.socket_id = rte_socket_id(); + params.name = "tx_spi_sai_hash"; + internals->tx_spi_sai_hash = rte_hash_create(¶ms); + } + + return 0; +} + + +static int +ixgbe_crypto_add_sa(struct ixgbe_crypto_session *sess) +{ + struct rte_eth_dev *dev = sess->dev; + struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct ixgbe_ipsec *priv = IXGBE_DEV_PRIVATE_TO_IPSEC( + dev->data->dev_private); + uint32_t reg; + int sa_index = -1; + + if (!(priv->flags & IS_INITIALIZED)) { + if (ixgbe_crypto_enable_ipsec(dev) == 0) + priv->flags |= IS_INITIALIZED; + } + + if (sess->op == IXGBE_OP_AUTHENTICATED_DECRYPTION) { + int i, ip_index = -1; + + /* Find a match in the IP table*/ + for (i = 0; i < IPSEC_MAX_RX_IP_COUNT; i++) { + if (CMP_IP(priv->rx_ip_table[i].ip, + sess->dst_ip)) { + ip_index = i; + break; + } + } + /* If no match, find a free entry in the IP table*/ + if (ip_index < 0) { + for (i = 0; i < IPSEC_MAX_RX_IP_COUNT; i++) { + if (priv->rx_ip_table[i].ref_count == 0) { + ip_index = i; + break; + } + } + } + + /* Fail if no match and no free entries*/ + if (ip_index < 0) { + PMD_DRV_LOG(ERR, "No free entry left " + "in the Rx IP table\n"); + return -1; + } + + /* Find a free entry in the SA table*/ + for (i = 0; i < IPSEC_MAX_SA_COUNT; i++) { + if (priv->rx_sa_table[i].used == 0) { + sa_index = i; + break; + } + } + /* Fail if no free entries*/ + if (sa_index < 0) { + PMD_DRV_LOG(ERR, "No free entry left in " + "the Rx SA table\n"); + return -1; + } + + priv->rx_ip_table[ip_index].ip.ipv6[0] = + rte_cpu_to_be_32(sess->dst_ip.ipv6[0]); + priv->rx_ip_table[ip_index].ip.ipv6[1] = + rte_cpu_to_be_32(sess->dst_ip.ipv6[1]); + priv->rx_ip_table[ip_index].ip.ipv6[2] = + rte_cpu_to_be_32(sess->dst_ip.ipv6[2]); + priv->rx_ip_table[ip_index].ip.ipv6[3] = + rte_cpu_to_be_32(sess->dst_ip.ipv6[3]); + priv->rx_ip_table[ip_index].ref_count++; + + priv->rx_sa_table[sa_index].spi = + rte_cpu_to_be_32(sess->spi); + priv->rx_sa_table[sa_index].ip_index = ip_index; + priv->rx_sa_table[sa_index].key[3] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[0]); + priv->rx_sa_table[sa_index].key[2] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[4]); + priv->rx_sa_table[sa_index].key[1] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[8]); + priv->rx_sa_table[sa_index].key[0] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[12]); + priv->rx_sa_table[sa_index].salt = + rte_cpu_to_be_32(sess->salt); + priv->rx_sa_table[sa_index].mode = IPSRXMOD_VALID; + if (sess->op == IXGBE_OP_AUTHENTICATED_DECRYPTION) + priv->rx_sa_table[sa_index].mode |= + (IPSRXMOD_PROTO | IPSRXMOD_DECRYPT); + if (sess->dst_ip.type == IPv6) + priv->rx_sa_table[sa_index].mode |= IPSRXMOD_IPV6; + priv->rx_sa_table[sa_index].used = 1; + + /* write IP table entry*/ + reg = IPSRXIDX_RX_EN | IPSRXIDX_WRITE + | IPSRXIDX_TABLE_IP | (ip_index << 3); + if (priv->rx_ip_table[ip_index].ip.type == IPv4) { + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(3), + priv->rx_ip_table[ip_index].ip.ipv4); + } else { + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(0), + priv->rx_ip_table[ip_index].ip.ipv6[0]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(1), + priv->rx_ip_table[ip_index].ip.ipv6[1]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(2), + priv->rx_ip_table[ip_index].ip.ipv6[2]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(3), + priv->rx_ip_table[ip_index].ip.ipv6[3]); + } + IXGBE_WAIT_RWRITE; + + /* write SPI table entry*/ + reg = IPSRXIDX_RX_EN | IPSRXIDX_WRITE + | IPSRXIDX_TABLE_SPI | (sa_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXSPI, + priv->rx_sa_table[sa_index].spi); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPIDX, + priv->rx_sa_table[sa_index].ip_index); + IXGBE_WAIT_RWRITE; + + /* write Key table entry*/ + reg = IPSRXIDX_RX_EN | IPSRXIDX_WRITE + | IPSRXIDX_TABLE_KEY | (sa_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(0), + priv->rx_sa_table[sa_index].key[0]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(1), + priv->rx_sa_table[sa_index].key[1]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(2), + priv->rx_sa_table[sa_index].key[2]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(3), + priv->rx_sa_table[sa_index].key[3]); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXSALT, + priv->rx_sa_table[sa_index].salt); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXMOD, + priv->rx_sa_table[sa_index].mode); + IXGBE_WAIT_RWRITE; + + } else { /* sess->dir == RTE_CRYPTO_OUTBOUND */ + int i; + + /* Find a free entry in the SA table*/ + for (i = 0; i < IPSEC_MAX_SA_COUNT; i++) { + if (priv->tx_sa_table[i].used == 0) { + sa_index = i; + break; + } + } + /* Fail if no free entries*/ + if (sa_index < 0) { + PMD_DRV_LOG(ERR, "No free entry left in " + "the Tx SA table\n"); + return -1; + } + + priv->tx_sa_table[sa_index].spi = + rte_cpu_to_be_32(sess->spi); + priv->tx_sa_table[sa_index].key[3] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[0]); + priv->tx_sa_table[sa_index].key[2] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[4]); + priv->tx_sa_table[sa_index].key[1] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[8]); + priv->tx_sa_table[sa_index].key[0] = + rte_cpu_to_be_32(*(uint32_t *)&sess->key[12]); + priv->tx_sa_table[sa_index].salt = + rte_cpu_to_be_32(sess->salt); + + reg = IPSRXIDX_RX_EN | IPSRXIDX_WRITE | (sa_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(0), + priv->tx_sa_table[sa_index].key[0]); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(1), + priv->tx_sa_table[sa_index].key[1]); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(2), + priv->tx_sa_table[sa_index].key[2]); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(3), + priv->tx_sa_table[sa_index].key[3]); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXSALT, + priv->tx_sa_table[sa_index].salt); + IXGBE_WAIT_TWRITE; + + rte_hash_add_key_data(priv->tx_spi_sai_hash, + &priv->tx_sa_table[sa_index].spi, + (void *)(uint64_t)sa_index); + priv->tx_sa_table[i].used = 1; + sess->sa_index = sa_index; + } + + return sa_index; +} + +static int +ixgbe_crypto_remove_sa(struct rte_eth_dev *dev, + struct ixgbe_crypto_session *sess) +{ + struct ixgbe_hw *hw = IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct ixgbe_ipsec *priv = IXGBE_DEV_PRIVATE_TO_IPSEC( + dev->data->dev_private); + uint32_t reg; + int sa_index = -1; + + if (sess->op == IXGBE_OP_AUTHENTICATED_DECRYPTION) { + int i, ip_index = -1; + + /* Find a match in the IP table*/ + for (i = 0; i < IPSEC_MAX_RX_IP_COUNT; i++) { + if (CMP_IP(priv->rx_ip_table[i].ip, sess->dst_ip)) { + ip_index = i; + break; + } + } + + /* Fail if no match*/ + if (ip_index < 0) { + PMD_DRV_LOG(ERR, "Entry not found in the Rx IP table\n"); + return -1; + } + + /* Find a free entry in the SA table*/ + for (i = 0; i < IPSEC_MAX_SA_COUNT; i++) { + if (priv->rx_sa_table[i].spi == + rte_cpu_to_be_32(sess->spi)) { + sa_index = i; + break; + } + } + /* Fail if no match*/ + if (sa_index < 0) { + PMD_DRV_LOG(ERR, "Entry not found in the Rx SA table\n"); + return -1; + } + + /* Disable and clear Rx SPI and key table table entryes*/ + reg = IPSRXIDX_WRITE | IPSRXIDX_TABLE_SPI | (sa_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXSPI, 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPIDX, 0); + IXGBE_WAIT_RWRITE; + reg = IPSRXIDX_WRITE | IPSRXIDX_TABLE_KEY | (sa_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXKEY(3), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXSALT, 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXMOD, 0); + IXGBE_WAIT_RWRITE; + priv->rx_sa_table[sa_index].used = 0; + + /* If last used then clear the IP table entry*/ + priv->rx_ip_table[ip_index].ref_count--; + if (priv->rx_ip_table[ip_index].ref_count == 0) { + reg = IPSRXIDX_WRITE | IPSRXIDX_TABLE_IP + | (ip_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSRXIPADDR(3), 0); + } + } else { /* sess->dir == RTE_CRYPTO_OUTBOUND */ + int i; + + /* Find a match in the SA table*/ + for (i = 0; i < IPSEC_MAX_SA_COUNT; i++) { + if (priv->tx_sa_table[i].spi == + rte_cpu_to_be_32(sess->spi)) { + sa_index = i; + break; + } + } + /* Fail if no match entries*/ + if (sa_index < 0) { + PMD_DRV_LOG(ERR, "Entry not found in the " + "Tx SA table\n"); + return -1; + } + reg = IPSRXIDX_WRITE | (sa_index << 3); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(0), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(1), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(2), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXKEY(3), 0); + IXGBE_WRITE_REG(hw, IXGBE_IPSTXSALT, 0); + IXGBE_WAIT_TWRITE; + + priv->tx_sa_table[sa_index].used = 0; + rte_hash_del_key(priv->tx_spi_sai_hash, + &priv->tx_sa_table[sa_index].spi); + } + + return 0; +} + +static int +ixgbe_crypto_create_session(void *dev, + struct rte_security_sess_conf *sess_conf, + struct rte_security_session *sess, + struct rte_mempool *mempool) +{ + struct ixgbe_crypto_session *session = NULL; + struct rte_security_ipsec_xform *ipsec_xform = sess_conf->ipsec_xform; + + if (rte_mempool_get(mempool, (void **)&session)) { + PMD_DRV_LOG(ERR, "Cannot get object from session mempool"); + return -ENOMEM; + } + if (ipsec_xform->aead_alg != RTE_CRYPTO_AEAD_AES_GCM) { + PMD_DRV_LOG(ERR, "Unsupported IPsec mode\n"); + return -ENOTSUP; + } + + session->op = (ipsec_xform->op == RTE_SECURITY_IPSEC_OP_DECAP) ? + IXGBE_OP_AUTHENTICATED_DECRYPTION : + IXGBE_OP_AUTHENTICATED_ENCRYPTION; + session->key = ipsec_xform->aead_key.data; + memcpy(&session->salt, + &ipsec_xform->aead_key.data[ipsec_xform->aead_key.length], 4); + session->spi = ipsec_xform->spi; + + if (ipsec_xform->tunnel.type == RTE_SECURITY_IPSEC_TUNNEL_IPV4) { + uint32_t sip = ipsec_xform->tunnel.ipv4.src_ip.s_addr; + uint32_t dip = ipsec_xform->tunnel.ipv4.dst_ip.s_addr; + session->src_ip.type = IPv4; + session->dst_ip.type = IPv4; + session->src_ip.ipv4 = rte_cpu_to_be_32(sip); + session->dst_ip.ipv4 = rte_cpu_to_be_32(dip); + + } else { + uint32_t *sip = (uint32_t *)&ipsec_xform->tunnel.ipv6.src_addr; + uint32_t *dip = (uint32_t *)&ipsec_xform->tunnel.ipv6.dst_addr; + session->src_ip.type = IPv6; + session->dst_ip.type = IPv6; + session->src_ip.ipv6[0] = rte_cpu_to_be_32(sip[0]); + session->src_ip.ipv6[1] = rte_cpu_to_be_32(sip[1]); + session->src_ip.ipv6[2] = rte_cpu_to_be_32(sip[2]); + session->src_ip.ipv6[3] = rte_cpu_to_be_32(sip[3]); + session->dst_ip.ipv6[0] = rte_cpu_to_be_32(dip[0]); + session->dst_ip.ipv6[1] = rte_cpu_to_be_32(dip[1]); + session->dst_ip.ipv6[2] = rte_cpu_to_be_32(dip[2]); + session->dst_ip.ipv6[3] = rte_cpu_to_be_32(dip[3]); + } + + session->dev = (struct rte_eth_dev *)dev; + set_sec_session_private_data(sess, 0, session); + + if (ixgbe_crypto_add_sa(session)) { + PMD_DRV_LOG(ERR, "Failed to add SA\n"); + return -EPERM; + } + + return 0; +} + +static void +ixgbe_crypto_remove_session(void *dev, + struct rte_security_session *session) +{ + struct ixgbe_crypto_session *sess = + (struct ixgbe_crypto_session *) + get_sec_session_private_data(session, 0); + if (dev != sess->dev) { + PMD_DRV_LOG(ERR, "Session not bound to this device\n"); + return; + } + + if (ixgbe_crypto_remove_sa(dev, sess)) { + PMD_DRV_LOG(ERR, "Failed to remove session\n"); + return; + } + + rte_free(session); +} + +uint64_t +ixgbe_crypto_get_txdesc_flags(uint16_t port_id, struct rte_mbuf *mb) { + struct rte_eth_dev *dev = &rte_eth_devices[port_id]; + struct ixgbe_ipsec *priv = + IXGBE_DEV_PRIVATE_TO_IPSEC(dev->data->dev_private); + struct ipv4_hdr *ip4 = + rte_pktmbuf_mtod_offset(mb, struct ipv4_hdr*, + sizeof(struct ether_hdr)); + uint32_t spi = 0; + uintptr_t sa_index; + struct ixgbe_crypto_tx_desc_metadata mdata = {0}; + + if (ip4->version_ihl == 0x45) + spi = *rte_pktmbuf_mtod_offset(mb, uint32_t*, + sizeof(struct ether_hdr) + + sizeof(struct ipv4_hdr)); + else + spi = *rte_pktmbuf_mtod_offset(mb, uint32_t*, + sizeof(struct ether_hdr) + + sizeof(struct ipv6_hdr)); + + if (priv->tx_spi_sai_hash && + rte_hash_lookup_data(priv->tx_spi_sai_hash, &spi, + (void **)&sa_index) == 0) { + mdata.enc = 1; + mdata.sa_idx = (uint32_t)sa_index; + mdata.pad_len = *rte_pktmbuf_mtod_offset(mb, uint8_t *, + rte_pktmbuf_pkt_len(mb) - 18); + } + + return mdata.data; +} + + +struct rte_security_ops ixgbe_security_ops = { + .session_configure = ixgbe_crypto_create_session, + .session_clear = ixgbe_crypto_remove_session, +}; diff --git a/drivers/net/ixgbe/ixgbe_ipsec.h b/drivers/net/ixgbe/ixgbe_ipsec.h new file mode 100644 index 0000000..fd479eb --- /dev/null +++ b/drivers/net/ixgbe/ixgbe_ipsec.h @@ -0,0 +1,142 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2010-2017 Intel Corporation. All rights reserved. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef IXGBE_IPSEC_H_ +#define IXGBE_IPSEC_H_ + +#include + +#define IPSRXIDX_RX_EN 0x00000001 +#define IPSRXIDX_TABLE_IP 0x00000002 +#define IPSRXIDX_TABLE_SPI 0x00000004 +#define IPSRXIDX_TABLE_KEY 0x00000006 +#define IPSRXIDX_WRITE 0x80000000 +#define IPSRXIDX_READ 0x40000000 +#define IPSRXMOD_VALID 0x00000001 +#define IPSRXMOD_PROTO 0x00000004 +#define IPSRXMOD_DECRYPT 0x00000008 +#define IPSRXMOD_IPV6 0x00000010 +#define IXGBE_ADVTXD_POPTS_IPSEC 0x00000400 +#define IXGBE_ADVTXD_TUCMD_IPSEC_TYPE_ESP 0x00002000 +#define IXGBE_ADVTXD_TUCMD_IPSEC_ENCRYPT_EN 0x00004000 +#define IXGBE_RXDADV_IPSEC_STATUS_SECP 0x00020000 +#define IXGBE_RXDADV_IPSEC_ERROR_BIT_MASK 0x18000000 +#define IXGBE_RXDADV_IPSEC_ERROR_INVALID_PROTOCOL 0x08000000 +#define IXGBE_RXDADV_IPSEC_ERROR_INVALID_LENGTH 0x10000000 +#define IXGBE_RXDADV_IPSEC_ERROR_AUTHENTICATION_FAILED 0x18000000 + +#define IPSEC_MAX_RX_IP_COUNT 128 +#define IPSEC_MAX_SA_COUNT 1024 + +enum ixgbe_operation { + IXGBE_OP_AUTHENTICATED_ENCRYPTION, IXGBE_OP_AUTHENTICATED_DECRYPTION +}; + +enum ixgbe_gcm_key { + IXGBE_GCM_KEY_128, IXGBE_GCM_KEY_256 +}; + +/** + * Generic IP address structure + * TODO: Find better location for this rte_net.h possibly. + **/ +struct ipaddr { + enum ipaddr_type { + IPv4, IPv6 + } type; + /**< IP Address Type - IPv4/IPv6 */ + + union { + uint32_t ipv4; + uint32_t ipv6[4]; + }; +}; + +/** inline crypto crypto private session structure */ +struct ixgbe_crypto_session { + enum ixgbe_operation op; + uint8_t *key; + uint32_t salt; + uint32_t sa_index; + uint32_t spi; + struct ipaddr src_ip; + struct ipaddr dst_ip; + struct rte_eth_dev *dev; +} __rte_cache_aligned; + +struct ixgbe_crypto_rx_ip_table { + struct ipaddr ip; + uint16_t ref_count; +}; +struct ixgbe_crypto_rx_sa_table { + uint32_t spi; + uint32_t ip_index; + uint32_t key[4]; + uint32_t salt; + uint8_t mode; + uint8_t used; +}; + +struct ixgbe_crypto_tx_sa_table { + uint32_t spi; + uint32_t key[4]; + uint32_t salt; + uint8_t used; +}; + +struct ixgbe_crypto_tx_desc_metadata { + union { + uint64_t data; + struct { + uint32_t sa_idx; + uint8_t pad_len; + uint8_t enc; + }; + }; +}; + +struct ixgbe_ipsec { +#define IS_INITIALIZED (1 << 0) + uint8_t flags; + struct ixgbe_crypto_rx_ip_table rx_ip_table[IPSEC_MAX_RX_IP_COUNT]; + struct ixgbe_crypto_rx_sa_table rx_sa_table[IPSEC_MAX_SA_COUNT]; + struct ixgbe_crypto_tx_sa_table tx_sa_table[IPSEC_MAX_SA_COUNT]; + struct rte_hash *tx_spi_sai_hash; +}; + +extern struct rte_security_ops ixgbe_security_ops; + +uint64_t ixgbe_crypto_get_txdesc_flags(uint16_t port_id, struct rte_mbuf *mb); + + +#endif /*IXGBE_IPSEC_H_*/ diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c index 64bff25..76be27a 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/ixgbe/ixgbe_rxtx.c @@ -395,7 +395,8 @@ ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, static inline void ixgbe_set_xmit_ctx(struct ixgbe_tx_queue *txq, volatile struct ixgbe_adv_tx_context_desc *ctx_txd, - uint64_t ol_flags, union ixgbe_tx_offload tx_offload) + uint64_t ol_flags, union ixgbe_tx_offload tx_offload, + __rte_unused struct rte_mbuf *mb) { uint32_t type_tucmd_mlhl; uint32_t mss_l4len_idx = 0; @@ -479,6 +480,20 @@ ixgbe_set_xmit_ctx(struct ixgbe_tx_queue *txq, seqnum_seed |= tx_offload.l2_len << IXGBE_ADVTXD_TUNNEL_LEN; } +#ifdef RTE_LIBRTE_IXGBE_IPSEC + if (mb->ol_flags & PKT_TX_SECURITY_OFFLOAD) { + struct ixgbe_crypto_tx_desc_metadata mdata = { + .data = ixgbe_crypto_get_txdesc_flags(txq->port_id, mb), + }; + seqnum_seed |= + (IXGBE_ADVTXD_IPSEC_SA_INDEX_MASK & mdata.sa_idx); + type_tucmd_mlhl |= mdata.enc ? + (IXGBE_ADVTXD_TUCMD_IPSEC_TYPE_ESP | + IXGBE_ADVTXD_TUCMD_IPSEC_ENCRYPT_EN) : 0; + type_tucmd_mlhl |= + (mdata.pad_len & IXGBE_ADVTXD_IPSEC_ESP_LEN_MASK); + } +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ txq->ctx_cache[ctx_idx].flags = ol_flags; txq->ctx_cache[ctx_idx].tx_offload.data[0] = @@ -855,7 +870,7 @@ ixgbe_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, } ixgbe_set_xmit_ctx(txq, ctx_txd, tx_ol_req, - tx_offload); + tx_offload, tx_pkt); txe->last_id = tx_last; tx_id = txe->next_id; @@ -872,7 +887,13 @@ ixgbe_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, olinfo_status |= ctx << IXGBE_ADVTXD_IDX_SHIFT; } +#ifdef RTE_LIBRTE_IXGBE_IPSEC + olinfo_status |= ((pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT) | + (((ol_flags & PKT_TX_SECURITY_OFFLOAD) != 0) + * IXGBE_ADVTXD_POPTS_IPSEC)); +#else /* RTE_LIBRTE_IXGBE_IPSEC */ olinfo_status |= (pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT); +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ m_seg = tx_pkt; do { @@ -1447,6 +1468,14 @@ rx_desc_error_to_pkt_flags(uint32_t rx_status) pkt_flags |= PKT_RX_EIP_CKSUM_BAD; } +#ifdef RTE_LIBRTE_IXGBE_IPSEC + if (rx_status & IXGBE_RXD_STAT_SECP) { + pkt_flags |= PKT_RX_SECURITY_OFFLOAD; + if (rx_status & IXGBE_RXDADV_LNKSEC_ERROR_BAD_SIG) + pkt_flags |= PKT_RX_SECURITY_OFFLOAD_FAILED; + } +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ + return pkt_flags; } diff --git a/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c index e704a7f..8673a01 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c @@ -128,6 +128,9 @@ desc_to_olflags_v(__m128i descs[4], __m128i mbuf_init, uint8_t vlan_flags, { __m128i ptype0, ptype1, vtag0, vtag1, csum; __m128i rearm0, rearm1, rearm2, rearm3; +#ifdef RTE_LIBRTE_IXGBE_IPSEC + __m128i sterr0, sterr1, sterr2, sterr3, tmp1, tmp2; +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ /* mask everything except rss type */ const __m128i rsstype_msk = _mm_set_epi16( @@ -174,6 +177,19 @@ desc_to_olflags_v(__m128i descs[4], __m128i mbuf_init, uint8_t vlan_flags, 0, PKT_RX_L4_CKSUM_GOOD >> sizeof(uint8_t), 0, PKT_RX_L4_CKSUM_GOOD >> sizeof(uint8_t)); +#ifdef RTE_LIBRTE_IXGBE_IPSEC + const __m128i ipsec_sterr_msk = _mm_set_epi32( + 0, IXGBE_RXD_STAT_SECP | IXGBE_RXDADV_LNKSEC_ERROR_BAD_SIG, + 0, 0); + const __m128i ipsec_proc_msk = _mm_set_epi32( + 0, IXGBE_RXD_STAT_SECP, 0, 0); + const __m128i ipsec_err_flag = _mm_set_epi32( + 0, PKT_RX_SECURITY_OFFLOAD_FAILED | PKT_RX_SECURITY_OFFLOAD, + 0, 0); + const __m128i ipsec_proc_flag = _mm_set_epi32( + 0, PKT_RX_SECURITY_OFFLOAD, 0, 0); +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ + ptype0 = _mm_unpacklo_epi16(descs[0], descs[1]); ptype1 = _mm_unpacklo_epi16(descs[2], descs[3]); vtag0 = _mm_unpackhi_epi16(descs[0], descs[1]); @@ -221,6 +237,34 @@ desc_to_olflags_v(__m128i descs[4], __m128i mbuf_init, uint8_t vlan_flags, rearm2 = _mm_blend_epi16(mbuf_init, _mm_slli_si128(vtag1, 4), 0x10); rearm3 = _mm_blend_epi16(mbuf_init, _mm_slli_si128(vtag1, 2), 0x10); +#ifdef RTE_LIBRTE_IXGBE_IPSEC + /*inline ipsec, extract the flags from the descriptors*/ + sterr0 = _mm_and_si128(descs[0], ipsec_sterr_msk); + sterr1 = _mm_and_si128(descs[1], ipsec_sterr_msk); + sterr2 = _mm_and_si128(descs[2], ipsec_sterr_msk); + sterr3 = _mm_and_si128(descs[3], ipsec_sterr_msk); + tmp1 = _mm_cmpeq_epi32(sterr0, ipsec_sterr_msk); + tmp2 = _mm_cmpeq_epi32(sterr0, ipsec_proc_msk); + sterr0 = _mm_or_si128(_mm_and_si128(tmp1, ipsec_err_flag), + _mm_and_si128(tmp2, ipsec_proc_flag)); + tmp1 = _mm_cmpeq_epi32(sterr1, ipsec_sterr_msk); + tmp2 = _mm_cmpeq_epi32(sterr1, ipsec_proc_msk); + sterr1 = _mm_or_si128(_mm_and_si128(tmp1, ipsec_err_flag), + _mm_and_si128(tmp2, ipsec_proc_flag)); + tmp1 = _mm_cmpeq_epi32(sterr2, ipsec_sterr_msk); + tmp2 = _mm_cmpeq_epi32(sterr2, ipsec_proc_msk); + sterr2 = _mm_or_si128(_mm_and_si128(tmp1, ipsec_err_flag), + _mm_and_si128(tmp2, ipsec_proc_flag)); + tmp1 = _mm_cmpeq_epi32(sterr3, ipsec_sterr_msk); + tmp2 = _mm_cmpeq_epi32(sterr3, ipsec_proc_msk); + sterr3 = _mm_or_si128(_mm_and_si128(tmp1, ipsec_err_flag), + _mm_and_si128(tmp2, ipsec_proc_flag)); + rearm0 = _mm_or_si128(rearm0, sterr0); + rearm1 = _mm_or_si128(rearm1, sterr1); + rearm2 = _mm_or_si128(rearm2, sterr2); + rearm3 = _mm_or_si128(rearm3, sterr3); +#endif /* RTE_LIBRTE_IXGBE_IPSEC */ + /* write the rearm data and the olflags in one write */ RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, ol_flags) != offsetof(struct rte_mbuf, rearm_data) + 8);