From patchwork Wed Apr 20 08:16:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 109907 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C721FA00BE; Wed, 20 Apr 2022 10:17:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 682DA410FB; Wed, 20 Apr 2022 10:17:02 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 88070410FB for ; Wed, 20 Apr 2022 10:17:01 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F2C451FB; Wed, 20 Apr 2022 01:17:00 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.114]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B5CC13F73B; Wed, 20 Apr 2022 01:16:58 -0700 (PDT) From: Feifei Wang To: Beilei Xing Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v1 1/5] net/i40e: remove redundant Dtype initialization Date: Wed, 20 Apr 2022 16:16:46 +0800 Message-Id: <20220420081650.2043183-2-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220420081650.2043183-1-feifei.wang2@arm.com> References: <20220420081650.2043183-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Dtype field is set to 0xf by the NIC to indicate DMA completion, only after the CPU requests to be informed by setting the RS bit. Hence, it is not required to set Dtype to 0xf during initialization. Not setting the Dtype field to 0xf helps to know that a given descriptor is not sent to the NIC yet after initialization. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli --- drivers/net/i40e/i40e_rxtx.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index 25a28ecea2..745734d5e4 100644 --- a/drivers/net/i40e/i40e_rxtx.c +++ b/drivers/net/i40e/i40e_rxtx.c @@ -2767,10 +2767,6 @@ i40e_reset_tx_queue(struct i40e_tx_queue *txq) prev = (uint16_t)(txq->nb_tx_desc - 1); for (i = 0; i < txq->nb_tx_desc; i++) { - volatile struct i40e_tx_desc *txd = &txq->tx_ring[i]; - - txd->cmd_type_offset_bsz = - rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE); txe[i].mbuf = NULL; txe[i].last_id = i; txe[prev].next_id = i; From patchwork Wed Apr 20 08:16:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 109908 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C1D2DA00BE; Wed, 20 Apr 2022 10:17:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5760C427E9; Wed, 20 Apr 2022 10:17:06 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id E9DE440687 for ; Wed, 20 Apr 2022 10:17:04 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 63AC61FB; Wed, 20 Apr 2022 01:17:04 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.114]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9ED413F73B; Wed, 20 Apr 2022 01:17:01 -0700 (PDT) From: Feifei Wang To: Beilei Xing , Bruce Richardson , Konstantin Ananyev , Ruifeng Wang Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli Subject: [PATCH v1 2/5] net/i40e: enable direct rearm mode Date: Wed, 20 Apr 2022 16:16:47 +0800 Message-Id: <20220420081650.2043183-3-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220420081650.2043183-1-feifei.wang2@arm.com> References: <20220420081650.2043183-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For i40e driver, enable direct re-arm mode. This patch supports the case of mapping Rx/Tx queues from the same single lcore. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli --- drivers/net/i40e/i40e_rxtx.h | 4 + drivers/net/i40e/i40e_rxtx_common_avx.h | 269 ++++++++++++++++++++++++ drivers/net/i40e/i40e_rxtx_vec_avx2.c | 14 +- drivers/net/i40e/i40e_rxtx_vec_avx512.c | 249 +++++++++++++++++++++- drivers/net/i40e/i40e_rxtx_vec_neon.c | 141 ++++++++++++- drivers/net/i40e/i40e_rxtx_vec_sse.c | 170 ++++++++++++++- 6 files changed, 839 insertions(+), 8 deletions(-) diff --git a/drivers/net/i40e/i40e_rxtx.h b/drivers/net/i40e/i40e_rxtx.h index 5e6eecc501..1fdf4305f4 100644 --- a/drivers/net/i40e/i40e_rxtx.h +++ b/drivers/net/i40e/i40e_rxtx.h @@ -102,6 +102,8 @@ struct i40e_rx_queue { uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ uint16_t rxrearm_start; /**< the idx we start the re-arming from */ + uint16_t direct_rxrearm_port; /** device TX port ID for direct re-arm mode */ + uint16_t direct_rxrearm_queue; /** TX queue index for direct re-arm mode */ uint64_t mbuf_initializer; /**< value to init mbufs */ uint16_t port_id; /**< device port ID */ @@ -121,6 +123,8 @@ struct i40e_rx_queue { uint16_t rx_using_sse; /**qrx_tail, rx_id); } + +static __rte_always_inline void +i40e_rxq_direct_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) +{ + struct rte_eth_dev *dev; + struct i40e_tx_queue *txq; + volatile union i40e_rx_desc *rxdp; + struct i40e_tx_entry *txep; + struct i40e_rx_entry *rxep; + struct rte_mbuf *m[RTE_I40E_RXQ_REARM_THRESH]; + uint16_t tx_port_id, tx_queue_id; + uint16_t rx_id; + uint16_t i, n; + uint16_t nb_rearm = 0; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + tx_port_id = rxq->direct_rxrearm_port; + tx_queue_id = rxq->direct_rxrearm_queue; + dev = &rte_eth_devices[tx_port_id]; + txq = dev->data->tx_queues[tx_queue_id]; + + /* check Rx queue is able to take in the whole + * batch of free mbufs from Tx queue + */ + if (rxq->rxrearm_nb > txq->tx_rs_thresh) { + /* check DD bits on threshold descriptor */ + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & + rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != + rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) { + goto mempool_bulk; + } + + if (txq->tx_rs_thresh != RTE_I40E_RXQ_REARM_THRESH) + goto mempool_bulk; + + n = txq->tx_rs_thresh; + + /* first buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1) + */ + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; + + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* directly put mbufs from Tx to Rx, + * and initialize the mbufs in vector + */ + for (i = 0; i < n; i++) + rxep[i].mbuf = txep[i].mbuf; + } else { + for (i = 0; i < n; i++) { + m[i] = rte_pktmbuf_prefree_seg(txep[i].mbuf); + /* ensure each Tx freed buffer is valid */ + if (m[i] != NULL) + nb_rearm++; + } + + if (nb_rearm != n) { + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + + goto mempool_bulk; + } else { + for (i = 0; i < n; i++) + rxep[i].mbuf = m[i]; + } + } + + /* update counters for Tx */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + } else { +mempool_bulk: + /* if TX did not free bufs into Rx sw-ring, + * get new bufs from mempool + */ + n = RTE_I40E_RXQ_REARM_THRESH; + + /* Pull 'n' more MBUFs into the software ring */ + if (rte_mempool_get_bulk(rxq->mp, + (void *)rxep, + RTE_I40E_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + rxq->nb_rx_desc) { + __m128i dma_addr0; + dma_addr0 = _mm_setzero_si128(); + for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + rxep[i].mbuf = &rxq->fake_mbuf; + _mm_store_si128((__m128i *)&rxdp[i].read, + dma_addr0); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += + RTE_I40E_RXQ_REARM_THRESH; + return; + } + } + +#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC + struct rte_mbuf *mb0, *mb1; + __m128i dma_addr0, dma_addr1; + __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, + RTE_PKTMBUF_HEADROOM); + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < n; i += 2, rxep += 2) { + __m128i vaddr0, vaddr1; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + + /* convert pa to dma_addr hdr/data */ + dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); + dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); + + /* add headroom to pa values */ + dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); + dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); + + /* flush desc with pa dma_addr */ + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr0); + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr1); + } +#else +#ifdef __AVX512VL__ + if (avx512) { + struct rte_mbuf *mb0, *mb1, *mb2, *mb3; + struct rte_mbuf *mb4, *mb5, *mb6, *mb7; + __m512i dma_addr0_3, dma_addr4_7; + __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); + /* Initialize the mbufs in vector, process 8 mbufs in one loop */ + for (i = 0; i < n; i += 8, rxep += 8, rxdp += 8) { + __m128i vaddr0, vaddr1, vaddr2, vaddr3; + __m128i vaddr4, vaddr5, vaddr6, vaddr7; + __m256i vaddr0_1, vaddr2_3; + __m256i vaddr4_5, vaddr6_7; + __m512i vaddr0_3, vaddr4_7; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + mb2 = rxep[2].mbuf; + mb3 = rxep[3].mbuf; + mb4 = rxep[4].mbuf; + mb5 = rxep[5].mbuf; + mb6 = rxep[6].mbuf; + mb7 = rxep[7].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); + vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); + vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); + vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); + vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 + * into the high lanes. Similarly for 2 & 3, and so on. + */ + vaddr0_1 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), + vaddr1, 1); + vaddr2_3 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), + vaddr3, 1); + vaddr4_5 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr4), + vaddr5, 1); + vaddr6_7 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr6), + vaddr7, 1); + vaddr0_3 = + _mm512_inserti64x4(_mm512_castsi256_si512(vaddr0_1), + vaddr2_3, 1); + vaddr4_7 = + _mm512_inserti64x4(_mm512_castsi256_si512(vaddr4_5), + vaddr6_7, 1); + + /* convert pa to dma_addr hdr/data */ + dma_addr0_3 = _mm512_unpackhi_epi64(vaddr0_3, vaddr0_3); + dma_addr4_7 = _mm512_unpackhi_epi64(vaddr4_7, vaddr4_7); + + /* add headroom to pa values */ + dma_addr0_3 = _mm512_add_epi64(dma_addr0_3, hdr_room); + dma_addr4_7 = _mm512_add_epi64(dma_addr4_7, hdr_room); + + /* flush desc with pa dma_addr */ + _mm512_store_si512((__m512i *)&rxdp->read, dma_addr0_3); + _mm512_store_si512((__m512i *)&(rxdp + 4)->read, dma_addr4_7); + } + } else { +#endif /* __AVX512VL__*/ + struct rte_mbuf *mb0, *mb1, *mb2, *mb3; + __m256i dma_addr0_1, dma_addr2_3; + __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); + /* Initialize the mbufs in vector, process 4 mbufs in one loop */ + for (i = 0; i < n; i += 4, rxep += 4, rxdp += 4) { + __m128i vaddr0, vaddr1, vaddr2, vaddr3; + __m256i vaddr0_1, vaddr2_3; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + mb2 = rxep[2].mbuf; + mb3 = rxep[3].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); + vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 + * into the high lanes. Similarly for 2 & 3 + */ + vaddr0_1 = _mm256_inserti128_si256 + (_mm256_castsi128_si256(vaddr0), vaddr1, 1); + vaddr2_3 = _mm256_inserti128_si256 + (_mm256_castsi128_si256(vaddr2), vaddr3, 1); + + /* convert pa to dma_addr hdr/data */ + dma_addr0_1 = _mm256_unpackhi_epi64(vaddr0_1, vaddr0_1); + dma_addr2_3 = _mm256_unpackhi_epi64(vaddr2_3, vaddr2_3); + + /* add headroom to pa values */ + dma_addr0_1 = _mm256_add_epi64(dma_addr0_1, hdr_room); + dma_addr2_3 = _mm256_add_epi64(dma_addr2_3, hdr_room); + + /* flush desc with pa dma_addr */ + _mm256_store_si256((__m256i *)&rxdp->read, dma_addr0_1); + _mm256_store_si256((__m256i *)&(rxdp + 2)->read, dma_addr2_3); + } + } + +#endif + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += n; + rx_id = rxq->rxrearm_start - 1; + + if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { + rxq->rxrearm_start = rxq->rxrearm_start - rxq->nb_rx_desc; + if (!rxq->rxrearm_start) + rx_id = rxq->nb_rx_desc - 1; + else + rx_id = rxq->rxrearm_start - 1; + } + + rxq->rxrearm_nb -= n; + + /* Update the tail pointer on the NIC */ + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); +} #endif /* __AVX2__*/ #endif /*_I40E_RXTX_COMMON_AVX_H_*/ diff --git a/drivers/net/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/i40e/i40e_rxtx_vec_avx2.c index c73b2a321b..fcb7ba0273 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/i40e/i40e_rxtx_vec_avx2.c @@ -25,6 +25,12 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) return i40e_rxq_rearm_common(rxq, false); } +static __rte_always_inline void +i40e_rxq_direct_rearm(struct i40e_rx_queue *rxq) +{ + return i40e_rxq_direct_rearm_common(rxq, false); +} + #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC /* Handles 32B descriptor FDIR ID processing: * rxdp: receive descriptor ring, required to load 2nd 16B half of each desc @@ -128,8 +134,12 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) - i40e_rxq_rearm(rxq); + if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) { + if (rxq->direct_rxrearm_enable) + i40e_rxq_direct_rearm(rxq); + else + i40e_rxq_rearm(rxq); + } /* Before we start moving massive data around, check to see if * there is actually a packet available diff --git a/drivers/net/i40e/i40e_rxtx_vec_avx512.c b/drivers/net/i40e/i40e_rxtx_vec_avx512.c index 2e8a3f0df6..d967095edc 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_avx512.c +++ b/drivers/net/i40e/i40e_rxtx_vec_avx512.c @@ -21,6 +21,12 @@ #define RTE_I40E_DESCS_PER_LOOP_AVX 8 +enum i40e_direct_rearm_type_value { + I40E_DIRECT_REARM_TYPE_NORMAL = 0x0, + I40E_DIRECT_REARM_TYPE_FAST_FREE = 0x1, + I40E_DIRECT_REARM_TYPE_PRE_FREE = 0x2, +}; + static __rte_always_inline void i40e_rxq_rearm(struct i40e_rx_queue *rxq) { @@ -150,6 +156,241 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } +static __rte_always_inline void +i40e_rxq_direct_rearm(struct i40e_rx_queue *rxq) +{ + struct rte_eth_dev *dev; + struct i40e_tx_queue *txq; + volatile union i40e_rx_desc *rxdp; + struct i40e_vec_tx_entry *txep; + struct i40e_rx_entry *rxep; + struct rte_mbuf *m[RTE_I40E_RXQ_REARM_THRESH]; + uint16_t tx_port_id, tx_queue_id; + uint16_t rx_id; + uint16_t i, n; + uint16_t j = 0; + uint16_t nb_rearm = 0; + enum i40e_direct_rearm_type_value type; + struct rte_mempool_cache *cache = NULL; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + tx_port_id = rxq->direct_rxrearm_port; + tx_queue_id = rxq->direct_rxrearm_queue; + dev = &rte_eth_devices[tx_port_id]; + txq = dev->data->tx_queues[tx_queue_id]; + + /* check Rx queue is able to take in the whole + * batch of free mbufs from Tx queue + */ + if (rxq->rxrearm_nb > txq->tx_rs_thresh) { + /* check DD bits on threshold descriptor */ + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & + rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != + rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) { + goto mempool_bulk; + } + + if (txq->tx_rs_thresh != RTE_I40E_RXQ_REARM_THRESH) + goto mempool_bulk; + + n = txq->tx_rs_thresh; + + /* first buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1) + */ + txep = (void *)txq->sw_ring; + txep += txq->tx_next_dd - (n - 1); + + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* directly put mbufs from Tx to Rx */ + uint32_t copied = 0; + /* n is multiple of 32 */ + while (copied < n) { + const __m512i a = _mm512_load_si512(&txep[copied]); + const __m512i b = _mm512_load_si512(&txep[copied + 8]); + const __m512i c = _mm512_load_si512(&txep[copied + 16]); + const __m512i d = _mm512_load_si512(&txep[copied + 24]); + + _mm512_storeu_si512(&rxep[copied], a); + _mm512_storeu_si512(&rxep[copied + 8], b); + _mm512_storeu_si512(&rxep[copied + 16], c); + _mm512_storeu_si512(&rxep[copied + 24], d); + copied += 32; + } + type = I40E_DIRECT_REARM_TYPE_FAST_FREE; + } else { + for (i = 0; i < n; i++) { + m[i] = rte_pktmbuf_prefree_seg(txep[i].mbuf); + /* ensure each Tx freed buffer is valid */ + if (m[i] != NULL) + nb_rearm++; + } + + if (nb_rearm != n) { + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + + goto mempool_bulk; + } else { + type = I40E_DIRECT_REARM_TYPE_PRE_FREE; + } + } + + /* update counters for Tx */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + } else { +mempool_bulk: + cache = rte_mempool_default_cache(rxq->mp, rte_lcore_id()); + + if (unlikely(!cache)) + return i40e_rxq_rearm_common(rxq, true); + + n = RTE_I40E_RXQ_REARM_THRESH; + + /* We need to pull 'n' more MBUFs into the software ring from mempool + * We inline the mempool function here, so we can vectorize the copy + * from the cache into the shadow ring. + */ + + if (cache->len < RTE_I40E_RXQ_REARM_THRESH) { + /* No. Backfill the cache first, and then fill from it */ + uint32_t req = RTE_I40E_RXQ_REARM_THRESH + (cache->size - + cache->len); + + /* How many do we require + * i.e. number to fill the cache + the request + */ + int ret = rte_mempool_ops_dequeue_bulk(rxq->mp, + &cache->objs[cache->len], req); + if (ret == 0) { + cache->len += req; + } else { + if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + rxq->nb_rx_desc) { + __m128i dma_addr0; + + dma_addr0 = _mm_setzero_si128(); + for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + rxep[i].mbuf = &rxq->fake_mbuf; + _mm_store_si128 + ((__m128i *)&rxdp[i].read, + dma_addr0); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += + RTE_I40E_RXQ_REARM_THRESH; + return; + } + } + + type = I40E_DIRECT_REARM_TYPE_NORMAL; + } + + const __m512i iova_offsets = _mm512_set1_epi64 + (offsetof(struct rte_mbuf, buf_iova)); + const __m512i headroom = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); + +#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC + /* to shuffle the addresses to correct slots. Values 4-7 will contain + * zeros, so use 7 for a zero-value. + */ + const __m512i permute_idx = _mm512_set_epi64(7, 7, 3, 1, 7, 7, 2, 0); +#else + const __m512i permute_idx = _mm512_set_epi64(7, 3, 6, 2, 5, 1, 4, 0); +#endif + + __m512i mbuf_ptrs; + + /* Initialize the mbufs in vector, process 8 mbufs in one loop, taking + * from mempool cache and populating both shadow and HW rings + */ + for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH / 8; i++) { + switch (type) { + case I40E_DIRECT_REARM_TYPE_FAST_FREE: + mbuf_ptrs = _mm512_loadu_si512(rxep); + break; + case I40E_DIRECT_REARM_TYPE_PRE_FREE: + mbuf_ptrs = _mm512_loadu_si512(&m[j]); + _mm512_store_si512(rxep, mbuf_ptrs); + j += 8; + break; + case I40E_DIRECT_REARM_TYPE_NORMAL: + mbuf_ptrs = _mm512_loadu_si512 + (&cache->objs[cache->len - 8]); + _mm512_store_si512(rxep, mbuf_ptrs); + cache->len -= 8; + break; + } + + /* gather iova of mbuf0-7 into one zmm reg */ + const __m512i iova_base_addrs = _mm512_i64gather_epi64 + (_mm512_add_epi64(mbuf_ptrs, iova_offsets), + 0, /* base */ + 1 /* scale */); + const __m512i iova_addrs = _mm512_add_epi64(iova_base_addrs, + headroom); +#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC + const __m512i iovas0 = _mm512_castsi256_si512 + (_mm512_extracti64x4_epi64(iova_addrs, 0)); + const __m512i iovas1 = _mm512_castsi256_si512 + (_mm512_extracti64x4_epi64(iova_addrs, 1)); + + /* permute leaves desc 2-3 addresses in header address slots 0-1 + * but these are ignored by driver since header split not + * enabled. Similarly for desc 4 & 5. + */ + const __m512i desc_rd_0_1 = _mm512_permutexvar_epi64 + (permute_idx, iovas0); + const __m512i desc_rd_2_3 = _mm512_bsrli_epi128(desc_rd_0_1, 8); + + const __m512i desc_rd_4_5 = _mm512_permutexvar_epi64 + (permute_idx, iovas1); + const __m512i desc_rd_6_7 = _mm512_bsrli_epi128(desc_rd_4_5, 8); + + _mm512_store_si512((void *)rxdp, desc_rd_0_1); + _mm512_store_si512((void *)(rxdp + 2), desc_rd_2_3); + _mm512_store_si512((void *)(rxdp + 4), desc_rd_4_5); + _mm512_store_si512((void *)(rxdp + 6), desc_rd_6_7); +#else + /* permute leaves desc 4-7 addresses in header address slots 0-3 + * but these are ignored by driver since header split not + * enabled. + */ + const __m512i desc_rd_0_3 = _mm512_permutexvar_epi64 + (permute_idx, iova_addrs); + const __m512i desc_rd_4_7 = _mm512_bsrli_epi128(desc_rd_0_3, 8); + + _mm512_store_si512((void *)rxdp, desc_rd_0_3); + _mm512_store_si512((void *)(rxdp + 4), desc_rd_4_7); +#endif + rxdp += 8, rxep += 8; + } + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += n; + rx_id = rxq->rxrearm_start - 1; + + if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { + rxq->rxrearm_start = rxq->rxrearm_start - rxq->nb_rx_desc; + if (!rxq->rxrearm_start) + rx_id = rxq->nb_rx_desc - 1; + else + rx_id = rxq->rxrearm_start - 1; + } + + rxq->rxrearm_nb -= n; + + /* Update the tail pointer on the NIC */ + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); +} + #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC /* Handles 32B descriptor FDIR ID processing: * rxdp: receive descriptor ring, required to load 2nd 16B half of each desc @@ -252,8 +493,12 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) - i40e_rxq_rearm(rxq); + if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) { + if (rxq->direct_rxrearm_enable) + i40e_rxq_direct_rearm(rxq); + else + i40e_rxq_rearm(rxq); + } /* Before we start moving massive data around, check to see if * there is actually a packet available diff --git a/drivers/net/i40e/i40e_rxtx_vec_neon.c b/drivers/net/i40e/i40e_rxtx_vec_neon.c index fa9e6582c5..dc78e3c90b 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_neon.c +++ b/drivers/net/i40e/i40e_rxtx_vec_neon.c @@ -77,6 +77,139 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); } +static inline void +i40e_rxq_direct_rearm(struct i40e_rx_queue *rxq) +{ + struct rte_eth_dev *dev; + struct i40e_tx_queue *txq; + volatile union i40e_rx_desc *rxdp; + struct i40e_tx_entry *txep; + struct i40e_rx_entry *rxep; + uint16_t tx_port_id, tx_queue_id; + uint16_t rx_id; + struct rte_mbuf *mb0, *mb1, *m; + uint64x2_t dma_addr0, dma_addr1; + uint64x2_t zero = vdupq_n_u64(0); + uint64_t paddr; + uint16_t i, n; + uint16_t nb_rearm = 0; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + tx_port_id = rxq->direct_rxrearm_port; + tx_queue_id = rxq->direct_rxrearm_queue; + dev = &rte_eth_devices[tx_port_id]; + txq = dev->data->tx_queues[tx_queue_id]; + + /* check Rx queue is able to take in the whole + * batch of free mbufs from Tx queue + */ + if (rxq->rxrearm_nb > txq->tx_rs_thresh) { + /* check DD bits on threshold descriptor */ + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & + rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != + rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) { + goto mempool_bulk; + } + + n = txq->tx_rs_thresh; + + /* first buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1) + */ + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; + + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* directly put mbufs from Tx to Rx, + * and initialize the mbufs in vector + */ + for (i = 0; i < n; i++, rxep++, txep++) { + rxep[0].mbuf = txep[0].mbuf; + + /* Initialize rxdp descs */ + mb0 = txep[0].mbuf; + + paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; + dma_addr0 = vdupq_n_u64(paddr); + /* flush desc with pa dma_addr */ + vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0); + } + } else { + for (i = 0; i < n; i++) { + m = rte_pktmbuf_prefree_seg(txep[i].mbuf); + if (m != NULL) { + rxep[i].mbuf = m; + + /* Initialize rxdp descs */ + paddr = m->buf_iova + RTE_PKTMBUF_HEADROOM; + dma_addr0 = vdupq_n_u64(paddr); + /* flush desc with pa dma_addr */ + vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0); + nb_rearm++; + } + } + n = nb_rearm; + } + + /* update counters for Tx */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + } else { +mempool_bulk: + /* if TX did not free bufs into Rx sw-ring, + * get new bufs from mempool + */ + n = RTE_I40E_RXQ_REARM_THRESH; + if (unlikely(rte_mempool_get_bulk(rxq->mp, (void *)rxep, n) < 0)) { + if (rxq->rxrearm_nb + n >= rxq->nb_rx_desc) { + for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + rxep[i].mbuf = &rxq->fake_mbuf; + vst1q_u64((uint64_t *)&rxdp[i].read, zero); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += n; + return; + } + + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < n; i += 2, rxep += 2) { + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + + paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; + dma_addr0 = vdupq_n_u64(paddr); + /* flush desc with pa dma_addr */ + vst1q_u64((uint64_t *)&rxdp++->read, dma_addr0); + + paddr = mb1->buf_iova + RTE_PKTMBUF_HEADROOM; + dma_addr1 = vdupq_n_u64(paddr); + /* flush desc with pa dma_addr */ + vst1q_u64((uint64_t *)&rxdp++->read, dma_addr1); + } + } + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += n; + rx_id = rxq->rxrearm_start - 1; + + if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { + rxq->rxrearm_start = rxq->rxrearm_start - rxq->nb_rx_desc; + if (!rxq->rxrearm_start) + rx_id = rxq->nb_rx_desc - 1; + else + rx_id = rxq->rxrearm_start - 1; + } + + rxq->rxrearm_nb -= n; + + rte_io_wmb(); + /* Update the tail pointer on the NIC */ + I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); +} + #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC /* NEON version of FDIR mark extraction for 4 32B descriptors at a time */ static inline uint32x4_t @@ -381,8 +514,12 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) - i40e_rxq_rearm(rxq); + if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) { + if (rxq->direct_rxrearm_enable) + i40e_rxq_direct_rearm(rxq); + else + i40e_rxq_rearm(rxq); + } /* Before we start moving massive data around, check to see if * there is actually a packet available diff --git a/drivers/net/i40e/i40e_rxtx_vec_sse.c b/drivers/net/i40e/i40e_rxtx_vec_sse.c index 3782e8052f..b2f1ab2c8d 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/i40e/i40e_rxtx_vec_sse.c @@ -89,6 +89,168 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } +static inline void +i40e_rxq_direct_rearm(struct i40e_rx_queue *rxq) +{ + struct rte_eth_dev *dev; + struct i40e_tx_queue *txq; + volatile union i40e_rx_desc *rxdp; + struct i40e_tx_entry *txep; + struct i40e_rx_entry *rxep; + uint16_t tx_port_id, tx_queue_id; + uint16_t rx_id; + struct rte_mbuf *mb0, *mb1, *m; + __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, + RTE_PKTMBUF_HEADROOM); + __m128i dma_addr0, dma_addr1; + __m128i vaddr0, vaddr1; + uint16_t i, n; + uint16_t nb_rearm = 0; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + tx_port_id = rxq->direct_rxrearm_port; + tx_queue_id = rxq->direct_rxrearm_queue; + dev = &rte_eth_devices[tx_port_id]; + txq = dev->data->tx_queues[tx_queue_id]; + + /* check Rx queue is able to take in the whole + * batch of free mbufs from Tx queue + */ + if (rxq->rxrearm_nb > txq->tx_rs_thresh) { + /* check DD bits on threshold descriptor */ + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & + rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != + rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) { + goto mempool_bulk; + } + + n = txq->tx_rs_thresh; + + /* first buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1) + */ + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; + + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* directly put mbufs from Tx to Rx, + * and initialize the mbufs in vector + */ + for (i = 0; i < n; i++, rxep++, txep++) { + rxep[0].mbuf = txep[0].mbuf; + + /* Initialize rxdp descs */ + mb0 = txep[0].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + + /* convert pa to dma_addr hdr/data */ + dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); + + /* add headroom to pa values */ + dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); + + /* flush desc with pa dma_addr */ + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr0); + } + } else { + for (i = 0; i < n; i++) { + m = rte_pktmbuf_prefree_seg(txep[i].mbuf); + if (m != NULL) { + rxep[i].mbuf = m; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&m->buf_addr); + + /* convert pa to dma_addr hdr/data */ + dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); + + /* add headroom to pa values */ + dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); + + /* flush desc with pa dma_addr */ + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr0); + nb_rearm++; + } + } + n = nb_rearm; + } + + /* update counters for Tx */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + } else { +mempool_bulk: + /* if TX did not free bufs into Rx sw-ring, + * get new bufs from mempool + */ + n = RTE_I40E_RXQ_REARM_THRESH; + /* Pull 'n' more MBUFs into the software ring */ + if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, n) < 0) { + if (rxq->rxrearm_nb + n >= rxq->nb_rx_desc) { + dma_addr0 = _mm_setzero_si128(); + for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + rxep[i].mbuf = &rxq->fake_mbuf; + _mm_store_si128((__m128i *)&rxdp[i].read, + dma_addr0); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += + RTE_I40E_RXQ_REARM_THRESH; + return; + } + + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; i += 2, rxep += 2) { + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + + /* convert pa to dma_addr hdr/data */ + dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); + dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); + + /* add headroom to pa values */ + dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); + dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); + + /* flush desc with pa dma_addr */ + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr0); + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr1); + } + } + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += n; + rx_id = rxq->rxrearm_start - 1; + + if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { + rxq->rxrearm_start = rxq->rxrearm_start - rxq->nb_rx_desc; + if (!rxq->rxrearm_start) + rx_id = rxq->nb_rx_desc - 1; + else + rx_id = rxq->rxrearm_start - 1; + } + + rxq->rxrearm_nb -= n; + + /* Update the tail pointer on the NIC */ + I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); +} + #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC /* SSE version of FDIR mark extraction for 4 32B descriptors at a time */ static inline __m128i @@ -394,8 +556,12 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) - i40e_rxq_rearm(rxq); + if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) { + if (rxq->direct_rxrearm_enable) + i40e_rxq_direct_rearm(rxq); + else + i40e_rxq_rearm(rxq); + } /* Before we start moving massive data around, check to see if * there is actually a packet available From patchwork Wed Apr 20 08:16:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 109909 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C3207A00BE; Wed, 20 Apr 2022 10:17:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 91383427F8; Wed, 20 Apr 2022 10:17:09 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 80095427F9 for ; Wed, 20 Apr 2022 10:17:08 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 141EC1FB; Wed, 20 Apr 2022 01:17:08 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.114]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 095673F73B; Wed, 20 Apr 2022 01:17:04 -0700 (PDT) From: Feifei Wang To: Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Ray Kinsella Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v1 3/5] ethdev: add API for direct rearm mode Date: Wed, 20 Apr 2022 16:16:48 +0800 Message-Id: <20220420081650.2043183-4-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220420081650.2043183-1-feifei.wang2@arm.com> References: <20220420081650.2043183-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add API for enabling direct rearm mode and for mapping RX and TX queues. Currently, the API supports 1:1(txq : rxq) mapping. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli --- lib/ethdev/ethdev_driver.h | 15 +++++++++++++++ lib/ethdev/rte_ethdev.c | 14 ++++++++++++++ lib/ethdev/rte_ethdev.h | 31 +++++++++++++++++++++++++++++++ lib/ethdev/version.map | 1 + 4 files changed, 61 insertions(+) diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h index 69d9dc21d8..22022f6da9 100644 --- a/lib/ethdev/ethdev_driver.h +++ b/lib/ethdev/ethdev_driver.h @@ -485,6 +485,16 @@ typedef int (*eth_rx_enable_intr_t)(struct rte_eth_dev *dev, typedef int (*eth_rx_disable_intr_t)(struct rte_eth_dev *dev, uint16_t rx_queue_id); +/** @internal Enable direct rearm of a receive queue of an Ethernet device. */ +typedef int (*eth_rx_direct_rearm_enable_t)(struct rte_eth_dev *dev, + uint16_t queue_id); + +/**< @internal map Rx/Tx queue of direct rearm mode */ +typedef int (*eth_rx_direct_rearm_map_t)(struct rte_eth_dev *dev, + uint16_t rx_queue_id, + uint16_t tx_port_id, + uint16_t tx_queue_id); + /** @internal Release memory resources allocated by given Rx/Tx queue. */ typedef void (*eth_queue_release_t)(struct rte_eth_dev *dev, uint16_t queue_id); @@ -1152,6 +1162,11 @@ struct eth_dev_ops { /** Disable Rx queue interrupt */ eth_rx_disable_intr_t rx_queue_intr_disable; + /** Enable Rx queue direct rearm mode */ + eth_rx_direct_rearm_enable_t rx_queue_direct_rearm_enable; + /** Map Rx/Tx queue for direct rearm mode */ + eth_rx_direct_rearm_map_t rx_queue_direct_rearm_map; + eth_tx_queue_setup_t tx_queue_setup;/**< Set up device Tx queue */ eth_queue_release_t tx_queue_release; /**< Release Tx queue */ eth_tx_done_cleanup_t tx_done_cleanup;/**< Free Tx ring mbufs */ diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index 29a3d80466..8e6f0284f4 100644 --- a/lib/ethdev/rte_ethdev.c +++ b/lib/ethdev/rte_ethdev.c @@ -2139,6 +2139,20 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, return eth_err(port_id, ret); } +int +rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id, + uint16_t tx_port_id, uint16_t tx_queue_id) +{ + struct rte_eth_dev *dev; + + dev = &rte_eth_devices[rx_port_id]; + (*dev->dev_ops->rx_queue_direct_rearm_enable)(dev, rx_queue_id); + (*dev->dev_ops->rx_queue_direct_rearm_map)(dev, rx_queue_id, + tx_port_id, tx_queue_id); + + return 0; +} + int rte_eth_hairpin_bind(uint16_t tx_port, uint16_t rx_port) { diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 04cff8ee10..4a431fcbed 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -5190,6 +5190,37 @@ __rte_experimental int rte_eth_dev_hairpin_capability_get(uint16_t port_id, struct rte_eth_hairpin_cap *cap); +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * Enable direct re-arm mode. In this mode the RX queue will be re-armed using + * buffers that have completed transmission on the transmit side. + * + * @note + * It is assumed that the buffers have completed transmission belong to the + * mempool used at the receive side, and have refcnt = 1. + * + * @param rx_port_id + * Port identifying the receive side. + * @param rx_queue_id + * The index of the receive queue identifying the receive side. + * The value must be in the range [0, nb_rx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * @param tx_port_id + * Port identifying the transmit side. + * @param tx_queue_id + * The index of the transmit queue identifying the transmit side. + * The value must be in the range [0, nb_tx_queue - 1] previously supplied + * to rte_eth_dev_configure(). + * + * @return + * - (0) if successful. + */ +__rte_experimental +int rte_eth_direct_rxrearm_map(uint16_t rx_port_id, uint16_t rx_queue_id, + uint16_t tx_port_id, uint16_t tx_queue_id); + /** * @warning * @b EXPERIMENTAL: this structure may change without prior notice. diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map index 20391ab29e..68d664498c 100644 --- a/lib/ethdev/version.map +++ b/lib/ethdev/version.map @@ -279,6 +279,7 @@ EXPERIMENTAL { rte_flow_async_action_handle_create; rte_flow_async_action_handle_destroy; rte_flow_async_action_handle_update; + rte_eth_direct_rxrearm_map; }; INTERNAL { From patchwork Wed Apr 20 08:16:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 109910 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D8DE0A00BE; Wed, 20 Apr 2022 10:17:25 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 81AE7427EF; Wed, 20 Apr 2022 10:17:12 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 73282427FE for ; Wed, 20 Apr 2022 10:17:11 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DFDE31424; Wed, 20 Apr 2022 01:17:10 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.114]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AAE6B3F73B; Wed, 20 Apr 2022 01:17:08 -0700 (PDT) From: Feifei Wang To: Beilei Xing Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v1 4/5] net/i40e: add direct rearm mode internal API Date: Wed, 20 Apr 2022 16:16:49 +0800 Message-Id: <20220420081650.2043183-5-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220420081650.2043183-1-feifei.wang2@arm.com> References: <20220420081650.2043183-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For direct rearm mode, add two internal functions. One is to enable direct rearm mode in Rx queue. The other is to map Tx queue with Rx queue to make Rx queue take buffers from the specific Tx queue. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli --- drivers/net/i40e/i40e_ethdev.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c index 755786dc10..9e1a523bcc 100644 --- a/drivers/net/i40e/i40e_ethdev.c +++ b/drivers/net/i40e/i40e_ethdev.c @@ -369,6 +369,13 @@ static int i40e_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, static int i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id); +static int i40e_dev_rx_queue_direct_rearm_enable(struct rte_eth_dev *dev, + uint16_t queue_id); +static int i40e_dev_rx_queue_direct_rearm_map(struct rte_eth_dev *dev, + uint16_t rx_queue_id, + uint16_t tx_port_id, + uint16_t tx_queue_id); + static int i40e_get_regs(struct rte_eth_dev *dev, struct rte_dev_reg_info *regs); @@ -477,6 +484,8 @@ static const struct eth_dev_ops i40e_eth_dev_ops = { .rx_queue_setup = i40e_dev_rx_queue_setup, .rx_queue_intr_enable = i40e_dev_rx_queue_intr_enable, .rx_queue_intr_disable = i40e_dev_rx_queue_intr_disable, + .rx_queue_direct_rearm_enable = i40e_dev_rx_queue_direct_rearm_enable, + .rx_queue_direct_rearm_map = i40e_dev_rx_queue_direct_rearm_map, .rx_queue_release = i40e_dev_rx_queue_release, .tx_queue_setup = i40e_dev_tx_queue_setup, .tx_queue_release = i40e_dev_tx_queue_release, @@ -11108,6 +11117,31 @@ i40e_dev_rx_queue_intr_disable(struct rte_eth_dev *dev, uint16_t queue_id) return 0; } +static int i40e_dev_rx_queue_direct_rearm_enable(struct rte_eth_dev *dev, + uint16_t queue_id) +{ + struct i40e_rx_queue *rxq; + + rxq = dev->data->rx_queues[queue_id]; + rxq->direct_rxrearm_enable = 1; + + return 0; +} + +static int i40e_dev_rx_queue_direct_rearm_map(struct rte_eth_dev *dev, + uint16_t rx_queue_id, uint16_t tx_port_id, + uint16_t tx_queue_id) +{ + struct i40e_rx_queue *rxq; + + rxq = dev->data->rx_queues[rx_queue_id]; + + rxq->direct_rxrearm_port = tx_port_id; + rxq->direct_rxrearm_queue = tx_queue_id; + + return 0; +} + /** * This function is used to check if the register is valid. * Below is the valid registers list for X722 only: From patchwork Wed Apr 20 08:16:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feifei Wang X-Patchwork-Id: 109911 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2B7ECA00BE; Wed, 20 Apr 2022 10:17:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5CE7742801; Wed, 20 Apr 2022 10:17:15 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id F020D427FF for ; Wed, 20 Apr 2022 10:17:13 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 742621477; Wed, 20 Apr 2022 01:17:13 -0700 (PDT) Received: from net-x86-dell-8268.shanghai.arm.com (net-x86-dell-8268.shanghai.arm.com [10.169.210.114]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 831263F73B; Wed, 20 Apr 2022 01:17:11 -0700 (PDT) From: Feifei Wang To: Cc: dev@dpdk.org, nd@arm.com, Feifei Wang , Honnappa Nagarahalli , Ruifeng Wang Subject: [PATCH v1 5/5] examples/l3fwd: enable direct rearm mode Date: Wed, 20 Apr 2022 16:16:50 +0800 Message-Id: <20220420081650.2043183-6-feifei.wang2@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220420081650.2043183-1-feifei.wang2@arm.com> References: <20220420081650.2043183-1-feifei.wang2@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable direct rearm mode. The mapping is decided in the data plane based on the first packet received. Suggested-by: Honnappa Nagarahalli Signed-off-by: Feifei Wang Reviewed-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli --- examples/l3fwd/l3fwd_lpm.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c index bec22c44cd..38ffdf4636 100644 --- a/examples/l3fwd/l3fwd_lpm.c +++ b/examples/l3fwd/l3fwd_lpm.c @@ -147,7 +147,7 @@ lpm_main_loop(__rte_unused void *dummy) unsigned lcore_id; uint64_t prev_tsc, diff_tsc, cur_tsc; int i, nb_rx; - uint16_t portid; + uint16_t portid, tx_portid; uint8_t queueid; struct lcore_conf *qconf; const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) / @@ -158,6 +158,8 @@ lpm_main_loop(__rte_unused void *dummy) const uint16_t n_rx_q = qconf->n_rx_queue; const uint16_t n_tx_p = qconf->n_tx_port; + int direct_rearm_map[n_rx_q]; + if (n_rx_q == 0) { RTE_LOG(INFO, L3FWD, "lcore %u has nothing to do\n", lcore_id); return 0; @@ -169,6 +171,7 @@ lpm_main_loop(__rte_unused void *dummy) portid = qconf->rx_queue_list[i].port_id; queueid = qconf->rx_queue_list[i].queue_id; + direct_rearm_map[i] = 0; RTE_LOG(INFO, L3FWD, " -- lcoreid=%u portid=%u rxqueueid=%hhu\n", lcore_id, portid, queueid); @@ -209,6 +212,17 @@ lpm_main_loop(__rte_unused void *dummy) if (nb_rx == 0) continue; + /* Determine the direct rearm mapping based on the first + * packet received on the rx queue + */ + if (direct_rearm_map[i] == 0) { + tx_portid = lpm_get_dst_port(qconf, pkts_burst[0], + portid); + rte_eth_direct_rxrearm_map(portid, queueid, + tx_portid, queueid); + direct_rearm_map[i] = 1; + } + #if defined RTE_ARCH_X86 || defined __ARM_NEON \ || defined RTE_ARCH_PPC_64 l3fwd_lpm_send_packets(nb_rx, pkts_burst,