From patchwork Mon Jun 9 15:36:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154198 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6FBF8468B7; Mon, 9 Jun 2025 17:37:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 57E3C41140; Mon, 9 Jun 2025 17:37:51 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 2232940EDF for ; Mon, 9 Jun 2025 17:37:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483470; x=1781019470; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IH5CQ/vLks2nbEU0CWXxWuRX2a66YX60lBNVt9cOXNU=; b=Ly0+ruKNVBqq/H4wvLx8gfa/KXG6t6NX6BdzQ5y8oChUI4H9Fuul8kxB CGr34HEwxw6iOAeEkcSagSlrQjgZ7r1v0b7ITXT1hh7wTq7jMDhPXXCxA n9decODuWmoZnxcRT91aZfT9Cd1SNp2DpIMiSvCkubxZQKO6AuG9jjqyP q0iCr3yYuqglmQ6l4LAEitBr7cY398/ldKEfPydJ0m1g1qcNONLNY8hfc LH3f6Orc37vdulZAYdd9hbFf42xuLm+S4qw1dJBlj56oDrZxfyDhD515h SkaU6uP8j8appgHks1PI1I1I6Ts7GINNEg7XWmnowBp5RNd/8rP2hQ3UM Q==; X-CSE-ConnectionGUID: hPCwu2qGRu+HYNGt6c15ww== X-CSE-MsgGUID: RmgH+3ApQyOAZC/eVG0voA== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012136" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012136" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:48 -0700 X-CSE-ConnectionGUID: 0vw6bvWzQPSJBQgishQYUQ== X-CSE-MsgGUID: 3+PPD1tDTfi9B2Si4vRWNg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419545" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:47 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 01/33] net/ixgbe: remove unused field in Rx queue struct Date: Mon, 9 Jun 2025 16:36:59 +0100 Message-ID: <5cd7f12fc0fc2c32d1b4280832432ee095758763.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The `rdh` (read head) field in the `ixgbe_rx_queue` struct is not used anywhere in the codebase, and can be removed. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/ixgbe/ixgbe_rxtx.c | 9 ++------- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 1 - 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 95c80ac1b8..0c07ce3186 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -3296,17 +3296,12 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, /* * Modified to setup VFRDT for Virtual Function */ - if (ixgbe_is_vf(dev)) { + if (ixgbe_is_vf(dev)) rxq->rdt_reg_addr = IXGBE_PCI_REG_ADDR(hw, IXGBE_VFRDT(queue_idx)); - rxq->rdh_reg_addr = - IXGBE_PCI_REG_ADDR(hw, IXGBE_VFRDH(queue_idx)); - } else { + else rxq->rdt_reg_addr = IXGBE_PCI_REG_ADDR(hw, IXGBE_RDT(rxq->reg_idx)); - rxq->rdh_reg_addr = - IXGBE_PCI_REG_ADDR(hw, IXGBE_RDH(rxq->reg_idx)); - } rxq->rx_ring_phys_addr = rz->iova; rxq->rx_ring = (union ixgbe_adv_rx_desc *) rz->addr; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index 641f982b01..20a5c5a0af 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -85,7 +85,6 @@ struct ixgbe_rx_queue { volatile union ixgbe_adv_rx_desc *rx_ring; /**< RX ring virtual address. */ uint64_t rx_ring_phys_addr; /**< RX ring DMA address. */ volatile uint32_t *rdt_reg_addr; /**< RDT register address. */ - volatile uint32_t *rdh_reg_addr; /**< RDH register address. */ struct ixgbe_rx_entry *sw_ring; /**< address of RX software ring. */ struct ixgbe_scattered_rx_entry *sw_sc_ring; /**< address of scattered Rx software ring. */ struct rte_mbuf *pkt_first_seg; /**< First segment of current packet. */ From patchwork Mon Jun 9 15:37:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154199 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 14A8A468B7; Mon, 9 Jun 2025 17:38:04 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 75573410FB; Mon, 9 Jun 2025 17:37:52 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id CA547410F1 for ; Mon, 9 Jun 2025 17:37:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483470; x=1781019470; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bkz9tlymLlmrAwDuhkcRWJw1kwySph8rHoseL3jHbBE=; b=cSw6Cxh2uICu4uiuWAcUAUny+8pTlK/00s0POGP6k++hV3cZ9Kgug9Pt HXXkrBtSuJ9mf+AeAyU7Ka5z83Qdhv603DZxgmGQ5QZIIAZA0Egv3ZUQE ouZVOvaRbJeYiCsy0BQQ4Eji2yc+a+Xv8LeGKtzkKD3UIx1ZjM+ynvT/X yw4Kflk/Xspztvo0VJLlsjhFfYmRqdYLhjtt2/AFsMuLJuWHblbQ7zlOJ nC/V4uu6zl4/Z426VRW145jyK4gxT6IgAGpxXOyeKQd27suSZuVVlKa9b 3ICRKUyctODaJp0C9t3Bt1iG0HBBnlnCPESa3HkoPJGODESjLTwwDaylh w==; X-CSE-ConnectionGUID: +e2hRY5bS+CG4ThoZ9Q3eQ== X-CSE-MsgGUID: NVHQ6Ze8SAibOv2Soz/ISw== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012139" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012139" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:50 -0700 X-CSE-ConnectionGUID: wEeqxMh+TbaaawmVd/st2g== X-CSE-MsgGUID: 37mW72QlQp2WcfDbn1CJDg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419551" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:48 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin , Ian Stokes Cc: bruce.richardson@intel.com Subject: [PATCH v6 02/33] net/iavf: make IPsec stats dynamically allocated Date: Mon, 9 Jun 2025 16:37:00 +0100 Message-ID: <278d88fca5688c1f0cce48141c52968c0557cd59.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, the stats structure is directly embedded in the queue structure. We're about to move iavf driver to a common Rx queue structure, so we can't have driver-specific structures that aren't pointers, inside the common queue structure. To prepare, we replace direct embedding into the queue structure with a pointer to the stats structure. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/iavf/iavf_ethdev.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.c | 21 ++++++++++++++++++--- drivers/net/intel/iavf/iavf_rxtx.h | 2 +- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/drivers/net/intel/iavf/iavf_ethdev.c b/drivers/net/intel/iavf/iavf_ethdev.c index b3dacbef84..5babd587b3 100644 --- a/drivers/net/intel/iavf/iavf_ethdev.c +++ b/drivers/net/intel/iavf/iavf_ethdev.c @@ -1870,7 +1870,7 @@ iavf_dev_update_ipsec_xstats(struct rte_eth_dev *ethdev, struct iavf_rx_queue *rxq; struct iavf_ipsec_crypto_stats *stats; rxq = (struct iavf_rx_queue *)ethdev->data->rx_queues[idx]; - stats = &rxq->stats.ipsec_crypto; + stats = &rxq->stats->ipsec_crypto; ips->icount += stats->icount; ips->ibytes += stats->ibytes; ips->ierrors.count += stats->ierrors.count; diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c index 5411eb6897..d23d2df807 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.c +++ b/drivers/net/intel/iavf/iavf_rxtx.c @@ -619,6 +619,18 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, return -ENOMEM; } + /* Allocate stats */ + rxq->stats = rte_zmalloc_socket("iavf rxq stats", + sizeof(struct iavf_rx_queue_stats), + RTE_CACHE_LINE_SIZE, + socket_id); + if (!rxq->stats) { + PMD_INIT_LOG(ERR, "Failed to allocate memory for " + "rx queue stats"); + rte_free(rxq); + return -ENOMEM; + } + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_RX_FLEX_DESC) { proto_xtr = vf->proto_xtr ? vf->proto_xtr[queue_idx] : IAVF_PROTO_XTR_NONE; @@ -677,6 +689,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, socket_id); if (!rxq->sw_ring) { PMD_INIT_LOG(ERR, "Failed to allocate memory for SW ring"); + rte_free(rxq->stats); rte_free(rxq); return -ENOMEM; } @@ -693,6 +706,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, if (!mz) { PMD_INIT_LOG(ERR, "Failed to reserve DMA memory for RX"); rte_free(rxq->sw_ring); + rte_free(rxq->stats); rte_free(rxq); return -ENOMEM; } @@ -1054,6 +1068,7 @@ iavf_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid) iavf_rxq_release_mbufs_ops[q->rel_mbufs_type].release_mbufs(q); rte_free(q->sw_ring); rte_memzone_free(q->mz); + rte_free(q->stats); rte_free(q); } @@ -1581,7 +1596,7 @@ iavf_recv_pkts_flex_rxd(void *rx_queue, rte_le_to_cpu_16(rxd.wb.ptype_flex_flags0)]; iavf_flex_rxd_to_vlan_tci(rxm, &rxd); iavf_flex_rxd_to_ipsec_crypto_status(rxm, &rxd, - &rxq->stats.ipsec_crypto); + &rxq->stats->ipsec_crypto); rxd_to_pkt_fields_ops[rxq->rxdid](rxq, rxm, &rxd); pkt_flags = iavf_flex_rxd_error_to_pkt_flags(rx_stat_err0); @@ -1750,7 +1765,7 @@ iavf_recv_scattered_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, rte_le_to_cpu_16(rxd.wb.ptype_flex_flags0)]; iavf_flex_rxd_to_vlan_tci(first_seg, &rxd); iavf_flex_rxd_to_ipsec_crypto_status(first_seg, &rxd, - &rxq->stats.ipsec_crypto); + &rxq->stats->ipsec_crypto); rxd_to_pkt_fields_ops[rxq->rxdid](rxq, first_seg, &rxd); pkt_flags = iavf_flex_rxd_error_to_pkt_flags(rx_stat_err0); @@ -2034,7 +2049,7 @@ iavf_rx_scan_hw_ring_flex_rxd(struct iavf_rx_queue *rxq, rte_le_to_cpu_16(rxdp[j].wb.ptype_flex_flags0)]; iavf_flex_rxd_to_vlan_tci(mb, &rxdp[j]); iavf_flex_rxd_to_ipsec_crypto_status(mb, &rxdp[j], - &rxq->stats.ipsec_crypto); + &rxq->stats->ipsec_crypto); rxd_to_pkt_fields_ops[rxq->rxdid](rxq, mb, &rxdp[j]); stat_err0 = rte_le_to_cpu_16(rxdp[j].wb.status_error0); pkt_flags = iavf_flex_rxd_error_to_pkt_flags(stat_err0); diff --git a/drivers/net/intel/iavf/iavf_rxtx.h b/drivers/net/intel/iavf/iavf_rxtx.h index 0b5d67e718..62b5a67c84 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.h +++ b/drivers/net/intel/iavf/iavf_rxtx.h @@ -268,7 +268,7 @@ struct iavf_rx_queue { uint8_t proto_xtr; /* protocol extraction type */ uint64_t xtr_ol_flag; /* flexible descriptor metadata extraction offload flag */ - struct iavf_rx_queue_stats stats; + struct iavf_rx_queue_stats *stats; uint64_t offloads; uint64_t phc_time; uint64_t hw_time_update; From patchwork Mon Jun 9 15:37:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154200 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C4E82468B7; Mon, 9 Jun 2025 17:38:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A20FF41149; Mon, 9 Jun 2025 17:37:53 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 7503F41143 for ; Mon, 9 Jun 2025 17:37:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483472; x=1781019472; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=09dg8NrAzZ41y7aeHduNJpsBbUKeg1wvlZRu7kZ9wH0=; b=giFteEdBmMV3vJ78lsiwAddjeMDZQgMWi2+ZjRv7sC8Y+m+76YCNYZ7u X6VfPOts1J5Oo4AYBlAfpAwT+oR3V7LR5xyQ5vol8ktDJFPUiGphl0V4B doGqa0E/V1VmnLi8a5ijGc+O7P5A/wjWtCTsg7eEx9tvZIbHJq+zciqab ApyR//955NQBCtBikhids0zfIya/UWuO5/gkxfcibnMNZJeSREb1/8OhX tleyJngoG+q1kTE6Hrz8XvAXspIYaOqQVe2gNRVIgsEO/Idg/rwqru/K9 ejcAMZasb0Vn1neZCoriPEYqPa3Z62oV1EwChDwj5X3j5Ax5r6oXbkopY A==; X-CSE-ConnectionGUID: ZZlaJnMATrOOveGngwlT4g== X-CSE-MsgGUID: Gk94ngESSm6z3hL9OsaKQg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012140" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012140" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:51 -0700 X-CSE-ConnectionGUID: avJndrU1SpaNOIIwffgXXg== X-CSE-MsgGUID: gqpPgExdTvmHHQxeBWPSwg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419561" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:49 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 03/33] net/ixgbe: match variable names to other drivers Date: Mon, 9 Jun 2025 16:37:01 +0100 Message-ID: <8cc442b025649d9bf764a2fc69fb05143caa5dac.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, the ixgbe driver has variables that have the same semantics as in other drivers, but have different names. Rename these variables to match ones in other drivers: - rdt_reg_addr -> qrx_tail (Rx ring tail register address) - rx_using_sse -> vector_rx (indicates if vectorized path is enabled) - mb_pool -> mp (other drivers use this name for mempool) Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Make this commit separate .../ixgbe/ixgbe_recycle_mbufs_vec_common.c | 2 +- drivers/net/intel/ixgbe/ixgbe_rxtx.c | 39 +++++++++---------- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 6 +-- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c | 4 +- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c | 4 +- 5 files changed, 27 insertions(+), 28 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c index c1b086ef6d..2ab7abbf4e 100644 --- a/drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c @@ -42,7 +42,7 @@ ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); } uint16_t diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 0c07ce3186..f75821029d 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -1679,7 +1679,7 @@ ixgbe_rx_alloc_bufs(struct ixgbe_rx_queue *rxq, bool reset_mbuf) /* allocate buffers in bulk directly into the S/W ring */ alloc_idx = rxq->rx_free_trigger - (rxq->rx_free_thresh - 1); rxep = &rxq->sw_ring[alloc_idx]; - diag = rte_mempool_get_bulk(rxq->mb_pool, (void *)rxep, + diag = rte_mempool_get_bulk(rxq->mp, (void *)rxep, rxq->rx_free_thresh); if (unlikely(diag != 0)) return -ENOMEM; @@ -1778,8 +1778,7 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, /* update tail pointer */ rte_wmb(); - IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->rdt_reg_addr, - cur_free_trigger); + IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->qrx_tail, cur_free_trigger); } if (rxq->rx_tail >= rxq->nb_rx_desc) @@ -1908,7 +1907,7 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, (unsigned) rx_id, (unsigned) staterr, (unsigned) rte_le_to_cpu_16(rxd.wb.upper.length)); - nmb = rte_mbuf_raw_alloc(rxq->mb_pool); + nmb = rte_mbuf_raw_alloc(rxq->mp); if (nmb == NULL) { PMD_RX_LOG(DEBUG, "RX mbuf alloc failed port_id=%u " "queue_id=%u", (unsigned) rxq->port_id, @@ -2017,7 +2016,7 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, (unsigned) nb_rx); rx_id = (uint16_t) ((rx_id == 0) ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2165,7 +2164,7 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, rte_le_to_cpu_16(rxd.wb.upper.length)); if (!bulk_alloc) { - nmb = rte_mbuf_raw_alloc(rxq->mb_pool); + nmb = rte_mbuf_raw_alloc(rxq->mp); if (nmb == NULL) { PMD_RX_LOG(DEBUG, "RX mbuf alloc failed " "port_id=%u queue_id=%u", @@ -2181,7 +2180,7 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, if (!ixgbe_rx_alloc_bufs(rxq, false)) { rte_wmb(); IXGBE_PCI_REG_WC_WRITE_RELAXED( - rxq->rdt_reg_addr, + rxq->qrx_tail, next_rdt); nb_hold -= rxq->rx_free_thresh; } else { @@ -2347,7 +2346,7 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, rxq->port_id, rxq->queue_id, rx_id, nb_hold, nb_rx); rte_wmb(); - IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->rdt_reg_addr, prev_id); + IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->qrx_tail, prev_id); nb_hold = 0; } @@ -2974,7 +2973,7 @@ ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq) unsigned i; /* SSE Vector driver has a different way of releasing mbufs. */ - if (rxq->rx_using_sse) { + if (rxq->vector_rx) { ixgbe_rx_queue_release_mbufs_vec(rxq); return; } @@ -3238,7 +3237,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, RTE_CACHE_LINE_SIZE, socket_id); if (rxq == NULL) return -ENOMEM; - rxq->mb_pool = mp; + rxq->mp = mp; rxq->nb_rx_desc = nb_desc; rxq->rx_free_thresh = rx_conf->rx_free_thresh; rxq->queue_id = queue_idx; @@ -3297,10 +3296,10 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, * Modified to setup VFRDT for Virtual Function */ if (ixgbe_is_vf(dev)) - rxq->rdt_reg_addr = + rxq->qrx_tail = IXGBE_PCI_REG_ADDR(hw, IXGBE_VFRDT(queue_idx)); else - rxq->rdt_reg_addr = + rxq->qrx_tail = IXGBE_PCI_REG_ADDR(hw, IXGBE_RDT(rxq->reg_idx)); rxq->rx_ring_phys_addr = rz->iova; @@ -3409,7 +3408,7 @@ ixgbe_dev_rx_descriptor_status(void *rx_queue, uint16_t offset) return -EINVAL; #if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM64) - if (rxq->rx_using_sse) + if (rxq->vector_rx) nb_hold = rxq->rxrearm_nb; else #endif @@ -4677,7 +4676,7 @@ ixgbe_alloc_rx_queue_mbufs(struct ixgbe_rx_queue *rxq) /* Initialize software ring entries */ for (i = 0; i < rxq->nb_rx_desc; i++) { volatile union ixgbe_adv_rx_desc *rxd; - struct rte_mbuf *mbuf = rte_mbuf_raw_alloc(rxq->mb_pool); + struct rte_mbuf *mbuf = rte_mbuf_raw_alloc(rxq->mp); if (mbuf == NULL) { PMD_INIT_LOG(ERR, "RX mbuf alloc failed queue_id=%u", @@ -5111,7 +5110,7 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) for (i = 0; i < dev->data->nb_rx_queues; i++) { struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; - rxq->rx_using_sse = rx_using_sse; + rxq->vector_rx = rx_using_sse; #ifdef RTE_LIB_SECURITY rxq->using_ipsec = !!(dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SECURITY); @@ -5217,7 +5216,7 @@ ixgbe_set_rsc(struct rte_eth_dev *dev) */ rscctl |= IXGBE_RSCCTL_RSCEN; - rscctl |= ixgbe_get_rscctl_maxdesc(rxq->mb_pool); + rscctl |= ixgbe_get_rscctl_maxdesc(rxq->mp); psrtype |= IXGBE_PSRTYPE_TCPHDR; /* @@ -5374,7 +5373,7 @@ ixgbe_dev_rx_init(struct rte_eth_dev *dev) * The value is in 1 KB resolution. Valid values can be from * 1 KB to 16 KB. */ - buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) - + buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mp) - RTE_PKTMBUF_HEADROOM); srrctl |= ((buf_size >> IXGBE_SRRCTL_BSIZEPKT_SHIFT) & IXGBE_SRRCTL_BSIZEPKT_MASK); @@ -5827,7 +5826,7 @@ ixgbe_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, rxq = dev->data->rx_queues[queue_id]; - qinfo->mp = rxq->mb_pool; + qinfo->mp = rxq->mp; qinfo->scattered_rx = dev->data->scattered_rx; qinfo->nb_desc = rxq->nb_rx_desc; @@ -5867,7 +5866,7 @@ ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, rxq = dev->data->rx_queues[queue_id]; recycle_rxq_info->mbuf_ring = (void *)rxq->sw_ring; - recycle_rxq_info->mp = rxq->mb_pool; + recycle_rxq_info->mp = rxq->mp; recycle_rxq_info->mbuf_ring_size = rxq->nb_rx_desc; recycle_rxq_info->receive_tail = &rxq->rx_tail; @@ -5972,7 +5971,7 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev) * The value is in 1 KB resolution. Valid values can be from * 1 KB to 16 KB. */ - buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mb_pool) - + buf_size = (uint16_t)(rte_pktmbuf_data_room_size(rxq->mp) - RTE_PKTMBUF_HEADROOM); srrctl |= ((buf_size >> IXGBE_SRRCTL_BSIZEPKT_SHIFT) & IXGBE_SRRCTL_BSIZEPKT_MASK); diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index 20a5c5a0af..c86714804f 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -81,10 +81,10 @@ struct ixgbe_scattered_rx_entry { * Structure associated with each RX queue. */ struct ixgbe_rx_queue { - struct rte_mempool *mb_pool; /**< mbuf pool to populate RX ring. */ + struct rte_mempool *mp; /**< mbuf pool to populate RX ring. */ volatile union ixgbe_adv_rx_desc *rx_ring; /**< RX ring virtual address. */ uint64_t rx_ring_phys_addr; /**< RX ring DMA address. */ - volatile uint32_t *rdt_reg_addr; /**< RDT register address. */ + volatile uint32_t *qrx_tail; /**< RDT register address. */ struct ixgbe_rx_entry *sw_ring; /**< address of RX software ring. */ struct ixgbe_scattered_rx_entry *sw_sc_ring; /**< address of scattered Rx software ring. */ struct rte_mbuf *pkt_first_seg; /**< First segment of current packet. */ @@ -96,7 +96,7 @@ struct ixgbe_rx_queue { uint16_t rx_nb_avail; /**< nr of staged pkts ready to ret to app */ uint16_t rx_next_avail; /**< idx of next staged pkt to ret to app */ uint16_t rx_free_trigger; /**< triggers rx buffer allocation */ - uint8_t rx_using_sse; + uint8_t vector_rx; /**< indicates that vector RX is in use */ #ifdef RTE_LIB_SECURITY uint8_t using_ipsec; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c index 9ccd8eba25..f8916d44e8 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c @@ -27,7 +27,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) rxdp = rxq->rx_ring + rxq->rxrearm_start; /* Pull 'n' more MBUFs into the software ring */ - if (unlikely(rte_mempool_get_bulk(rxq->mb_pool, + if (unlikely(rte_mempool_get_bulk(rxq->mp, (void *)rxep, RTE_IXGBE_RXQ_REARM_THRESH) < 0)) { if (rxq->rxrearm_nb + RTE_IXGBE_RXQ_REARM_THRESH >= @@ -76,7 +76,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); } static inline void diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c index e125f52cc5..9417e5b11f 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c @@ -29,7 +29,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) rxdp = rxq->rx_ring + rxq->rxrearm_start; /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mb_pool, + if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, RTE_IXGBE_RXQ_REARM_THRESH) < 0) { if (rxq->rxrearm_nb + RTE_IXGBE_RXQ_REARM_THRESH >= @@ -86,7 +86,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } #ifdef RTE_LIB_SECURITY From patchwork Mon Jun 9 15:37:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154201 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E85E4468B7; Mon, 9 Jun 2025 17:38:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D3FA841611; Mon, 9 Jun 2025 17:37:54 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 0C36641149 for ; Mon, 9 Jun 2025 17:37:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483473; x=1781019473; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=lHUrwgpnwncGFY35KX44hdp1tUY4qngnWncOM0sIzPM=; b=RdHbokMN6KBtQLXFtC7sZrYtvrNW7Y13TlwMs0UMJ4bVGAHmtdVf/QzW t4Mod6NOHJs7jqTeJ1HMCcBEJTsdvx1xcPULMJ2DxJNR7nZ8ckdIUC5r0 utDURz/aieC+vIaT4g7hUKo8MoI0W0a+qc580tr7BfwH+/TrUMx08Bcoi hgsnJZs+hAFYJzbNErch0xm06O3MiCIbizvtgkehjF5e4YCaWaXS8Aikj De51Li6HVsuqHwarzwq4PFgKqleycWcOVz/DSYVWPbkiJPqhLZJ+Z2YHD GmTXdfq7GtGdZBAffmMCpUBNpMSEJHdfX2qxTJHqBNwctuKW9d+GvWNt2 w==; X-CSE-ConnectionGUID: aIAiM4MvTgi03qN111zOdg== X-CSE-MsgGUID: gK3eBqOgT1CJzyWoNzYa0w== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012141" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012141" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:53 -0700 X-CSE-ConnectionGUID: r2n1gYc6S465/5JEW1hGmg== X-CSE-MsgGUID: C2TwIGYASoimKwo3zt5Vag== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419571" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:51 -0700 From: Anatoly Burakov To: dev@dpdk.org, Ian Stokes , Bruce Richardson Subject: [PATCH v6 04/33] net/i40e: match variable name to other drivers Date: Mon, 9 Jun 2025 16:37:02 +0100 Message-ID: <780f4f3e38edd67fc4161c23464afa936704b22a.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, the i40e driver has a variable that has the same semantics as in other drivers, but has a different name. Rename `rx_using_sse` to `vector_rx` to match it to other drivers. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Make this commit separate drivers/net/intel/i40e/i40e_rxtx.c | 8 ++++---- drivers/net/intel/i40e/i40e_rxtx.h | 2 +- drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c | 2 +- drivers/net/intel/i40e/i40e_rxtx_vec_neon.c | 2 +- drivers/net/intel/i40e/i40e_rxtx_vec_sse.c | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/net/intel/i40e/i40e_rxtx.c b/drivers/net/intel/i40e/i40e_rxtx.c index c3ff2e05c3..b4caa3bdd5 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.c +++ b/drivers/net/intel/i40e/i40e_rxtx.c @@ -2633,7 +2633,7 @@ i40e_rx_queue_release_mbufs(struct i40e_rx_queue *rxq) uint16_t i; /* SSE Vector driver has a different way of releasing mbufs. */ - if (rxq->rx_using_sse) { + if (rxq->vector_rx) { i40e_rx_queue_release_mbufs_vec(rxq); return; } @@ -3316,7 +3316,7 @@ i40e_set_rx_function(struct rte_eth_dev *dev) { struct i40e_adapter *ad = I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); - uint16_t rx_using_sse, i; + uint16_t vector_rx, i; /* In order to allow Vector Rx there are a few configuration * conditions to be met and Rx Bulk Allocation should be allowed. */ @@ -3427,7 +3427,7 @@ i40e_set_rx_function(struct rte_eth_dev *dev) /* Propagate information about RX function choice through all queues. */ if (rte_eal_process_type() == RTE_PROC_PRIMARY) { - rx_using_sse = + vector_rx = (dev->rx_pkt_burst == i40e_recv_scattered_pkts_vec || dev->rx_pkt_burst == i40e_recv_pkts_vec || #ifdef CC_AVX512_SUPPORT @@ -3441,7 +3441,7 @@ i40e_set_rx_function(struct rte_eth_dev *dev) struct i40e_rx_queue *rxq = dev->data->rx_queues[i]; if (rxq) - rxq->rx_using_sse = rx_using_sse; + rxq->vector_rx = vector_rx; } } } diff --git a/drivers/net/intel/i40e/i40e_rxtx.h b/drivers/net/intel/i40e/i40e_rxtx.h index 2f32fc5686..9db044f280 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.h +++ b/drivers/net/intel/i40e/i40e_rxtx.h @@ -118,7 +118,7 @@ struct i40e_rx_queue { uint8_t hs_mode; /* Header Split mode */ bool q_set; /**< indicate if rx queue has been configured */ bool rx_deferred_start; /**< don't start this queue in dev start */ - uint16_t rx_using_sse; /**rx_using_sse = 1; + rxq->vector_rx = 1; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); return 0; } diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c index d16ceb6b5d..317a0323bb 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c @@ -746,7 +746,7 @@ i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) int __rte_cold i40e_rxq_vec_setup(struct i40e_rx_queue *rxq) { - rxq->rx_using_sse = 1; + rxq->vector_rx = 1; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); return 0; } diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c index 774519265b..25a3ef7352 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c @@ -763,7 +763,7 @@ i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) int __rte_cold i40e_rxq_vec_setup(struct i40e_rx_queue *rxq) { - rxq->rx_using_sse = 1; + rxq->vector_rx = 1; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); return 0; } From patchwork Mon Jun 9 15:37:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154202 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BF82D468B7; Mon, 9 Jun 2025 17:38:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D392741148; Mon, 9 Jun 2025 17:37:56 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 536AE41101 for ; Mon, 9 Jun 2025 17:37:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483476; x=1781019476; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=0xtASUpwkqhIocKD8lBVjvyd5daBw1x3EhMUmG67eAQ=; b=JV8nKRYL6Vlt+v3O0Q1CyqDAjn5Ze3CUuOGCBZTyXQMfgIh9x8iwuo7T 98yhHvYF6wv0JYtbB2Yb+cYitURbcDN5GSjzrkeE8RpJEnlIeHsgUKBmp 7QH0GZr7IDKOu7it686iMbMXuYcCeZqdU2vBbsCJWGuwFVWcpZR1sjgfn q/W+fFHGDbB0N6x3gGKlMXtj5mtTG9ILwYmjoupgNVzSBhEvtrD9g5bBo AFTDjDRxc7PsXt+md/2vPBaty1yPTjyCADRx4y/8Fbto3YZqPqtOJrk3l HiR0ElrG3ZbhdgD+88GaDCjTIc0/OvsYSOdqj7vQ82umlD6GL1DByEwQx Q==; X-CSE-ConnectionGUID: GbLxq39rRWSW4NLEO5RMtg== X-CSE-MsgGUID: EKMZzCVVTxuSFlAUtONqaw== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012144" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012144" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:55 -0700 X-CSE-ConnectionGUID: jHnnq5rXSkqnjGlUfHZJvw== X-CSE-MsgGUID: jKR+tQpwTV+9KMimTuHu7Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419578" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:53 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson Subject: [PATCH v6 05/33] net/ice: match variable name to other drivers Date: Mon, 9 Jun 2025 16:37:03 +0100 Message-ID: <49231070d5bce3de481e3b880e6e80915b2942b7.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, the ice driver has a variable that have the same semantics as in other drivers, but has a different name. Rename `rx_ring_dma` to `rx_ring_phys_addr` for consistency with other drivers. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Make this commit separate drivers/net/intel/ice/ice_dcf.c | 2 +- drivers/net/intel/ice/ice_rxtx.c | 8 ++++---- drivers/net/intel/ice/ice_rxtx.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/net/intel/ice/ice_dcf.c b/drivers/net/intel/ice/ice_dcf.c index 65c18921f4..fa95aaaba6 100644 --- a/drivers/net/intel/ice/ice_dcf.c +++ b/drivers/net/intel/ice/ice_dcf.c @@ -1211,7 +1211,7 @@ ice_dcf_configure_queues(struct ice_dcf_hw *hw) vc_qp->rxq.max_pkt_size = rxq[i]->max_pkt_len; vc_qp->rxq.ring_len = rxq[i]->nb_rx_desc; - vc_qp->rxq.dma_ring_addr = rxq[i]->rx_ring_dma; + vc_qp->rxq.dma_ring_addr = rxq[i]->rx_ring_phys_addr; vc_qp->rxq.databuffer_size = rxq[i]->rx_buf_len; #ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC diff --git a/drivers/net/intel/ice/ice_rxtx.c b/drivers/net/intel/ice/ice_rxtx.c index ba1435b9de..81962a1f9a 100644 --- a/drivers/net/intel/ice/ice_rxtx.c +++ b/drivers/net/intel/ice/ice_rxtx.c @@ -370,7 +370,7 @@ ice_program_hw_rx_queue(struct ice_rx_queue *rxq) rx_ctx.dtype = 0; /* No Protocol Based Buffer Split mode */ } - rx_ctx.base = rxq->rx_ring_dma / ICE_QUEUE_BASE_ADDR_UNIT; + rx_ctx.base = rxq->rx_ring_phys_addr / ICE_QUEUE_BASE_ADDR_UNIT; rx_ctx.qlen = rxq->nb_rx_desc; rx_ctx.dbuf = rxq->rx_buf_len >> ICE_RLAN_CTX_DBUF_S; rx_ctx.hbuf = rxq->rx_hdr_len >> ICE_RLAN_CTX_HBUF_S; @@ -847,7 +847,7 @@ ice_fdir_program_hw_rx_queue(struct ice_rx_queue *rxq) memset(&rx_ctx, 0, sizeof(rx_ctx)); - rx_ctx.base = rxq->rx_ring_dma / ICE_QUEUE_BASE_ADDR_UNIT; + rx_ctx.base = rxq->rx_ring_phys_addr / ICE_QUEUE_BASE_ADDR_UNIT; rx_ctx.qlen = rxq->nb_rx_desc; rx_ctx.dbuf = rxq->rx_buf_len >> ICE_RLAN_CTX_DBUF_S; rx_ctx.hbuf = rxq->rx_hdr_len >> ICE_RLAN_CTX_HBUF_S; @@ -1273,7 +1273,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, /* Zero all the descriptors in the ring. */ memset(rz->addr, 0, ring_size); - rxq->rx_ring_dma = rz->iova; + rxq->rx_ring_phys_addr = rz->iova; rxq->rx_ring = rz->addr; /* always reserve more for bulk alloc */ @@ -2500,7 +2500,7 @@ ice_fdir_setup_rx_resources(struct ice_pf *pf) rxq->reg_idx = pf->fdir.fdir_vsi->base_queue; rxq->vsi = pf->fdir.fdir_vsi; - rxq->rx_ring_dma = rz->iova; + rxq->rx_ring_phys_addr = rz->iova; memset(rz->addr, 0, ICE_FDIR_NUM_RX_DESC * sizeof(union ice_32byte_rx_desc)); rxq->rx_ring = (union ice_rx_flex_desc *)rz->addr; diff --git a/drivers/net/intel/ice/ice_rxtx.h b/drivers/net/intel/ice/ice_rxtx.h index 500d630679..3c5c014b41 100644 --- a/drivers/net/intel/ice/ice_rxtx.h +++ b/drivers/net/intel/ice/ice_rxtx.h @@ -93,7 +93,7 @@ enum ice_rx_dtype { struct ice_rx_queue { struct rte_mempool *mp; /* mbuf pool to populate RX ring */ volatile union ice_rx_flex_desc *rx_ring;/* RX ring virtual address */ - rte_iova_t rx_ring_dma; /* RX ring DMA address */ + rte_iova_t rx_ring_phys_addr; /* RX ring DMA address */ struct ice_rx_entry *sw_ring; /* address of RX soft ring */ uint16_t nb_rx_desc; /* number of RX descriptors */ uint16_t rx_free_thresh; /* max free RX desc to hold */ From patchwork Mon Jun 9 15:37:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154203 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 926E2468B7; Mon, 9 Jun 2025 17:38:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 04EDC42709; Mon, 9 Jun 2025 17:37:59 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id AC8C44275A for ; Mon, 9 Jun 2025 17:37:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483477; x=1781019477; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=WPLoE8xndOh8GmklcV7m3fVc9XMNQqcA0Xm/3K4rFgA=; b=d0jwA/k1oYfTVqWD3Q1HDAgzXBSS0UhwUYbgiYsdpsq44ezfDed9WWzF 7oogGV6zvGxF+RZCwynNSP6SEOcKX9vQ1V7DxscuRfXQB5VT8I0IR4vEM QiyrPEOFSiTZeaCvObq4bY7pXA7+WmsHHvl6QHMtMpSVyHFxng9dFBSHu qU3lRGKQwXQSSkwfi8I3toPf5FA54reF4xWibdlbbwrmyud/aDKMWJXzr JNJRAsdJOJJusRAX6AeVzJFJJ3OcJpymauwE5j6OhQDB9Yyd06KUnreqc 0Lwiq32fw+Eczz33pj0+JJHR3nVDfGu58UEHky2yzM9FKjKVDTvcJerNe Q==; X-CSE-ConnectionGUID: WaUfxTS+TQCE5IcfzB7TXQ== X-CSE-MsgGUID: mImXn1GsSBqFFZezUApqvA== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012146" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012146" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:56 -0700 X-CSE-ConnectionGUID: gFk1LUeiQ1WPPh7WqZu5Ow== X-CSE-MsgGUID: aulCGCQkQU207PJRObkKlg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419590" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:55 -0700 From: Anatoly Burakov To: dev@dpdk.org, Aman Singh , Bruce Richardson , Ian Stokes Subject: [PATCH v6 06/33] net/i40e: rename 16-byte descriptor define Date: Mon, 9 Jun 2025 16:37:04 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In preparation for having a common definition for 16-byte and 32-byte Rx descriptors, rename `RTE_LIBRTE_I40E_16BYTE_RX_DESC` to `RTE_NET_INTEL_USE_16BYTE_DESC``. Suggested-by: Bruce Richardson Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit app/test-pmd/config.c | 4 ++-- config/rte_config.h | 2 +- doc/guides/nics/i40e.rst | 4 +++- drivers/net/intel/i40e/i40e_fdir.c | 2 +- drivers/net/intel/i40e/i40e_pf.c | 2 +- drivers/net/intel/i40e/i40e_rxtx.c | 10 +++++----- drivers/net/intel/i40e/i40e_rxtx.h | 2 +- drivers/net/intel/i40e/i40e_rxtx_common_avx.h | 2 +- drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c | 8 ++++---- drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c | 8 ++++---- drivers/net/intel/i40e/i40e_rxtx_vec_neon.c | 4 ++-- drivers/net/intel/i40e/i40e_rxtx_vec_sse.c | 4 ++-- 12 files changed, 27 insertions(+), 25 deletions(-) diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c index e89af21cec..1f43f3bbd8 100644 --- a/app/test-pmd/config.c +++ b/app/test-pmd/config.c @@ -4481,7 +4481,7 @@ ring_rxd_display_dword(union igb_ring_dword dword) static void ring_rx_descriptor_display(const struct rte_memzone *ring_mz, -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC portid_t port_id, #else __rte_unused portid_t port_id, @@ -4490,7 +4490,7 @@ ring_rx_descriptor_display(const struct rte_memzone *ring_mz, { struct igb_ring_desc_16_bytes *ring = (struct igb_ring_desc_16_bytes *)ring_mz->addr; -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC int ret; struct rte_eth_dev_info dev_info; diff --git a/config/rte_config.h b/config/rte_config.h index 86897de75e..6191ba3ae0 100644 --- a/config/rte_config.h +++ b/config/rte_config.h @@ -137,7 +137,7 @@ /* i40e defines */ #define RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC 1 -// RTE_LIBRTE_I40E_16BYTE_RX_DESC is not set +/* RTE_NET_INTEL_USE_16BYTE_DESC is not set */ #define RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF 64 #define RTE_LIBRTE_I40E_QUEUE_NUM_PER_VF 4 #define RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM 4 diff --git a/doc/guides/nics/i40e.rst b/doc/guides/nics/i40e.rst index ba592d23fe..234757cd7a 100644 --- a/doc/guides/nics/i40e.rst +++ b/doc/guides/nics/i40e.rst @@ -961,7 +961,9 @@ Use 16 Bytes RX Descriptor Size As i40e PMD supports both 16 and 32 bytes RX descriptor sizes, and 16 bytes size can provide helps to high performance of small packets. In ``config/rte_config.h`` set the following to use 16 bytes size RX descriptors:: - #define RTE_LIBRTE_I40E_16BYTE_RX_DESC 1 + #define RTE_NET_INTEL_USE_16BYTE_DESC 1 + +Note however that setting this up will make it so that all PMD's supporting this definition will also use 16-byte descriptors. Input set requirement of each pctype for FDIR ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/net/intel/i40e/i40e_fdir.c b/drivers/net/intel/i40e/i40e_fdir.c index 94e3ab44e3..734218b67d 100644 --- a/drivers/net/intel/i40e/i40e_fdir.c +++ b/drivers/net/intel/i40e/i40e_fdir.c @@ -112,7 +112,7 @@ i40e_fdir_rx_queue_init(struct i40e_rx_queue *rxq) rx_ctx.hbuff = 0; rx_ctx.base = rxq->rx_ring_phys_addr / I40E_QUEUE_BASE_ADDR_UNIT; rx_ctx.qlen = rxq->nb_rx_desc; -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rx_ctx.dsize = 1; #endif rx_ctx.dtype = i40e_header_split_none; diff --git a/drivers/net/intel/i40e/i40e_pf.c b/drivers/net/intel/i40e/i40e_pf.c index 4a47a8f7ee..ebe1deeade 100644 --- a/drivers/net/intel/i40e/i40e_pf.c +++ b/drivers/net/intel/i40e/i40e_pf.c @@ -401,7 +401,7 @@ i40e_pf_host_hmc_config_rxq(struct i40e_hw *hw, rx_ctx.hbuff = rxq->hdr_size >> I40E_RXQ_CTX_HBUFF_SHIFT; rx_ctx.base = rxq->dma_ring_addr / I40E_QUEUE_BASE_ADDR_UNIT; rx_ctx.qlen = rxq->ring_len; -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rx_ctx.dsize = 1; #endif diff --git a/drivers/net/intel/i40e/i40e_rxtx.c b/drivers/net/intel/i40e/i40e_rxtx.c index b4caa3bdd5..5f54bcc225 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.c +++ b/drivers/net/intel/i40e/i40e_rxtx.c @@ -125,7 +125,7 @@ i40e_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union i40e_rx_desc *rxdp) } else { mb->vlan_tci = 0; } -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (rte_le_to_cpu_16(rxdp->wb.qword2.ext_status) & (1 << I40E_RX_DESC_EXT_STATUS_L2TAG2P_SHIFT)) { mb->ol_flags |= RTE_MBUF_F_RX_QINQ_STRIPPED | RTE_MBUF_F_RX_QINQ | @@ -217,7 +217,7 @@ static inline uint64_t i40e_rxd_build_fdir(volatile union i40e_rx_desc *rxdp, struct rte_mbuf *mb) { uint64_t flags = 0; -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC uint16_t flexbh, flexbl; flexbh = (rte_le_to_cpu_32(rxdp->wb.qword2.ext_status) >> @@ -2925,10 +2925,10 @@ i40e_alloc_rx_queue_mbufs(struct i40e_rx_queue *rxq) rxd = &rxq->rx_ring[i]; rxd->read.pkt_addr = dma_addr; rxd->read.hdr_addr = 0; -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rxd->read.rsvd1 = 0; rxd->read.rsvd2 = 0; -#endif /* RTE_LIBRTE_I40E_16BYTE_RX_DESC */ +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ rxe[i].mbuf = mbuf; } @@ -3010,7 +3010,7 @@ i40e_rx_queue_init(struct i40e_rx_queue *rxq) rx_ctx.base = rxq->rx_ring_phys_addr / I40E_QUEUE_BASE_ADDR_UNIT; rx_ctx.qlen = rxq->nb_rx_desc; -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rx_ctx.dsize = 1; #endif rx_ctx.dtype = rxq->hs_mode; diff --git a/drivers/net/intel/i40e/i40e_rxtx.h b/drivers/net/intel/i40e/i40e_rxtx.h index 9db044f280..568f0536ac 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.h +++ b/drivers/net/intel/i40e/i40e_rxtx.h @@ -68,7 +68,7 @@ enum i40e_header_split_mode { I40E_HEADER_SPLIT_SCTP) /* HW desc structure, both 16-byte and 32-byte types are supported */ -#ifdef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC #define i40e_rx_desc i40e_16byte_rx_desc #else #define i40e_rx_desc i40e_32byte_rx_desc diff --git a/drivers/net/intel/i40e/i40e_rxtx_common_avx.h b/drivers/net/intel/i40e/i40e_rxtx_common_avx.h index b66a808f9f..7d2bda624b 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_common_avx.h +++ b/drivers/net/intel/i40e/i40e_rxtx_common_avx.h @@ -41,7 +41,7 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) return; } -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC struct rte_mbuf *mb0, *mb1; __m128i dma_addr0, dma_addr1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c index 9c406e7a6f..4469c73c56 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c @@ -21,7 +21,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) i40e_rxq_rearm_common(rxq, false); } -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /* Handles 32B descriptor FDIR ID processing: * rxdp: receive descriptor ring, required to load 2nd 16B half of each desc * rx_pkts: required to store metadata back to mbufs @@ -99,7 +99,7 @@ desc_fdir_processing_32b(volatile union i40e_rx_desc *rxdp, /* NOT REACHED, see above switch returns */ return _mm256_setzero_si256(); } -#endif /* RTE_LIBRTE_I40E_16BYTE_RX_DESC */ +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ #define PKTLEN_SHIFT 10 @@ -398,7 +398,7 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, * not always performed. Branch over the code when not enabled. */ if (rxq->fdir_enabled) { -#ifdef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC /* 16B descriptor code path: * RSS and FDIR ID use the same offset in the desc, so * only one can be present at a time. The code below @@ -490,7 +490,7 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, fdir_add_flags = desc_fdir_processing_32b(rxdp, rx_pkts, i, 6); mbuf_flags = _mm256_or_si256(mbuf_flags, fdir_add_flags); /* End 32B desc handling */ -#endif /* RTE_LIBRTE_I40E_16BYTE_RX_DESC */ +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ } /* if() on FDIR enabled */ diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c index d8244556c0..bb25acf398 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c @@ -23,7 +23,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) i40e_rxq_rearm_common(rxq, true); } -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /* Handles 32B descriptor FDIR ID processing: * rxdp: receive descriptor ring, required to load 2nd 16B half of each desc * rx_pkts: required to store metadata back to mbufs @@ -102,7 +102,7 @@ desc_fdir_processing_32b(volatile union i40e_rx_desc *rxdp, /* NOT REACHED, see above switch returns */ return _mm256_setzero_si256(); } -#endif /* RTE_LIBRTE_I40E_16BYTE_RX_DESC */ +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ #define PKTLEN_SHIFT 10 @@ -419,7 +419,7 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, * not always performed. Branch over the code when not enabled. */ if (rxq->fdir_enabled) { -#ifdef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC /* 16B descriptor code path: * RSS and FDIR ID use the same offset in the desc, so * only one can be present at a time. The code below @@ -539,7 +539,7 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, mbuf_flags = _mm256_or_si256(mbuf_flags, fdir_add_flags); /* End 32B desc handling */ -#endif /* RTE_LIBRTE_I40E_16BYTE_RX_DESC */ +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ } /* if() on FDIR enabled */ diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c index 317a0323bb..695b4e1040 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c @@ -77,7 +77,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); } -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /* NEON version of FDIR mark extraction for 4 32B descriptors at a time */ static inline uint32x4_t descs_to_fdir_32b(volatile union i40e_rx_desc *rxdp, struct rte_mbuf **rx_pkt) @@ -284,7 +284,7 @@ desc_to_olflags_v(struct i40e_rx_queue *rxq, volatile union i40e_rx_desc *rxdp, /* Extract FDIR ID only if FDIR is enabled to avoid useless work */ if (rxq->fdir_enabled) { -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC uint32x4_t v_fdir_ol_flags = descs_to_fdir_32b(rxdp, rx_pkts); #else (void)rxdp; /* rxdp not required for 16B desc mode */ diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c index 25a3ef7352..920089fe3e 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c @@ -86,7 +86,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /* SSE version of FDIR mark extraction for 4 32B descriptors at a time */ static inline __m128i descs_to_fdir_32b(volatile union i40e_rx_desc *rxdp, struct rte_mbuf **rx_pkt) @@ -285,7 +285,7 @@ desc_to_olflags_v(struct i40e_rx_queue *rxq, volatile union i40e_rx_desc *rxdp, /* Extract FDIR ID only if FDIR is enabled to avoid useless work */ if (rxq->fdir_enabled) { -#ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC __m128i v_fdir_ol_flags = descs_to_fdir_32b(rxdp, rx_pkts); #else (void)rxdp; /* rxdp not required for 16B desc mode */ From patchwork Mon Jun 9 15:37:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154204 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E043B468B7; Mon, 9 Jun 2025 17:38:43 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 22C0B42707; Mon, 9 Jun 2025 17:38:02 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 3D4F542709 for ; Mon, 9 Jun 2025 17:37:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483479; x=1781019479; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=wtIG0noST226Jd94jK7Nmar74R5oIVzAHc/Ap0nS5uQ=; b=aT8IHxpU5Dq1QgQB9o0HiFEubZOJEYFymxOw6gQrgT8Yv62EWh9mtpEB mLRPfVQkSDj0y8U99yOkHiMa6GWx2Zb3+oqR5ZYqErH/5oOGSHTI9nJYO plWc5SWDv02recE5feI8GfFI6c4/b9RbweTL7cggJEEl5SIZ449y6dhxy grrMW35shk1Yb8kLgAhsu7CxD1wY50X27+/BTLRp8xERW8xrScXMLnpen qhX/KzWo7lTEmL9ICh+s5f6vLEda2j/IJr55+Gu4J/f2ed+0dGtMa1q8U a69AOndOg+QSYeziT9qzStwXh2I4F9yLTprGFnKS35hcDnnvKXV5imKxT A==; X-CSE-ConnectionGUID: vrTgOtmNQ1Ow3zCqZrc0aw== X-CSE-MsgGUID: AYZNsHFWT0qBjVYUdbGNlA== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012150" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012150" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:37:58 -0700 X-CSE-ConnectionGUID: +4bObQWlRl2Gwz+LDxxumA== X-CSE-MsgGUID: hbZbwM8CRiu7UwQW/GRxDw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419599" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:56 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson Subject: [PATCH v6 07/33] net/ice: rename 16-byte descriptor define Date: Mon, 9 Jun 2025 16:37:05 +0100 Message-ID: <9bd75a32af442afc3961f16b4124aa7a43804e5a.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In preparation for having a common definition for 16-byte and 32-byte Rx descriptors, rename RTE_LIBRTE_ICE_16BYTE_RX_DESC to RTE_NET_INTEL_USE_16BYTE_DESC. Suggested-by: Bruce Richardson Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit drivers/net/intel/ice/ice_dcf.c | 2 +- drivers/net/intel/ice/ice_dcf_ethdev.c | 2 +- drivers/net/intel/ice/ice_rxtx.c | 30 ++++++++++----------- drivers/net/intel/ice/ice_rxtx.h | 2 +- drivers/net/intel/ice/ice_rxtx_common_avx.h | 2 +- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 2 +- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 2 +- drivers/net/intel/ice/ice_rxtx_vec_sse.c | 2 +- 8 files changed, 22 insertions(+), 22 deletions(-) diff --git a/drivers/net/intel/ice/ice_dcf.c b/drivers/net/intel/ice/ice_dcf.c index fa95aaaba6..2f7c239491 100644 --- a/drivers/net/intel/ice/ice_dcf.c +++ b/drivers/net/intel/ice/ice_dcf.c @@ -1214,7 +1214,7 @@ ice_dcf_configure_queues(struct ice_dcf_hw *hw) vc_qp->rxq.dma_ring_addr = rxq[i]->rx_ring_phys_addr; vc_qp->rxq.databuffer_size = rxq[i]->rx_buf_len; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (hw->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_RX_FLEX_DESC && hw->supported_rxdid & diff --git a/drivers/net/intel/ice/ice_dcf_ethdev.c b/drivers/net/intel/ice/ice_dcf_ethdev.c index efff76afa8..d3fd5d7122 100644 --- a/drivers/net/intel/ice/ice_dcf_ethdev.c +++ b/drivers/net/intel/ice/ice_dcf_ethdev.c @@ -308,7 +308,7 @@ alloc_rxq_mbufs(struct ice_rx_queue *rxq) rxd = &rxq->rx_ring[i]; rxd->read.pkt_addr = dma_addr; rxd->read.hdr_addr = 0; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rxd->read.rsvd1 = 0; rxd->read.rsvd2 = 0; #endif diff --git a/drivers/net/intel/ice/ice_rxtx.c b/drivers/net/intel/ice/ice_rxtx.c index 81962a1f9a..19569b6a38 100644 --- a/drivers/net/intel/ice/ice_rxtx.c +++ b/drivers/net/intel/ice/ice_rxtx.c @@ -86,7 +86,7 @@ ice_rxd_to_pkt_fields_by_comms_generic(__rte_unused struct ice_rx_queue *rxq, mb->hash.rss = rte_le_to_cpu_32(desc->rss_hash); } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (desc->flow_id != 0xFFFFFFFF) { mb->ol_flags |= RTE_MBUF_F_RX_FDIR | RTE_MBUF_F_RX_FDIR_ID; mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); @@ -101,7 +101,7 @@ ice_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct ice_rx_queue *rxq, { volatile struct ice_32b_rx_flex_desc_comms_ovs *desc = (volatile struct ice_32b_rx_flex_desc_comms_ovs *)rxdp; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC uint16_t stat_err; #endif @@ -110,7 +110,7 @@ ice_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct ice_rx_queue *rxq, mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC stat_err = rte_le_to_cpu_16(desc->status_error0); if (likely(stat_err & (1 << ICE_RX_FLEX_DESC_STATUS0_RSS_VALID_S))) { mb->ol_flags |= RTE_MBUF_F_RX_RSS_HASH; @@ -134,7 +134,7 @@ ice_rxd_to_pkt_fields_by_comms_aux_v1(struct ice_rx_queue *rxq, mb->hash.rss = rte_le_to_cpu_32(desc->rss_hash); } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (desc->flow_id != 0xFFFFFFFF) { mb->ol_flags |= RTE_MBUF_F_RX_FDIR | RTE_MBUF_F_RX_FDIR_ID; mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); @@ -178,7 +178,7 @@ ice_rxd_to_pkt_fields_by_comms_aux_v2(struct ice_rx_queue *rxq, mb->hash.rss = rte_le_to_cpu_32(desc->rss_hash); } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (desc->flow_id != 0xFFFFFFFF) { mb->ol_flags |= RTE_MBUF_F_RX_FDIR | RTE_MBUF_F_RX_FDIR_ID; mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); @@ -374,7 +374,7 @@ ice_program_hw_rx_queue(struct ice_rx_queue *rxq) rx_ctx.qlen = rxq->nb_rx_desc; rx_ctx.dbuf = rxq->rx_buf_len >> ICE_RLAN_CTX_DBUF_S; rx_ctx.hbuf = rxq->rx_hdr_len >> ICE_RLAN_CTX_HBUF_S; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rx_ctx.dsize = 1; /* 32B descriptors */ #endif rx_ctx.rxmax = rxq->max_pkt_len; @@ -501,7 +501,7 @@ ice_alloc_rx_queue_mbufs(struct ice_rx_queue *rxq) rxd->read.pkt_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(mbuf_pay)); } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC rxd->read.rsvd1 = 0; rxd->read.rsvd2 = 0; #endif @@ -1668,7 +1668,7 @@ ice_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union ice_rx_flex_desc *rxdp) mb->vlan_tci = 0; } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (rte_le_to_cpu_16(rxdp->wb.status_error1) & (1 << ICE_RX_FLEX_DESC_STATUS1_L2TAG2P_S)) { mb->ol_flags |= RTE_MBUF_F_RX_QINQ_STRIPPED | RTE_MBUF_F_RX_QINQ | @@ -1705,7 +1705,7 @@ ice_rx_scan_hw_ring(struct ice_rx_queue *rxq) int32_t i, j, nb_rx = 0; uint64_t pkt_flags = 0; uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC bool is_tsinit = false; uint64_t ts_ns; struct ice_vsi *vsi = rxq->vsi; @@ -1721,7 +1721,7 @@ ice_rx_scan_hw_ring(struct ice_rx_queue *rxq) if (!(stat_err0 & (1 << ICE_RX_FLEX_DESC_STATUS0_DD_S))) return 0; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); @@ -1783,7 +1783,7 @@ ice_rx_scan_hw_ring(struct ice_rx_queue *rxq) rte_le_to_cpu_16(rxdp[j].wb.ptype_flex_flags0)]; ice_rxd_to_vlan_tci(mb, &rxdp[j]); rxd_to_pkt_fields_ops[rxq->rxdid](rxq, mb, &rxdp[j]); -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (rxq->ts_flag > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) { rxq->time_high = @@ -2023,7 +2023,7 @@ ice_recv_scattered_pkts(void *rx_queue, uint64_t dma_addr; uint64_t pkt_flags; uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC bool is_tsinit = false; uint64_t ts_ns; struct ice_vsi *vsi = rxq->vsi; @@ -2151,7 +2151,7 @@ ice_recv_scattered_pkts(void *rx_queue, ice_rxd_to_vlan_tci(first_seg, &rxd); rxd_to_pkt_fields_ops[rxq->rxdid](rxq, first_seg, &rxd); pkt_flags = ice_rxd_error_to_pkt_flags(rx_stat_err0); -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (rxq->ts_flag > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) { rxq->time_high = @@ -2540,7 +2540,7 @@ ice_recv_pkts(void *rx_queue, uint64_t dma_addr; uint64_t pkt_flags; uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC bool is_tsinit = false; uint64_t ts_ns; struct ice_vsi *vsi = rxq->vsi; @@ -2649,7 +2649,7 @@ ice_recv_pkts(void *rx_queue, ice_rxd_to_vlan_tci(rxm, &rxd); rxd_to_pkt_fields_ops[rxq->rxdid](rxq, rxm, &rxd); pkt_flags = ice_rxd_error_to_pkt_flags(rx_stat_err0); -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC if (rxq->ts_flag > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) { rxq->time_high = diff --git a/drivers/net/intel/ice/ice_rxtx.h b/drivers/net/intel/ice/ice_rxtx.h index 3c5c014b41..d2d521c4f5 100644 --- a/drivers/net/intel/ice/ice_rxtx.h +++ b/drivers/net/intel/ice/ice_rxtx.h @@ -23,7 +23,7 @@ #define ICE_CHK_Q_ENA_COUNT 100 #define ICE_CHK_Q_ENA_INTERVAL_US 100 -#ifdef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC #define ice_rx_flex_desc ice_16b_rx_flex_desc #else #define ice_rx_flex_desc ice_32b_rx_flex_desc diff --git a/drivers/net/intel/ice/ice_rxtx_common_avx.h b/drivers/net/intel/ice/ice_rxtx_common_avx.h index c62e60c70e..a68cf8512d 100644 --- a/drivers/net/intel/ice/ice_rxtx_common_avx.h +++ b/drivers/net/intel/ice/ice_rxtx_common_avx.h @@ -38,7 +38,7 @@ ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) return; } -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC struct rte_mbuf *mb0, *mb1; __m128i dma_addr0, dma_addr1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c index 0c54b325c6..6fe5ffa6f4 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c @@ -440,7 +440,7 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, } /* if() on fdir_enabled */ if (offload) { -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /** * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c index bd49be07c9..490d1ae059 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c @@ -462,7 +462,7 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, } /* if() on fdir_enabled */ if (do_offload) { -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /** * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. diff --git a/drivers/net/intel/ice/ice_rxtx_vec_sse.c b/drivers/net/intel/ice/ice_rxtx_vec_sse.c index 97f05ba45e..719b37645e 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_sse.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_sse.c @@ -477,7 +477,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, pkt_mb1 = _mm_add_epi16(pkt_mb1, crc_adjust); pkt_mb0 = _mm_add_epi16(pkt_mb0, crc_adjust); -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC /** * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. From patchwork Mon Jun 9 15:37:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154205 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EE0D4468B7; Mon, 9 Jun 2025 17:38:51 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1E3F44275A; Mon, 9 Jun 2025 17:38:03 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 6120142707 for ; Mon, 9 Jun 2025 17:38:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483481; x=1781019481; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=keaIKEHLB8ZrFiSpsOh2GGbdGE5rQBF24uMsFr5mjks=; b=hNHpWQZywb3xzJQTzw91pyJA7+jZpAKR5FcUKUsTCqKP42RzzWazKFMg xq3dU+YA13h4XHzVlz0/Uk+uhuV3rRGmulKmOl6ElSsQgyaU874082A0o Uo8Vlm3SemmCsHze3BXQrimAexctCsa9VTkPINlADQMOsLI8L+KbLOBtK MVgi1huZcPPuBhoRWClmObAs8VqKtQIszrCnkCkNp7maJ+sGKlhr7ecKp DxrEoVu4a3iBMBz6vHv10aNdUosBYOZ6S78vBG2/PKn1hia5UYO9t8Eyj XZMOZYm4xuneYNeTNYMZJk1YCyldk6NprA24nfmGozJT0fWPjp1U+V1EO A==; X-CSE-ConnectionGUID: GX+/eIp8QV6ABkPDK8OEpA== X-CSE-MsgGUID: 4d1CbBS+Qpu7hi+LgxSDrA== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012162" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012162" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:00 -0700 X-CSE-ConnectionGUID: hWRjHZhFQPSIIYjrQ0ReRg== X-CSE-MsgGUID: bRQREpx9S3S1V5IeY5/0ug== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419612" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:37:58 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin , Ian Stokes Cc: bruce.richardson@intel.com Subject: [PATCH v6 08/33] net/iavf: remove 16-byte descriptor define Date: Mon, 9 Jun 2025 16:37:06 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There are no physical functions that support having VF use 16-byte descriptors, so remove all 16-byte descriptor related code from the driver. Suggested-by: Bruce Richardson Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v4 -> v5: - Remove 16-byte descriptor related codepaths v3 -> v4: - Add this commit drivers/net/intel/iavf/iavf_rxtx.c | 18 --- drivers/net/intel/iavf/iavf_rxtx.h | 38 ------ drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 11 -- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 10 -- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 124 ------------------ drivers/net/intel/iavf/iavf_rxtx_vec_sse.c | 25 ---- drivers/net/intel/iavf/iavf_vchnl.c | 14 -- 7 files changed, 240 deletions(-) diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c index d23d2df807..7b10c0314f 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.c +++ b/drivers/net/intel/iavf/iavf_rxtx.c @@ -345,10 +345,8 @@ alloc_rxq_mbufs(struct iavf_rx_queue *rxq) rxd = &rxq->rx_ring[i]; rxd->read.pkt_addr = dma_addr; rxd->read.hdr_addr = 0; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC rxd->read.rsvd1 = 0; rxd->read.rsvd2 = 0; -#endif rxq->sw_ring[i] = mbuf; } @@ -401,22 +399,18 @@ iavf_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct iavf_rx_queue *rxq, { volatile struct iavf_32b_rx_flex_desc_comms_ovs *desc = (volatile struct iavf_32b_rx_flex_desc_comms_ovs *)rxdp; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint16_t stat_err; -#endif if (desc->flow_id != 0xFFFFFFFF) { mb->ol_flags |= RTE_MBUF_F_RX_FDIR | RTE_MBUF_F_RX_FDIR_ID; mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC stat_err = rte_le_to_cpu_16(desc->status_error0); if (likely(stat_err & (1 << IAVF_RX_FLEX_DESC_STATUS0_RSS_VALID_S))) { mb->ol_flags |= RTE_MBUF_F_RX_RSS_HASH; mb->hash.rss = rte_le_to_cpu_32(desc->rss_hash); } -#endif } static inline void @@ -434,7 +428,6 @@ iavf_rxd_to_pkt_fields_by_comms_aux_v1(struct iavf_rx_queue *rxq, mb->hash.rss = rte_le_to_cpu_32(desc->rss_hash); } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (desc->flow_id != 0xFFFFFFFF) { mb->ol_flags |= RTE_MBUF_F_RX_FDIR | RTE_MBUF_F_RX_FDIR_ID; mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); @@ -458,7 +451,6 @@ iavf_rxd_to_pkt_fields_by_comms_aux_v1(struct iavf_rx_queue *rxq, *RTE_PMD_IFD_DYNF_PROTO_XTR_METADATA(mb) = metadata; } } -#endif } static inline void @@ -476,7 +468,6 @@ iavf_rxd_to_pkt_fields_by_comms_aux_v2(struct iavf_rx_queue *rxq, mb->hash.rss = rte_le_to_cpu_32(desc->rss_hash); } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (desc->flow_id != 0xFFFFFFFF) { mb->ol_flags |= RTE_MBUF_F_RX_FDIR | RTE_MBUF_F_RX_FDIR_ID; mb->hash.fdir.hi = rte_le_to_cpu_32(desc->flow_id); @@ -496,7 +487,6 @@ iavf_rxd_to_pkt_fields_by_comms_aux_v2(struct iavf_rx_queue *rxq, *RTE_PMD_IFD_DYNF_PROTO_XTR_METADATA(mb) = metadata; } } -#endif } static const @@ -1177,7 +1167,6 @@ iavf_flex_rxd_to_vlan_tci(struct rte_mbuf *mb, mb->vlan_tci = 0; } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (rte_le_to_cpu_16(rxdp->wb.status_error1) & (1 << IAVF_RX_FLEX_DESC_STATUS1_L2TAG2P_S)) { mb->ol_flags |= RTE_MBUF_F_RX_QINQ_STRIPPED | @@ -1192,7 +1181,6 @@ iavf_flex_rxd_to_vlan_tci(struct rte_mbuf *mb, } else { mb->vlan_tci_outer = 0; } -#endif } static inline void @@ -1301,7 +1289,6 @@ static inline uint64_t iavf_rxd_build_fdir(volatile union iavf_rx_desc *rxdp, struct rte_mbuf *mb) { uint64_t flags = 0; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint16_t flexbh; flexbh = (rte_le_to_cpu_32(rxdp->wb.qword2.ext_status) >> @@ -1313,11 +1300,6 @@ iavf_rxd_build_fdir(volatile union iavf_rx_desc *rxdp, struct rte_mbuf *mb) rte_le_to_cpu_32(rxdp->wb.qword3.hi_dword.fd_id); flags |= RTE_MBUF_F_RX_FDIR_ID; } -#else - mb->hash.fdir.hi = - rte_le_to_cpu_32(rxdp->wb.qword0.hi_dword.fd_id); - flags |= RTE_MBUF_F_RX_FDIR_ID; -#endif return flags; } diff --git a/drivers/net/intel/iavf/iavf_rxtx.h b/drivers/net/intel/iavf/iavf_rxtx.h index 62b5a67c84..a0e1fd8667 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.h +++ b/drivers/net/intel/iavf/iavf_rxtx.h @@ -126,30 +126,6 @@ extern int rte_pmd_iavf_tx_lldp_dynfield_offset; * Rx Flex Descriptors * These descriptors are used instead of the legacy version descriptors */ -union iavf_16b_rx_flex_desc { - struct { - __le64 pkt_addr; /* Packet buffer address */ - __le64 hdr_addr; /* Header buffer address */ - /* bit 0 of hdr_addr is DD bit */ - } read; - struct { - /* Qword 0 */ - u8 rxdid; /* descriptor builder profile ID */ - u8 mir_id_umb_cast; /* mirror=[5:0], umb=[7:6] */ - __le16 ptype_flex_flags0; /* ptype=[9:0], ff0=[15:10] */ - __le16 pkt_len; /* [15:14] are reserved */ - __le16 hdr_len_sph_flex_flags1; /* header=[10:0] */ - /* sph=[11:11] */ - /* ff1/ext=[15:12] */ - - /* Qword 1 */ - __le16 status_error0; - __le16 l2tag1; - __le16 flex_meta0; - __le16 flex_meta1; - } wb; /* writeback */ -}; - union iavf_32b_rx_flex_desc { struct { __le64 pkt_addr; /* Packet buffer address */ @@ -194,14 +170,8 @@ union iavf_32b_rx_flex_desc { } wb; /* writeback */ }; -/* HW desc structure, both 16-byte and 32-byte types are supported */ -#ifdef RTE_LIBRTE_IAVF_16BYTE_RX_DESC -#define iavf_rx_desc iavf_16byte_rx_desc -#define iavf_rx_flex_desc iavf_16b_rx_flex_desc -#else #define iavf_rx_desc iavf_32byte_rx_desc #define iavf_rx_flex_desc iavf_32b_rx_flex_desc -#endif typedef void (*iavf_rxd_to_pkt_fields_t)(struct iavf_rx_queue *rxq, struct rte_mbuf *mb, @@ -740,20 +710,12 @@ void iavf_dump_rx_descriptor(struct iavf_rx_queue *rxq, const volatile void *desc, uint16_t rx_id) { -#ifdef RTE_LIBRTE_IAVF_16BYTE_RX_DESC - const volatile union iavf_16byte_rx_desc *rx_desc = desc; - - printf("Queue %d Rx_desc %d: QW0: 0x%016"PRIx64" QW1: 0x%016"PRIx64"\n", - rxq->queue_id, rx_id, rx_desc->read.pkt_addr, - rx_desc->read.hdr_addr); -#else const volatile union iavf_32byte_rx_desc *rx_desc = desc; printf("Queue %d Rx_desc %d: QW0: 0x%016"PRIx64" QW1: 0x%016"PRIx64 " QW2: 0x%016"PRIx64" QW3: 0x%016"PRIx64"\n", rxq->queue_id, rx_id, rx_desc->read.pkt_addr, rx_desc->read.hdr_addr, rx_desc->read.rsvd1, rx_desc->read.rsvd2); -#endif } /* All the descriptors are 16 bytes, so just use one of them diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c index 88e35dc3e9..c7dc5bbe3e 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c @@ -495,10 +495,7 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, #define IAVF_DESCS_PER_LOOP_AVX 8 struct iavf_adapter *adapter = rxq->vsi->adapter; - -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; -#endif const uint32_t *type_table = adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, @@ -524,7 +521,6 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, if (!(rxdp->wb.status_error0 & rte_cpu_to_le_32(1 << IAVF_RX_FLEX_DESC_STATUS0_DD_S))) return 0; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC bool is_tsinit = false; uint8_t inflection_point = 0; __m256i hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, rxq->phc_time); @@ -538,7 +534,6 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, rxq->phc_time); } } -#endif /* constants used in processing loop */ const __m256i crc_adjust = @@ -946,7 +941,6 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, } /* if() on fdir_enabled */ if (offload) { -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC /** * needs to load 2nd 16B of each desc, * will cause performance drop to get into this context. @@ -1229,7 +1223,6 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, mbuf_flags = _mm256_or_si256(mbuf_flags, _mm256_set1_epi32(iavf_timestamp_dynflag)); } /* if() on Timestamp parsing */ } -#endif } /** @@ -1360,7 +1353,6 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { inflection_point = (inflection_point <= burst) ? inflection_point : 0; switch (inflection_point) { @@ -1406,15 +1398,12 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); } -#endif if (burst != IAVF_DESCS_PER_LOOP_AVX) break; } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (received > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) rxq->phc_time = *RTE_MBUF_DYNFIELD(rx_pkts[received - 1], iavf_timestamp_dynfield_offset, rte_mbuf_timestamp_t *); -#endif /* update tail pointers */ rxq->rx_tail += received; diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c index f2af028bef..51a2dc12bf 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c @@ -585,9 +585,7 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, bool offload) { struct iavf_adapter *adapter = rxq->vsi->adapter; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; -#endif #ifdef IAVF_RX_PTYPE_OFFLOAD const uint32_t *type_table = adapter->ptype_tbl; #endif @@ -616,7 +614,6 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, rte_cpu_to_le_32(1 << IAVF_RX_FLEX_DESC_STATUS0_DD_S))) return 0; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC #ifdef IAVF_RX_TS_OFFLOAD uint8_t inflection_point = 0; bool is_tsinit = false; @@ -632,7 +629,6 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, (uint32_t)rxq->phc_time); } } -#endif #endif /* constants used in processing loop */ @@ -1096,7 +1092,6 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, __m256i mb0_1 = _mm512_extracti64x4_epi64(mb0_3, 0); __m256i mb2_3 = _mm512_extracti64x4_epi64(mb0_3, 1); -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (offload) { #if defined(IAVF_RX_RSS_OFFLOAD) || defined(IAVF_RX_TS_OFFLOAD) /** @@ -1418,7 +1413,6 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, } /* if() on RSS hash or RX timestamp parsing */ #endif } -#endif /** * At this point, we have the 8 sets of flags in the low 16-bits @@ -1548,7 +1542,6 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC #ifdef IAVF_RX_TS_OFFLOAD if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { inflection_point = (inflection_point <= burst) ? inflection_point : 0; @@ -1595,18 +1588,15 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); } -#endif #endif if (burst != IAVF_DESCS_PER_LOOP_AVX) break; } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC #ifdef IAVF_RX_TS_OFFLOAD if (received > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) rxq->phc_time = *RTE_MBUF_DYNFIELD(rx_pkts[received - 1], iavf_timestamp_dynfield_offset, rte_mbuf_timestamp_t *); -#endif #endif /* update tail pointers */ diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h index 38e9a206d9..326b8b07ba 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h @@ -269,7 +269,6 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) return; } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC struct rte_mbuf *mb0, *mb1; __m128i dma_addr0, dma_addr1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, @@ -299,129 +298,6 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); } -#else -#ifdef CC_AVX512_SUPPORT - if (avx512) { - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - struct rte_mbuf *mb4, *mb5, *mb6, *mb7; - __m512i dma_addr0_3, dma_addr4_7; - __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 8 mbufs in one loop */ - for (i = 0; i < IAVF_RXQ_REARM_THRESH; - i += 8, rxp += 8, rxdp += 8) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m128i vaddr4, vaddr5, vaddr6, vaddr7; - __m256i vaddr0_1, vaddr2_3; - __m256i vaddr4_5, vaddr6_7; - __m512i vaddr0_3, vaddr4_7; - - mb0 = rxp[0]; - mb1 = rxp[1]; - mb2 = rxp[2]; - mb3 = rxp[3]; - mb4 = rxp[4]; - mb5 = rxp[5]; - mb6 = rxp[6]; - mb7 = rxp[7]; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); - vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); - vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); - vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3, and so on. - */ - vaddr0_1 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), - vaddr1, 1); - vaddr2_3 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), - vaddr3, 1); - vaddr4_5 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr4), - vaddr5, 1); - vaddr6_7 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr6), - vaddr7, 1); - vaddr0_3 = - _mm512_inserti64x4(_mm512_castsi256_si512(vaddr0_1), - vaddr2_3, 1); - vaddr4_7 = - _mm512_inserti64x4(_mm512_castsi256_si512(vaddr4_5), - vaddr6_7, 1); - - /* convert pa to dma_addr hdr/data */ - dma_addr0_3 = _mm512_unpackhi_epi64(vaddr0_3, vaddr0_3); - dma_addr4_7 = _mm512_unpackhi_epi64(vaddr4_7, vaddr4_7); - - /* add headroom to pa values */ - dma_addr0_3 = _mm512_add_epi64(dma_addr0_3, hdr_room); - dma_addr4_7 = _mm512_add_epi64(dma_addr4_7, hdr_room); - - /* flush desc with pa dma_addr */ - _mm512_store_si512((__m512i *)&rxdp->read, dma_addr0_3); - _mm512_store_si512((__m512i *)&(rxdp + 4)->read, dma_addr4_7); - } - } else -#endif - { - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - __m256i dma_addr0_1, dma_addr2_3; - __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < IAVF_RXQ_REARM_THRESH; - i += 4, rxp += 4, rxdp += 4) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m256i vaddr0_1, vaddr2_3; - - mb0 = rxp[0]; - mb1 = rxp[1]; - mb2 = rxp[2]; - mb3 = rxp[3]; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3 - */ - vaddr0_1 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), - vaddr1, 1); - vaddr2_3 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), - vaddr3, 1); - - /* convert pa to dma_addr hdr/data */ - dma_addr0_1 = _mm256_unpackhi_epi64(vaddr0_1, vaddr0_1); - dma_addr2_3 = _mm256_unpackhi_epi64(vaddr2_3, vaddr2_3); - - /* add headroom to pa values */ - dma_addr0_1 = _mm256_add_epi64(dma_addr0_1, hdr_room); - dma_addr2_3 = _mm256_add_epi64(dma_addr2_3, hdr_room); - - /* flush desc with pa dma_addr */ - _mm256_store_si256((__m256i *)&rxdp->read, dma_addr0_1); - _mm256_store_si256((__m256i *)&(rxdp + 2)->read, dma_addr2_3); - } - } - -#endif rxq->rxrearm_start += IAVF_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c index 2e41079e88..9c1f8276d0 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c @@ -204,15 +204,9 @@ flex_rxd_to_fdir_flags_vec(const __m128i fdir_id0_3) return fdir_flags; } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC static inline void flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], __m128i descs_bh[4], struct rte_mbuf **rx_pkts) -#else -static inline void -flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], - struct rte_mbuf **rx_pkts) -#endif { const __m128i mbuf_init = _mm_set_epi64x(0, rxq->mbuf_initializer); __m128i rearm0, rearm1, rearm2, rearm3; @@ -325,7 +319,6 @@ flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], /* merge the flags */ flags = _mm_or_si128(flags, rss_vlan); -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (rxq->rx_flags & IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2) { const __m128i l2tag2_mask = _mm_set1_epi32(1 << IAVF_RX_FLEX_DESC_STATUS1_L2TAG2P_S); @@ -356,7 +349,6 @@ flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], /* merge with vlan_flags */ flags = _mm_or_si128(flags, vlan_flags); } -#endif if (rxq->fdir_enabled) { const __m128i fdir_id0_1 = @@ -388,10 +380,8 @@ flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], _mm_extract_epi32(fdir_id0_3, 3); } /* if() on fdir_enabled */ -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) flags = _mm_or_si128(flags, _mm_set1_epi32(iavf_timestamp_dynflag)); -#endif /** * At this point, we have the 4 sets of flags in the low 16-bits @@ -724,9 +714,7 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, int pos; uint64_t var; struct iavf_adapter *adapter = rxq->vsi->adapter; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; -#endif const uint32_t *ptype_tbl = adapter->ptype_tbl; __m128i crc_adjust = _mm_set_epi16 (0, 0, 0, /* ignore non-length fields */ @@ -796,7 +784,6 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, rte_cpu_to_le_32(1 << IAVF_RX_FLEX_DESC_STATUS0_DD_S))) return 0; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint8_t inflection_point = 0; bool is_tsinit = false; __m128i hw_low_last = _mm_set_epi32(0, 0, 0, (uint32_t)rxq->phc_time); @@ -812,8 +799,6 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, } } -#endif - /** * Compile-time verify the shuffle mask * NOTE: some field positions already verified above, but duplicated @@ -845,9 +830,7 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, pos += IAVF_VPMD_DESCS_PER_LOOP, rxdp += IAVF_VPMD_DESCS_PER_LOOP) { __m128i descs[IAVF_VPMD_DESCS_PER_LOOP]; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC __m128i descs_bh[IAVF_VPMD_DESCS_PER_LOOP] = {_mm_setzero_si128()}; -#endif __m128i pkt_mb0, pkt_mb1, pkt_mb2, pkt_mb3; __m128i staterr, sterr_tmp1, sterr_tmp2; /* 2 64 bit or 4 32 bit mbuf pointers in one XMM reg. */ @@ -914,7 +897,6 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, pkt_mb1 = _mm_add_epi16(pkt_mb1, crc_adjust); pkt_mb0 = _mm_add_epi16(pkt_mb0, crc_adjust); -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC /** * needs to load 2nd 16B of each desc, * will cause performance drop to get into this context. @@ -1076,9 +1058,6 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, } /* if() on Timestamp parsing */ flex_desc_to_olflags_v(rxq, descs, descs_bh, &rx_pkts[pos]); -#else - flex_desc_to_olflags_v(rxq, descs, &rx_pkts[pos]); -#endif /* C.2 get 4 pkts staterr value */ staterr = _mm_unpacklo_epi32(sterr_tmp1, sterr_tmp2); @@ -1121,7 +1100,6 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, var = rte_popcount64(_mm_cvtsi128_si64(staterr)); nb_pkts_recd += var; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { inflection_point = (inflection_point <= var) ? inflection_point : 0; switch (inflection_point) { @@ -1151,18 +1129,15 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); } -#endif if (likely(var != IAVF_VPMD_DESCS_PER_LOOP)) break; } -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC #ifdef IAVF_RX_TS_OFFLOAD if (nb_pkts_recd > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) rxq->phc_time = *RTE_MBUF_DYNFIELD(rx_pkts[nb_pkts_recd - 1], iavf_timestamp_dynfield_offset, uint32_t *); -#endif #endif /* Update our internal tail pointer */ diff --git a/drivers/net/intel/iavf/iavf_vchnl.c b/drivers/net/intel/iavf/iavf_vchnl.c index 6feca8435e..2302d2bcf1 100644 --- a/drivers/net/intel/iavf/iavf_vchnl.c +++ b/drivers/net/intel/iavf/iavf_vchnl.c @@ -1260,7 +1260,6 @@ iavf_configure_queues(struct iavf_adapter *adapter, vc_qp->rxq.dma_ring_addr = rxq[i]->rx_ring_phys_addr; vc_qp->rxq.databuffer_size = rxq[i]->rx_buf_len; vc_qp->rxq.crc_disable = rxq[i]->crc_len != 0 ? 1 : 0; -#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_RX_FLEX_DESC) { if (vf->supported_rxdid & RTE_BIT64(rxq[i]->rxdid)) { @@ -1279,19 +1278,6 @@ iavf_configure_queues(struct iavf_adapter *adapter, rxq[i]->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) vc_qp->rxq.flags |= VIRTCHNL_PTP_RX_TSTAMP; } -#else - if (vf->vf_res->vf_cap_flags & - VIRTCHNL_VF_OFFLOAD_RX_FLEX_DESC && - vf->supported_rxdid & BIT(IAVF_RXDID_LEGACY_0)) { - vc_qp->rxq.rxdid = IAVF_RXDID_LEGACY_0; - PMD_DRV_LOG(NOTICE, "request RXDID[%d] in Queue[%d]", - vc_qp->rxq.rxdid, i); - } else { - PMD_DRV_LOG(ERR, "RXDID[%d] is not supported", - IAVF_RXDID_LEGACY_0); - return -1; - } -#endif } memset(&args, 0, sizeof(args)); From patchwork Mon Jun 9 15:37:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154206 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A400E468B7; Mon, 9 Jun 2025 17:38:59 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3DB7142797; Mon, 9 Jun 2025 17:38:04 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 82EAE415D7 for ; Mon, 9 Jun 2025 17:38:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483482; x=1781019482; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e65+CW+oRr+mmfLVocqYXU65BN2lSvN45H3HYpJDGlw=; b=fcDYTWl5gKRC7GnBzjFpYC7c4GW3oo7PgI3EzAo5VqcD6CdbzVKmi1Ak RDSm8NlHQRt7h4Hj7Yw2cjDIhjv0QULcwZJRu9kjOrbXu97sLCq0PQ0eP GXwNunAqlFErX+M3ifaQ2xpxbMUjlu+RNzVtkHlR/4ZE3TG0I79+jt9hS IB/jqZDwFkjoLo38+cbobs+UZlgReBCi50xq+ZRS1YWlfMj9xaOhSa3Bi GB6UiNnXTVwvK9bdSTOXBDx/AeF2s5SzAsrFpzDduE3CVt7/5QlFS6nVN xd2l0jOcwFVrvwP3rOfXkh8V0gyzDlWBDk+4rEz+0EsI67hZ2YyHMOpOx Q==; X-CSE-ConnectionGUID: 7NxTfX77Q/m7nD4PsxzoRA== X-CSE-MsgGUID: DAMta4rNRkqs+56Ad+2mpg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012170" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012170" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:01 -0700 X-CSE-ConnectionGUID: ZoSQlxVqSyGsSiGafgb80g== X-CSE-MsgGUID: yxGEG2DSTVawQSr16XmDtg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419618" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:00 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 09/33] net/ixgbe: simplify packet type support check Date: Mon, 9 Jun 2025 16:37:07 +0100 Message-ID: <45137f4f58c1eb9e5fd3e5cc31ed802209d491ad.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There are no differences between scalar and vector paths when it comes to packet type support, and the only data path currently not covered by the check is the VF representor path, because it's not meant to be used directly anyway. Simplify the check to reflect that fact. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_ethdev.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_ethdev.c b/drivers/net/intel/ixgbe/ixgbe_ethdev.c index f1fd271a0a..928ac57a93 100644 --- a/drivers/net/intel/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/intel/ixgbe/ixgbe_ethdev.c @@ -4067,21 +4067,14 @@ ixgbe_dev_supported_ptypes_get(struct rte_eth_dev *dev, size_t *no_of_elements) RTE_PTYPE_INNER_L4_UDP, }; - if (dev->rx_pkt_burst == ixgbe_recv_pkts || - dev->rx_pkt_burst == ixgbe_recv_pkts_lro_single_alloc || - dev->rx_pkt_burst == ixgbe_recv_pkts_lro_bulk_alloc || - dev->rx_pkt_burst == ixgbe_recv_pkts_bulk_alloc) { + /* + * Currently, all Rx functions support all packet types, except for VF representor Rx + * function which has no data path and is not meant to be used directly. + */ + if (dev->rx_pkt_burst != NULL && dev->rx_pkt_burst != ixgbe_vf_representor_rx_burst) { *no_of_elements = RTE_DIM(ptypes); return ptypes; } - -#if defined(RTE_ARCH_X86) || defined(__ARM_NEON) - if (dev->rx_pkt_burst == ixgbe_recv_pkts_vec || - dev->rx_pkt_burst == ixgbe_recv_scattered_pkts_vec) { - *no_of_elements = RTE_DIM(ptypes); - return ptypes; - } -#endif return NULL; } From patchwork Mon Jun 9 15:37:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154207 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 366E1468B7; Mon, 9 Jun 2025 17:39:07 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 59FE942794; Mon, 9 Jun 2025 17:38:05 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id E80054278E for ; Mon, 9 Jun 2025 17:38:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483483; x=1781019483; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WkeOksPvEHN3GAJlCulS0FoxRGdAjYoRDWA3vbh8ZqE=; b=NlB5Ggm4bsHbQq2MuKvFaivqJuTsVq2mOXI6Vi+Enmpfd9iENbd5WlXO YlF33mwu++WzA804xicpH++jfSnn+RBvfKl1C8W8Nefv8lWtnrwbTXIzO i3tqsRpndxDTbxFsf7wSYqSteePEUs2Wy8jz9u232mv8Zhgd+xlPb1YjH cNSmeKmTNuDdkUBE+mKxHQ45jGRxxpfWjk2sk+JbIX6hAx286xbJN8kW3 o75pa7RO3/uuxdTYs+7Oj8etdCFcFnYq3GrhLwB/js6mBeP88jV98gx6S JuDobqA3ZYu7ND0Y+rfuXDdvZz3oiJdTIioN6XLY+1s5KZY5F57PapsQY g==; X-CSE-ConnectionGUID: xdi8ycptQvmo6fbUPiXqng== X-CSE-MsgGUID: JxG31dOlQGO2cT8KtdEVYA== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012173" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012173" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:03 -0700 X-CSE-ConnectionGUID: Zg3Mi2esQROyNGpcwIMwXA== X-CSE-MsgGUID: kjZdDMIDRWWTd7PvoCUGMg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419628" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:01 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 10/33] net/ixgbe: adjust indentation Date: Mon, 9 Jun 2025 16:37:08 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is wrong indentation in `ixgbe_set_tx_function`. Fix it. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_rxtx.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index f75821029d..8fbb795097 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -2682,8 +2682,9 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ci_tx_queue *txq) dev->recycle_tx_mbufs_reuse = ixgbe_recycle_tx_mbufs_reuse_vec; #endif dev->tx_pkt_burst = ixgbe_xmit_pkts_vec; - } else - dev->tx_pkt_burst = ixgbe_xmit_pkts_simple; + } else { + dev->tx_pkt_burst = ixgbe_xmit_pkts_simple; + } } else { PMD_INIT_LOG(DEBUG, "Using full-featured tx code path"); PMD_INIT_LOG(DEBUG, From patchwork Mon Jun 9 15:37:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154208 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AE32D468B7; Mon, 9 Jun 2025 17:39:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 56C924279E; Mon, 9 Jun 2025 17:38:06 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 70C2E42799 for ; Mon, 9 Jun 2025 17:38:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483485; x=1781019485; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fv0D3bnitWr6JsPlQH8ExnM9PO2C0SCuj6C+dQXxbSA=; b=dlMx5AbQV/Nl+fHa+8NxyGQRNk7sgFJUz3UtIPKisNO8av+rPqD466WG BWTSgRjmeepK9rHeU76FdAxC4BI1mMl+MGqHeEJr/BTb4zqawssz/8zal vqrGZ7Qr2V+sy7jNUyLf401wRKmcQtUSiTEsrval0c53EDLvsost8eb07 YDO+mO2z5o/8l5CPTcDQiYBGIj/lhnmoAUeyNZo1zRPQlT7rKIa0JntWF lZTheIUGgkeMaawYSp90GoE/HphWgAWuPTe+U6+V7MIAV0D6NArhXeNLq Yjzarpw04QhVKK5omPmCW1FDE6/TKAHkFgU36LOeIT9rXzLdcgh/clb4F g==; X-CSE-ConnectionGUID: 7a4ksUXXTXKt0GczwACG8w== X-CSE-MsgGUID: tuCyxJKTSyyyY12/zymaZQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012176" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012176" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:04 -0700 X-CSE-ConnectionGUID: mEHW9GrHSfujIeFp5PjkfA== X-CSE-MsgGUID: vdhR7Es3RROk39U6C0wPPQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419633" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:03 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 11/33] net/ixgbe: remove unnecessary platform checks Date: Mon, 9 Jun 2025 16:37:09 +0100 Message-ID: <303054ab51d9a1a14adc851d59657f97bb695e9e.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Some of the things that are only used by vector drivers is guarded by platform define checks, but there is actually no real reason to have those checks, because this code won't run anyway. Remove all of the unnecessary platform checks, and provide function stubs where necessary to make compilation work. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_rxtx.c | 26 +++++++++++++------------- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 4 ---- 2 files changed, 13 insertions(+), 17 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 8fbb795097..0777f70a4b 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -2678,9 +2678,7 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ci_tx_queue *txq) (rte_eal_process_type() != RTE_PROC_PRIMARY || ixgbe_txq_vec_setup(txq) == 0)) { PMD_INIT_LOG(DEBUG, "Vector tx enabled."); -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) dev->recycle_tx_mbufs_reuse = ixgbe_recycle_tx_mbufs_reuse_vec; -#endif dev->tx_pkt_burst = ixgbe_xmit_pkts_vec; } else { dev->tx_pkt_burst = ixgbe_xmit_pkts_simple; @@ -3112,11 +3110,8 @@ ixgbe_reset_rx_queue(struct ixgbe_adapter *adapter, struct ixgbe_rx_queue *rxq) rxq->pkt_first_seg = NULL; rxq->pkt_last_seg = NULL; - -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM64) rxq->rxrearm_start = 0; rxq->rxrearm_nb = 0; -#endif } static int @@ -3408,11 +3403,9 @@ ixgbe_dev_rx_descriptor_status(void *rx_queue, uint16_t offset) if (unlikely(offset >= rxq->nb_rx_desc)) return -EINVAL; -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM64) if (rxq->vector_rx) nb_hold = rxq->rxrearm_nb; else -#endif nb_hold = rxq->nb_rx_hold; if (offset >= rxq->nb_rx_desc - nb_hold) return RTE_ETH_RX_DESC_UNAVAIL; @@ -5050,10 +5043,8 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) PMD_INIT_LOG(DEBUG, "Using Vector Scattered Rx " "callback (port=%d).", dev->data->port_id); -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) dev->recycle_rx_descriptors_refill = ixgbe_recycle_rx_descriptors_refill_vec; -#endif dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec; } else if (adapter->rx_bulk_alloc_allowed) { PMD_INIT_LOG(DEBUG, "Using a Scattered with bulk " @@ -5082,9 +5073,7 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) "burst size no less than %d (port=%d).", RTE_IXGBE_DESCS_PER_LOOP, dev->data->port_id); -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) dev->recycle_rx_descriptors_refill = ixgbe_recycle_rx_descriptors_refill_vec; -#endif dev->rx_pkt_burst = ixgbe_recv_pkts_vec; } else if (adapter->rx_bulk_alloc_allowed) { PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions are " @@ -5872,10 +5861,8 @@ ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, recycle_rxq_info->receive_tail = &rxq->rx_tail; if (adapter->rx_vec_allowed) { -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) recycle_rxq_info->refill_requirement = RTE_IXGBE_RXQ_REARM_THRESH; recycle_rxq_info->refill_head = &rxq->rxrearm_start; -#endif } else { recycle_rxq_info->refill_requirement = rxq->rx_free_thresh; recycle_rxq_info->refill_head = &rxq->rx_free_trigger; @@ -6251,6 +6238,19 @@ ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev __rte_unused *dev) return -1; } +void +ixgbe_recycle_rx_descriptors_refill_vec(void __rte_unused * rx_queue, + uint16_t __rte_unused nb_mbufs) +{ +} + +uint16_t +ixgbe_recycle_tx_mbufs_reuse_vec(void __rte_unused * tx_queue, + struct rte_eth_recycle_rxq_info __rte_unused * recycle_rxq_info) +{ + return 0; +} + uint16_t ixgbe_recv_pkts_vec( void __rte_unused *rx_queue, diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index c86714804f..bcd5db87e8 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -35,10 +35,8 @@ #define RTE_IXGBE_DESCS_PER_LOOP 4 -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) #define RTE_IXGBE_RXQ_REARM_THRESH 32 #define RTE_IXGBE_MAX_RX_BURST RTE_IXGBE_RXQ_REARM_THRESH -#endif #define RX_RING_SZ ((IXGBE_MAX_RING_DESC + RTE_PMD_IXGBE_RX_MAX_BURST) * \ sizeof(union ixgbe_adv_rx_desc)) @@ -102,10 +100,8 @@ struct ixgbe_rx_queue { uint8_t using_ipsec; /**< indicates that IPsec RX feature is in use */ #endif -#if defined(RTE_ARCH_X86) || defined(RTE_ARCH_ARM) uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ uint16_t rxrearm_start; /**< the idx we start the re-arming from */ -#endif uint16_t rx_free_thresh; /**< max free RX desc to hold. */ uint16_t queue_id; /**< RX queue index. */ uint16_t reg_idx; /**< RX queue register index. */ From patchwork Mon Jun 9 15:37:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154209 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D8E2C468B7; Mon, 9 Jun 2025 17:39:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 201F1427B1; Mon, 9 Jun 2025 17:38:08 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id CE3754279A for ; Mon, 9 Jun 2025 17:38:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483486; x=1781019486; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TTMLDqKqRhp4/hssFeJAMZsGfC8mQkjWRlun1ysqRzI=; b=XuYjexsoG4zhkZO7bipXs1u0mgmnnIRipHYGtTR4sk7DDVhAxqmpKCRg tRXo6EB1m5krEa57IFJdwydiB3FTihEpX+JjrwmqML8ug2GY5wi3JbwDX eMp6mkAgDDGuS0ruHZmEGE4rlq+gkOd5/7f8pi1ZRo7goRpSTNSfR9jsm otZMBBN5JaTcLhuZp8Cy4A1aAK6RTt4BtLI6zrxgV89RrIv16f6xd1pJz Mtc3tabr79BT+o9QEdKGyZs3GwKrCEi0AyTDqm7Oud6LP90GbrsLFgUGY x8luNVdGc+qPMjnLj9qaz2CS0dGY4SGpuXA/8moj48HOqazSkHpV7rn4C Q==; X-CSE-ConnectionGUID: m11tmkuBTcSIHnkM/8PPFQ== X-CSE-MsgGUID: KCKrjEE8T4yYymRZl2PKLg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012179" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012179" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:06 -0700 X-CSE-ConnectionGUID: 08Z2prqXQUOeMTFVDL51ag== X-CSE-MsgGUID: cNL9amCSTVOMwAeZoGBIAQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419644" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:04 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 12/33] net/ixgbe: decouple scalar and vec rxq free mbufs Date: Mon, 9 Jun 2025 16:37:10 +0100 Message-ID: <738618797c9573e2946b20e87abcbefd4fcdc684.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, vector Rx queue release mbufs function is only called from inside the scalar variant. Decouple them to allow both to be defined separately from each other, and provide a common function that picks between the two when necessary. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5 -> v6: - Rename functions to _vec and _non_vec, and keep common name as is v5: - Add this commit drivers/net/intel/ixgbe/ixgbe_rxtx.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 0777f70a4b..0b949c3cfc 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -2967,16 +2967,10 @@ ixgbe_free_sc_cluster(struct rte_mbuf *m) } static void __rte_cold -ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq) +ixgbe_rx_queue_release_mbufs_non_vec(struct ixgbe_rx_queue *rxq) { unsigned i; - /* SSE Vector driver has a different way of releasing mbufs. */ - if (rxq->vector_rx) { - ixgbe_rx_queue_release_mbufs_vec(rxq); - return; - } - if (rxq->sw_ring != NULL) { for (i = 0; i < rxq->nb_rx_desc; i++) { if (rxq->sw_ring[i].mbuf != NULL) { @@ -3003,6 +2997,15 @@ ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq) } } +static void __rte_cold +ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq) +{ + if (rxq->vector_rx) + ixgbe_rx_queue_release_mbufs_vec(rxq); + else + ixgbe_rx_queue_release_mbufs_non_vec(rxq); +} + static void __rte_cold ixgbe_rx_queue_release(struct ixgbe_rx_queue *rxq) { From patchwork Mon Jun 9 15:37:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154210 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3B8BE468B7; Mon, 9 Jun 2025 17:39:30 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0AB80427B6; Mon, 9 Jun 2025 17:38:09 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 187A2427A1 for ; Mon, 9 Jun 2025 17:38:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483487; x=1781019487; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zR2iShE9V074Wy0A2hrbtfLO8z6qRHJT4gU7Qe2ZqOE=; b=Rt9vbzlzuGnVUidYm7VZ0Vff93A9rC4Oyl+FJQGAJ0p9ovqF9AHEo0Ce /c4TMaixzQntgSVwphCsnhblVYpvWki1+MPFMQEaToUHu7vVfKVIYP4on /65yZPzClKkRe4otCSYGSJJ5difoNvsO4jvq5eVNMz5zrW7v36vUE4sgY fgBGpBhcAyyt32VeS0zNrOLDSk7q8FDojDL/3cBXsNF4jtaQjM7ncx/Cy hDqS/ZIe2knEah3bIVTI4cipEUgZnbccUdu4M1JypZ2Zh0i2eNzRYC8F+ 9naVp7ZIU+H4Z1DHxRa86jin2hIxAuzrazRD6wHtHF5NLWDuJMIABniGZ g==; X-CSE-ConnectionGUID: u7j8OrQCSyGD06kwLxsSBw== X-CSE-MsgGUID: 3sDKc66yTw+It18J4WMetw== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012183" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012183" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:07 -0700 X-CSE-ConnectionGUID: 4rmphkBEQO6yxYbxDzrQtQ== X-CSE-MsgGUID: tdouAKq7S72blDAkz0IJTw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419651" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:05 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 13/33] net/ixgbe: rename vector txq free mbufs Date: Mon, 9 Jun 2025 16:37:11 +0100 Message-ID: <8b66b094e3a32e39c031c1bdeb7d0ca7808c82ce.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is no reason why scalar and vector implementations of tx free mbufs function have to share the same name, in fact it is counter-productive because including `ixgbe_rxtx_vec_common.h` from `ixgbe_rxtx.c` will cause naming clashes. Rename the vector implementation to avoid naming clashes. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h | 2 +- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c | 2 +- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h index 018010820f..9e1abf4449 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h @@ -12,7 +12,7 @@ #include "ixgbe_rxtx.h" static __rte_always_inline int -ixgbe_tx_free_bufs(struct ci_tx_queue *txq) +ixgbe_tx_free_bufs_vec(struct ci_tx_queue *txq) { struct ci_tx_entry_vec *txep; uint32_t status; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c index f8916d44e8..3a0b2909a7 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c @@ -583,7 +583,7 @@ ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, nb_pkts = RTE_MIN(nb_pkts, txq->tx_rs_thresh); if (txq->nb_tx_free < txq->tx_free_thresh) - ixgbe_tx_free_bufs(txq); + ixgbe_tx_free_bufs_vec(txq); nb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c index 9417e5b11f..1e063bb243 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c @@ -703,7 +703,7 @@ ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, nb_pkts = RTE_MIN(nb_pkts, txq->tx_rs_thresh); if (txq->nb_tx_free < txq->tx_free_thresh) - ixgbe_tx_free_bufs(txq); + ixgbe_tx_free_bufs_vec(txq); nb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) From patchwork Mon Jun 9 15:37:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154211 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2CEAE468B7; Mon, 9 Jun 2025 17:39:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ED5DD427B4; Mon, 9 Jun 2025 17:38:10 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id CD9C0427B4 for ; Mon, 9 Jun 2025 17:38:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483489; x=1781019489; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4L235uJPuEV4nHW42nFCZ4UE0H7mvvksDbtLvrEjYvg=; b=b3YBdQsWW+OFBXAVHvdHAZH0pLS7cM9Y5NFkg+H4io4ATLIkU7z+9cht sfccDIN095Gosm33VOVkBsuo30UWjuwJk8iHXjiDoRpyGZNzZBzcqoxmv xILY5nEfeDDYcjsj0rFINYZZIxZ4mJjaytdGWFclqT3UYReCN6hyOmAfc 2mKJPOjM+1AjFuV8g3atubKM7h5Prziby7YmS66Bz+AXxO8e2e0F0bxCf NV2OQ+QfuUFQEN5SLLYdnMSGAEEFlMp+vVBHM05Q2tydMxaSrHkot4IPL AMPeWss5Wobf2yeYBDVMw0a4AImBw5/rkv47g544/gViBWbVMJybRAVRf Q==; X-CSE-ConnectionGUID: WeI+QfCtS0q56Iil9RmOvw== X-CSE-MsgGUID: 33JnVKYKSFuIDsqDFptgUQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012187" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012187" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:09 -0700 X-CSE-ConnectionGUID: w8/Cf5kwRAO5rz3ylhjKlA== X-CSE-MsgGUID: +oJLipgoTsKXakWadrbUlg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419660" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:07 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 14/33] net/ixgbe: refactor vector common code Date: Mon, 9 Jun 2025 16:37:12 +0100 Message-ID: <092f5c8de0b241346e02e55210be8a87e28c186e.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Each vector driver provides its own Rx queue setup etc. functions, but actually they're entirely identical, and can be merged. Rename the `ixgbe_recycle_mbufs_vec_common.c` to `ixgbe_rxtx_vec_common.c` and move all common code there from each respective vector driver. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_rxtx.c | 2 + drivers/net/intel/ixgbe/ixgbe_rxtx.h | 4 - ...s_vec_common.c => ixgbe_rxtx_vec_common.c} | 137 +++++++++++++++++- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.h | 127 ++-------------- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c | 42 ------ drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c | 42 ------ drivers/net/intel/ixgbe/meson.build | 4 +- 7 files changed, 148 insertions(+), 210 deletions(-) rename drivers/net/intel/ixgbe/{ixgbe_recycle_mbufs_vec_common.c => ixgbe_rxtx_vec_common.c} (56%) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 0b949c3cfc..79b3d4b71f 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -52,6 +52,8 @@ #include "base/ixgbe_common.h" #include "ixgbe_rxtx.h" +#include "ixgbe_rxtx_vec_common.h" + #ifdef RTE_LIBRTE_IEEE1588 #define IXGBE_TX_IEEE1588_TMST RTE_MBUF_F_TX_IEEE1588_TMST #else diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index bcd5db87e8..cd0015be9c 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -225,9 +225,6 @@ uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); uint16_t ixgbe_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); -int ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev); -int ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq); -void ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq); int ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt); extern const uint32_t ptype_table[IXGBE_PACKET_TYPE_MAX]; @@ -239,7 +236,6 @@ void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs); uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); -int ixgbe_txq_vec_setup(struct ci_tx_queue *txq); uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev); uint64_t ixgbe_get_rx_queue_offloads(struct rte_eth_dev *dev); diff --git a/drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c similarity index 56% rename from drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c rename to drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c index 2ab7abbf4e..be422ee238 100644 --- a/drivers/net/intel/ixgbe/ixgbe_recycle_mbufs_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c @@ -1,12 +1,143 @@ /* SPDX-License-Identifier: BSD-3-Clause * Copyright (c) 2023 Arm Limited. + * Copyright (c) 2025 Intel Corporation */ -#include -#include +#include +#include +#include -#include "ixgbe_ethdev.h" #include "ixgbe_rxtx.h" +#include "ixgbe_rxtx_vec_common.h" + +void __rte_cold +ixgbe_tx_free_swring_vec(struct ci_tx_queue *txq) +{ + if (txq == NULL) + return; + + if (txq->sw_ring != NULL) { + rte_free(txq->sw_ring_vec - 1); + txq->sw_ring_vec = NULL; + } +} + +void __rte_cold +ixgbe_reset_tx_queue_vec(struct ci_tx_queue *txq) +{ + static const union ixgbe_adv_tx_desc zeroed_desc = { { 0 } }; + struct ci_tx_entry_vec *txe = txq->sw_ring_vec; + uint16_t i; + + /* Zero out HW ring memory */ + for (i = 0; i < txq->nb_tx_desc; i++) + txq->ixgbe_tx_ring[i] = zeroed_desc; + + /* Initialize SW ring entries */ + for (i = 0; i < txq->nb_tx_desc; i++) { + volatile union ixgbe_adv_tx_desc *txd = &txq->ixgbe_tx_ring[i]; + + txd->wb.status = IXGBE_TXD_STAT_DD; + txe[i].mbuf = NULL; + } + + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1); + + txq->tx_tail = 0; + txq->nb_tx_used = 0; + /* + * Always allow 1 descriptor to be un-allocated to avoid + * a H/W race condition + */ + txq->last_desc_cleaned = (uint16_t)(txq->nb_tx_desc - 1); + txq->nb_tx_free = (uint16_t)(txq->nb_tx_desc - 1); + txq->ctx_curr = 0; + memset(txq->ctx_cache, 0, IXGBE_CTX_NUM * sizeof(struct ixgbe_advctx_info)); + + /* for PF, we do not need to initialize the context descriptor */ + if (!txq->is_vf) + txq->vf_ctx_initialized = 1; +} + +void __rte_cold +ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq) +{ + unsigned int i; + + if (rxq->sw_ring == NULL || rxq->rxrearm_nb >= rxq->nb_rx_desc) + return; + + /* free all mbufs that are valid in the ring */ + if (rxq->rxrearm_nb == 0) { + for (i = 0; i < rxq->nb_rx_desc; i++) { + if (rxq->sw_ring[i].mbuf != NULL) + rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); + } + } else { + for (i = rxq->rx_tail; + i != rxq->rxrearm_start; + i = (i + 1) % rxq->nb_rx_desc) { + if (rxq->sw_ring[i].mbuf != NULL) + rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); + } + } + + rxq->rxrearm_nb = rxq->nb_rx_desc; + + /* set all entries to NULL */ + memset(rxq->sw_ring, 0, sizeof(rxq->sw_ring[0]) * rxq->nb_rx_desc); +} + +int __rte_cold +ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq) +{ + rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); + return 0; +} + +static const struct ixgbe_txq_ops vec_txq_ops = { + .free_swring = ixgbe_tx_free_swring_vec, + .reset = ixgbe_reset_tx_queue_vec, +}; + +int __rte_cold +ixgbe_txq_vec_setup(struct ci_tx_queue *txq) +{ + if (txq->sw_ring_vec == NULL) + return -1; + + /* leave the first one for overflow */ + txq->sw_ring_vec = txq->sw_ring_vec + 1; + txq->ops = &vec_txq_ops; + txq->vector_tx = 1; + + return 0; +} + +int __rte_cold +ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev) +{ +#ifndef RTE_LIBRTE_IEEE1588 + struct rte_eth_fdir_conf *fconf = IXGBE_DEV_FDIR_CONF(dev); + + /* no fdir support */ + if (fconf->mode != RTE_FDIR_MODE_NONE) + return -1; + + for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) { + struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; + if (!rxq) + continue; + if (!ci_rxq_vec_capable(rxq->nb_rx_desc, rxq->rx_free_thresh, rxq->offloads)) + return -1; + } + return 0; +#else + RTE_SET_USED(dev); + return -1; +#endif +} void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h index 9e1abf4449..d5a051e024 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h @@ -11,6 +11,16 @@ #include "ixgbe_ethdev.h" #include "ixgbe_rxtx.h" +int ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev); +int ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq); +int ixgbe_txq_vec_setup(struct ci_tx_queue *txq); +void ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq); +void ixgbe_reset_tx_queue_vec(struct ci_tx_queue *txq); +void ixgbe_tx_free_swring_vec(struct ci_tx_queue *txq); +void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs); +uint16_t ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, + struct rte_eth_recycle_rxq_info *recycle_rxq_info); + static __rte_always_inline int ixgbe_tx_free_bufs_vec(struct ci_tx_queue *txq) { @@ -68,121 +78,4 @@ ixgbe_tx_free_bufs_vec(struct ci_tx_queue *txq) return txq->tx_rs_thresh; } -static inline void -_ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq) -{ - unsigned int i; - - if (rxq->sw_ring == NULL || rxq->rxrearm_nb >= rxq->nb_rx_desc) - return; - - /* free all mbufs that are valid in the ring */ - if (rxq->rxrearm_nb == 0) { - for (i = 0; i < rxq->nb_rx_desc; i++) { - if (rxq->sw_ring[i].mbuf != NULL) - rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); - } - } else { - for (i = rxq->rx_tail; - i != rxq->rxrearm_start; - i = (i + 1) % rxq->nb_rx_desc) { - if (rxq->sw_ring[i].mbuf != NULL) - rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); - } - } - - rxq->rxrearm_nb = rxq->nb_rx_desc; - - /* set all entries to NULL */ - memset(rxq->sw_ring, 0, sizeof(rxq->sw_ring[0]) * rxq->nb_rx_desc); -} - -static inline void -_ixgbe_tx_free_swring_vec(struct ci_tx_queue *txq) -{ - if (txq == NULL) - return; - - if (txq->sw_ring != NULL) { - rte_free(txq->sw_ring_vec - 1); - txq->sw_ring_vec = NULL; - } -} - -static inline void -_ixgbe_reset_tx_queue_vec(struct ci_tx_queue *txq) -{ - static const union ixgbe_adv_tx_desc zeroed_desc = { { 0 } }; - struct ci_tx_entry_vec *txe = txq->sw_ring_vec; - uint16_t i; - - /* Zero out HW ring memory */ - for (i = 0; i < txq->nb_tx_desc; i++) - txq->ixgbe_tx_ring[i] = zeroed_desc; - - /* Initialize SW ring entries */ - for (i = 0; i < txq->nb_tx_desc; i++) { - volatile union ixgbe_adv_tx_desc *txd = &txq->ixgbe_tx_ring[i]; - - txd->wb.status = IXGBE_TXD_STAT_DD; - txe[i].mbuf = NULL; - } - - txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); - txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1); - - txq->tx_tail = 0; - txq->nb_tx_used = 0; - /* - * Always allow 1 descriptor to be un-allocated to avoid - * a H/W race condition - */ - txq->last_desc_cleaned = (uint16_t)(txq->nb_tx_desc - 1); - txq->nb_tx_free = (uint16_t)(txq->nb_tx_desc - 1); - txq->ctx_curr = 0; - memset(txq->ctx_cache, 0, IXGBE_CTX_NUM * sizeof(struct ixgbe_advctx_info)); - - /* for PF, we do not need to initialize the context descriptor */ - if (!txq->is_vf) - txq->vf_ctx_initialized = 1; -} - -static inline int -ixgbe_txq_vec_setup_default(struct ci_tx_queue *txq, - const struct ixgbe_txq_ops *txq_ops) -{ - if (txq->sw_ring_vec == NULL) - return -1; - - /* leave the first one for overflow */ - txq->sw_ring_vec = txq->sw_ring_vec + 1; - txq->ops = txq_ops; - txq->vector_tx = 1; - - return 0; -} - -static inline int -ixgbe_rx_vec_dev_conf_condition_check_default(struct rte_eth_dev *dev) -{ -#ifndef RTE_LIBRTE_IEEE1588 - struct rte_eth_fdir_conf *fconf = IXGBE_DEV_FDIR_CONF(dev); - - /* no fdir support */ - if (fconf->mode != RTE_FDIR_MODE_NONE) - return -1; - - for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) { - struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; - if (!rxq) - continue; - if (!ci_rxq_vec_capable(rxq->nb_rx_desc, rxq->rx_free_thresh, rxq->offloads)) - return -1; - } - return 0; -#else - RTE_SET_USED(dev); - return -1; -#endif -} #endif diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c index 3a0b2909a7..ba213ccc67 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c @@ -632,45 +632,3 @@ ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, return nb_pkts; } - -void __rte_cold -ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq) -{ - _ixgbe_rx_queue_release_mbufs_vec(rxq); -} - -static void __rte_cold -ixgbe_tx_free_swring(struct ci_tx_queue *txq) -{ - _ixgbe_tx_free_swring_vec(txq); -} - -static void __rte_cold -ixgbe_reset_tx_queue(struct ci_tx_queue *txq) -{ - _ixgbe_reset_tx_queue_vec(txq); -} - -static const struct ixgbe_txq_ops vec_txq_ops = { - .free_swring = ixgbe_tx_free_swring, - .reset = ixgbe_reset_tx_queue, -}; - -int __rte_cold -ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq) -{ - rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); - return 0; -} - -int __rte_cold -ixgbe_txq_vec_setup(struct ci_tx_queue *txq) -{ - return ixgbe_txq_vec_setup_default(txq, &vec_txq_ops); -} - -int __rte_cold -ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev) -{ - return ixgbe_rx_vec_dev_conf_condition_check_default(dev); -} diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c index 1e063bb243..e1516a943d 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c @@ -753,45 +753,3 @@ ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, return nb_pkts; } - -void __rte_cold -ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq) -{ - _ixgbe_rx_queue_release_mbufs_vec(rxq); -} - -static void __rte_cold -ixgbe_tx_free_swring(struct ci_tx_queue *txq) -{ - _ixgbe_tx_free_swring_vec(txq); -} - -static void __rte_cold -ixgbe_reset_tx_queue(struct ci_tx_queue *txq) -{ - _ixgbe_reset_tx_queue_vec(txq); -} - -static const struct ixgbe_txq_ops vec_txq_ops = { - .free_swring = ixgbe_tx_free_swring, - .reset = ixgbe_reset_tx_queue, -}; - -int __rte_cold -ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq) -{ - rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); - return 0; -} - -int __rte_cold -ixgbe_txq_vec_setup(struct ci_tx_queue *txq) -{ - return ixgbe_txq_vec_setup_default(txq, &vec_txq_ops); -} - -int __rte_cold -ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev) -{ - return ixgbe_rx_vec_dev_conf_condition_check_default(dev); -} diff --git a/drivers/net/intel/ixgbe/meson.build b/drivers/net/intel/ixgbe/meson.build index d1122bb9cd..e6f0fd135e 100644 --- a/drivers/net/intel/ixgbe/meson.build +++ b/drivers/net/intel/ixgbe/meson.build @@ -24,11 +24,11 @@ testpmd_sources = files('ixgbe_testpmd.c') deps += ['hash', 'security'] if arch_subdir == 'x86' + sources += files('ixgbe_rxtx_vec_common.c') sources += files('ixgbe_rxtx_vec_sse.c') - sources += files('ixgbe_recycle_mbufs_vec_common.c') elif arch_subdir == 'arm' + sources += files('ixgbe_rxtx_vec_common.c') sources += files('ixgbe_rxtx_vec_neon.c') - sources += files('ixgbe_recycle_mbufs_vec_common.c') endif headers = files('rte_pmd_ixgbe.h') From patchwork Mon Jun 9 15:37:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154212 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0DF05468B7; Mon, 9 Jun 2025 17:39:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 83E00427BD; Mon, 9 Jun 2025 17:38:12 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 38502415D7 for ; Mon, 9 Jun 2025 17:38:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483491; x=1781019491; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vq1wX5uxxhomOCJ2NVMr0Uw+zyNs1ZaxfAYybeXuPG4=; b=lwhv/1ks666KSxRdBSZ7v9jpgrYE4a6Ay3xv51WlVHq8/+zl7JBThX3j PkOrp2OzwFptQi0cb/mViGeuK9y3vbFhy+NEVsWJ/sYrcKpPrptdzu/JC wTpGJB0Ha1rCDPuXRzry5VveEBQHFpsIUg6GfFUevzjmjZifOFhE3wJRP A0LA/L2U5PIogUryDdnL+zUuX9F3NjuwBnuMLGLSHwjI7qTA3WFCtkX7o PnzaHx03c9kQUFo480jqNG0Abm4iRx1P/tyPaXwcz3Y1GYku1lMaMxrKO eJQ9RlX/aunHIfuUFf3vpWDpJjtOwk5OQFg4iNY5fvLVdtsq2FRyYf8JJ g==; X-CSE-ConnectionGUID: ByegHVawSTitvrv+xCJ7iA== X-CSE-MsgGUID: HtBrfb52RyG3nnm33VsfLg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012191" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012191" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:10 -0700 X-CSE-ConnectionGUID: gIV5k/K6SVqfifuk9W9DNQ== X-CSE-MsgGUID: 6I9s1iwGTTGYTtr9Db/Zng== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419666" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:08 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 15/33] net/ixgbe: move vector Rx/Tx code to vec common Date: Mon, 9 Jun 2025 16:37:13 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is no reason why bits and pieces of vectorized code should be defined in `ixgbe_rxtx.c`, so move them to the vec common file. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_rxtx.c | 41 ++++--------------- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 6 +-- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.c | 31 ++++++++++++++ .../net/intel/ixgbe/ixgbe_rxtx_vec_common.h | 4 ++ 4 files changed, 45 insertions(+), 37 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 79b3d4b71f..ace21396f8 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -91,7 +91,6 @@ /* forward-declare some functions */ static int ixgbe_is_vf(struct rte_eth_dev *dev); -static int ixgbe_write_default_ctx_desc(struct ci_tx_queue *txq, struct rte_mempool *mp, bool vec); /********************************************************************* * @@ -361,37 +360,6 @@ ixgbe_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_pkts, return nb_tx; } -static uint16_t -ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts) -{ - uint16_t nb_tx = 0; - struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; - - /* we might check first packet's mempool */ - if (unlikely(nb_pkts == 0)) - return nb_pkts; - - /* check if we need to initialize default context descriptor */ - if (unlikely(!txq->vf_ctx_initialized) && - ixgbe_write_default_ctx_desc(txq, tx_pkts[0]->pool, true)) - return 0; - - while (nb_pkts) { - uint16_t ret, num; - - num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh); - ret = ixgbe_xmit_fixed_burst_vec(tx_queue, &tx_pkts[nb_tx], - num); - nb_tx += ret; - nb_pkts -= ret; - if (ret < num) - break; - } - - return nb_tx; -} - static inline void ixgbe_set_xmit_ctx(struct ci_tx_queue *txq, volatile struct ixgbe_adv_tx_context_desc *ctx_txd, @@ -2376,7 +2344,7 @@ ixgbe_recv_pkts_lro_bulk_alloc(void *rx_queue, struct rte_mbuf **rx_pkts, * **********************************************************************/ -static inline int +int ixgbe_write_default_ctx_desc(struct ci_tx_queue *txq, struct rte_mempool *mp, bool vec) { volatile struct ixgbe_adv_tx_context_desc *ctx_txd; @@ -6280,6 +6248,13 @@ ixgbe_rxq_vec_setup(struct ixgbe_rx_queue __rte_unused *rxq) return -1; } +uint16_t +ixgbe_xmit_pkts_vec(void __rte_unused * tx_queue, struct rte_mbuf __rte_unused * *tx_pkts, + __rte_unused uint16_t nb_pkts) +{ + return 0; +} + uint16_t ixgbe_xmit_fixed_burst_vec(void __rte_unused *tx_queue, struct rte_mbuf __rte_unused **tx_pkts, diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index cd0015be9c..6fcc5ee1e6 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -221,21 +221,19 @@ int ixgbe_rx_burst_mode_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_burst_mode *mode); int ixgbe_check_supported_loopback_mode(struct rte_eth_dev *dev); -uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, - uint16_t nb_pkts); -uint16_t ixgbe_recv_scattered_pkts_vec(void *rx_queue, - struct rte_mbuf **rx_pkts, uint16_t nb_pkts); int ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt); extern const uint32_t ptype_table[IXGBE_PACKET_TYPE_MAX]; extern const uint32_t ptype_table_tn[IXGBE_PACKET_TYPE_TN_MAX]; +int ixgbe_write_default_ctx_desc(struct ci_tx_queue *txq, struct rte_mempool *mp, bool vec); uint16_t ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, struct rte_eth_recycle_rxq_info *recycle_rxq_info); void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs); uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); +int ixgbe_write_default_ctx_desc(struct ci_tx_queue *txq, struct rte_mempool *mp, bool vec); uint64_t ixgbe_get_tx_port_offloads(struct rte_eth_dev *dev); uint64_t ixgbe_get_rx_queue_offloads(struct rte_eth_dev *dev); diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c index be422ee238..cf6d3e4914 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c @@ -139,6 +139,37 @@ ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev) #endif } +uint16_t +ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts) +{ + uint16_t nb_tx = 0; + struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; + + /* we might check first packet's mempool */ + if (unlikely(nb_pkts == 0)) + return nb_pkts; + + /* check if we need to initialize default context descriptor */ + if (unlikely(!txq->vf_ctx_initialized) && + ixgbe_write_default_ctx_desc(txq, tx_pkts[0]->pool, true)) + return 0; + + while (nb_pkts) { + uint16_t ret, num; + + num = (uint16_t)RTE_MIN(nb_pkts, txq->tx_rs_thresh); + ret = ixgbe_xmit_fixed_burst_vec(tx_queue, &tx_pkts[nb_tx], + num); + nb_tx += ret; + nb_pkts -= ret; + if (ret < num) + break; + } + + return nb_tx; +} + void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) { diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h index d5a051e024..4678a5dfd9 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h @@ -17,6 +17,10 @@ int ixgbe_txq_vec_setup(struct ci_tx_queue *txq); void ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq); void ixgbe_reset_tx_queue_vec(struct ci_tx_queue *txq); void ixgbe_tx_free_swring_vec(struct ci_tx_queue *txq); +uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); +uint16_t ixgbe_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); +uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); +uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs); uint16_t ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, struct rte_eth_recycle_rxq_info *recycle_rxq_info); From patchwork Mon Jun 9 15:37:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154213 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B9C3F468B7; Mon, 9 Jun 2025 17:39:51 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A1960427C3; Mon, 9 Jun 2025 17:38:13 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 94D15427A1 for ; Mon, 9 Jun 2025 17:38:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483492; x=1781019492; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2A0rzHo/QornRD4+VH7Cg6KgMxiVklTDC9ccPTktPUA=; b=Lfgrk3Y3eafmvPyPLOAUzCAnWQJ9Vdba/ELh/gf6wT5FJnJzKkLyLhr4 1wW/uXVUOw9f+adu173D1N4TukQ0qSogd0eRo2jsj6hZrP5jONCNTKXOp 2rH2dOQT6geZJ+I24fGtUZqTy2FpLnSJYlftTFB7/QSdOd/9z8WSF7XbR M4qTuyewZ11QzhEfz+Imrpv99QMfJwFpVbUBjyBwHaw88QvX/GEu7lUGs DWAMo4gXrqytbFtysxZJ1wy3KGlwH1IJpphSveX5shWqcN4lWN7CEYaeS 7ton21ukU5BT1SGyhPjB7ruF5Ph2nuuyWLVBSD9JdtOuwBD/uqrSqZDMr w==; X-CSE-ConnectionGUID: CYm+utrDTd6G2ROS6CY0Vg== X-CSE-MsgGUID: UGVg/AtYR0aLIEo4pITbXQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012196" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012196" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:11 -0700 X-CSE-ConnectionGUID: 9JUatYyfSQq+dRmzeA1Y2A== X-CSE-MsgGUID: rfS6/f7TTd6blJ3FTHX3Pg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419675" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:10 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 16/33] net/ixgbe: simplify vector PMD compilation Date: Mon, 9 Jun 2025 16:37:14 +0100 Message-ID: <416c762b57ba5b46a6c3b90afce848cfd9d3c3b3.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, vector code is guarded by platform checks, and unused functions are implemented using stubs. Simplify things the following way: - Define a compilation flag `IXGBE_VPMD_SUPPORTED` that will enable or disable vector code support regardless of platform - Wrap platform checks with said definition - Remove all stubs and replace them with macros that alias either to existing scalar implementations, or NULL Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Add this patch drivers/net/intel/ixgbe/ixgbe_rxtx.c | 96 ++++++---------------------- drivers/net/intel/ixgbe/meson.build | 2 + 2 files changed, 22 insertions(+), 76 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index ace21396f8..48675ab964 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -52,7 +52,23 @@ #include "base/ixgbe_common.h" #include "ixgbe_rxtx.h" +#ifdef IXGBE_VPMD_SUPPORTED #include "ixgbe_rxtx_vec_common.h" +#else +/* alias unsupported Rx/Tx vector functions to scalar implementations */ +#define ixgbe_recv_pkts_vec ixgbe_recv_pkts +#define ixgbe_recv_scattered_pkts_vec ixgbe_recv_pkts_lro_single_alloc +#define ixgbe_xmit_pkts_vec ixgbe_xmit_pkts_simple +/* ensure all vector checks/setup always fail */ +#define ixgbe_rx_vec_dev_conf_condition_check(unused) (RTE_SET_USED(unused), -1) +#define ixgbe_rxq_vec_setup(unused) RTE_SET_USED(unused) +#define ixgbe_txq_vec_setup(unused) (RTE_SET_USED(unused), -1) +/* use scalar mbuf release function */ +#define ixgbe_rx_queue_release_mbufs_vec ixgbe_rx_queue_release_mbufs_non_vec +/* these are not applicable to scalar paths */ +#define ixgbe_recycle_rx_descriptors_refill_vec NULL +#define ixgbe_recycle_tx_mbufs_reuse_vec NULL +#endif #ifdef RTE_LIBRTE_IEEE1588 #define IXGBE_TX_IEEE1588_TMST RTE_MBUF_F_TX_IEEE1588_TMST @@ -2602,11 +2618,13 @@ static const struct { { ixgbe_xmit_pkts, "Scalar"}, { ixgbe_xmit_pkts_simple, "Scalar simple"}, { ixgbe_vf_representor_tx_burst, "Scalar representor"}, +#ifdef IXGBE_VPMD_SUPPORTED #ifdef RTE_ARCH_X86 { ixgbe_xmit_pkts_vec, "Vector SSE"}, #elif defined(RTE_ARCH_ARM) { ixgbe_xmit_pkts_vec, "Vector NEON"}, #endif +#endif }; int @@ -4942,6 +4960,7 @@ static const struct { { ixgbe_recv_pkts_lro_bulk_alloc, "Scalar LRO bulk alloc"}, { ixgbe_recv_pkts_lro_single_alloc, "Scalar LRO single alloc"}, { ixgbe_vf_representor_rx_burst, "Scalar representor"}, +#ifdef IXGBE_VPMD_SUPPORTED #ifdef RTE_ARCH_X86 { ixgbe_recv_pkts_vec, "Vector SSE"}, { ixgbe_recv_scattered_pkts_vec, "Vector SSE scattered"}, @@ -4949,6 +4968,7 @@ static const struct { { ixgbe_recv_pkts_vec, "Vector NEON"}, { ixgbe_recv_scattered_pkts_vec, "Vector NEON scattered"}, #endif +#endif }; int @@ -6199,79 +6219,3 @@ ixgbe_config_rss_filter(struct rte_eth_dev *dev, return 0; } - -/* Stubs needed for linkage when RTE_ARCH_PPC_64, RTE_ARCH_RISCV or - * RTE_ARCH_LOONGARCH is set. - */ -#if defined(RTE_ARCH_PPC_64) || defined(RTE_ARCH_RISCV) || \ - defined(RTE_ARCH_LOONGARCH) -int -ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev __rte_unused *dev) -{ - return -1; -} - -void -ixgbe_recycle_rx_descriptors_refill_vec(void __rte_unused * rx_queue, - uint16_t __rte_unused nb_mbufs) -{ -} - -uint16_t -ixgbe_recycle_tx_mbufs_reuse_vec(void __rte_unused * tx_queue, - struct rte_eth_recycle_rxq_info __rte_unused * recycle_rxq_info) -{ - return 0; -} - -uint16_t -ixgbe_recv_pkts_vec( - void __rte_unused *rx_queue, - struct rte_mbuf __rte_unused **rx_pkts, - uint16_t __rte_unused nb_pkts) -{ - return 0; -} - -uint16_t -ixgbe_recv_scattered_pkts_vec( - void __rte_unused *rx_queue, - struct rte_mbuf __rte_unused **rx_pkts, - uint16_t __rte_unused nb_pkts) -{ - return 0; -} - -int -ixgbe_rxq_vec_setup(struct ixgbe_rx_queue __rte_unused *rxq) -{ - return -1; -} - -uint16_t -ixgbe_xmit_pkts_vec(void __rte_unused * tx_queue, struct rte_mbuf __rte_unused * *tx_pkts, - __rte_unused uint16_t nb_pkts) -{ - return 0; -} - -uint16_t -ixgbe_xmit_fixed_burst_vec(void __rte_unused *tx_queue, - struct rte_mbuf __rte_unused **tx_pkts, - uint16_t __rte_unused nb_pkts) -{ - return 0; -} - -int -ixgbe_txq_vec_setup(struct ci_tx_queue *txq __rte_unused) -{ - return -1; -} - -void -ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue __rte_unused *rxq) -{ - return; -} -#endif diff --git a/drivers/net/intel/ixgbe/meson.build b/drivers/net/intel/ixgbe/meson.build index e6f0fd135e..7e737ee7b4 100644 --- a/drivers/net/intel/ixgbe/meson.build +++ b/drivers/net/intel/ixgbe/meson.build @@ -26,9 +26,11 @@ deps += ['hash', 'security'] if arch_subdir == 'x86' sources += files('ixgbe_rxtx_vec_common.c') sources += files('ixgbe_rxtx_vec_sse.c') + cflags += ['-DIXGBE_VPMD_SUPPORTED'] elif arch_subdir == 'arm' sources += files('ixgbe_rxtx_vec_common.c') sources += files('ixgbe_rxtx_vec_neon.c') + cflags += ['-DIXGBE_VPMD_SUPPORTED'] endif headers = files('rte_pmd_ixgbe.h') From patchwork Mon Jun 9 15:37:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154214 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 77B27468B7; Mon, 9 Jun 2025 17:39:59 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C6FDC427A5; Mon, 9 Jun 2025 17:38:15 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id E3AA7427BF for ; Mon, 9 Jun 2025 17:38:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483493; x=1781019493; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Xjo7scfY+F3t4cxV5hL9XfijA2cQX/4e1M53BtP98Ts=; b=Es7pvZfBUBZlvApuSppe0YEY3atqGGRK+6HycTTT3l2libd6BgxmZGcR 0MUd/doSTAjAJhJxHeIMAJ7LRdFim4EytFEKOyW13yEsTWUWT0S7/0Ko6 DRI9qh3ZeIqhKw1cvnk8XldD+Xh33eKkFUjYdkKAtbWhmxeobK5LpaRW+ pDNeL7OvHjHSh+UOwL8mVRXNVokO+iKkJDYh6FcUrMx+Qqk84EknyZl4y /zkxbbiwsmJPO6VmhXWKNEs1pwoBLLX1Y+Gfybh/vd0leCRzpybbZDabA HRxBfXkyAQB4lNBYxAM3GC0rzTGZsmFsgMWvie7POnbx0pt1l7g7eFd/F g==; X-CSE-ConnectionGUID: aN0xLTX1Tj+TMXPrJ1TBAQ== X-CSE-MsgGUID: vtctPZZQQE67SskPJRfzRQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012201" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012201" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:13 -0700 X-CSE-ConnectionGUID: 6+8DQXvpSviCwkNzfspWEg== X-CSE-MsgGUID: yatOJ/yWQXOz7j9NQ81mcw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419679" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:11 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 17/33] net/ixgbe: replace always-true check Date: Mon, 9 Jun 2025 16:37:15 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is an option `RTE_PMD_PACKET_PREFETCH` in `rte_config.h` that is always set to 1 by default, and that controls some prefetch behavior in the driver. However, there's another prefetch behavior that is controlled by `RTE_PMD_USE_PREFETCH`, which is always defined unconditionally (literally using `#if 1` construct). Replace the check to also use `RTE_PMD_PACKET_PREFETCH` to allow turning it off at build time. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit drivers/net/intel/ixgbe/ixgbe_rxtx.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 48675ab964..cdf0d33955 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -92,11 +92,7 @@ #define IXGBE_TX_OFFLOAD_NOTSUP_MASK \ (RTE_MBUF_F_TX_OFFLOAD_MASK ^ IXGBE_TX_OFFLOAD_MASK) -#if 1 -#define RTE_PMD_USE_PREFETCH -#endif - -#ifdef RTE_PMD_USE_PREFETCH +#ifdef RTE_PMD_PACKET_PREFETCH /* * Prefetch a cache line into all cache levels. */ From patchwork Mon Jun 9 15:37:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154215 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CD0AC468B7; Mon, 9 Jun 2025 17:40:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B426E427CA; Mon, 9 Jun 2025 17:38:16 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 59238427BA for ; Mon, 9 Jun 2025 17:38:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483495; x=1781019495; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YJ749Jfl929FQ2wWL8GwPFN2OHC+lH5G6OyfYtGTSaw=; b=Cm9ZaxrqGH5ZOYC3qaVGHrZV218i3Xg3LR5kvKN8KOW5csGq96bUG985 eiDtflI2a8sR9tq8YIoZJXq6MiP5PUt0gXhZy8Oc6W+wmxn4NxEamUT3p 7EFlqMgbEFYQtk7QoY8jBwNxzIbXFzqzIaM3/N+stiwWxv9o4pJUM+Lvn QfqHvDQ7nkvcF21SHrqTf0cKOiDzC1HOoFNDmG3z5M56/u2/dgxQ/hmyV TkVnR6FAhoC2xebddJNmgHixcFaxWVOmZn3DMsPSH6rZvBzhWwp7ZmmGt 2xkAjrocau55zsGd+AQpwFzvHIe2llhMT88JR/Ts6cGPLxyHZ7y/rcgTB Q==; X-CSE-ConnectionGUID: YP1vH19yQcmsgX7AQrc2FA== X-CSE-MsgGUID: WQH2nTxSROKPb1AOjbC4hg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012206" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012206" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:14 -0700 X-CSE-ConnectionGUID: zTRsht0PTB+p3m1cEv3isQ== X-CSE-MsgGUID: MpO1uNVLQv6yUp4DqlV7FQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419682" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:13 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 18/33] net/ixgbe: add a desc done function Date: Mon, 9 Jun 2025 16:37:16 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add a function to check DD bit status, and use it everywhere we do these checks. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v6: - Move ixgbe_tx_desc_done() to ixgbe_rxtx.h to avoid compile errors on platforms that do not support vector driver v5: - Add this commit drivers/net/intel/ixgbe/ixgbe_rxtx.c | 8 ++----- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 21 +++++++++++++++++++ .../net/intel/ixgbe/ixgbe_rxtx_vec_common.c | 4 +--- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.h | 4 +--- 4 files changed, 25 insertions(+), 12 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index cdf0d33955..f32c2be988 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -118,13 +118,11 @@ static __rte_always_inline int ixgbe_tx_free_bufs(struct ci_tx_queue *txq) { struct ci_tx_entry *txep; - uint32_t status; int i, nb_free = 0; struct rte_mbuf *m, *free[RTE_IXGBE_TX_MAX_FREE_BUF_SZ]; /* check DD bit on threshold descriptor */ - status = txq->ixgbe_tx_ring[txq->tx_next_dd].wb.status; - if (!(status & rte_cpu_to_le_32(IXGBE_ADVTXD_STAT_DD))) + if (!ixgbe_tx_desc_done(txq, txq->tx_next_dd)) return 0; /* @@ -3412,7 +3410,6 @@ int ixgbe_dev_tx_descriptor_status(void *tx_queue, uint16_t offset) { struct ci_tx_queue *txq = tx_queue; - volatile uint32_t *status; uint32_t desc; if (unlikely(offset >= txq->nb_tx_desc)) @@ -3428,8 +3425,7 @@ ixgbe_dev_tx_descriptor_status(void *tx_queue, uint16_t offset) desc -= txq->nb_tx_desc; } - status = &txq->ixgbe_tx_ring[desc].wb.status; - if (*status & rte_cpu_to_le_32(IXGBE_ADVTXD_STAT_DD)) + if (ixgbe_tx_desc_done(txq, desc)) return RTE_ETH_TX_DESC_DONE; return RTE_ETH_TX_DESC_FULL; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index 6fcc5ee1e6..4887a81c4a 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -5,6 +5,8 @@ #ifndef _IXGBE_RXTX_H_ #define _IXGBE_RXTX_H_ +#include "ixgbe_type.h" + #include "../common/tx.h" /* @@ -241,4 +243,23 @@ uint64_t ixgbe_get_rx_port_offloads(struct rte_eth_dev *dev); uint64_t ixgbe_get_tx_queue_offloads(struct rte_eth_dev *dev); int ixgbe_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc); +/** + * Check if the Tx descriptor DD bit is set. + * + * @param txq + * Pointer to the Tx queue structure. + * @param idx + * Index of the Tx descriptor to check. + * + * @return + * 1 if the Tx descriptor is done, 0 otherwise. + */ +static inline int +ixgbe_tx_desc_done(struct ci_tx_queue *txq, uint16_t idx) +{ + const uint32_t status = txq->ixgbe_tx_ring[idx].wb.status; + + return !!(status & rte_cpu_to_le_32(IXGBE_ADVTXD_STAT_DD)); +} + #endif /* _IXGBE_RXTX_H_ */ diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c index cf6d3e4914..707dc7f5f9 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c @@ -215,7 +215,6 @@ ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, struct ci_tx_entry *txep; struct rte_mbuf **rxep; int i, n; - uint32_t status; uint16_t nb_recycle_mbufs; uint16_t avail = 0; uint16_t mbuf_ring_size = recycle_rxq_info->mbuf_ring_size; @@ -232,8 +231,7 @@ ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, return 0; /* check DD bits on threshold descriptor */ - status = txq->ixgbe_tx_ring[txq->tx_next_dd].wb.status; - if (!(status & IXGBE_ADVTXD_STAT_DD)) + if (!ixgbe_tx_desc_done(txq, txq->tx_next_dd)) return 0; n = txq->tx_rs_thresh; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h index 4678a5dfd9..e5b16af694 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h @@ -29,15 +29,13 @@ static __rte_always_inline int ixgbe_tx_free_bufs_vec(struct ci_tx_queue *txq) { struct ci_tx_entry_vec *txep; - uint32_t status; uint32_t n; uint32_t i; int nb_free = 0; struct rte_mbuf *m, *free[RTE_IXGBE_TX_MAX_FREE_BUF_SZ]; /* check DD bit on threshold descriptor */ - status = txq->ixgbe_tx_ring[txq->tx_next_dd].wb.status; - if (!(status & IXGBE_ADVTXD_STAT_DD)) + if (!ixgbe_tx_desc_done(txq, txq->tx_next_dd)) return 0; n = txq->tx_rs_thresh; From patchwork Mon Jun 9 15:37:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154216 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52152468B7; Mon, 9 Jun 2025 17:40:11 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B6F69427CE; Mon, 9 Jun 2025 17:38:17 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 55640427C5 for ; Mon, 9 Jun 2025 17:38:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483497; x=1781019497; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=A5wvRK7yDaMyZU/2eWjUDTUqPzeoLy923K2FyByFqtE=; b=TUmfkTzxCiPKWkqlnEMp4fXXquwdqtd/BrB0f6+jmBvy3kpafiA7ct2Z ABKNgufQh4A+90f+gwz3rERkV4/bTHSTgdUQwR1JaqtCAnlu8MULkz7uw ZkmvAmmnk/ihvmbPIvZS7VHOBAKGno5x/3v95LZxU5bgLxzmhoQbwnUUH 9mRU/xM4OqrO8q21rUBoCMYxSlGAQ2YFjH8njt+indzLazviD6emvH1Dq b9Gjac6elzYfaXlLobzvBnjiPr/LmTS+p3LILJREjGSYItnxdxpkP9ebC ZvznIGvm2u0eevVXW+cl31Mc6ymoGMywGPe7EnQ5n6oRkn4bcL8PGJTdw Q==; X-CSE-ConnectionGUID: k66OXVPRQditi+4aL1beFQ== X-CSE-MsgGUID: h/Y6PUb4SmatTDqDd21j/w== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012210" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012210" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:16 -0700 X-CSE-ConnectionGUID: DOtuzHjyRIOfibd/B4fo7g== X-CSE-MsgGUID: Mdyjxu2jRky4YIQyMVRCKA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419692" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:14 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin Cc: bruce.richardson@intel.com Subject: [PATCH v6 19/33] net/ixgbe: clean up definitions Date: Mon, 9 Jun 2025 16:37:17 +0100 Message-ID: <67b6ad60d6e01717ffb3fc2a4339d7172979447d.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch does the following cleanups: - Remove RTE_ and RTE_PMD_ prefix from internal definitions - Mark vector-PMD related definitions with IXGBE_VPMD_ prefix Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit drivers/net/intel/ixgbe/ixgbe_ipsec.c | 10 ++-- drivers/net/intel/ixgbe/ixgbe_rxtx.c | 60 +++++++++---------- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 22 +++---- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.h | 2 +- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c | 56 ++++++++--------- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c | 60 +++++++++---------- 6 files changed, 105 insertions(+), 105 deletions(-) diff --git a/drivers/net/intel/ixgbe/ixgbe_ipsec.c b/drivers/net/intel/ixgbe/ixgbe_ipsec.c index 778004cbe4..df0964a51d 100644 --- a/drivers/net/intel/ixgbe/ixgbe_ipsec.c +++ b/drivers/net/intel/ixgbe/ixgbe_ipsec.c @@ -15,20 +15,20 @@ #include "ixgbe_ethdev.h" #include "ixgbe_ipsec.h" -#define RTE_IXGBE_REGISTER_POLL_WAIT_5_MS 5 +#define IXGBE_REGISTER_POLL_WAIT_5_MS 5 #define IXGBE_WAIT_RREAD \ IXGBE_WRITE_REG_THEN_POLL_MASK(hw, IXGBE_IPSRXIDX, reg_val, \ - IPSRXIDX_READ, RTE_IXGBE_REGISTER_POLL_WAIT_5_MS) + IPSRXIDX_READ, IXGBE_REGISTER_POLL_WAIT_5_MS) #define IXGBE_WAIT_RWRITE \ IXGBE_WRITE_REG_THEN_POLL_MASK(hw, IXGBE_IPSRXIDX, reg_val, \ - IPSRXIDX_WRITE, RTE_IXGBE_REGISTER_POLL_WAIT_5_MS) + IPSRXIDX_WRITE, IXGBE_REGISTER_POLL_WAIT_5_MS) #define IXGBE_WAIT_TREAD \ IXGBE_WRITE_REG_THEN_POLL_MASK(hw, IXGBE_IPSTXIDX, reg_val, \ - IPSRXIDX_READ, RTE_IXGBE_REGISTER_POLL_WAIT_5_MS) + IPSRXIDX_READ, IXGBE_REGISTER_POLL_WAIT_5_MS) #define IXGBE_WAIT_TWRITE \ IXGBE_WRITE_REG_THEN_POLL_MASK(hw, IXGBE_IPSTXIDX, reg_val, \ - IPSRXIDX_WRITE, RTE_IXGBE_REGISTER_POLL_WAIT_5_MS) + IPSRXIDX_WRITE, IXGBE_REGISTER_POLL_WAIT_5_MS) #define CMP_IP(a, b) (\ (a).ipv6[0] == (b).ipv6[0] && \ diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index f32c2be988..5b2067bc0e 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -119,7 +119,7 @@ ixgbe_tx_free_bufs(struct ci_tx_queue *txq) { struct ci_tx_entry *txep; int i, nb_free = 0; - struct rte_mbuf *m, *free[RTE_IXGBE_TX_MAX_FREE_BUF_SZ]; + struct rte_mbuf *m, *free[IXGBE_TX_MAX_FREE_BUF_SZ]; /* check DD bit on threshold descriptor */ if (!ixgbe_tx_desc_done(txq, txq->tx_next_dd)) @@ -139,7 +139,7 @@ ixgbe_tx_free_bufs(struct ci_tx_queue *txq) if (unlikely(m == NULL)) continue; - if (nb_free >= RTE_IXGBE_TX_MAX_FREE_BUF_SZ || + if (nb_free >= IXGBE_TX_MAX_FREE_BUF_SZ || (nb_free > 0 && m->pool != free[0]->pool)) { rte_mempool_put_bulk(free[0]->pool, (void **)free, nb_free); @@ -351,7 +351,7 @@ ixgbe_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_pkts, return 0; /* Try to transmit at least chunks of TX_MAX_BURST pkts */ - if (likely(nb_pkts <= RTE_PMD_IXGBE_TX_MAX_BURST)) + if (likely(nb_pkts <= IXGBE_TX_MAX_BURST)) return tx_xmit_pkts(tx_queue, tx_pkts, nb_pkts); /* transmit more than the max burst, in chunks of TX_MAX_BURST */ @@ -359,7 +359,7 @@ ixgbe_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_pkts, while (nb_pkts) { uint16_t ret, n; - n = (uint16_t)RTE_MIN(nb_pkts, RTE_PMD_IXGBE_TX_MAX_BURST); + n = (uint16_t)RTE_MIN(nb_pkts, IXGBE_TX_MAX_BURST); ret = tx_xmit_pkts(tx_queue, &(tx_pkts[nb_tx]), n); nb_tx = (uint16_t)(nb_tx + ret); nb_pkts = (uint16_t)(nb_pkts - ret); @@ -1574,7 +1574,7 @@ ixgbe_rx_scan_hw_ring(struct ixgbe_rx_queue *rxq) * Scan LOOK_AHEAD descriptors at a time to determine which descriptors * reference packets that are ready to be received. */ - for (i = 0; i < RTE_PMD_IXGBE_RX_MAX_BURST; + for (i = 0; i < IXGBE_RX_MAX_BURST; i += LOOK_AHEAD, rxdp += LOOK_AHEAD, rxep += LOOK_AHEAD) { /* Read desc statuses backwards to avoid race condition */ for (j = 0; j < LOOK_AHEAD; j++) @@ -1771,7 +1771,7 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, return 0; } -/* split requests into chunks of size RTE_PMD_IXGBE_RX_MAX_BURST */ +/* split requests into chunks of size IXGBE_RX_MAX_BURST */ uint16_t ixgbe_recv_pkts_bulk_alloc(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) @@ -1781,7 +1781,7 @@ ixgbe_recv_pkts_bulk_alloc(void *rx_queue, struct rte_mbuf **rx_pkts, if (unlikely(nb_pkts == 0)) return 0; - if (likely(nb_pkts <= RTE_PMD_IXGBE_RX_MAX_BURST)) + if (likely(nb_pkts <= IXGBE_RX_MAX_BURST)) return rx_recv_pkts(rx_queue, rx_pkts, nb_pkts); /* request is relatively large, chunk it up */ @@ -1789,7 +1789,7 @@ ixgbe_recv_pkts_bulk_alloc(void *rx_queue, struct rte_mbuf **rx_pkts, while (nb_pkts) { uint16_t ret, n; - n = (uint16_t)RTE_MIN(nb_pkts, RTE_PMD_IXGBE_RX_MAX_BURST); + n = (uint16_t)RTE_MIN(nb_pkts, IXGBE_RX_MAX_BURST); ret = rx_recv_pkts(rx_queue, &rx_pkts[nb_rx], n); nb_rx = (uint16_t)(nb_rx + ret); nb_pkts = (uint16_t)(nb_pkts - ret); @@ -2494,8 +2494,8 @@ ixgbe_dev_tx_done_cleanup(void *tx_queue, uint32_t free_cnt) #ifdef RTE_LIB_SECURITY !(txq->using_ipsec) && #endif - txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST) { - if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ && + txq->tx_rs_thresh >= IXGBE_TX_MAX_BURST) { + if (txq->tx_rs_thresh <= IXGBE_TX_MAX_FREE_BUF_SZ && rte_vect_get_max_simd_bitwidth() >= RTE_VECT_SIMD_128 && (rte_eal_process_type() != RTE_PROC_PRIMARY || txq->sw_ring_vec != NULL)) { @@ -2652,10 +2652,10 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ci_tx_queue *txq) #ifdef RTE_LIB_SECURITY !(txq->using_ipsec) && #endif - (txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST)) { + (txq->tx_rs_thresh >= IXGBE_TX_MAX_BURST)) { PMD_INIT_LOG(DEBUG, "Using simple tx code path"); dev->tx_pkt_prepare = NULL; - if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ && + if (txq->tx_rs_thresh <= IXGBE_TX_MAX_FREE_BUF_SZ && rte_vect_get_max_simd_bitwidth() >= RTE_VECT_SIMD_128 && (rte_eal_process_type() != RTE_PROC_PRIMARY || ixgbe_txq_vec_setup(txq) == 0)) { @@ -2671,9 +2671,9 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct ci_tx_queue *txq) " - offloads = 0x%" PRIx64, txq->offloads); PMD_INIT_LOG(DEBUG, - " - tx_rs_thresh = %lu " "[RTE_PMD_IXGBE_TX_MAX_BURST=%lu]", + " - tx_rs_thresh = %lu [IXGBE_TX_MAX_BURST=%lu]", (unsigned long)txq->tx_rs_thresh, - (unsigned long)RTE_PMD_IXGBE_TX_MAX_BURST); + (unsigned long)IXGBE_TX_MAX_BURST); dev->tx_pkt_burst = ixgbe_xmit_pkts; dev->tx_pkt_prepare = ixgbe_prep_pkts; } @@ -3021,17 +3021,17 @@ check_rx_burst_bulk_alloc_preconditions(struct ixgbe_rx_queue *rxq) /* * Make sure the following pre-conditions are satisfied: - * rxq->rx_free_thresh >= RTE_PMD_IXGBE_RX_MAX_BURST + * rxq->rx_free_thresh >= IXGBE_RX_MAX_BURST * rxq->rx_free_thresh < rxq->nb_rx_desc * (rxq->nb_rx_desc % rxq->rx_free_thresh) == 0 * Scattered packets are not supported. This should be checked * outside of this function. */ - if (!(rxq->rx_free_thresh >= RTE_PMD_IXGBE_RX_MAX_BURST)) { + if (!(rxq->rx_free_thresh >= IXGBE_RX_MAX_BURST)) { PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions: " "rxq->rx_free_thresh=%d, " - "RTE_PMD_IXGBE_RX_MAX_BURST=%d", - rxq->rx_free_thresh, RTE_PMD_IXGBE_RX_MAX_BURST); + "IXGBE_RX_MAX_BURST=%d", + rxq->rx_free_thresh, IXGBE_RX_MAX_BURST); ret = -EINVAL; } else if (!(rxq->rx_free_thresh < rxq->nb_rx_desc)) { PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions: " @@ -3065,7 +3065,7 @@ ixgbe_reset_rx_queue(struct ixgbe_adapter *adapter, struct ixgbe_rx_queue *rxq) */ if (adapter->rx_bulk_alloc_allowed) /* zero out extra memory */ - len += RTE_PMD_IXGBE_RX_MAX_BURST; + len += IXGBE_RX_MAX_BURST; /* * Zero out HW ring memory. Zero out extra memory at the end of @@ -3306,7 +3306,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, */ len = nb_desc; if (adapter->rx_bulk_alloc_allowed) - len += RTE_PMD_IXGBE_RX_MAX_BURST; + len += IXGBE_RX_MAX_BURST; rxq->sw_ring = rte_zmalloc_socket("rxq->sw_ring", sizeof(struct ixgbe_rx_entry) * len, @@ -4600,7 +4600,7 @@ ixgbe_vmdq_rx_hw_configure(struct rte_eth_dev *dev) /* PFDMA Tx General Switch Control Enables VMDQ loopback */ if (cfg->enable_loop_back) { IXGBE_WRITE_REG(hw, IXGBE_PFDTXGSWC, IXGBE_PFDTXGSWC_VT_LBEN); - for (i = 0; i < RTE_IXGBE_VMTXSW_REGISTER_COUNT; i++) + for (i = 0; i < IXGBE_VMTXSW_REGISTER_COUNT; i++) IXGBE_WRITE_REG(hw, IXGBE_VMTXSW(i), UINT32_MAX); } @@ -5056,7 +5056,7 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) } else if (adapter->rx_vec_allowed) { PMD_INIT_LOG(DEBUG, "Vector rx enabled, please make sure RX " "burst size no less than %d (port=%d).", - RTE_IXGBE_DESCS_PER_LOOP, + IXGBE_VPMD_DESCS_PER_LOOP, dev->data->port_id); dev->recycle_rx_descriptors_refill = ixgbe_recycle_rx_descriptors_refill_vec; dev->rx_pkt_burst = ixgbe_recv_pkts_vec; @@ -5640,7 +5640,7 @@ ixgbe_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(rxq->reg_idx), rxdctl); /* Wait until RX Enable ready */ - poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS; + poll_ms = IXGBE_REGISTER_POLL_WAIT_10_MS; do { rte_delay_ms(1); rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx)); @@ -5677,7 +5677,7 @@ ixgbe_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(rxq->reg_idx), rxdctl); /* Wait until RX Enable bit clear */ - poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS; + poll_ms = IXGBE_REGISTER_POLL_WAIT_10_MS; do { rte_delay_ms(1); rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(rxq->reg_idx)); @@ -5685,7 +5685,7 @@ ixgbe_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) if (!poll_ms) PMD_INIT_LOG(ERR, "Could not disable Rx Queue %d", rx_queue_id); - rte_delay_us(RTE_IXGBE_WAIT_100_US); + rte_delay_us(IXGBE_WAIT_100_US); ixgbe_rx_queue_release_mbufs(rxq); ixgbe_reset_rx_queue(adapter, rxq); @@ -5717,7 +5717,7 @@ ixgbe_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id) /* Wait until TX Enable ready */ if (hw->mac.type == ixgbe_mac_82599EB) { - poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS; + poll_ms = IXGBE_REGISTER_POLL_WAIT_10_MS; do { rte_delay_ms(1); txdctl = IXGBE_READ_REG(hw, @@ -5753,9 +5753,9 @@ ixgbe_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id) /* Wait until TX queue is empty */ if (hw->mac.type == ixgbe_mac_82599EB) { - poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS; + poll_ms = IXGBE_REGISTER_POLL_WAIT_10_MS; do { - rte_delay_us(RTE_IXGBE_WAIT_100_US); + rte_delay_us(IXGBE_WAIT_100_US); txtdh = IXGBE_READ_REG(hw, IXGBE_TDH(txq->reg_idx)); txtdt = IXGBE_READ_REG(hw, @@ -5773,7 +5773,7 @@ ixgbe_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id) /* Wait until TX Enable bit clear */ if (hw->mac.type == ixgbe_mac_82599EB) { - poll_ms = RTE_IXGBE_REGISTER_POLL_WAIT_10_MS; + poll_ms = IXGBE_REGISTER_POLL_WAIT_10_MS; do { rte_delay_ms(1); txdctl = IXGBE_READ_REG(hw, @@ -5846,7 +5846,7 @@ ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, recycle_rxq_info->receive_tail = &rxq->rx_tail; if (adapter->rx_vec_allowed) { - recycle_rxq_info->refill_requirement = RTE_IXGBE_RXQ_REARM_THRESH; + recycle_rxq_info->refill_requirement = IXGBE_VPMD_RXQ_REARM_THRESH; recycle_rxq_info->refill_head = &rxq->rxrearm_start; } else { recycle_rxq_info->refill_requirement = rxq->rx_free_thresh; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index 4887a81c4a..9047ee4763 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -31,16 +31,16 @@ #define IXGBE_MIN_RING_DESC 64 #define IXGBE_MAX_RING_DESC 8192 -#define RTE_PMD_IXGBE_TX_MAX_BURST 32 -#define RTE_PMD_IXGBE_RX_MAX_BURST 32 -#define RTE_IXGBE_TX_MAX_FREE_BUF_SZ 64 +#define IXGBE_TX_MAX_BURST 32 +#define IXGBE_RX_MAX_BURST 32 +#define IXGBE_TX_MAX_FREE_BUF_SZ 64 -#define RTE_IXGBE_DESCS_PER_LOOP 4 +#define IXGBE_VPMD_DESCS_PER_LOOP 4 -#define RTE_IXGBE_RXQ_REARM_THRESH 32 -#define RTE_IXGBE_MAX_RX_BURST RTE_IXGBE_RXQ_REARM_THRESH +#define IXGBE_VPMD_RXQ_REARM_THRESH 32 +#define IXGBE_VPMD_RX_BURST IXGBE_VPMD_RXQ_REARM_THRESH -#define RX_RING_SZ ((IXGBE_MAX_RING_DESC + RTE_PMD_IXGBE_RX_MAX_BURST) * \ +#define RX_RING_SZ ((IXGBE_MAX_RING_DESC + IXGBE_RX_MAX_BURST) * \ sizeof(union ixgbe_adv_rx_desc)) #ifdef RTE_PMD_PACKET_PREFETCH @@ -49,9 +49,9 @@ #define rte_packet_prefetch(p) do {} while(0) #endif -#define RTE_IXGBE_REGISTER_POLL_WAIT_10_MS 10 -#define RTE_IXGBE_WAIT_100_US 100 -#define RTE_IXGBE_VMTXSW_REGISTER_COUNT 2 +#define IXGBE_REGISTER_POLL_WAIT_10_MS 10 +#define IXGBE_WAIT_100_US 100 +#define IXGBE_VMTXSW_REGISTER_COUNT 2 #define IXGBE_TX_MAX_SEG 40 @@ -120,7 +120,7 @@ struct ixgbe_rx_queue { /** need to alloc dummy mbuf, for wraparound when scanning hw ring */ struct rte_mbuf fake_mbuf; /** hold packets to return to application */ - struct rte_mbuf *rx_stage[RTE_PMD_IXGBE_RX_MAX_BURST*2]; + struct rte_mbuf *rx_stage[IXGBE_RX_MAX_BURST * 2]; const struct rte_memzone *mz; }; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h index e5b16af694..e05696f584 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h @@ -32,7 +32,7 @@ ixgbe_tx_free_bufs_vec(struct ci_tx_queue *txq) uint32_t n; uint32_t i; int nb_free = 0; - struct rte_mbuf *m, *free[RTE_IXGBE_TX_MAX_FREE_BUF_SZ]; + struct rte_mbuf *m, *free[IXGBE_TX_MAX_FREE_BUF_SZ]; /* check DD bit on threshold descriptor */ if (!ixgbe_tx_desc_done(txq, txq->tx_next_dd)) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c index ba213ccc67..2d42b7b1c1 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c @@ -29,24 +29,24 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (unlikely(rte_mempool_get_bulk(rxq->mp, (void *)rxep, - RTE_IXGBE_RXQ_REARM_THRESH) < 0)) { - if (rxq->rxrearm_nb + RTE_IXGBE_RXQ_REARM_THRESH >= + IXGBE_VPMD_RXQ_REARM_THRESH) < 0)) { + if (rxq->rxrearm_nb + IXGBE_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { - for (i = 0; i < RTE_IXGBE_DESCS_PER_LOOP; i++) { + for (i = 0; i < IXGBE_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i].read), zero); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - RTE_IXGBE_RXQ_REARM_THRESH; + IXGBE_VPMD_RXQ_REARM_THRESH; return; } p = vld1_u8((uint8_t *)&rxq->mbuf_initializer); /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < RTE_IXGBE_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < IXGBE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { mb0 = rxep[0].mbuf; mb1 = rxep[1].mbuf; @@ -66,11 +66,11 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += RTE_IXGBE_RXQ_REARM_THRESH; + rxq->rxrearm_start += IXGBE_VPMD_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) rxq->rxrearm_start = 0; - rxq->rxrearm_nb -= RTE_IXGBE_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= IXGBE_VPMD_RXQ_REARM_THRESH; rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); @@ -275,11 +275,11 @@ desc_to_ptype_v(uint64x2_t descs[4], uint16_t pkt_type_mask, } /** - * vPMD raw receive routine, only accept(nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP) + * vPMD raw receive routine, only accept(nb_pkts >= IXGBE_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_IXGBE_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_IXGBE_DESC_PER_LOOP power-of-two + * - nb_pkts < IXGBE_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a IXGBE_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, @@ -303,8 +303,8 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint8_t vlan_flags; uint16_t udp_p_flag = 0; /* Rx Descriptor UDP header present */ - /* nb_pkts has to be floor-aligned to RTE_IXGBE_DESCS_PER_LOOP */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_IXGBE_DESCS_PER_LOOP); + /* nb_pkts has to be floor-aligned to IXGBE_VPMD_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IXGBE_VPMD_DESCS_PER_LOOP); /* Just the act of getting into the function from the application is * going to cost about 7 cycles @@ -316,7 +316,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_IXGBE_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IXGBE_VPMD_RXQ_REARM_THRESH) ixgbe_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -345,9 +345,9 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, * D. fill info. from desc to mbuf */ for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; - pos += RTE_IXGBE_DESCS_PER_LOOP, - rxdp += RTE_IXGBE_DESCS_PER_LOOP) { - uint64x2_t descs[RTE_IXGBE_DESCS_PER_LOOP]; + pos += IXGBE_VPMD_DESCS_PER_LOOP, + rxdp += IXGBE_VPMD_DESCS_PER_LOOP) { + uint64x2_t descs[IXGBE_VPMD_DESCS_PER_LOOP]; uint8x16_t pkt_mb1, pkt_mb2, pkt_mb3, pkt_mb4; uint8x16x2_t sterr_tmp1, sterr_tmp2; uint64x2_t mbp1, mbp2; @@ -426,7 +426,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* and with mask to extract bits, flipping 1-0 */ *(int *)split_packet = ~stat & IXGBE_VPMD_DESC_EOP_MASK; - split_packet += RTE_IXGBE_DESCS_PER_LOOP; + split_packet += IXGBE_VPMD_DESCS_PER_LOOP; } /* C.4 expand DD bit to saturate UINT8 */ @@ -436,7 +436,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, IXGBE_UINT8_BIT - 1)); stat = ~vgetq_lane_u32(vreinterpretq_u32_u8(staterr), 0); - rte_prefetch_non_temporal(rxdp + RTE_IXGBE_DESCS_PER_LOOP); + rte_prefetch_non_temporal(rxdp + IXGBE_VPMD_DESCS_PER_LOOP); /* D.3 copy final 1,2 data to rx_pkts */ vst1q_u8((uint8_t *)&rx_pkts[pos + 1]->rx_descriptor_fields1, @@ -448,7 +448,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* C.5 calc available number of desc */ if (unlikely(stat == 0)) { - nb_pkts_recd += RTE_IXGBE_DESCS_PER_LOOP; + nb_pkts_recd += IXGBE_VPMD_DESCS_PER_LOOP; } else { nb_pkts_recd += rte_ctz32(stat) / IXGBE_UINT8_BIT; break; @@ -464,11 +464,11 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, } /** - * vPMD receive routine, only accept(nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP) + * vPMD receive routine, only accept(nb_pkts >= IXGBE_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_IXGBE_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_IXGBE_DESC_PER_LOOP power-of-two + * - nb_pkts < IXGBE_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a IXGBE_VPMD_DESC_PER_LOOP power-of-two */ uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -481,15 +481,15 @@ ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles scattered packets * * Notice: - * - nb_pkts < RTE_IXGBE_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_IXGBE_DESC_PER_LOOP power-of-two + * - nb_pkts < IXGBE_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a IXGBE_VPMD_DESCS_PER_LOOP power-of-two */ static uint16_t ixgbe_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { struct ixgbe_rx_queue *rxq = rx_queue; - uint8_t split_flags[RTE_IXGBE_MAX_RX_BURST] = {0}; + uint8_t split_flags[IXGBE_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _recv_raw_pkts_vec(rxq, rx_pkts, nb_pkts, @@ -527,15 +527,15 @@ ixgbe_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { uint16_t retval = 0; - while (nb_pkts > RTE_IXGBE_MAX_RX_BURST) { + while (nb_pkts > IXGBE_VPMD_RX_BURST) { uint16_t burst; burst = ixgbe_recv_scattered_burst_vec(rx_queue, rx_pkts + retval, - RTE_IXGBE_MAX_RX_BURST); + IXGBE_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_IXGBE_MAX_RX_BURST) + if (burst < IXGBE_VPMD_RX_BURST) return retval; } diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c index e1516a943d..f5bb7eb0bd 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c @@ -31,23 +31,23 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, - RTE_IXGBE_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + RTE_IXGBE_RXQ_REARM_THRESH >= + IXGBE_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + IXGBE_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < RTE_IXGBE_DESCS_PER_LOOP; i++) { + for (i = 0; i < IXGBE_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - RTE_IXGBE_RXQ_REARM_THRESH; + IXGBE_VPMD_RXQ_REARM_THRESH; return; } /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < RTE_IXGBE_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < IXGBE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { __m128i vaddr0, vaddr1; mb0 = rxep[0].mbuf; @@ -76,11 +76,11 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += RTE_IXGBE_RXQ_REARM_THRESH; + rxq->rxrearm_start += IXGBE_VPMD_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) rxq->rxrearm_start = 0; - rxq->rxrearm_nb -= RTE_IXGBE_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= IXGBE_VPMD_RXQ_REARM_THRESH; rx_id = (uint16_t) ((rxq->rxrearm_start == 0) ? (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); @@ -262,10 +262,10 @@ static inline uint32_t get_packet_type(int index, uint32_t etqf_check, uint32_t tunnel_check) { - if (etqf_check & (0x02 << (index * RTE_IXGBE_DESCS_PER_LOOP))) + if (etqf_check & (0x02 << (index * IXGBE_VPMD_DESCS_PER_LOOP))) return RTE_PTYPE_UNKNOWN; - if (tunnel_check & (0x02 << (index * RTE_IXGBE_DESCS_PER_LOOP))) { + if (tunnel_check & (0x02 << (index * IXGBE_VPMD_DESCS_PER_LOOP))) { pkt_info &= IXGBE_PACKET_TYPE_MASK_TUNNEL; return ptype_table_tn[pkt_info]; } @@ -320,11 +320,11 @@ desc_to_ptype_v(__m128i descs[4], uint16_t pkt_type_mask, } /** - * vPMD raw receive routine, only accept(nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP) + * vPMD raw receive routine, only accept(nb_pkts >= IXGBE_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_IXGBE_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_IXGBE_DESC_PER_LOOP power-of-two + * - nb_pkts < IXGBE_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a IXGBE_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, @@ -369,10 +369,10 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, * So we need to make some restrictions to ensure that * `rx_tail` will not exceed `rxrearm_start`. */ - nb_pkts = RTE_MIN(nb_pkts, RTE_IXGBE_RXQ_REARM_THRESH); + nb_pkts = RTE_MIN(nb_pkts, IXGBE_VPMD_RXQ_REARM_THRESH); - /* nb_pkts has to be floor-aligned to RTE_IXGBE_DESCS_PER_LOOP */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_IXGBE_DESCS_PER_LOOP); + /* nb_pkts has to be floor-aligned to IXGBE_VPMD_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IXGBE_VPMD_DESCS_PER_LOOP); /* Just the act of getting into the function from the application is * going to cost about 7 cycles @@ -384,7 +384,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_IXGBE_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IXGBE_VPMD_RXQ_REARM_THRESH) ixgbe_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -446,9 +446,9 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, * D. fill info. from desc to mbuf */ for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; - pos += RTE_IXGBE_DESCS_PER_LOOP, - rxdp += RTE_IXGBE_DESCS_PER_LOOP) { - __m128i descs[RTE_IXGBE_DESCS_PER_LOOP]; + pos += IXGBE_VPMD_DESCS_PER_LOOP, + rxdp += IXGBE_VPMD_DESCS_PER_LOOP) { + __m128i descs[IXGBE_VPMD_DESCS_PER_LOOP]; __m128i pkt_mb1, pkt_mb2, pkt_mb3, pkt_mb4; __m128i zero, staterr, sterr_tmp1, sterr_tmp2; /* 2 64 bit or 4 32 bit mbuf pointers in one XMM reg. */ @@ -554,7 +554,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, eop_bits = _mm_shuffle_epi8(eop_bits, eop_shuf_mask); /* store the resulting 32-bit value */ *(int *)split_packet = _mm_cvtsi128_si32(eop_bits); - split_packet += RTE_IXGBE_DESCS_PER_LOOP; + split_packet += IXGBE_VPMD_DESCS_PER_LOOP; } /* C.3 calc available number of desc */ @@ -572,7 +572,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* C.4 calc available number of desc */ var = rte_popcount64(_mm_cvtsi128_si64(staterr)); nb_pkts_recd += var; - if (likely(var != RTE_IXGBE_DESCS_PER_LOOP)) + if (likely(var != IXGBE_VPMD_DESCS_PER_LOOP)) break; } @@ -585,11 +585,11 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, } /** - * vPMD receive routine, only accept(nb_pkts >= RTE_IXGBE_DESCS_PER_LOOP) + * vPMD receiIXGBE_VPMD_RX_BURSTt(nb_pkts >= IXGBE_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_IXGBE_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_IXGBE_DESC_PER_LOOP power-of-two + * - nb_pkts RTE_IXGBE_MAX_RX_BURST) { + while (nb_pkts > IXGBE_VPMD_RX_BURST) { uint16_t burst; burst = ixgbe_recv_scattered_burst_vec(rx_queue, rx_pkts + retval, - RTE_IXGBE_MAX_RX_BURST); + IXGBE_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_IXGBE_MAX_RX_BURST) + if (burst < IXGBE_VPMD_RX_BURST) return retval; } From patchwork Mon Jun 9 15:37:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154217 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CAD42468B7; Mon, 9 Jun 2025 17:40:20 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1EA9D427D5; Mon, 9 Jun 2025 17:38:20 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 14A6941104 for ; Mon, 9 Jun 2025 17:38:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483498; x=1781019498; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=DBlCO8foK3lANkwyvEFQ7gdDkxJJtpBzRNd7kUkV8XU=; b=cEmzr4B38OwFcBYWHvlvhcqGGXMi8Uiyij/CQoEjhzBrpowawYmG5lEK xeae07OoS8B1nQKk2uRDH1gS4UgCul/OhFdQOhHcRahSHjRcrFM5ghmCf sk04ZJFz8u9hAtBQUNS2MhPodHdRv8yxLsFuGmZi9hLMTCRM1yhfKjiT0 VcOQJVI0Se5paf/G8rpazu7lQnW+66hsyYKkA/GOJrsqLbF1Bhotzf8eQ 0RJWLK18dorWWvfwSvpl76Ja7+PVT2BimZrHAtdlCz/pYw4xF31hwjRsO TSt9/qBuzdr7IZYaZLC4g2Yn+Jih0p22lzaLniKqgevc3UO7BaB6HA9MP w==; X-CSE-ConnectionGUID: Ocsd3zHyQ1mxzTroNFxKEQ== X-CSE-MsgGUID: ruXIA1grTFmKGxVCnt5EqQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012219" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012219" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:18 -0700 X-CSE-ConnectionGUID: BHDbvGPZReKy6HF0w0Asew== X-CSE-MsgGUID: X3adMwVmTeOt9vS4mKJEGw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419698" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:16 -0700 From: Anatoly Burakov To: dev@dpdk.org, Ian Stokes , Bruce Richardson Subject: [PATCH v6 20/33] net/i40e: clean up definitions Date: Mon, 9 Jun 2025 16:37:18 +0100 Message-ID: <33eca289d39a4beabb8dd0c98ca3262e5443e6f6.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This commit does the following cleanups: - Remove RTE_ and RTE_PMD_ prefix from internal definitions - Mark vector-PMD related definitions with I40E_VPMD_ prefix - Remove unused definitions - Create "descriptors per loop" for different vector implementations (regular for SSE, Neon, AltiVec, wide for AVX2, AVX512) Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit drivers/net/intel/i40e/i40e_rxtx.c | 42 +++++++-------- drivers/net/intel/i40e/i40e_rxtx.h | 17 +++--- drivers/net/intel/i40e/i40e_rxtx_common_avx.h | 18 +++---- .../net/intel/i40e/i40e_rxtx_vec_altivec.c | 48 ++++++++--------- drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c | 32 ++++++----- drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c | 32 ++++++----- drivers/net/intel/i40e/i40e_rxtx_vec_neon.c | 53 +++++++++---------- drivers/net/intel/i40e/i40e_rxtx_vec_sse.c | 48 ++++++++--------- 8 files changed, 142 insertions(+), 148 deletions(-) diff --git a/drivers/net/intel/i40e/i40e_rxtx.c b/drivers/net/intel/i40e/i40e_rxtx.c index 5f54bcc225..2e61076378 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.c +++ b/drivers/net/intel/i40e/i40e_rxtx.c @@ -424,11 +424,11 @@ check_rx_burst_bulk_alloc_preconditions(__rte_unused struct i40e_rx_queue *rxq) int ret = 0; #ifdef RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC - if (!(rxq->rx_free_thresh >= RTE_PMD_I40E_RX_MAX_BURST)) { + if (!(rxq->rx_free_thresh >= I40E_RX_MAX_BURST)) { PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions: " "rxq->rx_free_thresh=%d, " - "RTE_PMD_I40E_RX_MAX_BURST=%d", - rxq->rx_free_thresh, RTE_PMD_I40E_RX_MAX_BURST); + "I40E_RX_MAX_BURST=%d", + rxq->rx_free_thresh, I40E_RX_MAX_BURST); ret = -EINVAL; } else if (!(rxq->rx_free_thresh < rxq->nb_rx_desc)) { PMD_INIT_LOG(DEBUG, "Rx Burst Bulk Alloc Preconditions: " @@ -484,7 +484,7 @@ i40e_rx_scan_hw_ring(struct i40e_rx_queue *rxq) * Scan LOOK_AHEAD descriptors at a time to determine which * descriptors reference packets that are ready to be received. */ - for (i = 0; i < RTE_PMD_I40E_RX_MAX_BURST; i+=I40E_LOOK_AHEAD, + for (i = 0; i < I40E_RX_MAX_BURST; i += I40E_LOOK_AHEAD, rxdp += I40E_LOOK_AHEAD, rxep += I40E_LOOK_AHEAD) { /* Read desc statuses backwards to avoid race condition */ for (j = I40E_LOOK_AHEAD - 1; j >= 0; j--) { @@ -680,11 +680,11 @@ i40e_recv_pkts_bulk_alloc(void *rx_queue, if (unlikely(nb_pkts == 0)) return 0; - if (likely(nb_pkts <= RTE_PMD_I40E_RX_MAX_BURST)) + if (likely(nb_pkts <= I40E_RX_MAX_BURST)) return rx_recv_pkts(rx_queue, rx_pkts, nb_pkts); while (nb_pkts) { - n = RTE_MIN(nb_pkts, RTE_PMD_I40E_RX_MAX_BURST); + n = RTE_MIN(nb_pkts, I40E_RX_MAX_BURST); count = rx_recv_pkts(rx_queue, &rx_pkts[nb_rx], n); nb_rx = (uint16_t)(nb_rx + count); nb_pkts = (uint16_t)(nb_pkts - count); @@ -1334,9 +1334,9 @@ i40e_tx_free_bufs(struct ci_tx_queue *txq) struct ci_tx_entry *txep; uint16_t tx_rs_thresh = txq->tx_rs_thresh; uint16_t i = 0, j = 0; - struct rte_mbuf *free[RTE_I40E_TX_MAX_FREE_BUF_SZ]; - const uint16_t k = RTE_ALIGN_FLOOR(tx_rs_thresh, RTE_I40E_TX_MAX_FREE_BUF_SZ); - const uint16_t m = tx_rs_thresh % RTE_I40E_TX_MAX_FREE_BUF_SZ; + struct rte_mbuf *free[I40E_TX_MAX_FREE_BUF_SZ]; + const uint16_t k = RTE_ALIGN_FLOOR(tx_rs_thresh, I40E_TX_MAX_FREE_BUF_SZ); + const uint16_t m = tx_rs_thresh % I40E_TX_MAX_FREE_BUF_SZ; if ((txq->i40e_tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != @@ -1350,13 +1350,13 @@ i40e_tx_free_bufs(struct ci_tx_queue *txq) if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { if (k) { - for (j = 0; j != k; j += RTE_I40E_TX_MAX_FREE_BUF_SZ) { - for (i = 0; i < RTE_I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) { + for (j = 0; j != k; j += I40E_TX_MAX_FREE_BUF_SZ) { + for (i = 0; i < I40E_TX_MAX_FREE_BUF_SZ; ++i, ++txep) { free[i] = txep->mbuf; txep->mbuf = NULL; } rte_mempool_put_bulk(free[0]->pool, (void **)free, - RTE_I40E_TX_MAX_FREE_BUF_SZ); + I40E_TX_MAX_FREE_BUF_SZ); } } @@ -2146,7 +2146,7 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, * Allocating a little more memory because vectorized/bulk_alloc Rx * functions doesn't check boundaries each time. */ - len += RTE_PMD_I40E_RX_MAX_BURST; + len += I40E_RX_MAX_BURST; ring_size = RTE_ALIGN(len * sizeof(union i40e_rx_desc), I40E_DMA_MEM_ALIGN); @@ -2166,7 +2166,7 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, rxq->rx_ring_phys_addr = rz->iova; rxq->rx_ring = (union i40e_rx_desc *)rz->addr; - len = (uint16_t)(nb_desc + RTE_PMD_I40E_RX_MAX_BURST); + len = (uint16_t)(nb_desc + I40E_RX_MAX_BURST); /* Allocate the software ring. */ rxq->sw_ring = @@ -2370,7 +2370,7 @@ i40e_dev_tx_queue_setup_runtime(struct rte_eth_dev *dev, /* check vector conflict */ if (ad->tx_vec_allowed) { - if (txq->tx_rs_thresh > RTE_I40E_TX_MAX_FREE_BUF_SZ || + if (txq->tx_rs_thresh > I40E_TX_MAX_FREE_BUF_SZ || i40e_txq_vec_setup(txq)) { PMD_DRV_LOG(ERR, "Failed vector tx setup."); return -EINVAL; @@ -2379,7 +2379,7 @@ i40e_dev_tx_queue_setup_runtime(struct rte_eth_dev *dev, /* check simple tx conflict */ if (ad->tx_simple_allowed) { if ((txq->offloads & ~RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) != 0 || - txq->tx_rs_thresh < RTE_PMD_I40E_TX_MAX_BURST) { + txq->tx_rs_thresh < I40E_TX_MAX_BURST) { PMD_DRV_LOG(ERR, "No-simple tx is required."); return -EINVAL; } @@ -2675,7 +2675,7 @@ i40e_reset_rx_queue(struct i40e_rx_queue *rxq) #ifdef RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC if (check_rx_burst_bulk_alloc_preconditions(rxq) == 0) - len = (uint16_t)(rxq->nb_rx_desc + RTE_PMD_I40E_RX_MAX_BURST); + len = (uint16_t)(rxq->nb_rx_desc + I40E_RX_MAX_BURST); else #endif /* RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC */ len = rxq->nb_rx_desc; @@ -2684,7 +2684,7 @@ i40e_reset_rx_queue(struct i40e_rx_queue *rxq) ((volatile char *)rxq->rx_ring)[i] = 0; memset(&rxq->fake_mbuf, 0x0, sizeof(rxq->fake_mbuf)); - for (i = 0; i < RTE_PMD_I40E_RX_MAX_BURST; ++i) + for (i = 0; i < I40E_RX_MAX_BURST; ++i) rxq->sw_ring[rxq->nb_rx_desc + i].mbuf = &rxq->fake_mbuf; #ifdef RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC @@ -3276,7 +3276,7 @@ i40e_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, recycle_rxq_info->receive_tail = &rxq->rx_tail; if (ad->rx_vec_allowed) { - recycle_rxq_info->refill_requirement = RTE_I40E_RXQ_REARM_THRESH; + recycle_rxq_info->refill_requirement = I40E_VPMD_RXQ_REARM_THRESH; recycle_rxq_info->refill_head = &rxq->rxrearm_start; } else { recycle_rxq_info->refill_requirement = rxq->rx_free_thresh; @@ -3501,9 +3501,9 @@ i40e_set_tx_function_flag(struct rte_eth_dev *dev, struct ci_tx_queue *txq) ad->tx_simple_allowed = (txq->offloads == (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) && - txq->tx_rs_thresh >= RTE_PMD_I40E_TX_MAX_BURST); + txq->tx_rs_thresh >= I40E_TX_MAX_BURST); ad->tx_vec_allowed = (ad->tx_simple_allowed && - txq->tx_rs_thresh <= RTE_I40E_TX_MAX_FREE_BUF_SZ); + txq->tx_rs_thresh <= I40E_TX_MAX_FREE_BUF_SZ); if (ad->tx_vec_allowed) PMD_INIT_LOG(DEBUG, "Vector Tx can be enabled on Tx queue %u.", diff --git a/drivers/net/intel/i40e/i40e_rxtx.h b/drivers/net/intel/i40e/i40e_rxtx.h index 568f0536ac..3dca32b1ba 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.h +++ b/drivers/net/intel/i40e/i40e_rxtx.h @@ -7,15 +7,14 @@ #include "../common/tx.h" -#define RTE_PMD_I40E_RX_MAX_BURST 32 -#define RTE_PMD_I40E_TX_MAX_BURST 32 +#define I40E_RX_MAX_BURST 32 +#define I40E_TX_MAX_BURST 32 -#define RTE_I40E_VPMD_RX_BURST 32 -#define RTE_I40E_VPMD_TX_BURST 32 -#define RTE_I40E_RXQ_REARM_THRESH 32 -#define RTE_I40E_MAX_RX_BURST RTE_I40E_RXQ_REARM_THRESH -#define RTE_I40E_TX_MAX_FREE_BUF_SZ 64 -#define RTE_I40E_DESCS_PER_LOOP 4 +#define I40E_VPMD_RX_BURST 32 +#define I40E_VPMD_RXQ_REARM_THRESH 32 +#define I40E_TX_MAX_FREE_BUF_SZ 64 +#define I40E_VPMD_DESCS_PER_LOOP 4 +#define I40E_VPMD_DESCS_PER_LOOP_WIDE 8 #define I40E_RXBUF_SZ_1024 1024 #define I40E_RXBUF_SZ_2048 2048 @@ -97,7 +96,7 @@ struct i40e_rx_queue { uint16_t rx_nb_avail; /**< number of staged packets ready */ uint16_t rx_next_avail; /**< index of next staged packets */ uint16_t rx_free_trigger; /**< triggers rx buffer allocation */ - struct rte_mbuf *rx_stage[RTE_PMD_I40E_RX_MAX_BURST * 2]; + struct rte_mbuf *rx_stage[I40E_RX_MAX_BURST * 2]; #endif uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ diff --git a/drivers/net/intel/i40e/i40e_rxtx_common_avx.h b/drivers/net/intel/i40e/i40e_rxtx_common_avx.h index 7d2bda624b..8fc7cd5bd4 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_common_avx.h +++ b/drivers/net/intel/i40e/i40e_rxtx_common_avx.h @@ -25,19 +25,19 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, - RTE_I40E_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + I40E_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { __m128i dma_addr0; dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - RTE_I40E_RXQ_REARM_THRESH; + I40E_VPMD_RXQ_REARM_THRESH; return; } @@ -47,7 +47,7 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { __m128i vaddr0, vaddr1; mb0 = rxep[0].mbuf; @@ -79,7 +79,7 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) __m512i dma_addr0_3, dma_addr4_7; __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 8 mbufs in one loop */ - for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; + for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 8, rxep += 8, rxdp += 8) { __m128i vaddr0, vaddr1, vaddr2, vaddr3; __m128i vaddr4, vaddr5, vaddr6, vaddr7; @@ -152,7 +152,7 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) __m256i dma_addr0_1, dma_addr2_3; __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; + for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 4, rxep += 4, rxdp += 4) { __m128i vaddr0, vaddr1, vaddr2, vaddr3; __m256i vaddr0_1, vaddr2_3; @@ -197,7 +197,7 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) #endif - rxq->rxrearm_start += RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; rx_id = rxq->rxrearm_start - 1; if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { @@ -205,7 +205,7 @@ i40e_rxq_rearm_common(struct i40e_rx_queue *rxq, __rte_unused bool avx512) rx_id = rxq->nb_rx_desc - 1; } - rxq->rxrearm_nb -= RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; /* Update the tail pointer on the NIC */ I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c b/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c index 01dee811ba..568891cfb2 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c @@ -35,23 +35,23 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, - RTE_I40E_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + I40E_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { dma_addr0 = (__vector unsigned long){}; - for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; vec_st(dma_addr0, 0, RTE_CAST_PTR(__vector unsigned long *, &rxdp[i].read)); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - RTE_I40E_RXQ_REARM_THRESH; + I40E_VPMD_RXQ_REARM_THRESH; return; } /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { __vector unsigned long vaddr0, vaddr1; uintptr_t p0, p1; @@ -86,7 +86,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) vec_st(dma_addr1, 0, RTE_CAST_PTR(__vector unsigned long *, &rxdp++->read)); } - rxq->rxrearm_start += RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; rx_id = rxq->rxrearm_start - 1; if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { @@ -94,7 +94,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) rx_id = rxq->nb_rx_desc - 1; } - rxq->rxrearm_nb -= RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; /* Update the tail pointer on the NIC */ I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); @@ -188,11 +188,11 @@ desc_to_ptype_v(__vector unsigned long descs[4], struct rte_mbuf **rx_pkts, } /** - * vPMD raw receive routine, only accept(nb_pkts >= RTE_I40E_DESCS_PER_LOOP) + * vPMD raw receive routine, only accept(nb_pkts >= I40E_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_I40E_DESCS_PER_LOOP power-of-two + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a I40E_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, @@ -215,8 +215,8 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, }; __vector unsigned long dd_check, eop_check; - /* nb_pkts has to be floor-aligned to RTE_I40E_DESCS_PER_LOOP */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_I40E_DESCS_PER_LOOP); + /* nb_pkts has to be floor-aligned to I40E_VPMD_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, I40E_VPMD_DESCS_PER_LOOP); /* Just the act of getting into the function from the application is * going to cost about 7 cycles @@ -228,7 +228,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > I40E_VPMD_RXQ_REARM_THRESH) i40e_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -271,9 +271,9 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, */ for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; - pos += RTE_I40E_DESCS_PER_LOOP, - rxdp += RTE_I40E_DESCS_PER_LOOP) { - __vector unsigned long descs[RTE_I40E_DESCS_PER_LOOP]; + pos += I40E_VPMD_DESCS_PER_LOOP, + rxdp += I40E_VPMD_DESCS_PER_LOOP) { + __vector unsigned long descs[I40E_VPMD_DESCS_PER_LOOP]; __vector unsigned char pkt_mb1, pkt_mb2, pkt_mb3, pkt_mb4; __vector unsigned short staterr, sterr_tmp1, sterr_tmp2; __vector unsigned long mbp1, mbp2; /* two mbuf pointer @@ -406,7 +406,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* store the resulting 32-bit value */ *split_packet = (vec_ld(0, (__vector unsigned int *)&eop_bits))[0]; - split_packet += RTE_I40E_DESCS_PER_LOOP; + split_packet += I40E_VPMD_DESCS_PER_LOOP; /* zero-out next pointers */ rx_pkts[pos]->next = NULL; @@ -433,7 +433,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, var = rte_popcount64((vec_ld(0, (__vector unsigned long *)&staterr)[0])); nb_pkts_recd += var; - if (likely(var != RTE_I40E_DESCS_PER_LOOP)) + if (likely(var != I40E_VPMD_DESCS_PER_LOOP)) break; } @@ -446,7 +446,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, } /* Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -459,14 +459,14 @@ i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles single burst of 32 scattered packets * * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { struct i40e_rx_queue *rxq = rx_queue; - uint8_t split_flags[RTE_I40E_VPMD_RX_BURST] = {0}; + uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _recv_raw_pkts_vec(rxq, rx_pkts, nb_pkts, @@ -505,15 +505,15 @@ i40e_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { uint16_t retval = 0; - while (nb_pkts > RTE_I40E_VPMD_RX_BURST) { + while (nb_pkts > I40E_VPMD_RX_BURST) { uint16_t burst; burst = i40e_recv_scattered_burst_vec(rx_queue, rx_pkts + retval, - RTE_I40E_VPMD_RX_BURST); + I40E_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_I40E_VPMD_RX_BURST) + if (burst < I40E_VPMD_RX_BURST) return retval; } diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c index 4469c73c56..a13dd9bc78 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c @@ -108,8 +108,6 @@ static __rte_always_inline uint16_t _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { -#define RTE_I40E_DESCS_PER_LOOP_AVX 8 - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); @@ -118,13 +116,13 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, const int avx_aligned = ((rxq->rx_tail & 1) == 0); rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to RTE_I40E_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_I40E_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to I40E_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, I40E_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > I40E_VPMD_RXQ_REARM_THRESH) i40e_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -262,8 +260,8 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += RTE_I40E_DESCS_PER_LOOP_AVX, - rxdp += RTE_I40E_DESCS_PER_LOOP_AVX) { + i += I40E_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += I40E_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -299,7 +297,7 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, if (split_packet) { int j; - for (j = 0; j < RTE_I40E_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < I40E_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -577,7 +575,7 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, 12, 4, 14, 6); split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += RTE_I40E_DESCS_PER_LOOP_AVX; + split_packet += I40E_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -590,7 +588,7 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, burst += rte_popcount64(_mm_cvtsi128_si64( _mm256_castsi256_si128(status0_7))); received += burst; - if (burst != RTE_I40E_DESCS_PER_LOOP_AVX) + if (burst != I40E_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -607,7 +605,7 @@ _recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -619,14 +617,14 @@ i40e_recv_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, /* * vPMD receive routine that reassembles single burst of 32 scattered packets * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t i40e_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { struct i40e_rx_queue *rxq = rx_queue; - uint8_t split_flags[RTE_I40E_VPMD_RX_BURST] = {0}; + uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _recv_raw_pkts_vec_avx2(rxq, rx_pkts, nb_pkts, @@ -661,19 +659,19 @@ i40e_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles scattered packets. * Main receive routine that can handle arbitrary burst sizes * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_scattered_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { uint16_t retval = 0; - while (nb_pkts > RTE_I40E_VPMD_RX_BURST) { + while (nb_pkts > I40E_VPMD_RX_BURST) { uint16_t burst = i40e_recv_scattered_burst_vec_avx2(rx_queue, - rx_pkts + retval, RTE_I40E_VPMD_RX_BURST); + rx_pkts + retval, I40E_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_I40E_VPMD_RX_BURST) + if (burst < I40E_VPMD_RX_BURST) return retval; } return retval + i40e_recv_scattered_burst_vec_avx2(rx_queue, diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c index bb25acf398..f0320a221c 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c @@ -15,8 +15,6 @@ #include -#define RTE_I40E_DESCS_PER_LOOP_AVX 8 - static __rte_always_inline void i40e_rxq_rearm(struct i40e_rx_queue *rxq) { @@ -119,13 +117,13 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to RTE_I40E_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_I40E_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to I40E_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, I40E_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > I40E_VPMD_RXQ_REARM_THRESH) i40e_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -245,8 +243,8 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += RTE_I40E_DESCS_PER_LOOP_AVX, - rxdp += RTE_I40E_DESCS_PER_LOOP_AVX) { + i += I40E_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += I40E_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -312,7 +310,7 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, if (split_packet) { int j; - for (j = 0; j < RTE_I40E_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < I40E_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -642,7 +640,7 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += RTE_I40E_DESCS_PER_LOOP_AVX; + split_packet += I40E_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -657,7 +655,7 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, burst += rte_popcount64(_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; - if (burst != RTE_I40E_DESCS_PER_LOOP_AVX) + if (burst != I40E_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -674,7 +672,7 @@ _recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /** * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -686,7 +684,7 @@ i40e_recv_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, /** * vPMD receive routine that reassembles single burst of 32 scattered packets * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t i40e_recv_scattered_burst_vec_avx512(void *rx_queue, @@ -694,7 +692,7 @@ i40e_recv_scattered_burst_vec_avx512(void *rx_queue, uint16_t nb_pkts) { struct i40e_rx_queue *rxq = rx_queue; - uint8_t split_flags[RTE_I40E_VPMD_RX_BURST] = {0}; + uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _recv_raw_pkts_vec_avx512(rxq, rx_pkts, nb_pkts, @@ -729,7 +727,7 @@ i40e_recv_scattered_burst_vec_avx512(void *rx_queue, * vPMD receive routine that reassembles scattered packets. * Main receive routine that can handle arbitrary burst sizes * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_scattered_pkts_vec_avx512(void *rx_queue, @@ -738,12 +736,12 @@ i40e_recv_scattered_pkts_vec_avx512(void *rx_queue, { uint16_t retval = 0; - while (nb_pkts > RTE_I40E_VPMD_RX_BURST) { + while (nb_pkts > I40E_VPMD_RX_BURST) { uint16_t burst = i40e_recv_scattered_burst_vec_avx512(rx_queue, - rx_pkts + retval, RTE_I40E_VPMD_RX_BURST); + rx_pkts + retval, I40E_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_I40E_VPMD_RX_BURST) + if (burst < I40E_VPMD_RX_BURST) return retval; } return retval + i40e_recv_scattered_burst_vec_avx512(rx_queue, diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c index 695b4e1040..955382652c 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c @@ -33,21 +33,21 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (unlikely(rte_mempool_get_bulk(rxq->mp, (void *)rxep, - RTE_I40E_RXQ_REARM_THRESH) < 0)) { - if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + I40E_VPMD_RXQ_REARM_THRESH) < 0)) { + if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { - for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i].read), zero); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - RTE_I40E_RXQ_REARM_THRESH; + I40E_VPMD_RXQ_REARM_THRESH; return; } /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { mb0 = rxep[0].mbuf; mb1 = rxep[1].mbuf; @@ -62,7 +62,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; rx_id = rxq->rxrearm_start - 1; if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { @@ -70,7 +70,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) rx_id = rxq->nb_rx_desc - 1; } - rxq->rxrearm_nb -= RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; rte_io_wmb(); /* Update the tail pointer on the NIC */ @@ -325,11 +325,11 @@ desc_to_ptype_v(uint64x2_t descs[4], struct rte_mbuf **__rte_restrict rx_pkts, } /** - * vPMD raw receive routine, only accept(nb_pkts >= RTE_I40E_DESCS_PER_LOOP) + * vPMD raw receive routine, only accept(nb_pkts >= I40E_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_I40E_DESCS_PER_LOOP power-of-two + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a I40E_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, @@ -368,8 +368,8 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, 0, 0, 0 /* ignore non-length fields */ }; - /* nb_pkts has to be floor-aligned to RTE_I40E_DESCS_PER_LOOP */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_I40E_DESCS_PER_LOOP); + /* nb_pkts has to be floor-aligned to I40E_VPMD_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, I40E_VPMD_DESCS_PER_LOOP); /* Just the act of getting into the function from the application is * going to cost about 7 cycles @@ -381,7 +381,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > I40E_VPMD_RXQ_REARM_THRESH) i40e_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -405,9 +405,9 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, */ for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; - pos += RTE_I40E_DESCS_PER_LOOP, - rxdp += RTE_I40E_DESCS_PER_LOOP) { - uint64x2_t descs[RTE_I40E_DESCS_PER_LOOP]; + pos += I40E_VPMD_DESCS_PER_LOOP, + rxdp += I40E_VPMD_DESCS_PER_LOOP) { + uint64x2_t descs[I40E_VPMD_DESCS_PER_LOOP]; uint8x16_t pkt_mb1, pkt_mb2, pkt_mb3, pkt_mb4; uint16x8x2_t sterr_tmp1, sterr_tmp2; uint64x2_t mbp1, mbp2; @@ -502,9 +502,8 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, desc_to_ptype_v(descs, &rx_pkts[pos], ptype_tbl); - if (likely(pos + RTE_I40E_DESCS_PER_LOOP < nb_pkts)) { - rte_prefetch_non_temporal(rxdp + RTE_I40E_DESCS_PER_LOOP); - } + if (likely(pos + I40E_VPMD_DESCS_PER_LOOP < nb_pkts)) + rte_prefetch_non_temporal(rxdp + I40E_VPMD_DESCS_PER_LOOP); /* C.1 4=>2 filter staterr info only */ sterr_tmp2 = vzipq_u16(vreinterpretq_u16_u64(descs[1]), @@ -538,7 +537,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, /* store the resulting 32-bit value */ vst1q_lane_u32((uint32_t *)split_packet, vreinterpretq_u32_u8(eop_bits), 0); - split_packet += RTE_I40E_DESCS_PER_LOOP; + split_packet += I40E_VPMD_DESCS_PER_LOOP; /* zero-out next pointers */ rx_pkts[pos]->next = NULL; @@ -555,7 +554,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, /* C.4 calc available number of desc */ if (unlikely(stat == 0)) { - nb_pkts_recd += RTE_I40E_DESCS_PER_LOOP; + nb_pkts_recd += I40E_VPMD_DESCS_PER_LOOP; } else { nb_pkts_recd += rte_ctz64(stat) / I40E_UINT16_BIT; break; @@ -572,7 +571,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, /* * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_pkts_vec(void *__rte_restrict rx_queue, @@ -585,7 +584,7 @@ i40e_recv_pkts_vec(void *__rte_restrict rx_queue, * vPMD receive routine that reassembles single burst of 32 scattered packets * * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -593,7 +592,7 @@ i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { struct i40e_rx_queue *rxq = rx_queue; - uint8_t split_flags[RTE_I40E_VPMD_RX_BURST] = {0}; + uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _recv_raw_pkts_vec(rxq, rx_pkts, nb_pkts, @@ -633,15 +632,15 @@ i40e_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { uint16_t retval = 0; - while (nb_pkts > RTE_I40E_VPMD_RX_BURST) { + while (nb_pkts > I40E_VPMD_RX_BURST) { uint16_t burst; burst = i40e_recv_scattered_burst_vec(rx_queue, rx_pkts + retval, - RTE_I40E_VPMD_RX_BURST); + I40E_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_I40E_VPMD_RX_BURST) + if (burst < I40E_VPMD_RX_BURST) return retval; } diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c index 920089fe3e..7e7f4c0895 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c @@ -31,23 +31,23 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, - RTE_I40E_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + I40E_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - RTE_I40E_RXQ_REARM_THRESH; + I40E_VPMD_RXQ_REARM_THRESH; return; } /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { __m128i vaddr0, vaddr1; mb0 = rxep[0].mbuf; @@ -72,7 +72,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; rx_id = rxq->rxrearm_start - 1; if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { @@ -80,7 +80,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) rx_id = rxq->nb_rx_desc - 1; } - rxq->rxrearm_nb -= RTE_I40E_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; /* Update the tail pointer on the NIC */ I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); @@ -340,11 +340,11 @@ desc_to_ptype_v(__m128i descs[4], struct rte_mbuf **rx_pkts, } /** - * vPMD raw receive routine, only accept(nb_pkts >= RTE_I40E_DESCS_PER_LOOP) + * vPMD raw receive routine, only accept(nb_pkts >= I40E_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a RTE_I40E_DESCS_PER_LOOP power-of-two + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a I40E_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, @@ -376,8 +376,8 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, offsetof(struct rte_mbuf, rx_descriptor_fields1) + 8); __m128i dd_check, eop_check; - /* nb_pkts has to be floor-aligned to RTE_I40E_DESCS_PER_LOOP */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_I40E_DESCS_PER_LOOP); + /* nb_pkts has to be floor-aligned to I40E_VPMD_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, I40E_VPMD_DESCS_PER_LOOP); /* Just the act of getting into the function from the application is * going to cost about 7 cycles @@ -389,7 +389,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > I40E_VPMD_RXQ_REARM_THRESH) i40e_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -443,9 +443,9 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, */ for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; - pos += RTE_I40E_DESCS_PER_LOOP, - rxdp += RTE_I40E_DESCS_PER_LOOP) { - __m128i descs[RTE_I40E_DESCS_PER_LOOP]; + pos += I40E_VPMD_DESCS_PER_LOOP, + rxdp += I40E_VPMD_DESCS_PER_LOOP) { + __m128i descs[I40E_VPMD_DESCS_PER_LOOP]; __m128i pkt_mb1, pkt_mb2, pkt_mb3, pkt_mb4; __m128i zero, staterr, sterr_tmp1, sterr_tmp2; /* 2 64 bit or 4 32 bit mbuf pointers in one XMM reg. */ @@ -559,7 +559,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, eop_bits = _mm_shuffle_epi8(eop_bits, eop_shuf_mask); /* store the resulting 32-bit value */ *(int *)split_packet = _mm_cvtsi128_si32(eop_bits); - split_packet += RTE_I40E_DESCS_PER_LOOP; + split_packet += I40E_VPMD_DESCS_PER_LOOP; } /* C.3 calc available number of desc */ @@ -575,7 +575,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* C.4 calc available number of desc */ var = rte_popcount64(_mm_cvtsi128_si64(staterr)); nb_pkts_recd += var; - if (likely(var != RTE_I40E_DESCS_PER_LOOP)) + if (likely(var != I40E_VPMD_DESCS_PER_LOOP)) break; } @@ -589,7 +589,7 @@ _recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -602,7 +602,7 @@ i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles single burst of 32 scattered packets * * Notice: - * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts < I40E_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -610,7 +610,7 @@ i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { struct i40e_rx_queue *rxq = rx_queue; - uint8_t split_flags[RTE_I40E_VPMD_RX_BURST] = {0}; + uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _recv_raw_pkts_vec(rxq, rx_pkts, nb_pkts, @@ -650,15 +650,15 @@ i40e_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { uint16_t retval = 0; - while (nb_pkts > RTE_I40E_VPMD_RX_BURST) { + while (nb_pkts > I40E_VPMD_RX_BURST) { uint16_t burst; burst = i40e_recv_scattered_burst_vec(rx_queue, rx_pkts + retval, - RTE_I40E_VPMD_RX_BURST); + I40E_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < RTE_I40E_VPMD_RX_BURST) + if (burst < I40E_VPMD_RX_BURST) return retval; } From patchwork Mon Jun 9 15:37:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154218 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A58E2468B7; Mon, 9 Jun 2025 17:40:26 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 16A4E427D1; Mon, 9 Jun 2025 17:38:21 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 7C11A41104 for ; Mon, 9 Jun 2025 17:38:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483500; x=1781019500; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=lp/k/YJJCIOp8AMZSGV3S26/vp7230O3azGVvZ9kVk8=; b=aqzNAd9LmMI4IdvhCW8K/x5FFFHS23E/EHEfBS8832lpwEcKbk4CJHEP KVpeB5F+eBtnseOlgO7kPQpDbye8ZBtTvxTXC8mcHZN1j3c+PzMvJmO8h osNllD+yEv6TQ7LN9Anf2JaGkKBjYGGQMG/rEZosGjuARICfPk3NlHZoM EVglF7x/wFDM65M1Wxr1TBxzediVdCX0+syCpPx3c5+OAbWJbqHTrEyev ARS5dqiej8mw1ZcpfaPyce7c3eASw+a+OI6XU4/PiZhVkkD8CHnIO6nq0 VkQ6LUm6+txKBdRjigFzGp95KcNU/bxsUmc6hblffld03SX/wRvVe6/GY Q==; X-CSE-ConnectionGUID: fTMyYy09S9q4blufiq7R9w== X-CSE-MsgGUID: Mm33DijGTnK6za4rpFZ9WQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012221" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012221" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:19 -0700 X-CSE-ConnectionGUID: lsuoCl0MTKWfh7nGkuYLQg== X-CSE-MsgGUID: QCZMIv0OSoaWqaZKOZ5klQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419704" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:18 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson Subject: [PATCH v6 21/33] net/ice: clean up definitions Date: Mon, 9 Jun 2025 16:37:19 +0100 Message-ID: <4f5d7974404975311bd4f0c4bf571c5f351a9a40.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This commit does the following cleanups: - Mark vector-PMD related definitions with ICE_VPMD_ prefix - Remove unused definitions - Create "descriptors per loop" for different vector implementations (regular for SSE, Neon, wide for AVX2, AVX512) Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit drivers/net/intel/ice/ice_rxtx.h | 6 ++-- drivers/net/intel/ice/ice_rxtx_common_avx.h | 18 +++++----- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 24 ++++++------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 30 ++++++++-------- drivers/net/intel/ice/ice_rxtx_vec_sse.c | 40 ++++++++++----------- 5 files changed, 57 insertions(+), 61 deletions(-) diff --git a/drivers/net/intel/ice/ice_rxtx.h b/drivers/net/intel/ice/ice_rxtx.h index d2d521c4f5..52c753ba7c 100644 --- a/drivers/net/intel/ice/ice_rxtx.h +++ b/drivers/net/intel/ice/ice_rxtx.h @@ -35,10 +35,10 @@ #define ICE_VPMD_RX_BURST 32 #define ICE_VPMD_TX_BURST 32 -#define ICE_RXQ_REARM_THRESH 64 -#define ICE_MAX_RX_BURST ICE_RXQ_REARM_THRESH +#define ICE_VPMD_RXQ_REARM_THRESH 64 #define ICE_TX_MAX_FREE_BUF_SZ 64 -#define ICE_DESCS_PER_LOOP 4 +#define ICE_VPMD_DESCS_PER_LOOP 4 +#define ICE_VPMD_DESCS_PER_LOOP_WIDE 8 #define ICE_FDIR_PKT_LEN 512 diff --git a/drivers/net/intel/ice/ice_rxtx_common_avx.h b/drivers/net/intel/ice/ice_rxtx_common_avx.h index a68cf8512d..d1c772bf06 100644 --- a/drivers/net/intel/ice/ice_rxtx_common_avx.h +++ b/drivers/net/intel/ice/ice_rxtx_common_avx.h @@ -21,20 +21,20 @@ ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, - ICE_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + ICE_RXQ_REARM_THRESH >= + ICE_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + ICE_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { __m128i dma_addr0; dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < ICE_DESCS_PER_LOOP; i++) { + for (i = 0; i < ICE_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - ICE_RXQ_REARM_THRESH; + ICE_VPMD_RXQ_REARM_THRESH; return; } @@ -44,7 +44,7 @@ ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < ICE_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { __m128i vaddr0, vaddr1; mb0 = rxep[0].mbuf; @@ -84,7 +84,7 @@ ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) __m512i dma_addr0_3, dma_addr4_7; __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 8 mbufs in one loop */ - for (i = 0; i < ICE_RXQ_REARM_THRESH; + for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; i += 8, rxep += 8, rxdp += 8) { __m128i vaddr0, vaddr1, vaddr2, vaddr3; __m128i vaddr4, vaddr5, vaddr6, vaddr7; @@ -163,7 +163,7 @@ ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) __m256i dma_addr0_1, dma_addr2_3; __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < ICE_RXQ_REARM_THRESH; + for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; i += 4, rxep += 4, rxdp += 4) { __m128i vaddr0, vaddr1, vaddr2, vaddr3; __m256i vaddr0_1, vaddr2_3; @@ -216,11 +216,11 @@ ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) #endif - rxq->rxrearm_start += ICE_RXQ_REARM_THRESH; + rxq->rxrearm_start += ICE_VPMD_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) rxq->rxrearm_start = 0; - rxq->rxrearm_nb -= ICE_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= ICE_VPMD_RXQ_REARM_THRESH; rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c index 6fe5ffa6f4..5ed669fc30 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c @@ -37,8 +37,6 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { -#define ICE_DESCS_PER_LOOP_AVX 8 - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); @@ -48,13 +46,13 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to ICE_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, ICE_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to ICE_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, ICE_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > ICE_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > ICE_VPMD_RXQ_REARM_THRESH) ice_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -239,8 +237,8 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += ICE_DESCS_PER_LOOP_AVX, - rxdp += ICE_DESCS_PER_LOOP_AVX) { + i += ICE_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += ICE_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -286,7 +284,7 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, if (split_packet) { int j; - for (j = 0; j < ICE_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < ICE_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -634,7 +632,7 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += ICE_DESCS_PER_LOOP_AVX; + split_packet += ICE_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -650,7 +648,7 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; - if (burst != ICE_DESCS_PER_LOOP_AVX) + if (burst != ICE_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -667,7 +665,7 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, /** * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t ice_recv_pkts_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -688,7 +686,7 @@ ice_recv_pkts_vec_avx2_offload(void *rx_queue, struct rte_mbuf **rx_pkts, /** * vPMD receive routine that reassembles single burst of 32 scattered packets * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ static __rte_always_inline uint16_t ice_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -730,7 +728,7 @@ ice_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles scattered packets. * Main receive routine that can handle arbitrary burst sizes * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ static __rte_always_inline uint16_t ice_recv_scattered_pkts_vec_avx2_common(void *rx_queue, diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c index 490d1ae059..e52e9e9ceb 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c @@ -7,8 +7,6 @@ #include -#define ICE_DESCS_PER_LOOP_AVX 8 - static __rte_always_inline void ice_rxq_rearm(struct ice_rx_queue *rxq) { @@ -49,13 +47,13 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to ICE_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, ICE_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to ICE_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, ICE_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > ICE_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > ICE_VPMD_RXQ_REARM_THRESH) ice_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -224,8 +222,8 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += ICE_DESCS_PER_LOOP_AVX, - rxdp += ICE_DESCS_PER_LOOP_AVX) { + i += ICE_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += ICE_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -292,7 +290,7 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, if (split_packet) { int j; - for (j = 0; j < ICE_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < ICE_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -660,7 +658,7 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += ICE_DESCS_PER_LOOP_AVX; + split_packet += ICE_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -676,7 +674,7 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; - if (burst != ICE_DESCS_PER_LOOP_AVX) + if (burst != ICE_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -693,7 +691,7 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, /** * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t ice_recv_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -704,7 +702,7 @@ ice_recv_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, /** * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t ice_recv_pkts_vec_avx512_offload(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -717,7 +715,7 @@ ice_recv_pkts_vec_avx512_offload(void *rx_queue, struct rte_mbuf **rx_pkts, /** * vPMD receive routine that reassembles single burst of 32 scattered packets * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t ice_recv_scattered_burst_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -758,7 +756,7 @@ ice_recv_scattered_burst_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, /** * vPMD receive routine that reassembles single burst of 32 scattered packets * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t ice_recv_scattered_burst_vec_avx512_offload(void *rx_queue, @@ -801,7 +799,7 @@ ice_recv_scattered_burst_vec_avx512_offload(void *rx_queue, * vPMD receive routine that reassembles scattered packets. * Main receive routine that can handle arbitrary burst sizes * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t ice_recv_scattered_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, @@ -825,7 +823,7 @@ ice_recv_scattered_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles scattered packets. * Main receive routine that can handle arbitrary burst sizes * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ uint16_t ice_recv_scattered_pkts_vec_avx512_offload(void *rx_queue, diff --git a/drivers/net/intel/ice/ice_rxtx_vec_sse.c b/drivers/net/intel/ice/ice_rxtx_vec_sse.c index 719b37645e..36da5b5d1b 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_sse.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_sse.c @@ -42,23 +42,23 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxep, - ICE_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + ICE_RXQ_REARM_THRESH >= + ICE_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + ICE_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < ICE_DESCS_PER_LOOP; i++) { + for (i = 0; i < ICE_VPMD_DESCS_PER_LOOP; i++) { rxep[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - ICE_RXQ_REARM_THRESH; + ICE_VPMD_RXQ_REARM_THRESH; return; } /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < ICE_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { __m128i vaddr0, vaddr1; mb0 = rxep[0].mbuf; @@ -91,11 +91,11 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += ICE_RXQ_REARM_THRESH; + rxq->rxrearm_start += ICE_VPMD_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) rxq->rxrearm_start = 0; - rxq->rxrearm_nb -= ICE_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= ICE_VPMD_RXQ_REARM_THRESH; rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); @@ -294,11 +294,11 @@ ice_rx_desc_to_ptype_v(__m128i descs[4], struct rte_mbuf **rx_pkts, } /** - * vPMD raw receive routine, only accept(nb_pkts >= ICE_DESCS_PER_LOOP) + * vPMD raw receive routine, only accept(nb_pkts >= ICE_VPMD_DESCS_PER_LOOP) * * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet - * - floor align nb_pkts to a ICE_DESCS_PER_LOOP power-of-two + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet + * - floor align nb_pkts to a ICE_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, @@ -355,8 +355,8 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, const __m128i eop_check = _mm_set_epi64x(0x0000000200000002LL, 0x0000000200000002LL); - /* nb_pkts has to be floor-aligned to ICE_DESCS_PER_LOOP */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, ICE_DESCS_PER_LOOP); + /* nb_pkts has to be floor-aligned to ICE_VPMD_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, ICE_VPMD_DESCS_PER_LOOP); /* Just the act of getting into the function from the application is * going to cost about 7 cycles @@ -368,7 +368,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > ICE_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > ICE_VPMD_RXQ_REARM_THRESH) ice_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -406,9 +406,9 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, */ for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; - pos += ICE_DESCS_PER_LOOP, - rxdp += ICE_DESCS_PER_LOOP) { - __m128i descs[ICE_DESCS_PER_LOOP]; + pos += ICE_VPMD_DESCS_PER_LOOP, + rxdp += ICE_VPMD_DESCS_PER_LOOP) { + __m128i descs[ICE_VPMD_DESCS_PER_LOOP]; __m128i pkt_mb0, pkt_mb1, pkt_mb2, pkt_mb3; __m128i staterr, sterr_tmp1, sterr_tmp2; /* 2 64 bit or 4 32 bit mbuf pointers in one XMM reg. */ @@ -556,7 +556,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, eop_bits = _mm_shuffle_epi8(eop_bits, eop_shuf_mask); /* store the resulting 32-bit value */ *(int *)split_packet = _mm_cvtsi128_si32(eop_bits); - split_packet += ICE_DESCS_PER_LOOP; + split_packet += ICE_VPMD_DESCS_PER_LOOP; } /* C.3 calc available number of desc */ @@ -573,7 +573,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* C.4 calc available number of desc */ var = rte_popcount64(_mm_cvtsi128_si64(staterr)); nb_pkts_recd += var; - if (likely(var != ICE_DESCS_PER_LOOP)) + if (likely(var != ICE_VPMD_DESCS_PER_LOOP)) break; } @@ -587,7 +587,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, /** * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet * - nb_pkts > ICE_VPMD_RX_BURST, only scan ICE_VPMD_RX_BURST * numbers of DD bits */ @@ -602,7 +602,7 @@ ice_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, * vPMD receive routine that reassembles single burst of 32 scattered packets * * Notice: - * - nb_pkts < ICE_DESCS_PER_LOOP, just return no packet + * - nb_pkts < ICE_VPMD_DESCS_PER_LOOP, just return no packet */ static uint16_t ice_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, From patchwork Mon Jun 9 15:37:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154219 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A4009468B7; Mon, 9 Jun 2025 17:40:36 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8FD1D427E7; Mon, 9 Jun 2025 17:38:23 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 7A2B2427D9 for ; Mon, 9 Jun 2025 17:38:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483502; x=1781019502; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kP05oIFxOUX6gaR/8AuLA/kNUkI66vLH8UzO81ks718=; b=h5387gEqioGF3eyyBzZd5NzS3UXhSgbU2UMTjfMUeNWIzJeg2k06n6kf NwXaTnLIAgJ4FG1utJqr7r5rguE+o10RQhQ3U6YWSk/yxe2IIgIAq+miv fnmE0i7tiBzYH/IqwADrwXiW6hnllnpngV+mwhxLkQgs33M6yZejvOOI0 lAU/gVYiumgbLdKQCTpVCyS8gDo0dD3Pt0xKz8Yi9sRRJqThz2/ng3IS/ RAOFC7dch25/NdQ1J5/TtJNpFzA/0SiGPVCxoFKXPQcZ673FFT2y79c9q Tf8WVars+FeFduLHp5aE1rCOSqZ+PlhQQ/koKaCx9XGCGhMiVGoRqsm9v w==; X-CSE-ConnectionGUID: imAF/9VHRMiHLFnrzYjRqg== X-CSE-MsgGUID: fplERol6SK6dI8WbE7IEdw== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012224" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012224" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:21 -0700 X-CSE-ConnectionGUID: p02O4wcPQ8ub+SWzYh9MbQ== X-CSE-MsgGUID: FMs/ygJUQfGj3R29MKU5Vg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419713" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:19 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin , Ian Stokes Cc: bruce.richardson@intel.com Subject: [PATCH v6 22/33] net/iavf: clean up definitions Date: Mon, 9 Jun 2025 16:37:20 +0100 Message-ID: <1a2cafc7701d1825362d35352433523aa06176fe.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This commit does the following cleanups: - Mark vector-PMD related definitions with IAVF_VPMD_ prefix - Create "descriptors per loop" for different vector implementations (regular for SSE, Neon, AltiVec, wide for AVX2, AVX512) - Make definitions' names match naming conventions used in other drivers Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Add this commit drivers/net/intel/iavf/iavf_rxtx.c | 2 +- drivers/net/intel/iavf/iavf_rxtx.h | 11 ++-- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 52 +++++++++---------- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 49 +++++++++-------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 16 +++--- drivers/net/intel/iavf/iavf_rxtx_vec_neon.c | 14 ++--- drivers/net/intel/iavf/iavf_rxtx_vec_sse.c | 20 +++---- 7 files changed, 80 insertions(+), 84 deletions(-) diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c index 7b10c0314f..5c798f2b6e 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.c +++ b/drivers/net/intel/iavf/iavf_rxtx.c @@ -212,7 +212,7 @@ static inline bool check_tx_vec_allow(struct ci_tx_queue *txq) { if (!(txq->offloads & IAVF_TX_NO_VECTOR_FLAGS) && - txq->tx_rs_thresh >= IAVF_VPMD_TX_MAX_BURST && + txq->tx_rs_thresh >= IAVF_VPMD_TX_BURST && txq->tx_rs_thresh <= IAVF_VPMD_TX_MAX_FREE_BUF) { PMD_INIT_LOG(DEBUG, "Vector tx can be enabled on this txq."); return true; diff --git a/drivers/net/intel/iavf/iavf_rxtx.h b/drivers/net/intel/iavf/iavf_rxtx.h index a0e1fd8667..258103e222 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.h +++ b/drivers/net/intel/iavf/iavf_rxtx.h @@ -23,11 +23,12 @@ #define IAVF_RX_MAX_DATA_BUF_SIZE (16 * 1024 - 128) /* used for Vector PMD */ -#define IAVF_VPMD_RX_MAX_BURST 32 -#define IAVF_VPMD_TX_MAX_BURST 32 -#define IAVF_RXQ_REARM_THRESH 32 -#define IAVF_VPMD_DESCS_PER_LOOP 4 -#define IAVF_VPMD_TX_MAX_FREE_BUF 64 +#define IAVF_VPMD_RX_BURST 32 +#define IAVF_VPMD_TX_BURST 32 +#define IAVF_VPMD_RXQ_REARM_THRESH 32 +#define IAVF_VPMD_DESCS_PER_LOOP 4 +#define IAVF_VPMD_DESCS_PER_LOOP_WIDE 8 +#define IAVF_VPMD_TX_MAX_FREE_BUF 64 #define IAVF_TX_NO_VECTOR_FLAGS ( \ RTE_ETH_TX_OFFLOAD_VLAN_INSERT | \ diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c index c7dc5bbe3e..b4fe77a98b 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c @@ -20,8 +20,6 @@ _iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { -#define IAVF_DESCS_PER_LOOP_AVX 8 - /* const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; */ const uint32_t *type_table = rxq->vsi->adapter->ptype_tbl; @@ -34,13 +32,13 @@ _iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to IAVF_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to IAVF_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > IAVF_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IAVF_VPMD_RXQ_REARM_THRESH) iavf_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -178,8 +176,8 @@ _iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += IAVF_DESCS_PER_LOOP_AVX, - rxdp += IAVF_DESCS_PER_LOOP_AVX) { + i += IAVF_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += IAVF_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -217,7 +215,7 @@ _iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, if (split_packet) { int j; - for (j = 0; j < IAVF_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < IAVF_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -436,7 +434,7 @@ _iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += IAVF_DESCS_PER_LOOP_AVX; + split_packet += IAVF_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -452,7 +450,7 @@ _iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; - if (burst != IAVF_DESCS_PER_LOOP_AVX) + if (burst != IAVF_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -492,8 +490,6 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { -#define IAVF_DESCS_PER_LOOP_AVX 8 - struct iavf_adapter *adapter = rxq->vsi->adapter; uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; const uint32_t *type_table = adapter->ptype_tbl; @@ -506,13 +502,13 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to IAVF_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to IAVF_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > IAVF_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IAVF_VPMD_RXQ_REARM_THRESH) iavf_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -720,8 +716,8 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += IAVF_DESCS_PER_LOOP_AVX, - rxdp += IAVF_DESCS_PER_LOOP_AVX) { + i += IAVF_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += IAVF_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -777,7 +773,7 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, if (split_packet) { int j; - for (j = 0; j < IAVF_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < IAVF_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -1337,7 +1333,7 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += IAVF_DESCS_PER_LOOP_AVX; + split_packet += IAVF_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -1398,7 +1394,7 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); } - if (burst != IAVF_DESCS_PER_LOOP_AVX) + if (burst != IAVF_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -1466,7 +1462,7 @@ iavf_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool offload) { struct iavf_rx_queue *rxq = rx_queue; - uint8_t split_flags[IAVF_VPMD_RX_MAX_BURST] = {0}; + uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _iavf_recv_raw_pkts_vec_avx2(rxq, rx_pkts, nb_pkts, @@ -1509,12 +1505,12 @@ iavf_recv_scattered_pkts_vec_avx2_common(void *rx_queue, struct rte_mbuf **rx_pk { uint16_t retval = 0; - while (nb_pkts > IAVF_VPMD_RX_MAX_BURST) { + while (nb_pkts > IAVF_VPMD_RX_BURST) { uint16_t burst = iavf_recv_scattered_burst_vec_avx2(rx_queue, - rx_pkts + retval, IAVF_VPMD_RX_MAX_BURST, offload); + rx_pkts + retval, IAVF_VPMD_RX_BURST, offload); retval += burst; nb_pkts -= burst; - if (burst < IAVF_VPMD_RX_MAX_BURST) + if (burst < IAVF_VPMD_RX_BURST) return retval; } return retval + iavf_recv_scattered_burst_vec_avx2(rx_queue, @@ -1555,7 +1551,7 @@ iavf_recv_scattered_burst_vec_avx2_flex_rxd(void *rx_queue, uint16_t nb_pkts, bool offload) { struct iavf_rx_queue *rxq = rx_queue; - uint8_t split_flags[IAVF_VPMD_RX_MAX_BURST] = {0}; + uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _iavf_recv_raw_pkts_vec_avx2_flex_rxd(rxq, @@ -1599,14 +1595,14 @@ iavf_recv_scattered_pkts_vec_avx2_flex_rxd_common(void *rx_queue, { uint16_t retval = 0; - while (nb_pkts > IAVF_VPMD_RX_MAX_BURST) { + while (nb_pkts > IAVF_VPMD_RX_BURST) { uint16_t burst = iavf_recv_scattered_burst_vec_avx2_flex_rxd - (rx_queue, rx_pkts + retval, IAVF_VPMD_RX_MAX_BURST, + (rx_queue, rx_pkts + retval, IAVF_VPMD_RX_BURST, offload); retval += burst; nb_pkts -= burst; - if (burst < IAVF_VPMD_RX_MAX_BURST) + if (burst < IAVF_VPMD_RX_BURST) return retval; } return retval + iavf_recv_scattered_burst_vec_avx2_flex_rxd(rx_queue, diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c index 51a2dc12bf..6eac24baf5 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c @@ -6,7 +6,6 @@ #include -#define IAVF_DESCS_PER_LOOP_AVX 8 #define PKTLEN_SHIFT 10 /****************************************************************************** @@ -51,13 +50,13 @@ _iavf_recv_raw_pkts_vec_avx512(struct iavf_rx_queue *rxq, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to IAVF_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to IAVF_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > IAVF_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IAVF_VPMD_RXQ_REARM_THRESH) iavf_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -148,8 +147,8 @@ _iavf_recv_raw_pkts_vec_avx512(struct iavf_rx_queue *rxq, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += IAVF_DESCS_PER_LOOP_AVX, - rxdp += IAVF_DESCS_PER_LOOP_AVX) { + i += IAVF_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += IAVF_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -196,7 +195,7 @@ _iavf_recv_raw_pkts_vec_avx512(struct iavf_rx_queue *rxq, if (split_packet) { int j; - for (j = 0; j < IAVF_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < IAVF_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -527,7 +526,7 @@ _iavf_recv_raw_pkts_vec_avx512(struct iavf_rx_queue *rxq, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += IAVF_DESCS_PER_LOOP_AVX; + split_packet += IAVF_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -543,7 +542,7 @@ _iavf_recv_raw_pkts_vec_avx512(struct iavf_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; - if (burst != IAVF_DESCS_PER_LOOP_AVX) + if (burst != IAVF_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -598,13 +597,13 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, rte_prefetch0(rxdp); - /* nb_pkts has to be floor-aligned to IAVF_DESCS_PER_LOOP_AVX */ - nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_DESCS_PER_LOOP_AVX); + /* nb_pkts has to be floor-aligned to IAVF_VPMD_DESCS_PER_LOOP_WIDE */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, IAVF_VPMD_DESCS_PER_LOOP_WIDE); /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > IAVF_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IAVF_VPMD_RXQ_REARM_THRESH) iavf_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if @@ -712,8 +711,8 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, uint16_t i, received; for (i = 0, received = 0; i < nb_pkts; - i += IAVF_DESCS_PER_LOOP_AVX, - rxdp += IAVF_DESCS_PER_LOOP_AVX) { + i += IAVF_VPMD_DESCS_PER_LOOP_WIDE, + rxdp += IAVF_VPMD_DESCS_PER_LOOP_WIDE) { /* step 1, copy over 8 mbuf pointers to rx_pkts array */ _mm256_storeu_si256((void *)&rx_pkts[i], _mm256_loadu_si256((void *)&sw_ring[i])); @@ -761,7 +760,7 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, if (split_packet) { int j; - for (j = 0; j < IAVF_DESCS_PER_LOOP_AVX; j++) + for (j = 0; j < IAVF_VPMD_DESCS_PER_LOOP_WIDE; j++) rte_mbuf_prefetch_part2(rx_pkts[i + j]); } @@ -1526,7 +1525,7 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, split_bits = _mm_shuffle_epi8(split_bits, eop_shuffle); *(uint64_t *)split_packet = _mm_cvtsi128_si64(split_bits); - split_packet += IAVF_DESCS_PER_LOOP_AVX; + split_packet += IAVF_VPMD_DESCS_PER_LOOP_WIDE; } /* perform dd_check */ @@ -1589,7 +1588,7 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); } #endif - if (burst != IAVF_DESCS_PER_LOOP_AVX) + if (burst != IAVF_VPMD_DESCS_PER_LOOP_WIDE) break; } @@ -1644,7 +1643,7 @@ iavf_recv_scattered_burst_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool offload) { struct iavf_rx_queue *rxq = rx_queue; - uint8_t split_flags[IAVF_VPMD_RX_MAX_BURST] = {0}; + uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _iavf_recv_raw_pkts_vec_avx512(rxq, rx_pkts, nb_pkts, @@ -1687,12 +1686,12 @@ iavf_recv_scattered_pkts_vec_avx512_cmn(void *rx_queue, struct rte_mbuf **rx_pkt { uint16_t retval = 0; - while (nb_pkts > IAVF_VPMD_RX_MAX_BURST) { + while (nb_pkts > IAVF_VPMD_RX_BURST) { uint16_t burst = iavf_recv_scattered_burst_vec_avx512(rx_queue, - rx_pkts + retval, IAVF_VPMD_RX_MAX_BURST, offload); + rx_pkts + retval, IAVF_VPMD_RX_BURST, offload); retval += burst; nb_pkts -= burst; - if (burst < IAVF_VPMD_RX_MAX_BURST) + if (burst < IAVF_VPMD_RX_BURST) return retval; } return retval + iavf_recv_scattered_burst_vec_avx512(rx_queue, @@ -1720,7 +1719,7 @@ iavf_recv_scattered_burst_vec_avx512_flex_rxd(void *rx_queue, bool offload) { struct iavf_rx_queue *rxq = rx_queue; - uint8_t split_flags[IAVF_VPMD_RX_MAX_BURST] = {0}; + uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ uint16_t nb_bufs = _iavf_recv_raw_pkts_vec_avx512_flex_rxd(rxq, @@ -1765,14 +1764,14 @@ iavf_recv_scattered_pkts_vec_avx512_flex_rxd_cmn(void *rx_queue, { uint16_t retval = 0; - while (nb_pkts > IAVF_VPMD_RX_MAX_BURST) { + while (nb_pkts > IAVF_VPMD_RX_BURST) { uint16_t burst = iavf_recv_scattered_burst_vec_avx512_flex_rxd (rx_queue, rx_pkts + retval, - IAVF_VPMD_RX_MAX_BURST, offload); + IAVF_VPMD_RX_BURST, offload); retval += burst; nb_pkts -= burst; - if (burst < IAVF_VPMD_RX_MAX_BURST) + if (burst < IAVF_VPMD_RX_BURST) return retval; } return retval + iavf_recv_scattered_burst_vec_avx512_flex_rxd(rx_queue, diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h index 326b8b07ba..8c31334570 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h @@ -59,7 +59,7 @@ iavf_rx_vec_queue_default(struct iavf_rx_queue *rxq) if (!rte_is_power_of_2(rxq->nb_rx_desc)) return -1; - if (rxq->rx_free_thresh < IAVF_VPMD_RX_MAX_BURST) + if (rxq->rx_free_thresh < IAVF_VPMD_RX_BURST) return -1; if (rxq->nb_rx_desc % rxq->rx_free_thresh) @@ -80,7 +80,7 @@ iavf_tx_vec_queue_default(struct ci_tx_queue *txq) if (!txq) return -1; - if (txq->tx_rs_thresh < IAVF_VPMD_TX_MAX_BURST || + if (txq->tx_rs_thresh < IAVF_VPMD_TX_BURST || txq->tx_rs_thresh > IAVF_VPMD_TX_MAX_FREE_BUF) return -1; @@ -252,8 +252,8 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, (void *)rxp, - IAVF_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + IAVF_RXQ_REARM_THRESH >= + IAVF_VPMD_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + IAVF_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { __m128i dma_addr0; @@ -265,7 +265,7 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - IAVF_RXQ_REARM_THRESH; + IAVF_VPMD_RXQ_REARM_THRESH; return; } @@ -274,7 +274,7 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < IAVF_RXQ_REARM_THRESH; i += 2, rxp += 2) { + for (i = 0; i < IAVF_VPMD_RXQ_REARM_THRESH; i += 2, rxp += 2) { __m128i vaddr0, vaddr1; mb0 = rxp[0]; @@ -299,11 +299,11 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += IAVF_RXQ_REARM_THRESH; + rxq->rxrearm_start += IAVF_VPMD_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) rxq->rxrearm_start = 0; - rxq->rxrearm_nb -= IAVF_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= IAVF_VPMD_RXQ_REARM_THRESH; rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c b/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c index a583340f15..86f3a7839d 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c @@ -31,8 +31,8 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) /* Pull 'n' more MBUFs into the software ring */ if (unlikely(rte_mempool_get_bulk(rxq->mp, (void *)rxep, - IAVF_RXQ_REARM_THRESH) < 0)) { - if (rxq->rxrearm_nb + IAVF_RXQ_REARM_THRESH >= + IAVF_VPMD_RXQ_REARM_THRESH) < 0)) { + if (rxq->rxrearm_nb + IAVF_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { rxep[i] = &rxq->fake_mbuf; @@ -40,12 +40,12 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) } } rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - IAVF_RXQ_REARM_THRESH; + IAVF_VPMD_RXQ_REARM_THRESH; return; } /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < IAVF_RXQ_REARM_THRESH; i += 2, rxep += 2) { + for (i = 0; i < IAVF_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { mb0 = rxep[0]; mb1 = rxep[1]; @@ -60,11 +60,11 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr1); } - rxq->rxrearm_start += IAVF_RXQ_REARM_THRESH; + rxq->rxrearm_start += IAVF_VPMD_RXQ_REARM_THRESH; if (rxq->rxrearm_start >= rxq->nb_rx_desc) rxq->rxrearm_start = 0; - rxq->rxrearm_nb -= IAVF_RXQ_REARM_THRESH; + rxq->rxrearm_nb -= IAVF_VPMD_RXQ_REARM_THRESH; rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); @@ -233,7 +233,7 @@ _recv_raw_pkts_vec(struct iavf_rx_queue *__rte_restrict rxq, /* See if we need to rearm the RX queue - gives the prefetch a bit * of time to act */ - if (rxq->rxrearm_nb > IAVF_RXQ_REARM_THRESH) + if (rxq->rxrearm_nb > IAVF_VPMD_RXQ_REARM_THRESH) iavf_rxq_rearm(rxq); /* Before we start moving massive data around, check to see if diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c index 9c1f8276d0..0633a0c33d 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c @@ -1150,7 +1150,7 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, /* Notice: * - nb_pkts < IAVF_DESCS_PER_LOOP, just return no packet - * - nb_pkts > IAVF_VPMD_RX_MAX_BURST, only scan IAVF_VPMD_RX_MAX_BURST + * - nb_pkts > IAVF_VPMD_RX_BURST, only scan IAVF_VPMD_RX_BURST * numbers of DD bits */ uint16_t @@ -1162,7 +1162,7 @@ iavf_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, /* Notice: * - nb_pkts < IAVF_DESCS_PER_LOOP, just return no packet - * - nb_pkts > IAVF_VPMD_RX_MAX_BURST, only scan IAVF_VPMD_RX_MAX_BURST + * - nb_pkts > IAVF_VPMD_RX_BURST, only scan IAVF_VPMD_RX_BURST * numbers of DD bits */ uint16_t @@ -1183,7 +1183,7 @@ iavf_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { struct iavf_rx_queue *rxq = rx_queue; - uint8_t split_flags[IAVF_VPMD_RX_MAX_BURST] = {0}; + uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; unsigned int i = 0; /* get some new buffers */ @@ -1222,15 +1222,15 @@ iavf_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, { uint16_t retval = 0; - while (nb_pkts > IAVF_VPMD_RX_MAX_BURST) { + while (nb_pkts > IAVF_VPMD_RX_BURST) { uint16_t burst; burst = iavf_recv_scattered_burst_vec(rx_queue, rx_pkts + retval, - IAVF_VPMD_RX_MAX_BURST); + IAVF_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < IAVF_VPMD_RX_MAX_BURST) + if (burst < IAVF_VPMD_RX_BURST) return retval; } @@ -1252,7 +1252,7 @@ iavf_recv_scattered_burst_vec_flex_rxd(void *rx_queue, uint16_t nb_pkts) { struct iavf_rx_queue *rxq = rx_queue; - uint8_t split_flags[IAVF_VPMD_RX_MAX_BURST] = {0}; + uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; unsigned int i = 0; /* get some new buffers */ @@ -1292,15 +1292,15 @@ iavf_recv_scattered_pkts_vec_flex_rxd(void *rx_queue, { uint16_t retval = 0; - while (nb_pkts > IAVF_VPMD_RX_MAX_BURST) { + while (nb_pkts > IAVF_VPMD_RX_BURST) { uint16_t burst; burst = iavf_recv_scattered_burst_vec_flex_rxd(rx_queue, rx_pkts + retval, - IAVF_VPMD_RX_MAX_BURST); + IAVF_VPMD_RX_BURST); retval += burst; nb_pkts -= burst; - if (burst < IAVF_VPMD_RX_MAX_BURST) + if (burst < IAVF_VPMD_RX_BURST) return retval; } From patchwork Mon Jun 9 15:37:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154220 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0A9F2468B7; Mon, 9 Jun 2025 17:40:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 76EE1427DF; Mon, 9 Jun 2025 17:38:25 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 4B4FC427DD for ; Mon, 9 Jun 2025 17:38:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483504; x=1781019504; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=TZgNVwoP9y7rve+vWkdQ5l86v9cAogSyMrV0WW/49Pw=; b=k1wNubQdGFFWruJ0p90qKfowGmt+4cmUHQzMV8/LBLKS/Z4gaKT13f0X lDoy7cu8XoSGeHa9ktz2U8NJ9947vHHYVtFZV5uG3zJYqkrz4ShzS3t34 OngASTdcgf/Qbk9ztAmbVYX9FoXQEJ5lXdIIZBMJgH4LQ0iYJGslF02VO GvgWvpj7SuPKTqEkoYaDoUbdoQuTSJDB7hiHm9C+4dZju+1iEdPQURcfl qvguwdGERdLIr55gTK981+FMMR+ysprSUnjnFzKVbGI+vUwBPpr+UZDco ZK7CG0kDfp4qzgfMqfZ0AC55etOqrGbk3nvFNoKvXVEZffK83v2aNum5A g==; X-CSE-ConnectionGUID: rsixDdzXSlq5XL/OyBXnQQ== X-CSE-MsgGUID: JftFDVuZTfazMzR5YVho0Q== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012227" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012227" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:23 -0700 X-CSE-ConnectionGUID: tJSV8Bg3R8aShMmcnWsY6w== X-CSE-MsgGUID: rTulYXlXRIaAjkbBn/6hiw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419721" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:21 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Vladimir Medvedkin Subject: [PATCH v6 23/33] net/ixgbe: create common Rx queue structure Date: Mon, 9 Jun 2025 16:37:21 +0100 Message-ID: <67e447a725410d7ef1af5317e3c7e1ee487792c3.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In preparation for deduplication effort, generalize the Rx queue structure. The entire Rx queue structure is moved to common/rx.h, clarifying the comments where necessary, and separating common parts from ixgbe-specific parts. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Sort ixgbe-specific fields by size v3 -> v4: - Separate out some of the changes from this commit into previous commits - Rename CI_RX_BURST to CI_RX_MAX_BURST to match the driver naming convention drivers/net/intel/common/rx.h | 67 ++++++++++- drivers/net/intel/ixgbe/ixgbe_ethdev.c | 8 +- drivers/net/intel/ixgbe/ixgbe_rxtx.c | 108 +++++++++--------- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 61 +--------- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.c | 12 +- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.h | 5 +- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c | 14 +-- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c | 14 +-- 8 files changed, 146 insertions(+), 143 deletions(-) diff --git a/drivers/net/intel/common/rx.h b/drivers/net/intel/common/rx.h index abb01ba5e7..b60ca24dfb 100644 --- a/drivers/net/intel/common/rx.h +++ b/drivers/net/intel/common/rx.h @@ -10,14 +10,75 @@ #include #include -#define CI_RX_BURST 32 +#define CI_RX_MAX_BURST 32 + +struct ci_rx_queue; + +struct ci_rx_entry { + struct rte_mbuf *mbuf; /* mbuf associated with RX descriptor. */ +}; + +struct ci_rx_entry_sc { + struct rte_mbuf *fbuf; /* First segment of the fragmented packet.*/ +}; + +/** + * Structure associated with each RX queue. + */ +struct ci_rx_queue { + struct rte_mempool *mp; /**< mbuf pool to populate RX ring. */ + union { /* RX ring virtual address */ + volatile union ixgbe_adv_rx_desc *ixgbe_rx_ring; + }; + volatile uint8_t *qrx_tail; /**< register address of tail */ + struct ci_rx_entry *sw_ring; /**< address of RX software ring. */ + struct ci_rx_entry_sc *sw_sc_ring; /**< address of scattered Rx software ring. */ + rte_iova_t rx_ring_phys_addr; /**< RX ring DMA address. */ + struct rte_mbuf *pkt_first_seg; /**< First segment of current packet. */ + struct rte_mbuf *pkt_last_seg; /**< Last segment of current packet. */ + /** hold packets to return to application */ + struct rte_mbuf *rx_stage[CI_RX_MAX_BURST * 2]; + uint16_t nb_rx_desc; /**< number of RX descriptors. */ + uint16_t rx_tail; /**< current value of tail register. */ + uint16_t rx_nb_avail; /**< nr of staged pkts ready to ret to app */ + uint16_t nb_rx_hold; /**< number of held free RX desc. */ + uint16_t rx_next_avail; /**< idx of next staged pkt to ret to app */ + uint16_t rx_free_thresh; /**< max free RX desc to hold. */ + uint16_t rx_free_trigger; /**< triggers rx buffer allocation */ + uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ + uint16_t rxrearm_start; /**< the idx we start the re-arming from */ + uint16_t queue_id; /**< RX queue index. */ + uint16_t port_id; /**< Device port identifier. */ + uint16_t reg_idx; /**< RX queue register index. */ + uint8_t crc_len; /**< 0 if CRC stripped, 4 otherwise. */ + bool rx_deferred_start; /**< queue is not started on dev start. */ + bool vector_rx; /**< indicates that vector RX is in use */ + bool drop_en; /**< if 1, drop packets if no descriptors are available. */ + uint64_t mbuf_initializer; /**< value to init mbufs */ + uint64_t offloads; /**< Rx offloads with RTE_ETH_RX_OFFLOAD_* */ + /** need to alloc dummy mbuf, for wraparound when scanning hw ring */ + struct rte_mbuf fake_mbuf; + const struct rte_memzone *mz; + union { + struct { /* ixgbe specific values */ + /** flags to set in mbuf when a vlan is detected. */ + uint64_t vlan_flags; + /** Packet type mask for different NICs. */ + uint16_t pkt_type_mask; + /** indicates that IPsec RX feature is in use */ + uint8_t using_ipsec; + /** UDP frames with a 0 checksum can be marked as checksum errors. */ + uint8_t rx_udp_csum_zero_err; + }; + }; +}; static inline uint16_t ci_rx_reassemble_packets(struct rte_mbuf **rx_bufs, uint16_t nb_bufs, uint8_t *split_flags, struct rte_mbuf **pkt_first_seg, struct rte_mbuf **pkt_last_seg, const uint8_t crc_len) { - struct rte_mbuf *pkts[CI_RX_BURST] = {0}; /*finished pkts*/ + struct rte_mbuf *pkts[CI_RX_MAX_BURST] = {0}; /*finished pkts*/ struct rte_mbuf *start = *pkt_first_seg; struct rte_mbuf *end = *pkt_last_seg; unsigned int pkt_idx, buf_idx; @@ -97,7 +158,7 @@ static inline bool ci_rxq_vec_capable(uint16_t nb_desc, uint16_t rx_free_thresh, uint64_t offloads) { if (!rte_is_power_of_2(nb_desc) || - rx_free_thresh < CI_RX_BURST || + rx_free_thresh < CI_RX_MAX_BURST || (nb_desc % rx_free_thresh) != 0) return false; diff --git a/drivers/net/intel/ixgbe/ixgbe_ethdev.c b/drivers/net/intel/ixgbe/ixgbe_ethdev.c index 928ac57a93..f8b99d4de5 100644 --- a/drivers/net/intel/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/intel/ixgbe/ixgbe_ethdev.c @@ -2022,7 +2022,7 @@ ixgbe_vlan_hw_strip_bitmap_set(struct rte_eth_dev *dev, uint16_t queue, bool on) { struct ixgbe_hwstrip *hwstrip = IXGBE_DEV_PRIVATE_TO_HWSTRIP_BITMAP(dev->data->dev_private); - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; if (queue >= IXGBE_MAX_RX_QUEUE_NUM) return; @@ -2157,7 +2157,7 @@ ixgbe_vlan_hw_strip_config(struct rte_eth_dev *dev) struct rte_eth_rxmode *rxmode = &dev->data->dev_conf.rxmode; uint32_t ctrl; uint16_t i; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; bool on; PMD_INIT_FUNC_TRACE(); @@ -2200,7 +2200,7 @@ ixgbe_config_vlan_strip_on_all_queues(struct rte_eth_dev *dev, int mask) { uint16_t i; struct rte_eth_rxmode *rxmode; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; if (mask & RTE_ETH_VLAN_STRIP_MASK) { rxmode = &dev->data->dev_conf.rxmode; @@ -5782,7 +5782,7 @@ ixgbevf_vlan_strip_queue_set(struct rte_eth_dev *dev, uint16_t queue, int on) static int ixgbevf_vlan_offload_config(struct rte_eth_dev *dev, int mask) { - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint16_t i; int on = 0; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.c b/drivers/net/intel/ixgbe/ixgbe_rxtx.c index 5b2067bc0e..bbe665a6ff 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.c @@ -1403,11 +1403,11 @@ int ixgbe_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc) { volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint16_t desc; desc = rxq->rx_tail; - rxdp = &rxq->rx_ring[desc]; + rxdp = &rxq->ixgbe_rx_ring[desc]; /* watch for changes in status bit */ pmc->addr = &rxdp->wb.upper.status_error; @@ -1547,10 +1547,10 @@ rx_desc_error_to_pkt_flags(uint32_t rx_status, uint16_t pkt_info, #error "PMD IXGBE: LOOK_AHEAD must be 8\n" #endif static inline int -ixgbe_rx_scan_hw_ring(struct ixgbe_rx_queue *rxq) +ixgbe_rx_scan_hw_ring(struct ci_rx_queue *rxq) { volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *rxep; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t pkt_len; uint64_t pkt_flags; @@ -1562,7 +1562,7 @@ ixgbe_rx_scan_hw_ring(struct ixgbe_rx_queue *rxq) uint64_t vlan_flags = rxq->vlan_flags; /* get references to current descriptor and S/W ring entry */ - rxdp = &rxq->rx_ring[rxq->rx_tail]; + rxdp = &rxq->ixgbe_rx_ring[rxq->rx_tail]; rxep = &rxq->sw_ring[rxq->rx_tail]; status = rxdp->wb.upper.status_error; @@ -1647,10 +1647,10 @@ ixgbe_rx_scan_hw_ring(struct ixgbe_rx_queue *rxq) } static inline int -ixgbe_rx_alloc_bufs(struct ixgbe_rx_queue *rxq, bool reset_mbuf) +ixgbe_rx_alloc_bufs(struct ci_rx_queue *rxq, bool reset_mbuf) { volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *rxep; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t alloc_idx; __le64 dma_addr; @@ -1664,7 +1664,7 @@ ixgbe_rx_alloc_bufs(struct ixgbe_rx_queue *rxq, bool reset_mbuf) if (unlikely(diag != 0)) return -ENOMEM; - rxdp = &rxq->rx_ring[alloc_idx]; + rxdp = &rxq->ixgbe_rx_ring[alloc_idx]; for (i = 0; i < rxq->rx_free_thresh; ++i) { /* populate the static rte mbuf fields */ mb = rxep[i].mbuf; @@ -1691,7 +1691,7 @@ ixgbe_rx_alloc_bufs(struct ixgbe_rx_queue *rxq, bool reset_mbuf) } static inline uint16_t -ixgbe_rx_fill_from_stage(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, +ixgbe_rx_fill_from_stage(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { struct rte_mbuf **stage = &rxq->rx_stage[rxq->rx_next_avail]; @@ -1715,7 +1715,7 @@ static inline uint16_t rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ixgbe_rx_queue *rxq = (struct ixgbe_rx_queue *)rx_queue; + struct ci_rx_queue *rxq = (struct ci_rx_queue *)rx_queue; uint16_t nb_rx = 0; /* Any previously recv'd pkts will be returned from the Rx stage */ @@ -1804,11 +1804,11 @@ uint16_t ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; volatile union ixgbe_adv_rx_desc *rx_ring; volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *sw_ring; - struct ixgbe_rx_entry *rxe; + struct ci_rx_entry *sw_ring; + struct ci_rx_entry *rxe; struct rte_mbuf *rxm; struct rte_mbuf *nmb; union ixgbe_adv_rx_desc rxd; @@ -1826,7 +1826,7 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, nb_hold = 0; rxq = rx_queue; rx_id = rxq->rx_tail; - rx_ring = rxq->rx_ring; + rx_ring = rxq->ixgbe_rx_ring; sw_ring = rxq->sw_ring; vlan_flags = rxq->vlan_flags; while (nb_rx < nb_pkts) { @@ -2031,7 +2031,7 @@ static inline void ixgbe_fill_cluster_head_buf( struct rte_mbuf *head, union ixgbe_adv_rx_desc *desc, - struct ixgbe_rx_queue *rxq, + struct ci_rx_queue *rxq, uint32_t staterr) { uint32_t pkt_info; @@ -2093,10 +2093,10 @@ static inline uint16_t ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool bulk_alloc) { - struct ixgbe_rx_queue *rxq = rx_queue; - volatile union ixgbe_adv_rx_desc *rx_ring = rxq->rx_ring; - struct ixgbe_rx_entry *sw_ring = rxq->sw_ring; - struct ixgbe_scattered_rx_entry *sw_sc_ring = rxq->sw_sc_ring; + struct ci_rx_queue *rxq = rx_queue; + volatile union ixgbe_adv_rx_desc *rx_ring = rxq->ixgbe_rx_ring; + struct ci_rx_entry *sw_ring = rxq->sw_ring; + struct ci_rx_entry_sc *sw_sc_ring = rxq->sw_sc_ring; uint16_t rx_id = rxq->rx_tail; uint16_t nb_rx = 0; uint16_t nb_hold = rxq->nb_rx_hold; @@ -2104,10 +2104,10 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, while (nb_rx < nb_pkts) { bool eop; - struct ixgbe_rx_entry *rxe; - struct ixgbe_scattered_rx_entry *sc_entry; - struct ixgbe_scattered_rx_entry *next_sc_entry = NULL; - struct ixgbe_rx_entry *next_rxe = NULL; + struct ci_rx_entry *rxe; + struct ci_rx_entry_sc *sc_entry; + struct ci_rx_entry_sc *next_sc_entry = NULL; + struct ci_rx_entry *next_rxe = NULL; struct rte_mbuf *first_seg; struct rte_mbuf *rxm; struct rte_mbuf *nmb = NULL; @@ -2949,7 +2949,7 @@ ixgbe_free_sc_cluster(struct rte_mbuf *m) } static void __rte_cold -ixgbe_rx_queue_release_mbufs_non_vec(struct ixgbe_rx_queue *rxq) +ixgbe_rx_queue_release_mbufs_non_vec(struct ci_rx_queue *rxq) { unsigned i; @@ -2980,7 +2980,7 @@ ixgbe_rx_queue_release_mbufs_non_vec(struct ixgbe_rx_queue *rxq) } static void __rte_cold -ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq) +ixgbe_rx_queue_release_mbufs(struct ci_rx_queue *rxq) { if (rxq->vector_rx) ixgbe_rx_queue_release_mbufs_vec(rxq); @@ -2989,7 +2989,7 @@ ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq) } static void __rte_cold -ixgbe_rx_queue_release(struct ixgbe_rx_queue *rxq) +ixgbe_rx_queue_release(struct ci_rx_queue *rxq) { if (rxq != NULL) { ixgbe_rx_queue_release_mbufs(rxq); @@ -3015,7 +3015,7 @@ ixgbe_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid) * function must be used. */ static inline int __rte_cold -check_rx_burst_bulk_alloc_preconditions(struct ixgbe_rx_queue *rxq) +check_rx_burst_bulk_alloc_preconditions(struct ci_rx_queue *rxq) { int ret = 0; @@ -3052,7 +3052,7 @@ check_rx_burst_bulk_alloc_preconditions(struct ixgbe_rx_queue *rxq) /* Reset dynamic ixgbe_rx_queue fields back to defaults */ static void __rte_cold -ixgbe_reset_rx_queue(struct ixgbe_adapter *adapter, struct ixgbe_rx_queue *rxq) +ixgbe_reset_rx_queue(struct ixgbe_adapter *adapter, struct ci_rx_queue *rxq) { static const union ixgbe_adv_rx_desc zeroed_desc = {{0}}; unsigned i; @@ -3073,7 +3073,7 @@ ixgbe_reset_rx_queue(struct ixgbe_adapter *adapter, struct ixgbe_rx_queue *rxq) * reads extra memory as zeros. */ for (i = 0; i < len; i++) { - rxq->rx_ring[i] = zeroed_desc; + rxq->ixgbe_rx_ring[i] = zeroed_desc; } /* @@ -3185,7 +3185,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, struct rte_mempool *mp) { const struct rte_memzone *rz; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; struct ixgbe_hw *hw; uint16_t len; struct ixgbe_adapter *adapter = dev->data->dev_private; @@ -3214,7 +3214,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, } /* First allocate the rx queue data structure */ - rxq = rte_zmalloc_socket("ethdev RX queue", sizeof(struct ixgbe_rx_queue), + rxq = rte_zmalloc_socket("ethdev RX queue", sizeof(struct ci_rx_queue), RTE_CACHE_LINE_SIZE, socket_id); if (rxq == NULL) return -ENOMEM; @@ -3284,7 +3284,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, IXGBE_PCI_REG_ADDR(hw, IXGBE_RDT(rxq->reg_idx)); rxq->rx_ring_phys_addr = rz->iova; - rxq->rx_ring = (union ixgbe_adv_rx_desc *) rz->addr; + rxq->ixgbe_rx_ring = (union ixgbe_adv_rx_desc *)rz->addr; /* * Certain constraints must be met in order to use the bulk buffer @@ -3309,7 +3309,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, len += IXGBE_RX_MAX_BURST; rxq->sw_ring = rte_zmalloc_socket("rxq->sw_ring", - sizeof(struct ixgbe_rx_entry) * len, + sizeof(struct ci_rx_entry) * len, RTE_CACHE_LINE_SIZE, socket_id); if (!rxq->sw_ring) { ixgbe_rx_queue_release(rxq); @@ -3326,7 +3326,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, */ rxq->sw_sc_ring = rte_zmalloc_socket("rxq->sw_sc_ring", - sizeof(struct ixgbe_scattered_rx_entry) * len, + sizeof(struct ci_rx_entry_sc) * len, RTE_CACHE_LINE_SIZE, socket_id); if (!rxq->sw_sc_ring) { ixgbe_rx_queue_release(rxq); @@ -3335,7 +3335,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev, PMD_INIT_LOG(DEBUG, "sw_ring=%p sw_sc_ring=%p hw_ring=%p " "dma_addr=0x%"PRIx64, - rxq->sw_ring, rxq->sw_sc_ring, rxq->rx_ring, + rxq->sw_ring, rxq->sw_sc_ring, rxq->ixgbe_rx_ring, rxq->rx_ring_phys_addr); if (!rte_is_power_of_2(nb_desc)) { @@ -3359,11 +3359,11 @@ ixgbe_dev_rx_queue_count(void *rx_queue) { #define IXGBE_RXQ_SCAN_INTERVAL 4 volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint32_t desc = 0; rxq = rx_queue; - rxdp = &(rxq->rx_ring[rxq->rx_tail]); + rxdp = &rxq->ixgbe_rx_ring[rxq->rx_tail]; while ((desc < rxq->nb_rx_desc) && (rxdp->wb.upper.status_error & @@ -3371,7 +3371,7 @@ ixgbe_dev_rx_queue_count(void *rx_queue) desc += IXGBE_RXQ_SCAN_INTERVAL; rxdp += IXGBE_RXQ_SCAN_INTERVAL; if (rxq->rx_tail + desc >= rxq->nb_rx_desc) - rxdp = &(rxq->rx_ring[rxq->rx_tail + + rxdp = &(rxq->ixgbe_rx_ring[rxq->rx_tail + desc - rxq->nb_rx_desc]); } @@ -3381,7 +3381,7 @@ ixgbe_dev_rx_queue_count(void *rx_queue) int ixgbe_dev_rx_descriptor_status(void *rx_queue, uint16_t offset) { - struct ixgbe_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; volatile uint32_t *status; uint32_t nb_hold, desc; @@ -3399,7 +3399,7 @@ ixgbe_dev_rx_descriptor_status(void *rx_queue, uint16_t offset) if (desc >= rxq->nb_rx_desc) desc -= rxq->nb_rx_desc; - status = &rxq->rx_ring[desc].wb.upper.status_error; + status = &rxq->ixgbe_rx_ring[desc].wb.upper.status_error; if (*status & rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD)) return RTE_ETH_RX_DESC_DONE; @@ -3482,7 +3482,7 @@ ixgbe_dev_clear_queues(struct rte_eth_dev *dev) } for (i = 0; i < dev->data->nb_rx_queues; i++) { - struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; if (rxq != NULL) { ixgbe_rx_queue_release_mbufs(rxq); @@ -4644,9 +4644,9 @@ ixgbe_vmdq_tx_hw_configure(struct ixgbe_hw *hw) } static int __rte_cold -ixgbe_alloc_rx_queue_mbufs(struct ixgbe_rx_queue *rxq) +ixgbe_alloc_rx_queue_mbufs(struct ci_rx_queue *rxq) { - struct ixgbe_rx_entry *rxe = rxq->sw_ring; + struct ci_rx_entry *rxe = rxq->sw_ring; uint64_t dma_addr; unsigned int i; @@ -4666,7 +4666,7 @@ ixgbe_alloc_rx_queue_mbufs(struct ixgbe_rx_queue *rxq) dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(mbuf)); - rxd = &rxq->rx_ring[i]; + rxd = &rxq->ixgbe_rx_ring[i]; rxd->read.hdr_addr = 0; rxd->read.pkt_addr = dma_addr; rxe[i].mbuf = mbuf; @@ -5083,7 +5083,7 @@ ixgbe_set_rx_function(struct rte_eth_dev *dev) dev->rx_pkt_burst == ixgbe_recv_pkts_vec); for (i = 0; i < dev->data->nb_rx_queues; i++) { - struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; rxq->vector_rx = rx_using_sse; #ifdef RTE_LIB_SECURITY @@ -5161,7 +5161,7 @@ ixgbe_set_rsc(struct rte_eth_dev *dev) /* Per-queue RSC configuration (chapter 4.6.7.2.2 of 82599 Spec) */ for (i = 0; i < dev->data->nb_rx_queues; i++) { - struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; uint32_t srrctl = IXGBE_READ_REG(hw, IXGBE_SRRCTL(rxq->reg_idx)); uint32_t rscctl = @@ -5237,7 +5237,7 @@ int __rte_cold ixgbe_dev_rx_init(struct rte_eth_dev *dev) { struct ixgbe_hw *hw; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint64_t bus_addr; uint32_t rxctrl; uint32_t fctrl; @@ -5533,7 +5533,7 @@ ixgbe_dev_rxtx_start(struct rte_eth_dev *dev) { struct ixgbe_hw *hw; struct ci_tx_queue *txq; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint32_t txdctl; uint32_t dmatxctl; uint32_t rxctrl; @@ -5620,7 +5620,7 @@ int __rte_cold ixgbe_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) { struct ixgbe_hw *hw; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint32_t rxdctl; int poll_ms; @@ -5663,7 +5663,7 @@ ixgbe_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) { struct ixgbe_hw *hw; struct ixgbe_adapter *adapter = dev->data->dev_private; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint32_t rxdctl; int poll_ms; @@ -5797,7 +5797,7 @@ void ixgbe_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_rxq_info *qinfo) { - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; rxq = dev->data->rx_queues[queue_id]; @@ -5835,7 +5835,7 @@ void ixgbe_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_recycle_rxq_info *recycle_rxq_info) { - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; struct ixgbe_adapter *adapter = dev->data->dev_private; rxq = dev->data->rx_queues[queue_id]; @@ -5861,7 +5861,7 @@ int __rte_cold ixgbevf_dev_rx_init(struct rte_eth_dev *dev) { struct ixgbe_hw *hw; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; struct rte_eth_rxmode *rxmode = &dev->data->dev_conf.rxmode; uint32_t frame_size = dev->data->mtu + IXGBE_ETH_OVERHEAD; uint64_t bus_addr; @@ -6048,7 +6048,7 @@ ixgbevf_dev_rxtx_start(struct rte_eth_dev *dev) { struct ixgbe_hw *hw; struct ci_tx_queue *txq; - struct ixgbe_rx_queue *rxq; + struct ci_rx_queue *rxq; uint32_t txdctl; uint32_t rxdctl; uint16_t i; diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index 9047ee4763..aad7ee81ee 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -7,6 +7,7 @@ #include "ixgbe_type.h" +#include "../common/rx.h" #include "../common/tx.h" /* @@ -32,7 +33,7 @@ #define IXGBE_MAX_RING_DESC 8192 #define IXGBE_TX_MAX_BURST 32 -#define IXGBE_RX_MAX_BURST 32 +#define IXGBE_RX_MAX_BURST CI_RX_MAX_BURST #define IXGBE_TX_MAX_FREE_BUF_SZ 64 #define IXGBE_VPMD_DESCS_PER_LOOP 4 @@ -66,64 +67,6 @@ #define IXGBE_PACKET_TYPE_TN_MAX 0X100 #define IXGBE_PACKET_TYPE_SHIFT 0X04 -/** - * Structure associated with each descriptor of the RX ring of a RX queue. - */ -struct ixgbe_rx_entry { - struct rte_mbuf *mbuf; /**< mbuf associated with RX descriptor. */ -}; - -struct ixgbe_scattered_rx_entry { - struct rte_mbuf *fbuf; /**< First segment of the fragmented packet. */ -}; - -/** - * Structure associated with each RX queue. - */ -struct ixgbe_rx_queue { - struct rte_mempool *mp; /**< mbuf pool to populate RX ring. */ - volatile union ixgbe_adv_rx_desc *rx_ring; /**< RX ring virtual address. */ - uint64_t rx_ring_phys_addr; /**< RX ring DMA address. */ - volatile uint32_t *qrx_tail; /**< RDT register address. */ - struct ixgbe_rx_entry *sw_ring; /**< address of RX software ring. */ - struct ixgbe_scattered_rx_entry *sw_sc_ring; /**< address of scattered Rx software ring. */ - struct rte_mbuf *pkt_first_seg; /**< First segment of current packet. */ - struct rte_mbuf *pkt_last_seg; /**< Last segment of current packet. */ - uint64_t mbuf_initializer; /**< value to init mbufs */ - uint16_t nb_rx_desc; /**< number of RX descriptors. */ - uint16_t rx_tail; /**< current value of RDT register. */ - uint16_t nb_rx_hold; /**< number of held free RX desc. */ - uint16_t rx_nb_avail; /**< nr of staged pkts ready to ret to app */ - uint16_t rx_next_avail; /**< idx of next staged pkt to ret to app */ - uint16_t rx_free_trigger; /**< triggers rx buffer allocation */ - uint8_t vector_rx; - /**< indicates that vector RX is in use */ -#ifdef RTE_LIB_SECURITY - uint8_t using_ipsec; - /**< indicates that IPsec RX feature is in use */ -#endif - uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ - uint16_t rxrearm_start; /**< the idx we start the re-arming from */ - uint16_t rx_free_thresh; /**< max free RX desc to hold. */ - uint16_t queue_id; /**< RX queue index. */ - uint16_t reg_idx; /**< RX queue register index. */ - uint16_t pkt_type_mask; /**< Packet type mask for different NICs. */ - uint16_t port_id; /**< Device port identifier. */ - uint8_t crc_len; /**< 0 if CRC stripped, 4 otherwise. */ - uint8_t drop_en; /**< If not 0, set SRRCTL.Drop_En. */ - uint8_t rx_deferred_start; /**< not in global dev start. */ - /** UDP frames with a 0 checksum can be marked as checksum errors. */ - uint8_t rx_udp_csum_zero_err; - /** flags to set in mbuf when a vlan is detected. */ - uint64_t vlan_flags; - uint64_t offloads; /**< Rx offloads with RTE_ETH_RX_OFFLOAD_* */ - /** need to alloc dummy mbuf, for wraparound when scanning hw ring */ - struct rte_mbuf fake_mbuf; - /** hold packets to return to application */ - struct rte_mbuf *rx_stage[IXGBE_RX_MAX_BURST * 2]; - const struct rte_memzone *mz; -}; - /** * IXGBE CTX Constants */ diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c index 707dc7f5f9..5f231b9012 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c @@ -61,7 +61,7 @@ ixgbe_reset_tx_queue_vec(struct ci_tx_queue *txq) } void __rte_cold -ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq) +ixgbe_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { unsigned int i; @@ -90,7 +90,7 @@ ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq) } int __rte_cold -ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq) +ixgbe_rxq_vec_setup(struct ci_rx_queue *rxq) { rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); return 0; @@ -126,7 +126,7 @@ ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev) return -1; for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) { - struct ixgbe_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; if (!rxq) continue; if (!ci_rxq_vec_capable(rxq->nb_rx_desc, rxq->rx_free_thresh, rxq->offloads)) @@ -173,15 +173,15 @@ ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) { - struct ixgbe_rx_queue *rxq = rx_queue; - struct ixgbe_rx_entry *rxep; + struct ci_rx_queue *rxq = rx_queue; + struct ci_rx_entry *rxep; volatile union ixgbe_adv_rx_desc *rxdp; uint16_t rx_id; uint64_t paddr; uint64_t dma_addr; uint16_t i; - rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxdp = rxq->ixgbe_rx_ring + rxq->rxrearm_start; rxep = &rxq->sw_ring[rxq->rxrearm_start]; for (i = 0; i < nb_mbufs; i++) { diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h index e05696f584..e54f532497 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.h @@ -12,9 +12,9 @@ #include "ixgbe_rxtx.h" int ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev); -int ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq); +int ixgbe_rxq_vec_setup(struct ci_rx_queue *rxq); int ixgbe_txq_vec_setup(struct ci_tx_queue *txq); -void ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq); +void ixgbe_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq); void ixgbe_reset_tx_queue_vec(struct ci_tx_queue *txq); void ixgbe_tx_free_swring_vec(struct ci_tx_queue *txq); uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); @@ -79,5 +79,4 @@ ixgbe_tx_free_bufs_vec(struct ci_tx_queue *txq) return txq->tx_rs_thresh; } - #endif diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c index 2d42b7b1c1..ce492f2ff1 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c @@ -12,19 +12,19 @@ #include "ixgbe_rxtx_vec_common.h" static inline void -ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) +ixgbe_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; uint64x2_t dma_addr0, dma_addr1; uint64x2_t zero = vdupq_n_u64(0); uint64_t paddr; uint8x8_t p; - rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxdp = rxq->ixgbe_rx_ring + rxq->rxrearm_start; /* Pull 'n' more MBUFs into the software ring */ if (unlikely(rte_mempool_get_bulk(rxq->mp, @@ -282,11 +282,11 @@ desc_to_ptype_v(uint64x2_t descs[4], uint16_t pkt_type_mask, * - floor align nb_pkts to a IXGBE_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *sw_ring; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; uint8x16_t shuf_msk = { @@ -309,7 +309,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* Just the act of getting into the function from the application is * going to cost about 7 cycles */ - rxdp = rxq->rx_ring + rxq->rx_tail; + rxdp = rxq->ixgbe_rx_ring + rxq->rx_tail; rte_prefetch_non_temporal(rxdp); @@ -488,7 +488,7 @@ static uint16_t ixgbe_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ixgbe_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IXGBE_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c index f5bb7eb0bd..f977489b95 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c @@ -13,12 +13,12 @@ #include static inline void -ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) +ixgbe_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); @@ -26,7 +26,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) const __m128i hba_msk = _mm_set_epi64x(0, UINT64_MAX); - rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxdp = rxq->ixgbe_rx_ring + rxq->rxrearm_start; /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, @@ -327,11 +327,11 @@ desc_to_ptype_v(__m128i descs[4], uint16_t pkt_type_mask, * - floor align nb_pkts to a IXGBE_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { volatile union ixgbe_adv_rx_desc *rxdp; - struct ixgbe_rx_entry *sw_ring; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; #ifdef RTE_LIB_SECURITY uint8_t use_ipsec = rxq->using_ipsec; @@ -377,7 +377,7 @@ _recv_raw_pkts_vec(struct ixgbe_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* Just the act of getting into the function from the application is * going to cost about 7 cycles */ - rxdp = rxq->rx_ring + rxq->rx_tail; + rxdp = rxq->ixgbe_rx_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -609,7 +609,7 @@ static uint16_t ixgbe_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ixgbe_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IXGBE_VPMD_RX_BURST] = {0}; /* get some new buffers */ From patchwork Mon Jun 9 15:37:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154221 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D15CE468B7; Mon, 9 Jun 2025 17:40:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1BBCD42D68; Mon, 9 Jun 2025 17:38:27 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 3A0D6427D9 for ; Mon, 9 Jun 2025 17:38:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483506; x=1781019506; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=sNVq4A2Yz0u67uHy1NmYD2yCjmBj/rRAVFpxJV/8q2I=; b=bd+4Rv98UkJozTiFSYc+6gmO54RtTV+1R7vCbNVqS/gt6CVxONVrHccF C0rsI1Ic2ZUgygL+V7H5lR8LR+FgdjpfbxSa2SxirQxFHUWf8KAGJ5HZr 8fnR0cqeYQY785ufRx9xmcB/4xbBplK5sflLCi2EmlOT14z33GieNGI/6 ToSURThERIsweicV8IHoM1dVjkceMKEEeUqlm3mEgaXnABmLyxe+QvQHi TexTYOBrhGuTZBDVGKP543+RxOfku42wzbY/8f2C4FcHFpvru2I4wcGWl UNiRFBCgeENHdx3HGA+wKIJ6mfYcSvFNjYtyIFY341mAgkl2df9DxYZxG w==; X-CSE-ConnectionGUID: WCmfIzqCS3ez3U+EWBVPOQ== X-CSE-MsgGUID: Aon1g0FoSzKnk8UXqDcnYw== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012228" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012228" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:25 -0700 X-CSE-ConnectionGUID: EdMSGle1TwWF93b38kNjLg== X-CSE-MsgGUID: tXWDNYZpQUiBetpecwEJoQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419727" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:23 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Ian Stokes Subject: [PATCH v6 24/33] net/i40e: use the common Rx queue structure Date: Mon, 9 Jun 2025 16:37:22 +0100 Message-ID: <420509396eec1b451a6b6d564d979b160b5c8b33.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Make the i40e driver use the new common Rx queue structure. The i40e driver supports 16-byte and 32-byte Rx descriptor formats, which is shared by other drivers. To have fewer driver-specific definitions in common structures, add a header file defining shared descriptor formats, and switch between 16-byte and 32-byte formats by way of the existing RTE_NET_INTEL_USE_16BYTE_DESC define. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Fix compilation issues for AltiVec drivers/net/intel/common/desc.h | 89 ++++++++++++ drivers/net/intel/common/rx.h | 15 ++ drivers/net/intel/i40e/i40e_ethdev.c | 4 +- drivers/net/intel/i40e/i40e_ethdev.h | 4 +- drivers/net/intel/i40e/i40e_fdir.c | 16 +-- .../i40e/i40e_recycle_mbufs_vec_common.c | 6 +- drivers/net/intel/i40e/i40e_rxtx.c | 134 +++++++++--------- drivers/net/intel/i40e/i40e_rxtx.h | 74 ++-------- drivers/net/intel/i40e/i40e_rxtx_common_avx.h | 6 +- .../net/intel/i40e/i40e_rxtx_vec_altivec.c | 20 +-- drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c | 14 +- drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c | 14 +- drivers/net/intel/i40e/i40e_rxtx_vec_common.h | 4 +- drivers/net/intel/i40e/i40e_rxtx_vec_neon.c | 24 ++-- drivers/net/intel/i40e/i40e_rxtx_vec_sse.c | 24 ++-- 15 files changed, 248 insertions(+), 200 deletions(-) create mode 100644 drivers/net/intel/common/desc.h diff --git a/drivers/net/intel/common/desc.h b/drivers/net/intel/common/desc.h new file mode 100644 index 0000000000..f9e7f27991 --- /dev/null +++ b/drivers/net/intel/common/desc.h @@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Intel Corporation + */ + + #ifndef _COMMON_INTEL_DESC_H_ + #define _COMMON_INTEL_DESC_H_ + +#include + +/* HW desc structures, both 16-byte and 32-byte types are supported */ +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC +union ci_rx_desc { + struct { + rte_le64_t pkt_addr; /* Packet buffer address */ + rte_le64_t hdr_addr; /* Header buffer address */ + } read; + struct { + struct { + struct { + union { + rte_le16_t mirroring_status; + rte_le16_t fcoe_ctx_id; + } mirr_fcoe; + rte_le16_t l2tag1; + } lo_dword; + union { + rte_le32_t rss; /* RSS Hash */ + rte_le32_t fd_id; /* Flow director filter id */ + rte_le32_t fcoe_param; /* FCoE DDP Context id */ + } hi_dword; + } qword0; + struct { + /* ext status/error/pktype/length */ + rte_le64_t status_error_len; + } qword1; + } wb; /* writeback */ +}; +#else +union ci_rx_desc { + struct { + rte_le64_t pkt_addr; /* Packet buffer address */ + rte_le64_t hdr_addr; /* Header buffer address */ + /* bit 0 of hdr_buffer_addr is DD bit */ + rte_le64_t rsvd1; + rte_le64_t rsvd2; + } read; + struct { + struct { + struct { + union { + rte_le16_t mirroring_status; + rte_le16_t fcoe_ctx_id; + } mirr_fcoe; + rte_le16_t l2tag1; + } lo_dword; + union { + rte_le32_t rss; /* RSS Hash */ + rte_le32_t fcoe_param; /* FCoE DDP Context id */ + /* Flow director filter id in case of + * Programming status desc WB + */ + rte_le32_t fd_id; + } hi_dword; + } qword0; + struct { + /* status/error/pktype/length */ + rte_le64_t status_error_len; + } qword1; + struct { + rte_le16_t ext_status; /* extended status */ + rte_le16_t rsvd; + rte_le16_t l2tag2_1; + rte_le16_t l2tag2_2; + } qword2; + struct { + union { + rte_le32_t flex_bytes_lo; + rte_le32_t pe_status; + } lo_dword; + union { + rte_le32_t flex_bytes_hi; + rte_le32_t fd_id; + } hi_dword; + } qword3; + } wb; /* writeback */ +}; +#endif + +#endif /* _COMMON_INTEL_DESC_H_ */ diff --git a/drivers/net/intel/common/rx.h b/drivers/net/intel/common/rx.h index b60ca24dfb..98e85406cf 100644 --- a/drivers/net/intel/common/rx.h +++ b/drivers/net/intel/common/rx.h @@ -10,6 +10,8 @@ #include #include +#include "desc.h" + #define CI_RX_MAX_BURST 32 struct ci_rx_queue; @@ -29,6 +31,7 @@ struct ci_rx_queue { struct rte_mempool *mp; /**< mbuf pool to populate RX ring. */ union { /* RX ring virtual address */ volatile union ixgbe_adv_rx_desc *ixgbe_rx_ring; + volatile union ci_rx_desc *rx_ring; }; volatile uint8_t *qrx_tail; /**< register address of tail */ struct ci_rx_entry *sw_ring; /**< address of RX software ring. */ @@ -50,14 +53,22 @@ struct ci_rx_queue { uint16_t queue_id; /**< RX queue index. */ uint16_t port_id; /**< Device port identifier. */ uint16_t reg_idx; /**< RX queue register index. */ + uint16_t rx_buf_len; /* The packet buffer size */ + uint16_t rx_hdr_len; /* The header buffer size */ + uint16_t max_pkt_len; /* Maximum packet length */ uint8_t crc_len; /**< 0 if CRC stripped, 4 otherwise. */ + bool q_set; /**< indicate if rx queue has been configured */ bool rx_deferred_start; /**< queue is not started on dev start. */ + bool fdir_enabled; /* 0 if FDIR disabled, 1 when enabled */ bool vector_rx; /**< indicates that vector RX is in use */ bool drop_en; /**< if 1, drop packets if no descriptors are available. */ uint64_t mbuf_initializer; /**< value to init mbufs */ uint64_t offloads; /**< Rx offloads with RTE_ETH_RX_OFFLOAD_* */ /** need to alloc dummy mbuf, for wraparound when scanning hw ring */ struct rte_mbuf fake_mbuf; + union { /* the VSI this queue belongs to */ + struct i40e_vsi *i40e_vsi; + }; const struct rte_memzone *mz; union { struct { /* ixgbe specific values */ @@ -70,6 +81,10 @@ struct ci_rx_queue { /** UDP frames with a 0 checksum can be marked as checksum errors. */ uint8_t rx_udp_csum_zero_err; }; + struct { /* i40e specific values */ + uint8_t hs_mode; /**< Header Split mode */ + uint8_t dcb_tc; /**< Traffic class of rx queue */ + }; }; }; diff --git a/drivers/net/intel/i40e/i40e_ethdev.c b/drivers/net/intel/i40e/i40e_ethdev.c index 90eba3419f..e0a865845b 100644 --- a/drivers/net/intel/i40e/i40e_ethdev.c +++ b/drivers/net/intel/i40e/i40e_ethdev.c @@ -6609,7 +6609,7 @@ i40e_dev_rx_init(struct i40e_pf *pf) struct rte_eth_dev_data *data = pf->dev_data; int ret = I40E_SUCCESS; uint16_t i; - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; i40e_pf_config_rss(pf); for (i = 0; i < data->nb_rx_queues; i++) { @@ -8974,7 +8974,7 @@ i40e_pf_calc_configured_queues_num(struct i40e_pf *pf) { struct rte_eth_dev_data *data = pf->dev_data; int i, num; - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; num = 0; for (i = 0; i < pf->lan_nb_qps; i++) { diff --git a/drivers/net/intel/i40e/i40e_ethdev.h b/drivers/net/intel/i40e/i40e_ethdev.h index ccc8732d7d..44864292d0 100644 --- a/drivers/net/intel/i40e/i40e_ethdev.h +++ b/drivers/net/intel/i40e/i40e_ethdev.h @@ -333,7 +333,7 @@ struct i40e_vsi_list { struct i40e_vsi *vsi; }; -struct i40e_rx_queue; +struct ci_rx_queue; struct ci_tx_queue; /* Bandwidth limit information */ @@ -739,7 +739,7 @@ struct i40e_fdir_info { struct i40e_vsi *fdir_vsi; /* pointer to fdir VSI structure */ uint16_t match_counter_index; /* Statistic counter index used for fdir*/ struct ci_tx_queue *txq; - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; void *prg_pkt[I40E_FDIR_PRG_PKT_CNT]; /* memory for fdir program packet */ uint64_t dma_addr[I40E_FDIR_PRG_PKT_CNT]; /* physic address of packet memory*/ /* diff --git a/drivers/net/intel/i40e/i40e_fdir.c b/drivers/net/intel/i40e/i40e_fdir.c index 734218b67d..a891819f47 100644 --- a/drivers/net/intel/i40e/i40e_fdir.c +++ b/drivers/net/intel/i40e/i40e_fdir.c @@ -100,9 +100,9 @@ i40e_flow_fdir_filter_programming(struct i40e_pf *pf, bool add, bool wait_status); static int -i40e_fdir_rx_queue_init(struct i40e_rx_queue *rxq) +i40e_fdir_rx_queue_init(struct ci_rx_queue *rxq) { - struct i40e_hw *hw = I40E_VSI_TO_HW(rxq->vsi); + struct i40e_hw *hw = I40E_VSI_TO_HW(rxq->i40e_vsi); struct i40e_hmc_obj_rxq rx_ctx; int err = I40E_SUCCESS; @@ -139,7 +139,7 @@ i40e_fdir_rx_queue_init(struct i40e_rx_queue *rxq) return err; } rxq->qrx_tail = hw->hw_addr + - I40E_QRX_TAIL(rxq->vsi->base_queue); + I40E_QRX_TAIL(rxq->i40e_vsi->base_queue); rte_wmb(); /* Init the RX tail register. */ @@ -382,7 +382,7 @@ i40e_fdir_rx_proc_enable(struct rte_eth_dev *dev, bool on) int32_t i; for (i = 0; i < dev->data->nb_rx_queues; i++) { - struct i40e_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; if (!rxq) continue; rxq->fdir_enabled = on; @@ -929,9 +929,9 @@ i40e_build_ctob(uint32_t td_cmd, * tx queue */ static inline int -i40e_check_fdir_programming_status(struct i40e_rx_queue *rxq) +i40e_check_fdir_programming_status(struct ci_rx_queue *rxq) { - volatile union i40e_rx_desc *rxdp; + volatile union ci_rx_desc *rxdp; uint64_t qword1; uint32_t rx_status; uint32_t len, id; @@ -987,7 +987,7 @@ i40e_check_fdir_programming_status(struct i40e_rx_queue *rxq) } static inline void -i40e_fdir_programming_status_cleanup(struct i40e_rx_queue *rxq) +i40e_fdir_programming_status_cleanup(struct ci_rx_queue *rxq) { uint16_t retry_count = 0; @@ -1627,7 +1627,7 @@ i40e_flow_fdir_filter_programming(struct i40e_pf *pf, bool add, bool wait_status) { struct ci_tx_queue *txq = pf->fdir.txq; - struct i40e_rx_queue *rxq = pf->fdir.rxq; + struct ci_rx_queue *rxq = pf->fdir.rxq; const struct i40e_fdir_action *fdir_action = &filter->action; volatile struct i40e_tx_desc *txdp; volatile struct i40e_filter_program_desc *fdirdp; diff --git a/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c b/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c index 2875c578af..20d9fd7b22 100644 --- a/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c +++ b/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c @@ -13,9 +13,9 @@ void i40e_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) { - struct i40e_rx_queue *rxq = rx_queue; - struct i40e_rx_entry *rxep; - volatile union i40e_rx_desc *rxdp; + struct ci_rx_queue *rxq = rx_queue; + struct ci_rx_entry *rxep; + volatile union ci_rx_desc *rxdp; uint16_t rx_id; uint64_t paddr; uint64_t dma_addr; diff --git a/drivers/net/intel/i40e/i40e_rxtx.c b/drivers/net/intel/i40e/i40e_rxtx.c index 2e61076378..0b06130fe5 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.c +++ b/drivers/net/intel/i40e/i40e_rxtx.c @@ -94,8 +94,8 @@ i40e_monitor_callback(const uint64_t value, int i40e_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc) { - struct i40e_rx_queue *rxq = rx_queue; - volatile union i40e_rx_desc *rxdp; + struct ci_rx_queue *rxq = rx_queue; + volatile union ci_rx_desc *rxdp; uint16_t desc; desc = rxq->rx_tail; @@ -113,7 +113,7 @@ i40e_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc) } static inline void -i40e_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union i40e_rx_desc *rxdp) +i40e_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union ci_rx_desc *rxdp) { if (rte_le_to_cpu_64(rxdp->wb.qword1.status_error_len) & (1 << I40E_RX_DESC_STATUS_L2TAG1P_SHIFT)) { @@ -214,7 +214,7 @@ i40e_get_iee15888_flags(struct rte_mbuf *mb, uint64_t qword) #endif static inline uint64_t -i40e_rxd_build_fdir(volatile union i40e_rx_desc *rxdp, struct rte_mbuf *mb) +i40e_rxd_build_fdir(volatile union ci_rx_desc *rxdp, struct rte_mbuf *mb) { uint64_t flags = 0; #ifndef RTE_NET_INTEL_USE_16BYTE_DESC @@ -416,9 +416,9 @@ i40e_xmit_cleanup(struct ci_tx_queue *txq) static inline int #ifdef RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC -check_rx_burst_bulk_alloc_preconditions(struct i40e_rx_queue *rxq) +check_rx_burst_bulk_alloc_preconditions(struct ci_rx_queue *rxq) #else -check_rx_burst_bulk_alloc_preconditions(__rte_unused struct i40e_rx_queue *rxq) +check_rx_burst_bulk_alloc_preconditions(__rte_unused struct ci_rx_queue *rxq) #endif { int ret = 0; @@ -456,10 +456,10 @@ check_rx_burst_bulk_alloc_preconditions(__rte_unused struct i40e_rx_queue *rxq) #error "PMD I40E: I40E_LOOK_AHEAD must be 8\n" #endif static inline int -i40e_rx_scan_hw_ring(struct i40e_rx_queue *rxq) +i40e_rx_scan_hw_ring(struct ci_rx_queue *rxq) { - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *rxep; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t pkt_len; uint64_t qword1; @@ -467,7 +467,7 @@ i40e_rx_scan_hw_ring(struct i40e_rx_queue *rxq) int32_t s[I40E_LOOK_AHEAD], var, nb_dd; int32_t i, j, nb_rx = 0; uint64_t pkt_flags; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; rxdp = &rxq->rx_ring[rxq->rx_tail]; rxep = &rxq->sw_ring[rxq->rx_tail]; @@ -558,7 +558,7 @@ i40e_rx_scan_hw_ring(struct i40e_rx_queue *rxq) } static inline uint16_t -i40e_rx_fill_from_stage(struct i40e_rx_queue *rxq, +i40e_rx_fill_from_stage(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { @@ -577,10 +577,10 @@ i40e_rx_fill_from_stage(struct i40e_rx_queue *rxq, } static inline int -i40e_rx_alloc_bufs(struct i40e_rx_queue *rxq) +i40e_rx_alloc_bufs(struct ci_rx_queue *rxq) { - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *rxep; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t alloc_idx, i; uint64_t dma_addr; @@ -629,7 +629,7 @@ i40e_rx_alloc_bufs(struct i40e_rx_queue *rxq) static inline uint16_t rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = (struct i40e_rx_queue *)rx_queue; + struct ci_rx_queue *rxq = (struct ci_rx_queue *)rx_queue; struct rte_eth_dev *dev; uint16_t nb_rx = 0; @@ -648,7 +648,7 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) if (i40e_rx_alloc_bufs(rxq) != 0) { uint16_t i, j; - dev = I40E_VSI_TO_ETH_DEV(rxq->vsi); + dev = I40E_VSI_TO_ETH_DEV(rxq->i40e_vsi); dev->data->rx_mbuf_alloc_failed += rxq->rx_free_thresh; @@ -707,12 +707,12 @@ i40e_recv_pkts_bulk_alloc(void __rte_unused *rx_queue, uint16_t i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq; - volatile union i40e_rx_desc *rx_ring; - volatile union i40e_rx_desc *rxdp; - union i40e_rx_desc rxd; - struct i40e_rx_entry *sw_ring; - struct i40e_rx_entry *rxe; + struct ci_rx_queue *rxq; + volatile union ci_rx_desc *rx_ring; + volatile union ci_rx_desc *rxdp; + union ci_rx_desc rxd; + struct ci_rx_entry *sw_ring; + struct ci_rx_entry *rxe; struct rte_eth_dev *dev; struct rte_mbuf *rxm; struct rte_mbuf *nmb; @@ -731,7 +731,7 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rx_id = rxq->rx_tail; rx_ring = rxq->rx_ring; sw_ring = rxq->sw_ring; - ptype_tbl = rxq->vsi->adapter->ptype_tbl; + ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; while (nb_rx < nb_pkts) { rxdp = &rx_ring[rx_id]; @@ -745,7 +745,7 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) nmb = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!nmb)) { - dev = I40E_VSI_TO_ETH_DEV(rxq->vsi); + dev = I40E_VSI_TO_ETH_DEV(rxq->i40e_vsi); dev->data->rx_mbuf_alloc_failed++; break; } @@ -837,12 +837,12 @@ i40e_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = rx_queue; - volatile union i40e_rx_desc *rx_ring = rxq->rx_ring; - volatile union i40e_rx_desc *rxdp; - union i40e_rx_desc rxd; - struct i40e_rx_entry *sw_ring = rxq->sw_ring; - struct i40e_rx_entry *rxe; + struct ci_rx_queue *rxq = rx_queue; + volatile union ci_rx_desc *rx_ring = rxq->rx_ring; + volatile union ci_rx_desc *rxdp; + union ci_rx_desc rxd; + struct ci_rx_entry *sw_ring = rxq->sw_ring; + struct ci_rx_entry *rxe; struct rte_mbuf *first_seg = rxq->pkt_first_seg; struct rte_mbuf *last_seg = rxq->pkt_last_seg; struct rte_mbuf *nmb, *rxm; @@ -853,7 +853,7 @@ i40e_recv_scattered_pkts(void *rx_queue, uint64_t qword1; uint64_t dma_addr; uint64_t pkt_flags; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; while (nb_rx < nb_pkts) { rxdp = &rx_ring[rx_id]; @@ -867,7 +867,7 @@ i40e_recv_scattered_pkts(void *rx_queue, nmb = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!nmb)) { - dev = I40E_VSI_TO_ETH_DEV(rxq->vsi); + dev = I40E_VSI_TO_ETH_DEV(rxq->i40e_vsi); dev->data->rx_mbuf_alloc_failed++; break; } @@ -1798,7 +1798,7 @@ i40e_get_queue_offset_by_qindex(struct i40e_pf *pf, uint16_t queue_idx) int i40e_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) { - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); @@ -1841,7 +1841,7 @@ i40e_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) int i40e_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) { - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); @@ -2004,7 +2004,7 @@ i40e_dev_first_queue(uint16_t idx, void **queues, int num) static int i40e_dev_rx_queue_setup_runtime(struct rte_eth_dev *dev, - struct i40e_rx_queue *rxq) + struct ci_rx_queue *rxq) { struct i40e_adapter *ad = I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); @@ -2081,7 +2081,7 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); struct i40e_vsi *vsi; struct i40e_pf *pf = NULL; - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; const struct rte_memzone *rz; uint32_t ring_size; uint16_t len, i; @@ -2116,7 +2116,7 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, /* Allocate the rx queue data structure */ rxq = rte_zmalloc_socket("i40e rx queue", - sizeof(struct i40e_rx_queue), + sizeof(struct ci_rx_queue), RTE_CACHE_LINE_SIZE, socket_id); if (!rxq) { @@ -2135,7 +2135,7 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, else rxq->crc_len = 0; rxq->drop_en = rx_conf->rx_drop_en; - rxq->vsi = vsi; + rxq->i40e_vsi = vsi; rxq->rx_deferred_start = rx_conf->rx_deferred_start; rxq->offloads = offloads; @@ -2148,7 +2148,7 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, */ len += I40E_RX_MAX_BURST; - ring_size = RTE_ALIGN(len * sizeof(union i40e_rx_desc), + ring_size = RTE_ALIGN(len * sizeof(union ci_rx_desc), I40E_DMA_MEM_ALIGN); rz = rte_eth_dma_zone_reserve(dev, "rx_ring", queue_idx, @@ -2164,14 +2164,14 @@ i40e_dev_rx_queue_setup(struct rte_eth_dev *dev, memset(rz->addr, 0, ring_size); rxq->rx_ring_phys_addr = rz->iova; - rxq->rx_ring = (union i40e_rx_desc *)rz->addr; + rxq->rx_ring = (union ci_rx_desc *)rz->addr; len = (uint16_t)(nb_desc + I40E_RX_MAX_BURST); /* Allocate the software ring. */ rxq->sw_ring = rte_zmalloc_socket("i40e rx sw ring", - sizeof(struct i40e_rx_entry) * len, + sizeof(struct ci_rx_entry) * len, RTE_CACHE_LINE_SIZE, socket_id); if (!rxq->sw_ring) { @@ -2242,7 +2242,7 @@ i40e_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid) void i40e_rx_queue_release(void *rxq) { - struct i40e_rx_queue *q = (struct i40e_rx_queue *)rxq; + struct ci_rx_queue *q = (struct ci_rx_queue *)rxq; if (!q) { PMD_DRV_LOG(DEBUG, "Pointer to rxq is NULL"); @@ -2259,8 +2259,8 @@ uint32_t i40e_dev_rx_queue_count(void *rx_queue) { #define I40E_RXQ_SCAN_INTERVAL 4 - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_queue *rxq; + volatile union ci_rx_desc *rxdp; + struct ci_rx_queue *rxq; uint16_t desc = 0; rxq = rx_queue; @@ -2287,7 +2287,7 @@ i40e_dev_rx_queue_count(void *rx_queue) int i40e_dev_rx_descriptor_status(void *rx_queue, uint16_t offset) { - struct i40e_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; volatile uint64_t *status; uint64_t mask; uint32_t desc; @@ -2628,7 +2628,7 @@ i40e_memzone_reserve(const char *name, uint32_t len, int socket_id) } void -i40e_rx_queue_release_mbufs(struct i40e_rx_queue *rxq) +i40e_rx_queue_release_mbufs(struct ci_rx_queue *rxq) { uint16_t i; @@ -2663,7 +2663,7 @@ i40e_rx_queue_release_mbufs(struct i40e_rx_queue *rxq) } void -i40e_reset_rx_queue(struct i40e_rx_queue *rxq) +i40e_reset_rx_queue(struct ci_rx_queue *rxq) { unsigned i; uint16_t len; @@ -2680,7 +2680,7 @@ i40e_reset_rx_queue(struct i40e_rx_queue *rxq) #endif /* RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC */ len = rxq->nb_rx_desc; - for (i = 0; i < len * sizeof(union i40e_rx_desc); i++) + for (i = 0; i < len * sizeof(union ci_rx_desc); i++) ((volatile char *)rxq->rx_ring)[i] = 0; memset(&rxq->fake_mbuf, 0x0, sizeof(rxq->fake_mbuf)); @@ -2898,14 +2898,14 @@ i40e_tx_queue_init(struct ci_tx_queue *txq) } int -i40e_alloc_rx_queue_mbufs(struct i40e_rx_queue *rxq) +i40e_alloc_rx_queue_mbufs(struct ci_rx_queue *rxq) { - struct i40e_rx_entry *rxe = rxq->sw_ring; + struct ci_rx_entry *rxe = rxq->sw_ring; uint64_t dma_addr; uint16_t i; for (i = 0; i < rxq->nb_rx_desc; i++) { - volatile union i40e_rx_desc *rxd; + volatile union ci_rx_desc *rxd; struct rte_mbuf *mbuf = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!mbuf)) { @@ -2941,10 +2941,10 @@ i40e_alloc_rx_queue_mbufs(struct i40e_rx_queue *rxq) * and maximum packet length. */ static int -i40e_rx_queue_config(struct i40e_rx_queue *rxq) +i40e_rx_queue_config(struct ci_rx_queue *rxq) { - struct i40e_pf *pf = I40E_VSI_TO_PF(rxq->vsi); - struct i40e_hw *hw = I40E_VSI_TO_HW(rxq->vsi); + struct i40e_pf *pf = I40E_VSI_TO_PF(rxq->i40e_vsi); + struct i40e_hw *hw = I40E_VSI_TO_HW(rxq->i40e_vsi); struct rte_eth_dev_data *data = pf->dev_data; uint16_t buf_size; @@ -2988,11 +2988,11 @@ i40e_rx_queue_config(struct i40e_rx_queue *rxq) /* Init the RX queue in hardware */ int -i40e_rx_queue_init(struct i40e_rx_queue *rxq) +i40e_rx_queue_init(struct ci_rx_queue *rxq) { int err = I40E_SUCCESS; - struct i40e_hw *hw = I40E_VSI_TO_HW(rxq->vsi); - struct rte_eth_dev_data *dev_data = I40E_VSI_TO_DEV_DATA(rxq->vsi); + struct i40e_hw *hw = I40E_VSI_TO_HW(rxq->i40e_vsi); + struct rte_eth_dev_data *dev_data = I40E_VSI_TO_DEV_DATA(rxq->i40e_vsi); uint16_t pf_q = rxq->reg_idx; uint16_t buf_size; struct i40e_hmc_obj_rxq rx_ctx; @@ -3166,7 +3166,7 @@ i40e_fdir_setup_tx_resources(struct i40e_pf *pf) enum i40e_status_code i40e_fdir_setup_rx_resources(struct i40e_pf *pf) { - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; const struct rte_memzone *rz = NULL; uint32_t ring_size; struct rte_eth_dev *dev; @@ -3180,7 +3180,7 @@ i40e_fdir_setup_rx_resources(struct i40e_pf *pf) /* Allocate the RX queue data structure. */ rxq = rte_zmalloc_socket("i40e fdir rx queue", - sizeof(struct i40e_rx_queue), + sizeof(struct ci_rx_queue), RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); if (!rxq) { @@ -3190,7 +3190,7 @@ i40e_fdir_setup_rx_resources(struct i40e_pf *pf) } /* Allocate RX hardware ring descriptors. */ - ring_size = sizeof(union i40e_rx_desc) * I40E_FDIR_NUM_RX_DESC; + ring_size = sizeof(union ci_rx_desc) * I40E_FDIR_NUM_RX_DESC; ring_size = RTE_ALIGN(ring_size, I40E_DMA_MEM_ALIGN); rz = rte_eth_dma_zone_reserve(dev, "fdir_rx_ring", @@ -3206,11 +3206,11 @@ i40e_fdir_setup_rx_resources(struct i40e_pf *pf) rxq->nb_rx_desc = I40E_FDIR_NUM_RX_DESC; rxq->queue_id = I40E_FDIR_QUEUE_ID; rxq->reg_idx = pf->fdir.fdir_vsi->base_queue; - rxq->vsi = pf->fdir.fdir_vsi; + rxq->i40e_vsi = pf->fdir.fdir_vsi; rxq->rx_ring_phys_addr = rz->iova; - memset(rz->addr, 0, I40E_FDIR_NUM_RX_DESC * sizeof(union i40e_rx_desc)); - rxq->rx_ring = (union i40e_rx_desc *)rz->addr; + memset(rz->addr, 0, I40E_FDIR_NUM_RX_DESC * sizeof(union ci_rx_desc)); + rxq->rx_ring = (union ci_rx_desc *)rz->addr; /* * Don't need to allocate software ring and reset for the fdir @@ -3226,7 +3226,7 @@ void i40e_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_rxq_info *qinfo) { - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; rxq = dev->data->rx_queues[queue_id]; @@ -3264,7 +3264,7 @@ void i40e_recycle_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_recycle_rxq_info *recycle_rxq_info) { - struct i40e_rx_queue *rxq; + struct ci_rx_queue *rxq; struct i40e_adapter *ad = I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); @@ -3335,7 +3335,7 @@ i40e_set_rx_function(struct rte_eth_dev *dev) } if (ad->rx_vec_allowed) { for (i = 0; i < dev->data->nb_rx_queues; i++) { - struct i40e_rx_queue *rxq = + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; if (rxq && i40e_rxq_vec_setup(rxq)) { @@ -3438,7 +3438,7 @@ i40e_set_rx_function(struct rte_eth_dev *dev) dev->rx_pkt_burst == i40e_recv_pkts_vec_avx2); for (i = 0; i < dev->data->nb_rx_queues; i++) { - struct i40e_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; if (rxq) rxq->vector_rx = vector_rx; diff --git a/drivers/net/intel/i40e/i40e_rxtx.h b/drivers/net/intel/i40e/i40e_rxtx.h index 3dca32b1ba..05c41d473e 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.h +++ b/drivers/net/intel/i40e/i40e_rxtx.h @@ -6,8 +6,9 @@ #define _I40E_RXTX_H_ #include "../common/tx.h" +#include "../common/rx.h" -#define I40E_RX_MAX_BURST 32 +#define I40E_RX_MAX_BURST CI_RX_MAX_BURST #define I40E_TX_MAX_BURST 32 #define I40E_VPMD_RX_BURST 32 @@ -66,63 +67,6 @@ enum i40e_header_split_mode { I40E_HEADER_SPLIT_UDP_TCP | \ I40E_HEADER_SPLIT_SCTP) -/* HW desc structure, both 16-byte and 32-byte types are supported */ -#ifdef RTE_NET_INTEL_USE_16BYTE_DESC -#define i40e_rx_desc i40e_16byte_rx_desc -#else -#define i40e_rx_desc i40e_32byte_rx_desc -#endif - -struct i40e_rx_entry { - struct rte_mbuf *mbuf; -}; - -/* - * Structure associated with each RX queue. - */ -struct i40e_rx_queue { - struct rte_mempool *mp; /**< mbuf pool to populate RX ring */ - volatile union i40e_rx_desc *rx_ring;/**< RX ring virtual address */ - uint64_t rx_ring_phys_addr; /**< RX ring DMA address */ - struct i40e_rx_entry *sw_ring; /**< address of RX soft ring */ - uint16_t nb_rx_desc; /**< number of RX descriptors */ - uint16_t rx_free_thresh; /**< max free RX desc to hold */ - uint16_t rx_tail; /**< current value of tail */ - uint16_t nb_rx_hold; /**< number of held free RX desc */ - struct rte_mbuf *pkt_first_seg; /**< first segment of current packet */ - struct rte_mbuf *pkt_last_seg; /**< last segment of current packet */ - struct rte_mbuf fake_mbuf; /**< dummy mbuf */ -#ifdef RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC - uint16_t rx_nb_avail; /**< number of staged packets ready */ - uint16_t rx_next_avail; /**< index of next staged packets */ - uint16_t rx_free_trigger; /**< triggers rx buffer allocation */ - struct rte_mbuf *rx_stage[I40E_RX_MAX_BURST * 2]; -#endif - - uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ - uint16_t rxrearm_start; /**< the idx we start the re-arming from */ - uint64_t mbuf_initializer; /**< value to init mbufs */ - - uint16_t port_id; /**< device port ID */ - uint8_t crc_len; /**< 0 if CRC stripped, 4 otherwise */ - uint8_t fdir_enabled; /**< 0 if FDIR disabled, 1 when enabled */ - uint16_t queue_id; /**< RX queue index */ - uint16_t reg_idx; /**< RX queue register index */ - uint8_t drop_en; /**< if not 0, set register bit */ - volatile uint8_t *qrx_tail; /**< register address of tail */ - struct i40e_vsi *vsi; /**< the VSI this queue belongs to */ - uint16_t rx_buf_len; /* The packet buffer size */ - uint16_t rx_hdr_len; /* The header buffer size */ - uint16_t max_pkt_len; /* Maximum packet length */ - uint8_t hs_mode; /* Header Split mode */ - bool q_set; /**< indicate if rx queue has been configured */ - bool rx_deferred_start; /**< don't start this queue in dev start */ - uint16_t vector_rx; /**sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; rxdp = rxq->rx_ring + rxq->rxrearm_start; diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c b/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c index 568891cfb2..a914ef20f4 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c @@ -16,13 +16,13 @@ #include static inline void -i40e_rxq_rearm(struct i40e_rx_queue *rxq) +i40e_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; - volatile union i40e_rx_desc *rxdp; + volatile union ci_rx_desc *rxdp; - struct i40e_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; __vector unsigned long hdr_room = (__vector unsigned long){ @@ -195,16 +195,16 @@ desc_to_ptype_v(__vector unsigned long descs[4], struct rte_mbuf **rx_pkts, * - floor align nb_pkts to a I40E_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *sw_ring; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; uint64_t var; __vector unsigned char shuf_msk; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; __vector unsigned short crc_adjust = (__vector unsigned short){ 0, 0, /* ignore pkt_type field */ @@ -465,7 +465,7 @@ static uint16_t i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -611,13 +611,13 @@ i40e_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, } void __rte_cold -i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) +i40e_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { _i40e_rx_queue_release_mbufs_vec(rxq); } int __rte_cold -i40e_rxq_vec_setup(struct i40e_rx_queue *rxq) +i40e_rxq_vec_setup(struct ci_rx_queue *rxq) { rxq->vector_rx = 1; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c index a13dd9bc78..fee2a6e670 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c @@ -16,7 +16,7 @@ #include static __rte_always_inline void -i40e_rxq_rearm(struct i40e_rx_queue *rxq) +i40e_rxq_rearm(struct ci_rx_queue *rxq) { i40e_rxq_rearm_common(rxq, false); } @@ -29,7 +29,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) * desc_idx: required to select the correct shift at compile time */ static inline __m256i -desc_fdir_processing_32b(volatile union i40e_rx_desc *rxdp, +desc_fdir_processing_32b(volatile union ci_rx_desc *rxdp, struct rte_mbuf **rx_pkts, const uint32_t pkt_idx, const uint32_t desc_idx) @@ -105,14 +105,14 @@ desc_fdir_processing_32b(volatile union i40e_rx_desc *rxdp, /* Force inline as some compilers will not inline by default. */ static __rte_always_inline uint16_t -_recv_raw_pkts_vec_avx2(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec_avx2(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct i40e_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union i40e_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; const int avx_aligned = ((rxq->rx_tail & 1) == 0); rte_prefetch0(rxdp); @@ -623,7 +623,7 @@ static uint16_t i40e_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c index f0320a221c..e609b7c411 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c @@ -16,7 +16,7 @@ #include static __rte_always_inline void -i40e_rxq_rearm(struct i40e_rx_queue *rxq) +i40e_rxq_rearm(struct ci_rx_queue *rxq) { i40e_rxq_rearm_common(rxq, true); } @@ -29,7 +29,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) * desc_idx: required to select the correct shift at compile time */ static inline __m256i -desc_fdir_processing_32b(volatile union i40e_rx_desc *rxdp, +desc_fdir_processing_32b(volatile union ci_rx_desc *rxdp, struct rte_mbuf **rx_pkts, const uint32_t pkt_idx, const uint32_t desc_idx) @@ -106,14 +106,14 @@ desc_fdir_processing_32b(volatile union i40e_rx_desc *rxdp, /* Force inline as some compilers will not inline by default. */ static __rte_always_inline uint16_t -_recv_raw_pkts_vec_avx512(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec_avx512(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct i40e_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union i40e_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -691,7 +691,7 @@ i40e_recv_scattered_burst_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_common.h b/drivers/net/intel/i40e/i40e_rxtx_vec_common.h index ba72df8e13..d19b9e4bf4 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_common.h +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_common.h @@ -21,7 +21,7 @@ i40e_tx_desc_done(struct ci_tx_queue *txq, uint16_t idx) } static inline void -_i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) +_i40e_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { const unsigned mask = rxq->nb_rx_desc - 1; unsigned i; @@ -68,7 +68,7 @@ i40e_rx_vec_dev_conf_condition_check_default(struct rte_eth_dev *dev) */ ad->rx_vec_allowed = true; for (uint16_t i = 0; i < dev->data->nb_rx_queues; i++) { - struct i40e_rx_queue *rxq = dev->data->rx_queues[i]; + struct ci_rx_queue *rxq = dev->data->rx_queues[i]; if (!rxq) continue; if (!ci_rxq_vec_capable(rxq->nb_rx_desc, rxq->rx_free_thresh, rxq->offloads)) { diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c index 955382652c..02ba03c290 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c @@ -17,12 +17,12 @@ #include "i40e_rxtx_vec_common.h" static inline void -i40e_rxq_rearm(struct i40e_rx_queue *rxq) +i40e_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; uint64x2_t dma_addr0, dma_addr1; uint64x2_t zero = vdupq_n_u64(0); @@ -80,7 +80,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) #ifndef RTE_NET_INTEL_USE_16BYTE_DESC /* NEON version of FDIR mark extraction for 4 32B descriptors at a time */ static inline uint32x4_t -descs_to_fdir_32b(volatile union i40e_rx_desc *rxdp, struct rte_mbuf **rx_pkt) +descs_to_fdir_32b(volatile union ci_rx_desc *rxdp, struct rte_mbuf **rx_pkt) { /* 32B descriptors: Load 2nd half of descriptors for FDIR ID data */ uint64x2_t desc0_qw23, desc1_qw23, desc2_qw23, desc3_qw23; @@ -203,7 +203,7 @@ descs_to_fdir_16b(uint32x4_t fltstat, uint64x2_t descs[4], struct rte_mbuf **rx_ #endif static inline void -desc_to_olflags_v(struct i40e_rx_queue *rxq, volatile union i40e_rx_desc *rxdp, +desc_to_olflags_v(struct ci_rx_queue *rxq, volatile union ci_rx_desc *rxdp, uint64x2_t descs[4], struct rte_mbuf **rx_pkts) { uint32x4_t vlan0, vlan1, rss, l3_l4e; @@ -332,15 +332,15 @@ desc_to_ptype_v(uint64x2_t descs[4], struct rte_mbuf **__rte_restrict rx_pkts, * - floor align nb_pkts to a I40E_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct i40e_rx_queue *__rte_restrict rxq, +_recv_raw_pkts_vec(struct ci_rx_queue *__rte_restrict rxq, struct rte_mbuf **__rte_restrict rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *sw_ring; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; /* mask to shuffle from desc. to mbuf */ uint8x16_t shuf_msk = { @@ -591,7 +591,7 @@ i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -737,13 +737,13 @@ i40e_xmit_fixed_burst_vec(void *__rte_restrict tx_queue, } void __rte_cold -i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) +i40e_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { _i40e_rx_queue_release_mbufs_vec(rxq); } int __rte_cold -i40e_rxq_vec_setup(struct i40e_rx_queue *rxq) +i40e_rxq_vec_setup(struct ci_rx_queue *rxq) { rxq->vector_rx = 1; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c index 7e7f4c0895..6bafd96797 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c @@ -15,12 +15,12 @@ #include static inline void -i40e_rxq_rearm(struct i40e_rx_queue *rxq) +i40e_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); @@ -89,7 +89,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) #ifndef RTE_NET_INTEL_USE_16BYTE_DESC /* SSE version of FDIR mark extraction for 4 32B descriptors at a time */ static inline __m128i -descs_to_fdir_32b(volatile union i40e_rx_desc *rxdp, struct rte_mbuf **rx_pkt) +descs_to_fdir_32b(volatile union ci_rx_desc *rxdp, struct rte_mbuf **rx_pkt) { /* 32B descriptors: Load 2nd half of descriptors for FDIR ID data */ __m128i desc0_qw23, desc1_qw23, desc2_qw23, desc3_qw23; @@ -207,7 +207,7 @@ descs_to_fdir_16b(__m128i fltstat, __m128i descs[4], struct rte_mbuf **rx_pkt) #endif static inline void -desc_to_olflags_v(struct i40e_rx_queue *rxq, volatile union i40e_rx_desc *rxdp, +desc_to_olflags_v(struct ci_rx_queue *rxq, volatile union ci_rx_desc *rxdp, __m128i descs[4], struct rte_mbuf **rx_pkts) { const __m128i mbuf_init = _mm_set_epi64x(0, rxq->mbuf_initializer); @@ -347,16 +347,16 @@ desc_to_ptype_v(__m128i descs[4], struct rte_mbuf **rx_pkts, * - floor align nb_pkts to a I40E_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - volatile union i40e_rx_desc *rxdp; - struct i40e_rx_entry *sw_ring; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; uint64_t var; __m128i shuf_msk; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->i40e_vsi->adapter->ptype_tbl; __m128i crc_adjust = _mm_set_epi16( 0, 0, 0, /* ignore non-length fields */ @@ -609,7 +609,7 @@ i40e_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct i40e_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[I40E_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -755,13 +755,13 @@ i40e_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, } void __rte_cold -i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) +i40e_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { _i40e_rx_queue_release_mbufs_vec(rxq); } int __rte_cold -i40e_rxq_vec_setup(struct i40e_rx_queue *rxq) +i40e_rxq_vec_setup(struct ci_rx_queue *rxq) { rxq->vector_rx = 1; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); From patchwork Mon Jun 9 15:37:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154222 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27E2A468B7; Mon, 9 Jun 2025 17:41:07 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9D1F642D72; Mon, 9 Jun 2025 17:38:29 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id EAA9742D2E for ; Mon, 9 Jun 2025 17:38:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483507; x=1781019507; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=iwaqQBrw+LoCjjcOjLxydlvIX2T2fv6X1rBCJQNfFv4=; b=RLfB+HsNZJ3WvwbZ3IZ7LQv0S0WTSdmjViuJqR8EF7apoxsIy8aoKwHV HyP55HbSVRMvJbmN5Xpr2jPpgaOR7JDMwBhJdClTFNdnpuxBL/fWEZfnV U1DqaULbCUPv5PbiZmiu+LM415oL7QdRY9IXiG2U49ePbG2qY0vRloLf4 DGgfAyjeRClY6aiRGEx2UMhWNfFWCCRsYn2dCBP8g/Ji7Q+xuOP89DtWv OsVd5G7DcJitD866FOrP0lVZvtBEAlCG8owZ4Hyos6Xi7RD/AEA3qNTWC qUtDF4XOkuTnf9BXiHea17vu9jP9A/JlwErzm59/b4PwcZMBZPLqZ/Nir w==; X-CSE-ConnectionGUID: sY+KhTJwTRafulpHecMTCA== X-CSE-MsgGUID: Yms0vPkiRAS1ZVFRS0fsvg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012229" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012229" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:27 -0700 X-CSE-ConnectionGUID: LCPwH2A0RNm8yWABosHKOg== X-CSE-MsgGUID: Dh9WJZ3LSxizjtHxGxK7ww== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419732" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:25 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson Subject: [PATCH v6 25/33] net/ice: use the common Rx queue structure Date: Mon, 9 Jun 2025 16:37:23 +0100 Message-ID: <993c1852caf5d0a5a0ee0b7d552b4d38900b0e8f.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Make the ice driver use the new common Rx queue structure. In addition to 16-byte and 32-byte descriptors supported by other drivers which we adjust ice driver to use where necessary, ice driver also supports flex descriptor format that is also shared by some of the other drivers, so add a common definition for the flex descriptor formats as well. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Separate some of the changes into other commits - Add a new common flex descriptor format definition v2: - Make xtr_field_offs of type ptrdiff_t instead of off_t to fix 32-bit compile issues drivers/net/intel/common/desc.h | 68 ++++++++ drivers/net/intel/common/rx.h | 23 +++ drivers/net/intel/ice/ice_dcf.c | 3 +- drivers/net/intel/ice/ice_dcf_ethdev.c | 25 ++- drivers/net/intel/ice/ice_ethdev.c | 2 +- drivers/net/intel/ice/ice_ethdev.h | 4 +- drivers/net/intel/ice/ice_rxtx.c | 184 ++++++++++---------- drivers/net/intel/ice/ice_rxtx.h | 80 +-------- drivers/net/intel/ice/ice_rxtx_common_avx.h | 8 +- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 14 +- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 16 +- drivers/net/intel/ice/ice_rxtx_vec_common.h | 6 +- drivers/net/intel/ice/ice_rxtx_vec_sse.c | 26 +-- 13 files changed, 241 insertions(+), 218 deletions(-) diff --git a/drivers/net/intel/common/desc.h b/drivers/net/intel/common/desc.h index f9e7f27991..dca265b5f6 100644 --- a/drivers/net/intel/common/desc.h +++ b/drivers/net/intel/common/desc.h @@ -35,6 +35,30 @@ union ci_rx_desc { } qword1; } wb; /* writeback */ }; + +union ci_rx_flex_desc { + struct { + rte_le64_t pkt_addr; /* Packet buffer address */ + rte_le64_t hdr_addr; /* Header buffer address */ + /* bit 0 of hdr_addr is DD bit */ + } read; + struct { + /* Qword 0 */ + uint8_t rxdid; /* descriptor builder profile ID */ + uint8_t mir_id_umb_cast; /* mirror=[5:0], umb=[7:6] */ + rte_le16_t ptype_flex_flags0; /* ptype=[9:0], ff0=[15:10] */ + rte_le16_t pkt_len; /* [15:14] are reserved */ + rte_le16_t hdr_len_sph_flex_flags1; /* header=[10:0] */ + /* sph=[11:11] */ + /* ff1/ext=[15:12] */ + + /* Qword 1 */ + rte_le16_t status_error0; + rte_le16_t l2tag1; + rte_le16_t flex_meta0; + rte_le16_t flex_meta1; + } wb; /* writeback */ +}; #else union ci_rx_desc { struct { @@ -84,6 +108,50 @@ union ci_rx_desc { } qword3; } wb; /* writeback */ }; + +union ci_rx_flex_desc { + struct { + rte_le64_t pkt_addr; /* Packet buffer address */ + rte_le64_t hdr_addr; /* Header buffer address */ + /* bit 0 of hdr_addr is DD bit */ + rte_le64_t rsvd1; + rte_le64_t rsvd2; + } read; + struct { + /* Qword 0 */ + uint8_t rxdid; /* descriptor builder profile ID */ + uint8_t mir_id_umb_cast; /* mirror=[5:0], umb=[7:6] */ + rte_le16_t ptype_flex_flags0; /* ptype=[9:0], ff0=[15:10] */ + rte_le16_t pkt_len; /* [15:14] are reserved */ + rte_le16_t hdr_len_sph_flex_flags1; /* header=[10:0] */ + /* sph=[11:11] */ + /* ff1/ext=[15:12] */ + + /* Qword 1 */ + rte_le16_t status_error0; + rte_le16_t l2tag1; + rte_le16_t flex_meta0; + rte_le16_t flex_meta1; + + /* Qword 2 */ + rte_le16_t status_error1; + uint8_t flex_flags2; + uint8_t time_stamp_low; + rte_le16_t l2tag2_1st; + rte_le16_t l2tag2_2nd; + + /* Qword 3 */ + rte_le16_t flex_meta2; + rte_le16_t flex_meta3; + union { + struct { + rte_le16_t flex_meta4; + rte_le16_t flex_meta5; + } flex; + rte_le32_t ts_high; + } flex_ts; + } wb; /* writeback */ +}; #endif #endif /* _COMMON_INTEL_DESC_H_ */ diff --git a/drivers/net/intel/common/rx.h b/drivers/net/intel/common/rx.h index 98e85406cf..c60d84e447 100644 --- a/drivers/net/intel/common/rx.h +++ b/drivers/net/intel/common/rx.h @@ -5,6 +5,7 @@ #ifndef _COMMON_INTEL_RX_H_ #define _COMMON_INTEL_RX_H_ +#include #include #include #include @@ -13,6 +14,7 @@ #include "desc.h" #define CI_RX_MAX_BURST 32 +#define CI_RX_MAX_NSEG 2 struct ci_rx_queue; @@ -24,6 +26,8 @@ struct ci_rx_entry_sc { struct rte_mbuf *fbuf; /* First segment of the fragmented packet.*/ }; +typedef void (*ci_rx_release_mbufs_t)(struct ci_rx_queue *rxq); + /** * Structure associated with each RX queue. */ @@ -32,6 +36,7 @@ struct ci_rx_queue { union { /* RX ring virtual address */ volatile union ixgbe_adv_rx_desc *ixgbe_rx_ring; volatile union ci_rx_desc *rx_ring; + volatile union ci_rx_flex_desc *rx_flex_ring; }; volatile uint8_t *qrx_tail; /**< register address of tail */ struct ci_rx_entry *sw_ring; /**< address of RX software ring. */ @@ -64,10 +69,16 @@ struct ci_rx_queue { bool drop_en; /**< if 1, drop packets if no descriptors are available. */ uint64_t mbuf_initializer; /**< value to init mbufs */ uint64_t offloads; /**< Rx offloads with RTE_ETH_RX_OFFLOAD_* */ + uint32_t rxdid; /**< RX descriptor format ID. */ + uint32_t proto_xtr; /* protocol extraction type */ + uint64_t xtr_ol_flag; /* flexible descriptor metadata extraction offload flag */ + ptrdiff_t xtr_field_offs; /* Protocol extraction matedata offset*/ + uint64_t hw_time_update; /**< Last time HW timestamp was updated */ /** need to alloc dummy mbuf, for wraparound when scanning hw ring */ struct rte_mbuf fake_mbuf; union { /* the VSI this queue belongs to */ struct i40e_vsi *i40e_vsi; + struct ice_vsi *ice_vsi; }; const struct rte_memzone *mz; union { @@ -85,6 +96,18 @@ struct ci_rx_queue { uint8_t hs_mode; /**< Header Split mode */ uint8_t dcb_tc; /**< Traffic class of rx queue */ }; + struct { /* ice specific values */ + ci_rx_release_mbufs_t rx_rel_mbufs; /**< release mbuf function */ + /** holds buffer split information */ + struct rte_eth_rxseg_split rxseg[CI_RX_MAX_NSEG]; + struct ci_rx_entry *sw_split_buf; /**< Buffer split SW ring */ + uint32_t rxseg_nb; /**< number of buffer split segments */ + uint32_t time_high; /* high 32 bits of hardware timestamp register */ + uint32_t hw_time_high; /* high 32 bits of timestamp */ + uint32_t hw_time_low; /* low 32 bits of timestamp */ + int ts_offset; /* dynamic mbuf timestamp field offset */ + uint64_t ts_flag; /* dynamic mbuf timestamp flag */ + }; }; }; diff --git a/drivers/net/intel/ice/ice_dcf.c b/drivers/net/intel/ice/ice_dcf.c index 2f7c239491..51716a4d5b 100644 --- a/drivers/net/intel/ice/ice_dcf.c +++ b/drivers/net/intel/ice/ice_dcf.c @@ -1175,8 +1175,7 @@ ice_dcf_init_rss(struct ice_dcf_hw *hw) int ice_dcf_configure_queues(struct ice_dcf_hw *hw) { - struct ice_rx_queue **rxq = - (struct ice_rx_queue **)hw->eth_dev->data->rx_queues; + struct ci_rx_queue **rxq = (struct ci_rx_queue **)hw->eth_dev->data->rx_queues; struct ci_tx_queue **txq = (struct ci_tx_queue **)hw->eth_dev->data->tx_queues; struct virtchnl_vsi_queue_config_info *vc_config; diff --git a/drivers/net/intel/ice/ice_dcf_ethdev.c b/drivers/net/intel/ice/ice_dcf_ethdev.c index d3fd5d7122..88d943d432 100644 --- a/drivers/net/intel/ice/ice_dcf_ethdev.c +++ b/drivers/net/intel/ice/ice_dcf_ethdev.c @@ -106,7 +106,7 @@ ice_dcf_xmit_pkts(__rte_unused void *tx_queue, } static int -ice_dcf_init_rxq(struct rte_eth_dev *dev, struct ice_rx_queue *rxq) +ice_dcf_init_rxq(struct rte_eth_dev *dev, struct ci_rx_queue *rxq) { struct ice_dcf_adapter *dcf_ad = dev->data->dev_private; struct rte_eth_dev_data *dev_data = dev->data; @@ -145,8 +145,7 @@ ice_dcf_init_rxq(struct rte_eth_dev *dev, struct ice_rx_queue *rxq) static int ice_dcf_init_rx_queues(struct rte_eth_dev *dev) { - struct ice_rx_queue **rxq = - (struct ice_rx_queue **)dev->data->rx_queues; + struct ci_rx_queue **rxq = (struct ci_rx_queue **)dev->data->rx_queues; int i, ret; for (i = 0; i < dev->data->nb_rx_queues; i++) { @@ -282,9 +281,9 @@ ice_dcf_config_rx_queues_irqs(struct rte_eth_dev *dev, } static int -alloc_rxq_mbufs(struct ice_rx_queue *rxq) +alloc_rxq_mbufs(struct ci_rx_queue *rxq) { - volatile union ice_rx_flex_desc *rxd; + volatile union ci_rx_flex_desc *rxd; struct rte_mbuf *mbuf = NULL; uint64_t dma_addr; uint16_t i; @@ -305,7 +304,7 @@ alloc_rxq_mbufs(struct ice_rx_queue *rxq) dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(mbuf)); - rxd = &rxq->rx_ring[i]; + rxd = &rxq->rx_flex_ring[i]; rxd->read.pkt_addr = dma_addr; rxd->read.hdr_addr = 0; #ifndef RTE_NET_INTEL_USE_16BYTE_DESC @@ -324,7 +323,7 @@ ice_dcf_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) { struct ice_dcf_adapter *ad = dev->data->dev_private; struct iavf_hw *hw = &ad->real_hw.avf; - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int err = 0; if (rx_queue_id >= dev->data->nb_rx_queues) @@ -358,7 +357,7 @@ ice_dcf_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) } static inline void -reset_rx_queue(struct ice_rx_queue *rxq) +reset_rx_queue(struct ci_rx_queue *rxq) { uint16_t len; uint32_t i; @@ -368,8 +367,8 @@ reset_rx_queue(struct ice_rx_queue *rxq) len = rxq->nb_rx_desc + ICE_RX_MAX_BURST; - for (i = 0; i < len * sizeof(union ice_rx_flex_desc); i++) - ((volatile char *)rxq->rx_ring)[i] = 0; + for (i = 0; i < len * sizeof(union ci_rx_flex_desc); i++) + ((volatile char *)rxq->rx_flex_ring)[i] = 0; memset(&rxq->fake_mbuf, 0x0, sizeof(rxq->fake_mbuf)); @@ -429,7 +428,7 @@ ice_dcf_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) { struct ice_dcf_adapter *ad = dev->data->dev_private; struct ice_dcf_hw *hw = &ad->real_hw; - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; if (rx_queue_id >= dev->data->nb_rx_queues) @@ -511,7 +510,7 @@ ice_dcf_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id) static int ice_dcf_start_queues(struct rte_eth_dev *dev) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; struct ci_tx_queue *txq; int nb_rxq = 0; int nb_txq, i; @@ -638,7 +637,7 @@ ice_dcf_stop_queues(struct rte_eth_dev *dev) { struct ice_dcf_adapter *ad = dev->data->dev_private; struct ice_dcf_hw *hw = &ad->real_hw; - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; struct ci_tx_queue *txq; int ret, i; diff --git a/drivers/net/intel/ice/ice_ethdev.c b/drivers/net/intel/ice/ice_ethdev.c index 7cc083ca32..938c89e773 100644 --- a/drivers/net/intel/ice/ice_ethdev.c +++ b/drivers/net/intel/ice/ice_ethdev.c @@ -6724,7 +6724,7 @@ ice_timesync_read_rx_timestamp(struct rte_eth_dev *dev, struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private); struct ice_adapter *ad = ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; uint32_t ts_high; uint64_t ts_ns; diff --git a/drivers/net/intel/ice/ice_ethdev.h b/drivers/net/intel/ice/ice_ethdev.h index bfe093afca..8e5799f8b4 100644 --- a/drivers/net/intel/ice/ice_ethdev.h +++ b/drivers/net/intel/ice/ice_ethdev.h @@ -257,7 +257,7 @@ struct ice_vsi_list { struct ice_vsi *vsi; }; -struct ice_rx_queue; +struct ci_rx_queue; struct ci_tx_queue; @@ -425,7 +425,7 @@ struct ice_fdir_counter_pool_container { struct ice_fdir_info { struct ice_vsi *fdir_vsi; /* pointer to fdir VSI structure */ struct ci_tx_queue *txq; - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; void *prg_pkt; /* memory for fdir program packet */ uint64_t dma_addr; /* physic address of packet memory*/ const struct rte_memzone *mz; diff --git a/drivers/net/intel/ice/ice_rxtx.c b/drivers/net/intel/ice/ice_rxtx.c index 19569b6a38..e2fcc31d0d 100644 --- a/drivers/net/intel/ice/ice_rxtx.c +++ b/drivers/net/intel/ice/ice_rxtx.c @@ -36,12 +36,12 @@ ice_monitor_callback(const uint64_t value, int ice_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc) { - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_queue *rxq = rx_queue; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_queue *rxq = rx_queue; uint16_t desc; desc = rxq->rx_tail; - rxdp = &rxq->rx_ring[desc]; + rxdp = &rxq->rx_flex_ring[desc]; /* watch for changes in status bit */ pmc->addr = &rxdp->wb.status_error0; @@ -73,9 +73,9 @@ ice_proto_xtr_type_to_rxdid(uint8_t xtr_type) } static inline void -ice_rxd_to_pkt_fields_by_comms_generic(__rte_unused struct ice_rx_queue *rxq, +ice_rxd_to_pkt_fields_by_comms_generic(__rte_unused struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union ice_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct ice_32b_rx_flex_desc_comms *desc = (volatile struct ice_32b_rx_flex_desc_comms *)rxdp; @@ -95,9 +95,9 @@ ice_rxd_to_pkt_fields_by_comms_generic(__rte_unused struct ice_rx_queue *rxq, } static inline void -ice_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct ice_rx_queue *rxq, +ice_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union ice_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct ice_32b_rx_flex_desc_comms_ovs *desc = (volatile struct ice_32b_rx_flex_desc_comms_ovs *)rxdp; @@ -120,9 +120,9 @@ ice_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct ice_rx_queue *rxq, } static inline void -ice_rxd_to_pkt_fields_by_comms_aux_v1(struct ice_rx_queue *rxq, +ice_rxd_to_pkt_fields_by_comms_aux_v1(struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union ice_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct ice_32b_rx_flex_desc_comms *desc = (volatile struct ice_32b_rx_flex_desc_comms *)rxdp; @@ -164,9 +164,9 @@ ice_rxd_to_pkt_fields_by_comms_aux_v1(struct ice_rx_queue *rxq, } static inline void -ice_rxd_to_pkt_fields_by_comms_aux_v2(struct ice_rx_queue *rxq, +ice_rxd_to_pkt_fields_by_comms_aux_v2(struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union ice_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct ice_32b_rx_flex_desc_comms *desc = (volatile struct ice_32b_rx_flex_desc_comms *)rxdp; @@ -215,7 +215,7 @@ static const ice_rxd_to_pkt_fields_t rxd_to_pkt_fields_ops[] = { }; void -ice_select_rxd_to_pkt_fields_handler(struct ice_rx_queue *rxq, uint32_t rxdid) +ice_select_rxd_to_pkt_fields_handler(struct ci_rx_queue *rxq, uint32_t rxdid) { rxq->rxdid = rxdid; @@ -243,17 +243,17 @@ ice_select_rxd_to_pkt_fields_handler(struct ice_rx_queue *rxq, uint32_t rxdid) } static int -ice_program_hw_rx_queue(struct ice_rx_queue *rxq) +ice_program_hw_rx_queue(struct ci_rx_queue *rxq) { - struct ice_vsi *vsi = rxq->vsi; + struct ice_vsi *vsi = rxq->ice_vsi; struct ice_hw *hw = ICE_VSI_TO_HW(vsi); struct ice_pf *pf = ICE_VSI_TO_PF(vsi); - struct rte_eth_dev_data *dev_data = rxq->vsi->adapter->pf.dev_data; + struct rte_eth_dev_data *dev_data = rxq->ice_vsi->adapter->pf.dev_data; struct ice_rlan_ctx rx_ctx; uint16_t buf_size; uint32_t rxdid = ICE_RXDID_COMMS_OVS; uint32_t regval; - struct ice_adapter *ad = rxq->vsi->adapter; + struct ice_adapter *ad = rxq->ice_vsi->adapter; uint32_t frame_size = dev_data->mtu + ICE_ETH_OVERHEAD; int err; @@ -451,15 +451,15 @@ ice_program_hw_rx_queue(struct ice_rx_queue *rxq) /* Allocate mbufs for all descriptors in rx queue */ static int -ice_alloc_rx_queue_mbufs(struct ice_rx_queue *rxq) +ice_alloc_rx_queue_mbufs(struct ci_rx_queue *rxq) { - struct ice_rx_entry *rxe = rxq->sw_ring; + struct ci_rx_entry *rxe = rxq->sw_ring; uint64_t dma_addr; uint16_t i; for (i = 0; i < rxq->nb_rx_desc; i++) { - volatile union ice_rx_flex_desc *rxd; - rxd = &rxq->rx_ring[i]; + volatile union ci_rx_flex_desc *rxd; + rxd = &rxq->rx_flex_ring[i]; struct rte_mbuf *mbuf = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!mbuf)) { @@ -513,7 +513,7 @@ ice_alloc_rx_queue_mbufs(struct ice_rx_queue *rxq) /* Free all mbufs for descriptors in rx queue */ static void -_ice_rx_queue_release_mbufs(struct ice_rx_queue *rxq) +_ice_rx_queue_release_mbufs(struct ci_rx_queue *rxq) { uint16_t i; @@ -590,7 +590,7 @@ ice_switch_rx_queue(struct ice_hw *hw, uint16_t q_idx, bool on) } static inline int -ice_check_rx_burst_bulk_alloc_preconditions(struct ice_rx_queue *rxq) +ice_check_rx_burst_bulk_alloc_preconditions(struct ci_rx_queue *rxq) { int ret = 0; @@ -617,9 +617,9 @@ ice_check_rx_burst_bulk_alloc_preconditions(struct ice_rx_queue *rxq) return ret; } -/* reset fields in ice_rx_queue back to default */ +/* reset fields in ci_rx_queue back to default */ static void -ice_reset_rx_queue(struct ice_rx_queue *rxq) +ice_reset_rx_queue(struct ci_rx_queue *rxq) { unsigned int i; uint16_t len; @@ -631,8 +631,8 @@ ice_reset_rx_queue(struct ice_rx_queue *rxq) len = (uint16_t)(rxq->nb_rx_desc + ICE_RX_MAX_BURST); - for (i = 0; i < len * sizeof(union ice_rx_flex_desc); i++) - ((volatile char *)rxq->rx_ring)[i] = 0; + for (i = 0; i < len * sizeof(union ci_rx_flex_desc); i++) + ((volatile char *)rxq->rx_flex_ring)[i] = 0; memset(&rxq->fake_mbuf, 0x0, sizeof(rxq->fake_mbuf)); for (i = 0; i < ICE_RX_MAX_BURST; ++i) @@ -654,7 +654,7 @@ ice_reset_rx_queue(struct ice_rx_queue *rxq) int ice_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private); @@ -714,7 +714,7 @@ ice_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) int ice_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private); @@ -833,9 +833,9 @@ ice_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id) } static int -ice_fdir_program_hw_rx_queue(struct ice_rx_queue *rxq) +ice_fdir_program_hw_rx_queue(struct ci_rx_queue *rxq) { - struct ice_vsi *vsi = rxq->vsi; + struct ice_vsi *vsi = rxq->ice_vsi; struct ice_hw *hw = ICE_VSI_TO_HW(vsi); uint32_t rxdid = ICE_RXDID_LEGACY_1; struct ice_rlan_ctx rx_ctx; @@ -908,7 +908,7 @@ ice_fdir_program_hw_rx_queue(struct ice_rx_queue *rxq) int ice_fdir_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private); struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private); @@ -1098,7 +1098,7 @@ ice_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id) int ice_fdir_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; struct ice_hw *hw = ICE_DEV_PRIVATE_TO_HW(dev->data->dev_private); struct ice_pf *pf = ICE_DEV_PRIVATE_TO_PF(dev->data->dev_private); @@ -1169,7 +1169,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, struct ice_adapter *ad = ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); struct ice_vsi *vsi = pf->main_vsi; - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; const struct rte_memzone *rz; uint32_t ring_size, tlen; uint16_t len; @@ -1205,7 +1205,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, /* Allocate the rx queue data structure */ rxq = rte_zmalloc_socket(NULL, - sizeof(struct ice_rx_queue), + sizeof(struct ci_rx_queue), RTE_CACHE_LINE_SIZE, socket_id); @@ -1239,7 +1239,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, rxq->crc_len = 0; rxq->drop_en = rx_conf->rx_drop_en; - rxq->vsi = vsi; + rxq->ice_vsi = vsi; rxq->rx_deferred_start = rx_conf->rx_deferred_start; rxq->proto_xtr = pf->proto_xtr != NULL ? pf->proto_xtr[queue_idx] : PROTO_XTR_NONE; @@ -1258,7 +1258,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, len += ICE_RX_MAX_BURST; /* Allocate the maximum number of RX ring hardware descriptor. */ - ring_size = sizeof(union ice_rx_flex_desc) * len; + ring_size = sizeof(union ci_rx_flex_desc) * len; ring_size = RTE_ALIGN(ring_size, ICE_DMA_MEM_ALIGN); rz = rte_eth_dma_zone_reserve(dev, "rx_ring", queue_idx, ring_size, ICE_RING_BASE_ALIGN, @@ -1274,7 +1274,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, memset(rz->addr, 0, ring_size); rxq->rx_ring_phys_addr = rz->iova; - rxq->rx_ring = rz->addr; + rxq->rx_flex_ring = rz->addr; /* always reserve more for bulk alloc */ len = (uint16_t)(nb_desc + ICE_RX_MAX_BURST); @@ -1286,7 +1286,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, /* Allocate the software ring. */ rxq->sw_ring = rte_zmalloc_socket(NULL, - sizeof(struct ice_rx_entry) * tlen, + sizeof(struct ci_rx_entry) * tlen, RTE_CACHE_LINE_SIZE, socket_id); if (!rxq->sw_ring) { @@ -1323,7 +1323,7 @@ ice_rx_queue_setup(struct rte_eth_dev *dev, void ice_rx_queue_release(void *rxq) { - struct ice_rx_queue *q = (struct ice_rx_queue *)rxq; + struct ci_rx_queue *q = (struct ci_rx_queue *)rxq; if (!q) { PMD_DRV_LOG(DEBUG, "Pointer to rxq is NULL"); @@ -1547,7 +1547,7 @@ void ice_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_rxq_info *qinfo) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; rxq = dev->data->rx_queues[queue_id]; @@ -1584,12 +1584,12 @@ uint32_t ice_rx_queue_count(void *rx_queue) { #define ICE_RXQ_SCAN_INTERVAL 4 - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_queue *rxq; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_queue *rxq; uint16_t desc = 0; rxq = rx_queue; - rxdp = &rxq->rx_ring[rxq->rx_tail]; + rxdp = &rxq->rx_flex_ring[rxq->rx_tail]; while ((desc < rxq->nb_rx_desc) && rte_le_to_cpu_16(rxdp->wb.status_error0) & (1 << ICE_RX_FLEX_DESC_STATUS0_DD_S)) { @@ -1601,8 +1601,7 @@ ice_rx_queue_count(void *rx_queue) desc += ICE_RXQ_SCAN_INTERVAL; rxdp += ICE_RXQ_SCAN_INTERVAL; if (rxq->rx_tail + desc >= rxq->nb_rx_desc) - rxdp = &(rxq->rx_ring[rxq->rx_tail + - desc - rxq->nb_rx_desc]); + rxdp = &rxq->rx_flex_ring[rxq->rx_tail + desc - rxq->nb_rx_desc]; } return desc; @@ -1655,7 +1654,7 @@ ice_rxd_error_to_pkt_flags(uint16_t stat_err0) } static inline void -ice_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union ice_rx_flex_desc *rxdp) +ice_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union ci_rx_flex_desc *rxdp) { if (rte_le_to_cpu_16(rxdp->wb.status_error0) & (1 << ICE_RX_FLEX_DESC_STATUS0_L2TAG1P_S)) { @@ -1694,25 +1693,25 @@ ice_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union ice_rx_flex_desc *rxdp) #define ICE_PTP_TS_VALID 0x1 static inline int -ice_rx_scan_hw_ring(struct ice_rx_queue *rxq) +ice_rx_scan_hw_ring(struct ci_rx_queue *rxq) { - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_entry *rxep; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t stat_err0; uint16_t pkt_len, hdr_len; int32_t s[ICE_LOOK_AHEAD], nb_dd; int32_t i, j, nb_rx = 0; uint64_t pkt_flags = 0; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->ice_vsi->adapter->ptype_tbl; #ifndef RTE_NET_INTEL_USE_16BYTE_DESC bool is_tsinit = false; uint64_t ts_ns; - struct ice_vsi *vsi = rxq->vsi; + struct ice_vsi *vsi = rxq->ice_vsi; struct ice_hw *hw = ICE_VSI_TO_HW(vsi); - struct ice_adapter *ad = rxq->vsi->adapter; + struct ice_adapter *ad = rxq->ice_vsi->adapter; #endif - rxdp = &rxq->rx_ring[rxq->rx_tail]; + rxdp = &rxq->rx_flex_ring[rxq->rx_tail]; rxep = &rxq->sw_ring[rxq->rx_tail]; stat_err0 = rte_le_to_cpu_16(rxdp->wb.status_error0); @@ -1842,7 +1841,7 @@ ice_rx_scan_hw_ring(struct ice_rx_queue *rxq) } static inline uint16_t -ice_rx_fill_from_stage(struct ice_rx_queue *rxq, +ice_rx_fill_from_stage(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { @@ -1861,10 +1860,10 @@ ice_rx_fill_from_stage(struct ice_rx_queue *rxq, } static inline int -ice_rx_alloc_bufs(struct ice_rx_queue *rxq) +ice_rx_alloc_bufs(struct ci_rx_queue *rxq) { - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_entry *rxep; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t alloc_idx, i; uint64_t dma_addr; @@ -1893,7 +1892,7 @@ ice_rx_alloc_bufs(struct ice_rx_queue *rxq) } } - rxdp = &rxq->rx_ring[alloc_idx]; + rxdp = &rxq->rx_flex_ring[alloc_idx]; for (i = 0; i < rxq->rx_free_thresh; i++) { if (likely(i < (rxq->rx_free_thresh - 1))) /* Prefetch next mbuf */ @@ -1932,7 +1931,7 @@ ice_rx_alloc_bufs(struct ice_rx_queue *rxq) static inline uint16_t rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ice_rx_queue *rxq = (struct ice_rx_queue *)rx_queue; + struct ci_rx_queue *rxq = (struct ci_rx_queue *)rx_queue; uint16_t nb_rx = 0; if (!nb_pkts) @@ -1950,7 +1949,7 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) if (ice_rx_alloc_bufs(rxq) != 0) { uint16_t i, j; - rxq->vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed += + rxq->ice_vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed += rxq->rx_free_thresh; PMD_RX_LOG(DEBUG, "Rx mbuf alloc failed for " "port_id=%u, queue_id=%u", @@ -2005,12 +2004,12 @@ ice_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ice_rx_queue *rxq = rx_queue; - volatile union ice_rx_flex_desc *rx_ring = rxq->rx_ring; - volatile union ice_rx_flex_desc *rxdp; - union ice_rx_flex_desc rxd; - struct ice_rx_entry *sw_ring = rxq->sw_ring; - struct ice_rx_entry *rxe; + struct ci_rx_queue *rxq = rx_queue; + volatile union ci_rx_flex_desc *rx_ring = rxq->rx_flex_ring; + volatile union ci_rx_flex_desc *rxdp; + union ci_rx_flex_desc rxd; + struct ci_rx_entry *sw_ring = rxq->sw_ring; + struct ci_rx_entry *rxe; struct rte_mbuf *first_seg = rxq->pkt_first_seg; struct rte_mbuf *last_seg = rxq->pkt_last_seg; struct rte_mbuf *nmb; /* new allocated mbuf */ @@ -2022,13 +2021,13 @@ ice_recv_scattered_pkts(void *rx_queue, uint16_t rx_stat_err0; uint64_t dma_addr; uint64_t pkt_flags; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->ice_vsi->adapter->ptype_tbl; #ifndef RTE_NET_INTEL_USE_16BYTE_DESC bool is_tsinit = false; uint64_t ts_ns; - struct ice_vsi *vsi = rxq->vsi; + struct ice_vsi *vsi = rxq->ice_vsi; struct ice_hw *hw = ICE_VSI_TO_HW(vsi); - struct ice_adapter *ad = rxq->vsi->adapter; + struct ice_adapter *ad = rxq->ice_vsi->adapter; if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); @@ -2049,7 +2048,7 @@ ice_recv_scattered_pkts(void *rx_queue, /* allocate mbuf */ nmb = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!nmb)) { - rxq->vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed++; + rxq->ice_vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed++; break; } rxd = *rxdp; /* copy descriptor in ring to temp variable*/ @@ -2317,8 +2316,8 @@ ice_dev_supported_ptypes_get(struct rte_eth_dev *dev, size_t *no_of_elements) int ice_rx_descriptor_status(void *rx_queue, uint16_t offset) { - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_queue *rxq = rx_queue; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_queue *rxq = rx_queue; uint32_t desc; if (unlikely(offset >= rxq->nb_rx_desc)) @@ -2331,7 +2330,7 @@ ice_rx_descriptor_status(void *rx_queue, uint16_t offset) if (desc >= rxq->nb_rx_desc) desc -= rxq->nb_rx_desc; - rxdp = &rxq->rx_ring[desc]; + rxdp = &rxq->rx_flex_ring[desc]; if (rte_le_to_cpu_16(rxdp->wb.status_error0) & (1 << ICE_RX_FLEX_DESC_STATUS0_DD_S)) return RTE_ETH_RX_DESC_DONE; @@ -2458,7 +2457,7 @@ ice_fdir_setup_tx_resources(struct ice_pf *pf) int ice_fdir_setup_rx_resources(struct ice_pf *pf) { - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; const struct rte_memzone *rz = NULL; uint32_t ring_size; struct rte_eth_dev *dev; @@ -2472,7 +2471,7 @@ ice_fdir_setup_rx_resources(struct ice_pf *pf) /* Allocate the RX queue data structure. */ rxq = rte_zmalloc_socket("ice fdir rx queue", - sizeof(struct ice_rx_queue), + sizeof(struct ci_rx_queue), RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); if (!rxq) { @@ -2498,12 +2497,12 @@ ice_fdir_setup_rx_resources(struct ice_pf *pf) rxq->nb_rx_desc = ICE_FDIR_NUM_RX_DESC; rxq->queue_id = ICE_FDIR_QUEUE_ID; rxq->reg_idx = pf->fdir.fdir_vsi->base_queue; - rxq->vsi = pf->fdir.fdir_vsi; + rxq->ice_vsi = pf->fdir.fdir_vsi; rxq->rx_ring_phys_addr = rz->iova; memset(rz->addr, 0, ICE_FDIR_NUM_RX_DESC * sizeof(union ice_32byte_rx_desc)); - rxq->rx_ring = (union ice_rx_flex_desc *)rz->addr; + rxq->rx_flex_ring = (union ci_rx_flex_desc *)rz->addr; /* * Don't need to allocate software ring and reset for the fdir @@ -2522,12 +2521,12 @@ ice_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ice_rx_queue *rxq = rx_queue; - volatile union ice_rx_flex_desc *rx_ring = rxq->rx_ring; - volatile union ice_rx_flex_desc *rxdp; - union ice_rx_flex_desc rxd; - struct ice_rx_entry *sw_ring = rxq->sw_ring; - struct ice_rx_entry *rxe; + struct ci_rx_queue *rxq = rx_queue; + volatile union ci_rx_flex_desc *rx_ring = rxq->rx_flex_ring; + volatile union ci_rx_flex_desc *rxdp; + union ci_rx_flex_desc rxd; + struct ci_rx_entry *sw_ring = rxq->sw_ring; + struct ci_rx_entry *rxe; struct rte_mbuf *nmb; /* new allocated mbuf */ struct rte_mbuf *nmb_pay; /* new allocated payload mbuf */ struct rte_mbuf *rxm; /* pointer to store old mbuf in SW ring */ @@ -2539,13 +2538,13 @@ ice_recv_pkts(void *rx_queue, uint16_t rx_stat_err0; uint64_t dma_addr; uint64_t pkt_flags; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->ice_vsi->adapter->ptype_tbl; #ifndef RTE_NET_INTEL_USE_16BYTE_DESC bool is_tsinit = false; uint64_t ts_ns; - struct ice_vsi *vsi = rxq->vsi; + struct ice_vsi *vsi = rxq->ice_vsi; struct ice_hw *hw = ICE_VSI_TO_HW(vsi); - struct ice_adapter *ad = rxq->vsi->adapter; + struct ice_adapter *ad = rxq->ice_vsi->adapter; if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); @@ -2566,7 +2565,7 @@ ice_recv_pkts(void *rx_queue, /* allocate header mbuf */ nmb = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!nmb)) { - rxq->vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed++; + rxq->ice_vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed++; break; } @@ -2593,7 +2592,7 @@ ice_recv_pkts(void *rx_queue, /* allocate payload mbuf */ nmb_pay = rte_mbuf_raw_alloc(rxq->rxseg[1].mp); if (unlikely(!nmb_pay)) { - rxq->vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed++; + rxq->ice_vsi->adapter->pf.dev_data->rx_mbuf_alloc_failed++; rxe->mbuf = NULL; nb_hold--; if (unlikely(rx_id == 0)) @@ -3471,7 +3470,7 @@ ice_set_rx_function(struct rte_eth_dev *dev) struct ice_adapter *ad = ICE_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); #ifdef RTE_ARCH_X86 - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int i; int rx_check_ret = -1; @@ -4633,7 +4632,7 @@ ice_set_default_ptype_table(struct rte_eth_dev *dev) * tx queue */ static inline int -ice_check_fdir_programming_status(struct ice_rx_queue *rxq) +ice_check_fdir_programming_status(struct ci_rx_queue *rxq) { volatile union ice_32byte_rx_desc *rxdp; uint64_t qword1; @@ -4642,8 +4641,7 @@ ice_check_fdir_programming_status(struct ice_rx_queue *rxq) uint32_t id; int ret = -EAGAIN; - rxdp = (volatile union ice_32byte_rx_desc *) - (&rxq->rx_ring[rxq->rx_tail]); + rxdp = (volatile union ice_32byte_rx_desc *)&rxq->rx_flex_ring[rxq->rx_tail]; qword1 = rte_le_to_cpu_64(rxdp->wb.qword1.status_error_len); rx_status = (qword1 & ICE_RXD_QW1_STATUS_M) >> ICE_RXD_QW1_STATUS_S; @@ -4688,7 +4686,7 @@ int ice_fdir_programming(struct ice_pf *pf, struct ice_fltr_desc *fdir_desc) { struct ci_tx_queue *txq = pf->fdir.txq; - struct ice_rx_queue *rxq = pf->fdir.rxq; + struct ci_rx_queue *rxq = pf->fdir.rxq; volatile struct ice_fltr_desc *fdirdp; volatile struct ice_tx_desc *txdp; uint32_t td_cmd; diff --git a/drivers/net/intel/ice/ice_rxtx.h b/drivers/net/intel/ice/ice_rxtx.h index 52c753ba7c..62f98579f5 100644 --- a/drivers/net/intel/ice/ice_rxtx.h +++ b/drivers/net/intel/ice/ice_rxtx.h @@ -5,6 +5,7 @@ #ifndef _ICE_RXTX_H_ #define _ICE_RXTX_H_ +#include "../common/rx.h" #include "../common/tx.h" #include "ice_ethdev.h" @@ -14,21 +15,15 @@ #define ICE_DMA_MEM_ALIGN 4096 #define ICE_RING_BASE_ALIGN 128 -#define ICE_RX_MAX_BURST 32 +#define ICE_RX_MAX_BURST CI_RX_MAX_BURST #define ICE_TX_MAX_BURST 32 /* Maximal number of segments to split. */ -#define ICE_RX_MAX_NSEG 2 +#define ICE_RX_MAX_NSEG CI_RX_MAX_NSEG #define ICE_CHK_Q_ENA_COUNT 100 #define ICE_CHK_Q_ENA_INTERVAL_US 100 -#ifdef RTE_NET_INTEL_USE_16BYTE_DESC -#define ice_rx_flex_desc ice_16b_rx_flex_desc -#else -#define ice_rx_flex_desc ice_32b_rx_flex_desc -#endif - #define ICE_SUPPORT_CHAIN_NUM 5 #define ICE_TD_CMD ICE_TX_DESC_CMD_EOP @@ -75,14 +70,9 @@ #define ICE_TX_MTU_SEG_MAX 8 -typedef void (*ice_rx_release_mbufs_t)(struct ice_rx_queue *rxq); -typedef void (*ice_rxd_to_pkt_fields_t)(struct ice_rx_queue *rxq, +typedef void (*ice_rxd_to_pkt_fields_t)(struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union ice_rx_flex_desc *rxdp); - -struct ice_rx_entry { - struct rte_mbuf *mbuf; -}; + volatile union ci_rx_flex_desc *rxdp); enum ice_rx_dtype { ICE_RX_DTYPE_NO_SPLIT = 0, @@ -90,60 +80,6 @@ enum ice_rx_dtype { ICE_RX_DTYPE_SPLIT_ALWAYS = 2, }; -struct ice_rx_queue { - struct rte_mempool *mp; /* mbuf pool to populate RX ring */ - volatile union ice_rx_flex_desc *rx_ring;/* RX ring virtual address */ - rte_iova_t rx_ring_phys_addr; /* RX ring DMA address */ - struct ice_rx_entry *sw_ring; /* address of RX soft ring */ - uint16_t nb_rx_desc; /* number of RX descriptors */ - uint16_t rx_free_thresh; /* max free RX desc to hold */ - uint16_t rx_tail; /* current value of tail */ - uint16_t nb_rx_hold; /* number of held free RX desc */ - struct rte_mbuf *pkt_first_seg; /**< first segment of current packet */ - struct rte_mbuf *pkt_last_seg; /**< last segment of current packet */ - uint16_t rx_nb_avail; /**< number of staged packets ready */ - uint16_t rx_next_avail; /**< index of next staged packets */ - uint16_t rx_free_trigger; /**< triggers rx buffer allocation */ - struct rte_mbuf fake_mbuf; /**< dummy mbuf */ - struct rte_mbuf *rx_stage[ICE_RX_MAX_BURST * 2]; - - uint16_t rxrearm_nb; /**< number of remaining to be re-armed */ - uint16_t rxrearm_start; /**< the idx we start the re-arming from */ - uint64_t mbuf_initializer; /**< value to init mbufs */ - - uint16_t port_id; /* device port ID */ - uint8_t crc_len; /* 0 if CRC stripped, 4 otherwise */ - uint8_t fdir_enabled; /* 0 if FDIR disabled, 1 when enabled */ - uint16_t queue_id; /* RX queue index */ - uint16_t reg_idx; /* RX queue register index */ - uint8_t drop_en; /* if not 0, set register bit */ - volatile uint8_t *qrx_tail; /* register address of tail */ - struct ice_vsi *vsi; /* the VSI this queue belongs to */ - uint16_t rx_buf_len; /* The packet buffer size */ - uint16_t rx_hdr_len; /* The header buffer size */ - uint16_t max_pkt_len; /* Maximum packet length */ - bool q_set; /* indicate if rx queue has been configured */ - bool rx_deferred_start; /* don't start this queue in dev start */ - uint8_t proto_xtr; /* Protocol extraction from flexible descriptor */ - int xtr_field_offs; /*Protocol extraction matedata offset*/ - uint64_t xtr_ol_flag; /* Protocol extraction offload flag */ - uint32_t rxdid; /* Receive Flex Descriptor profile ID */ - ice_rx_release_mbufs_t rx_rel_mbufs; - uint64_t offloads; - uint32_t time_high; - uint32_t hw_register_set; - const struct rte_memzone *mz; - uint32_t hw_time_high; /* high 32 bits of timestamp */ - uint32_t hw_time_low; /* low 32 bits of timestamp */ - uint64_t hw_time_update; /* SW time of HW record updating */ - struct ice_rx_entry *sw_split_buf; - /* address of temp buffer for RX split mbufs */ - struct rte_eth_rxseg_split rxseg[ICE_RX_MAX_NSEG]; - uint32_t rxseg_nb; - int ts_offset; /* dynamic mbuf timestamp field offset */ - uint64_t ts_flag; /* dynamic mbuf timestamp flag */ -}; - /* Offload features */ union ice_tx_offload { uint64_t data; @@ -247,12 +183,12 @@ int ice_tx_descriptor_status(void *tx_queue, uint16_t offset); void ice_set_default_ptype_table(struct rte_eth_dev *dev); const uint32_t *ice_dev_supported_ptypes_get(struct rte_eth_dev *dev, size_t *no_of_elements); -void ice_select_rxd_to_pkt_fields_handler(struct ice_rx_queue *rxq, +void ice_select_rxd_to_pkt_fields_handler(struct ci_rx_queue *rxq, uint32_t rxdid); int ice_rx_vec_dev_check(struct rte_eth_dev *dev); int ice_tx_vec_dev_check(struct rte_eth_dev *dev); -int ice_rxq_vec_setup(struct ice_rx_queue *rxq); +int ice_rxq_vec_setup(struct ci_rx_queue *rxq); int ice_txq_vec_setup(struct ci_tx_queue *txq); uint16_t ice_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); @@ -297,7 +233,7 @@ int ice_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc); #define FDIR_PARSING_ENABLE_PER_QUEUE(ad, on) do { \ int i; \ for (i = 0; i < (ad)->pf.dev_data->nb_rx_queues; i++) { \ - struct ice_rx_queue *rxq = (ad)->pf.dev_data->rx_queues[i]; \ + struct ci_rx_queue *rxq = (ad)->pf.dev_data->rx_queues[i]; \ if (!rxq) \ continue; \ rxq->fdir_enabled = on; \ diff --git a/drivers/net/intel/ice/ice_rxtx_common_avx.h b/drivers/net/intel/ice/ice_rxtx_common_avx.h index d1c772bf06..7c65e7ed4d 100644 --- a/drivers/net/intel/ice/ice_rxtx_common_avx.h +++ b/drivers/net/intel/ice/ice_rxtx_common_avx.h @@ -9,14 +9,14 @@ #ifdef __AVX2__ static __rte_always_inline void -ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) +ice_rxq_rearm_common(struct ci_rx_queue *rxq, __rte_unused bool avx512) { int i; uint16_t rx_id; - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxdp = rxq->rx_flex_ring + rxq->rxrearm_start; /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c index 5ed669fc30..5b1a13dd22 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c @@ -8,7 +8,7 @@ #include static __rte_always_inline void -ice_rxq_rearm(struct ice_rx_queue *rxq) +ice_rxq_rearm(struct ci_rx_queue *rxq) { ice_rxq_rearm_common(rxq, false); } @@ -33,15 +33,15 @@ ice_flex_rxd_to_fdir_flags_vec_avx2(const __m256i fdir_id0_7) } static __rte_always_inline uint16_t -_ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_ice_recv_raw_pkts_vec_avx2(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->ice_vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct ice_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union ice_rx_flex_desc *rxdp = rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_flex_desc *rxdp = rxq->rx_flex_ring + rxq->rx_tail; const int avx_aligned = ((rxq->rx_tail & 1) == 0); rte_prefetch0(rxdp); @@ -443,7 +443,7 @@ _ice_recv_raw_pkts_vec_avx2(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. */ - if (rxq->vsi->adapter->pf.dev_data->dev_conf.rxmode.offloads & + if (rxq->ice_vsi->adapter->pf.dev_data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH) { /* load bottom half of every 32B desc */ const __m128i raw_desc_bh7 = _mm_load_si128 @@ -692,7 +692,7 @@ static __rte_always_inline uint16_t ice_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool offload) { - struct ice_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[ICE_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c index e52e9e9ceb..b943caf0f0 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c @@ -8,7 +8,7 @@ #include static __rte_always_inline void -ice_rxq_rearm(struct ice_rx_queue *rxq) +ice_rxq_rearm(struct ci_rx_queue *rxq) { ice_rxq_rearm_common(rxq, true); } @@ -33,17 +33,17 @@ ice_flex_rxd_to_fdir_flags_vec_avx512(const __m256i fdir_id0_7) } static __rte_always_inline uint16_t -_ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, +_ice_recv_raw_pkts_vec_avx512(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool do_offload) { - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->ice_vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct ice_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union ice_rx_flex_desc *rxdp = rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_flex_desc *rxdp = rxq->rx_flex_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -465,7 +465,7 @@ _ice_recv_raw_pkts_vec_avx512(struct ice_rx_queue *rxq, * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. */ - if (rxq->vsi->adapter->pf.dev_data->dev_conf.rxmode.offloads & + if (rxq->ice_vsi->adapter->pf.dev_data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH) { /* load bottom half of every 32B desc */ const __m128i raw_desc_bh7 = _mm_load_si128 @@ -721,7 +721,7 @@ static uint16_t ice_recv_scattered_burst_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ice_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[ICE_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -763,7 +763,7 @@ ice_recv_scattered_burst_vec_avx512_offload(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ice_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[ICE_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/ice/ice_rxtx_vec_common.h b/drivers/net/intel/ice/ice_rxtx_vec_common.h index 7933c26366..9430a99ba5 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_common.h +++ b/drivers/net/intel/ice/ice_rxtx_vec_common.h @@ -17,7 +17,7 @@ ice_tx_desc_done(struct ci_tx_queue *txq, uint16_t idx) } static inline void -_ice_rx_queue_release_mbufs_vec(struct ice_rx_queue *rxq) +_ice_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { const unsigned int mask = rxq->nb_rx_desc - 1; unsigned int i; @@ -79,7 +79,7 @@ _ice_rx_queue_release_mbufs_vec(struct ice_rx_queue *rxq) #define ICE_VECTOR_OFFLOAD_PATH 1 static inline int -ice_rx_vec_queue_default(struct ice_rx_queue *rxq) +ice_rx_vec_queue_default(struct ci_rx_queue *rxq) { if (!rxq) return -1; @@ -119,7 +119,7 @@ static inline int ice_rx_vec_dev_check_default(struct rte_eth_dev *dev) { int i; - struct ice_rx_queue *rxq; + struct ci_rx_queue *rxq; int ret = 0; int result = 0; diff --git a/drivers/net/intel/ice/ice_rxtx_vec_sse.c b/drivers/net/intel/ice/ice_rxtx_vec_sse.c index 36da5b5d1b..cae2188279 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_sse.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_sse.c @@ -26,18 +26,18 @@ ice_flex_rxd_to_fdir_flags_vec(const __m128i fdir_id0_3) } static inline void -ice_rxq_rearm(struct ice_rx_queue *rxq) +ice_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); __m128i dma_addr0, dma_addr1; - rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxdp = rxq->rx_flex_ring + rxq->rxrearm_start; /* Pull 'n' more MBUFs into the software ring */ if (rte_mempool_get_bulk(rxq->mp, @@ -105,7 +105,7 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) } static inline void -ice_rx_desc_to_olflags_v(struct ice_rx_queue *rxq, __m128i descs[4], +ice_rx_desc_to_olflags_v(struct ci_rx_queue *rxq, __m128i descs[4], struct rte_mbuf **rx_pkts) { const __m128i mbuf_init = _mm_set_epi64x(0, rxq->mbuf_initializer); @@ -301,15 +301,15 @@ ice_rx_desc_to_ptype_v(__m128i descs[4], struct rte_mbuf **rx_pkts, * - floor align nb_pkts to a ICE_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_ice_recv_raw_pkts_vec(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_entry *sw_ring; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; uint64_t var; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->ice_vsi->adapter->ptype_tbl; __m128i crc_adjust = _mm_set_epi16 (0, 0, 0, /* ignore non-length fields */ -rxq->crc_len, /* sub crc on data_len */ @@ -361,7 +361,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, /* Just the act of getting into the function from the application is * going to cost about 7 cycles */ - rxdp = rxq->rx_ring + rxq->rx_tail; + rxdp = rxq->rx_flex_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -482,7 +482,7 @@ _ice_recv_raw_pkts_vec(struct ice_rx_queue *rxq, struct rte_mbuf **rx_pkts, * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. */ - if (rxq->vsi->adapter->pf.dev_data->dev_conf.rxmode.offloads & + if (rxq->ice_vsi->adapter->pf.dev_data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH) { /* load bottom half of every 32B desc */ const __m128i raw_desc_bh3 = @@ -608,7 +608,7 @@ static uint16_t ice_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct ice_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[ICE_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -779,7 +779,7 @@ ice_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, } int __rte_cold -ice_rxq_vec_setup(struct ice_rx_queue *rxq) +ice_rxq_vec_setup(struct ci_rx_queue *rxq) { if (!rxq) return -1; From patchwork Mon Jun 9 15:37:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154223 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4FA7D468B7; Mon, 9 Jun 2025 17:41:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6149542DA3; Mon, 9 Jun 2025 17:38:32 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 203B542831 for ; Mon, 9 Jun 2025 17:38:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483510; x=1781019510; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=uNv6KhWLHomMu7bH0vikpGv8cN0Gm9M0WissLlGny+0=; b=BEmwEga9qHIqcM1K+h+okzokahNm+jP68Z9/yKAuYCelmafLi8nfY+1M GGEB7KC/HmjKVDOYeqb0Nuuh7krpREnr/a9YZA8xw7/rDf0oXdGDCaEWM NNRPLNh6Y48YuiiWVHluqYzOjjpmd7LfLGAHkIDUMwrmKi9QzWInNXuYg 8YhE+WpXLwxgvua5oJffQ3taIYWmOrYEiq7SDX1KLhhn/2rcqBC5uCIZP Q/rGV6dqhlJMQcAOf0oCrmZylMbD8eeZM3FF7VDhUO/oYK1nb3lKzwhGt hPIaPx1l6QOUWpdRXnlZvB39FzexBskLfMwsUfAFEIzPSmTlJSvaoMJAX Q==; X-CSE-ConnectionGUID: mRBK/gv3So6jOGZDhK9ZAw== X-CSE-MsgGUID: +tsbDu4DQS6V40X7BOz9eQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012232" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012232" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:29 -0700 X-CSE-ConnectionGUID: WyIMFH0ySbuxLAB0Ol0sgg== X-CSE-MsgGUID: iFHVGgD+RlG0sg1xycBlUw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419741" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:27 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Vladimir Medvedkin , Ian Stokes Subject: [PATCH v6 26/33] net/iavf: use the common Rx queue structure Date: Mon, 9 Jun 2025 16:37:24 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Make the iavf driver use the new common Rx queue structure. The iavf driver only supports 32-byte descriptors but they are of a common format, so replace all usages of iavf-specific descriptors with the common ones, and force the common queue structure to only use 32-byte descriptor formats for IAVF driver. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Force to always use 32-byte descriptor format v3 -> v4: - Use the common descriptor format v2: - Fix compile issues for Arm drivers/net/intel/common/rx.h | 10 + drivers/net/intel/iavf/iavf.h | 4 +- drivers/net/intel/iavf/iavf_ethdev.c | 11 +- drivers/net/intel/iavf/iavf_rxtx.c | 228 +++++++++--------- drivers/net/intel/iavf/iavf_rxtx.h | 129 ++-------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 26 +- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 23 +- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 27 +-- drivers/net/intel/iavf/iavf_rxtx_vec_neon.c | 30 +-- drivers/net/intel/iavf/iavf_rxtx_vec_sse.c | 46 ++-- drivers/net/intel/iavf/iavf_vchnl.c | 6 +- 11 files changed, 222 insertions(+), 318 deletions(-) diff --git a/drivers/net/intel/common/rx.h b/drivers/net/intel/common/rx.h index c60d84e447..3e3fea76a7 100644 --- a/drivers/net/intel/common/rx.h +++ b/drivers/net/intel/common/rx.h @@ -79,6 +79,7 @@ struct ci_rx_queue { union { /* the VSI this queue belongs to */ struct i40e_vsi *i40e_vsi; struct ice_vsi *ice_vsi; + struct iavf_vsi *iavf_vsi; }; const struct rte_memzone *mz; union { @@ -108,6 +109,15 @@ struct ci_rx_queue { int ts_offset; /* dynamic mbuf timestamp field offset */ uint64_t ts_flag; /* dynamic mbuf timestamp flag */ }; + struct { /* iavf specific values */ + const struct iavf_rxq_ops *ops; /**< queue ops */ + struct iavf_rx_queue_stats *stats; /**< per-queue stats */ + uint64_t phc_time; /**< HW timestamp */ + uint8_t rel_mbufs_type; /**< type of release mbuf function */ + uint8_t rx_flags; /**< Rx VLAN tag location flags */ +#define IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG1 BIT(0) +#define IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2 BIT(1) + }; }; }; diff --git a/drivers/net/intel/iavf/iavf.h b/drivers/net/intel/iavf/iavf.h index 97e6b243fb..f81c939c96 100644 --- a/drivers/net/intel/iavf/iavf.h +++ b/drivers/net/intel/iavf/iavf.h @@ -97,7 +97,7 @@ #define IAVF_L2TPV2_FLAGS_LEN 0x4000 struct iavf_adapter; -struct iavf_rx_queue; +struct ci_rx_queue; struct ci_tx_queue; @@ -555,7 +555,7 @@ int iavf_ipsec_crypto_request(struct iavf_adapter *adapter, uint8_t *resp_msg, size_t resp_msg_len); extern const struct rte_tm_ops iavf_tm_ops; int iavf_get_ptp_cap(struct iavf_adapter *adapter); -int iavf_get_phc_time(struct iavf_rx_queue *rxq); +int iavf_get_phc_time(struct ci_rx_queue *rxq); int iavf_flow_sub(struct iavf_adapter *adapter, struct iavf_fsub_conf *filter); int iavf_flow_unsub(struct iavf_adapter *adapter, diff --git a/drivers/net/intel/iavf/iavf_ethdev.c b/drivers/net/intel/iavf/iavf_ethdev.c index 5babd587b3..02649c19b2 100644 --- a/drivers/net/intel/iavf/iavf_ethdev.c +++ b/drivers/net/intel/iavf/iavf_ethdev.c @@ -728,7 +728,7 @@ iavf_dev_configure(struct rte_eth_dev *dev) } static int -iavf_init_rxq(struct rte_eth_dev *dev, struct iavf_rx_queue *rxq) +iavf_init_rxq(struct rte_eth_dev *dev, struct ci_rx_queue *rxq) { struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private); struct rte_eth_dev_data *dev_data = dev->data; @@ -779,8 +779,7 @@ iavf_init_rxq(struct rte_eth_dev *dev, struct iavf_rx_queue *rxq) static int iavf_init_queues(struct rte_eth_dev *dev) { - struct iavf_rx_queue **rxq = - (struct iavf_rx_queue **)dev->data->rx_queues; + struct ci_rx_queue **rxq = (struct ci_rx_queue **)dev->data->rx_queues; int i, ret = IAVF_SUCCESS; for (i = 0; i < dev->data->nb_rx_queues; i++) { @@ -955,7 +954,7 @@ static int iavf_config_rx_queues_irqs(struct rte_eth_dev *dev, static int iavf_start_queues(struct rte_eth_dev *dev) { - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; struct ci_tx_queue *txq; int i; uint16_t nb_txq, nb_rxq; @@ -1867,9 +1866,9 @@ iavf_dev_update_ipsec_xstats(struct rte_eth_dev *ethdev, { uint16_t idx; for (idx = 0; idx < ethdev->data->nb_rx_queues; idx++) { - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; struct iavf_ipsec_crypto_stats *stats; - rxq = (struct iavf_rx_queue *)ethdev->data->rx_queues[idx]; + rxq = (struct ci_rx_queue *)ethdev->data->rx_queues[idx]; stats = &rxq->stats->ipsec_crypto; ips->icount += stats->icount; ips->ibytes += stats->ibytes; diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c index 5c798f2b6e..7033a74610 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.c +++ b/drivers/net/intel/iavf/iavf_rxtx.c @@ -128,8 +128,8 @@ iavf_monitor_callback(const uint64_t value, int iavf_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc) { - struct iavf_rx_queue *rxq = rx_queue; - volatile union iavf_rx_desc *rxdp; + struct ci_rx_queue *rxq = rx_queue; + volatile union ci_rx_desc *rxdp; uint16_t desc; desc = rxq->rx_tail; @@ -222,7 +222,7 @@ check_tx_vec_allow(struct ci_tx_queue *txq) } static inline bool -check_rx_bulk_allow(struct iavf_rx_queue *rxq) +check_rx_bulk_allow(struct ci_rx_queue *rxq) { int ret = true; @@ -243,7 +243,7 @@ check_rx_bulk_allow(struct iavf_rx_queue *rxq) } static inline void -reset_rx_queue(struct iavf_rx_queue *rxq) +reset_rx_queue(struct ci_rx_queue *rxq) { uint16_t len; uint32_t i; @@ -253,13 +253,13 @@ reset_rx_queue(struct iavf_rx_queue *rxq) len = rxq->nb_rx_desc + IAVF_RX_MAX_BURST; - for (i = 0; i < len * sizeof(union iavf_rx_desc); i++) + for (i = 0; i < len * sizeof(union ci_rx_desc); i++) ((volatile char *)rxq->rx_ring)[i] = 0; memset(&rxq->fake_mbuf, 0x0, sizeof(rxq->fake_mbuf)); for (i = 0; i < IAVF_RX_MAX_BURST; i++) - rxq->sw_ring[rxq->nb_rx_desc + i] = &rxq->fake_mbuf; + rxq->sw_ring[rxq->nb_rx_desc + i].mbuf = &rxq->fake_mbuf; /* for rx bulk */ rxq->rx_nb_avail = 0; @@ -315,9 +315,9 @@ reset_tx_queue(struct ci_tx_queue *txq) } static int -alloc_rxq_mbufs(struct iavf_rx_queue *rxq) +alloc_rxq_mbufs(struct ci_rx_queue *rxq) { - volatile union iavf_rx_desc *rxd; + volatile union ci_rx_desc *rxd; struct rte_mbuf *mbuf = NULL; uint64_t dma_addr; uint16_t i, j; @@ -326,8 +326,8 @@ alloc_rxq_mbufs(struct iavf_rx_queue *rxq) mbuf = rte_mbuf_raw_alloc(rxq->mp); if (unlikely(!mbuf)) { for (j = 0; j < i; j++) { - rte_pktmbuf_free_seg(rxq->sw_ring[j]); - rxq->sw_ring[j] = NULL; + rte_pktmbuf_free_seg(rxq->sw_ring[j].mbuf); + rxq->sw_ring[j].mbuf = NULL; } PMD_DRV_LOG(ERR, "Failed to allocate mbuf for RX"); return -ENOMEM; @@ -348,14 +348,14 @@ alloc_rxq_mbufs(struct iavf_rx_queue *rxq) rxd->read.rsvd1 = 0; rxd->read.rsvd2 = 0; - rxq->sw_ring[i] = mbuf; + rxq->sw_ring[i].mbuf = mbuf; } return 0; } static inline void -release_rxq_mbufs(struct iavf_rx_queue *rxq) +release_rxq_mbufs(struct ci_rx_queue *rxq) { uint16_t i; @@ -363,9 +363,9 @@ release_rxq_mbufs(struct iavf_rx_queue *rxq) return; for (i = 0; i < rxq->nb_rx_desc; i++) { - if (rxq->sw_ring[i]) { - rte_pktmbuf_free_seg(rxq->sw_ring[i]); - rxq->sw_ring[i] = NULL; + if (rxq->sw_ring[i].mbuf) { + rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); + rxq->sw_ring[i].mbuf = NULL; } } @@ -393,9 +393,9 @@ struct iavf_rxq_ops iavf_rxq_release_mbufs_ops[] = { }; static inline void -iavf_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct iavf_rx_queue *rxq, +iavf_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct iavf_32b_rx_flex_desc_comms_ovs *desc = (volatile struct iavf_32b_rx_flex_desc_comms_ovs *)rxdp; @@ -414,9 +414,9 @@ iavf_rxd_to_pkt_fields_by_comms_ovs(__rte_unused struct iavf_rx_queue *rxq, } static inline void -iavf_rxd_to_pkt_fields_by_comms_aux_v1(struct iavf_rx_queue *rxq, +iavf_rxd_to_pkt_fields_by_comms_aux_v1(struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct iavf_32b_rx_flex_desc_comms *desc = (volatile struct iavf_32b_rx_flex_desc_comms *)rxdp; @@ -454,9 +454,9 @@ iavf_rxd_to_pkt_fields_by_comms_aux_v1(struct iavf_rx_queue *rxq, } static inline void -iavf_rxd_to_pkt_fields_by_comms_aux_v2(struct iavf_rx_queue *rxq, +iavf_rxd_to_pkt_fields_by_comms_aux_v2(struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct iavf_32b_rx_flex_desc_comms *desc = (volatile struct iavf_32b_rx_flex_desc_comms *)rxdp; @@ -507,7 +507,7 @@ iavf_rxd_to_pkt_fields_t rxd_to_pkt_fields_ops[IAVF_RXDID_LAST + 1] = { }; static void -iavf_select_rxd_to_pkt_fields_handler(struct iavf_rx_queue *rxq, uint32_t rxdid) +iavf_select_rxd_to_pkt_fields_handler(struct ci_rx_queue *rxq, uint32_t rxdid) { rxq->rxdid = rxdid; @@ -562,7 +562,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); struct iavf_vsi *vsi = &vf->vsi; - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; const struct rte_memzone *mz; uint32_t ring_size; uint8_t proto_xtr; @@ -600,7 +600,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, /* Allocate the rx queue data structure */ rxq = rte_zmalloc_socket("iavf rxq", - sizeof(struct iavf_rx_queue), + sizeof(struct ci_rx_queue), RTE_CACHE_LINE_SIZE, socket_id); if (!rxq) { @@ -658,7 +658,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, rxq->port_id = dev->data->port_id; rxq->rx_deferred_start = rx_conf->rx_deferred_start; rxq->rx_hdr_len = 0; - rxq->vsi = vsi; + rxq->iavf_vsi = vsi; rxq->offloads = offloads; if (dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_KEEP_CRC) @@ -688,7 +688,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, * a little more to support bulk allocate. */ len = IAVF_MAX_RING_DESC + IAVF_RX_MAX_BURST; - ring_size = RTE_ALIGN(len * sizeof(union iavf_rx_desc), + ring_size = RTE_ALIGN(len * sizeof(union ci_rx_desc), IAVF_DMA_MEM_ALIGN); mz = rte_eth_dma_zone_reserve(dev, "rx_ring", queue_idx, ring_size, IAVF_RING_BASE_ALIGN, @@ -703,7 +703,7 @@ iavf_dev_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, /* Zero all the descriptors in the ring. */ memset(mz->addr, 0, ring_size); rxq->rx_ring_phys_addr = mz->iova; - rxq->rx_ring = (union iavf_rx_desc *)mz->addr; + rxq->rx_ring = (union ci_rx_desc *)mz->addr; rxq->mz = mz; reset_rx_queue(rxq); @@ -895,7 +895,7 @@ iavf_dev_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private); - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; int err = 0; PMD_DRV_FUNC_TRACE(); @@ -987,7 +987,7 @@ iavf_dev_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) struct iavf_adapter *adapter = IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; int err; PMD_DRV_FUNC_TRACE(); @@ -1050,7 +1050,7 @@ iavf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id) void iavf_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid) { - struct iavf_rx_queue *q = dev->data->rx_queues[qid]; + struct ci_rx_queue *q = dev->data->rx_queues[qid]; if (!q) return; @@ -1079,7 +1079,7 @@ iavf_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid) static void iavf_reset_queues(struct rte_eth_dev *dev) { - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; struct ci_tx_queue *txq; int i; @@ -1141,7 +1141,7 @@ iavf_stop_queues(struct rte_eth_dev *dev) (1 << IAVF_RX_FLEX_DESC_STATUS0_RXE_S)) static inline void -iavf_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union iavf_rx_desc *rxdp) +iavf_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union ci_rx_desc *rxdp) { if (rte_le_to_cpu_64(rxdp->wb.qword1.status_error_len) & (1 << IAVF_RX_DESC_STATUS_L2TAG1P_SHIFT)) { @@ -1155,7 +1155,7 @@ iavf_rxd_to_vlan_tci(struct rte_mbuf *mb, volatile union iavf_rx_desc *rxdp) static inline void iavf_flex_rxd_to_vlan_tci(struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { if (rte_le_to_cpu_64(rxdp->wb.status_error0) & (1 << IAVF_RX_FLEX_DESC_STATUS0_L2TAG1P_S)) { @@ -1185,7 +1185,7 @@ iavf_flex_rxd_to_vlan_tci(struct rte_mbuf *mb, static inline void iavf_flex_rxd_to_ipsec_crypto_said_get(struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp) + volatile union ci_rx_flex_desc *rxdp) { volatile struct iavf_32b_rx_flex_desc_comms_ipsec *desc = (volatile struct iavf_32b_rx_flex_desc_comms_ipsec *)rxdp; @@ -1196,7 +1196,7 @@ iavf_flex_rxd_to_ipsec_crypto_said_get(struct rte_mbuf *mb, static inline void iavf_flex_rxd_to_ipsec_crypto_status(struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp, + volatile union ci_rx_flex_desc *rxdp, struct iavf_ipsec_crypto_stats *stats) { uint16_t status1 = rte_le_to_cpu_64(rxdp->wb.status_error1); @@ -1286,7 +1286,7 @@ iavf_rxd_to_pkt_flags(uint64_t qword) } static inline uint64_t -iavf_rxd_build_fdir(volatile union iavf_rx_desc *rxdp, struct rte_mbuf *mb) +iavf_rxd_build_fdir(volatile union ci_rx_desc *rxdp, struct rte_mbuf *mb) { uint64_t flags = 0; uint16_t flexbh; @@ -1357,7 +1357,7 @@ iavf_flex_rxd_error_to_pkt_flags(uint16_t stat_err0) * from the hardware point of view. */ static inline void -iavf_update_rx_tail(struct iavf_rx_queue *rxq, uint16_t nb_hold, uint16_t rx_id) +iavf_update_rx_tail(struct ci_rx_queue *rxq, uint16_t nb_hold, uint16_t rx_id) { nb_hold = (uint16_t)(nb_hold + rxq->nb_rx_hold); @@ -1377,11 +1377,11 @@ iavf_update_rx_tail(struct iavf_rx_queue *rxq, uint16_t nb_hold, uint16_t rx_id) uint16_t iavf_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - volatile union iavf_rx_desc *rx_ring; - volatile union iavf_rx_desc *rxdp; - struct iavf_rx_queue *rxq; - union iavf_rx_desc rxd; - struct rte_mbuf *rxe; + volatile union ci_rx_desc *rx_ring; + volatile union ci_rx_desc *rxdp; + struct ci_rx_queue *rxq; + union ci_rx_desc rxd; + struct ci_rx_entry rxe; struct rte_eth_dev *dev; struct rte_mbuf *rxm; struct rte_mbuf *nmb; @@ -1399,7 +1399,7 @@ iavf_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rxq = rx_queue; rx_id = rxq->rx_tail; rx_ring = rxq->rx_ring; - ptype_tbl = rxq->vsi->adapter->ptype_tbl; + ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; while (nb_rx < nb_pkts) { rxdp = &rx_ring[rx_id]; @@ -1424,13 +1424,13 @@ iavf_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rxd = *rxdp; nb_hold++; rxe = rxq->sw_ring[rx_id]; - rxq->sw_ring[rx_id] = nmb; + rxq->sw_ring[rx_id].mbuf = nmb; rx_id++; if (unlikely(rx_id == rxq->nb_rx_desc)) rx_id = 0; /* Prefetch next mbuf */ - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); /* When next RX descriptor is on a cache line boundary, * prefetch the next 4 RX descriptors and next 8 pointers @@ -1438,9 +1438,9 @@ iavf_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) */ if ((rx_id & 0x3) == 0) { rte_prefetch0(&rx_ring[rx_id]); - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); } - rxm = rxe; + rxm = rxe.mbuf; dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(nmb)); rxdp->read.hdr_addr = 0; @@ -1486,11 +1486,11 @@ uint16_t iavf_recv_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - volatile union iavf_rx_desc *rx_ring; - volatile union iavf_rx_flex_desc *rxdp; - struct iavf_rx_queue *rxq; - union iavf_rx_flex_desc rxd; - struct rte_mbuf *rxe; + volatile union ci_rx_flex_desc *rx_ring; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_queue *rxq; + union ci_rx_flex_desc rxd; + struct ci_rx_entry rxe; struct rte_eth_dev *dev; struct rte_mbuf *rxm; struct rte_mbuf *nmb; @@ -1507,8 +1507,8 @@ iavf_recv_pkts_flex_rxd(void *rx_queue, nb_hold = 0; rxq = rx_queue; rx_id = rxq->rx_tail; - rx_ring = rxq->rx_ring; - ptype_tbl = rxq->vsi->adapter->ptype_tbl; + rx_ring = rxq->rx_flex_ring; + ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); @@ -1521,7 +1521,7 @@ iavf_recv_pkts_flex_rxd(void *rx_queue, } while (nb_rx < nb_pkts) { - rxdp = (volatile union iavf_rx_flex_desc *)&rx_ring[rx_id]; + rxdp = &rx_ring[rx_id]; rx_stat_err0 = rte_le_to_cpu_16(rxdp->wb.status_error0); /* Check the DD bit first */ @@ -1541,13 +1541,13 @@ iavf_recv_pkts_flex_rxd(void *rx_queue, rxd = *rxdp; nb_hold++; rxe = rxq->sw_ring[rx_id]; - rxq->sw_ring[rx_id] = nmb; + rxq->sw_ring[rx_id].mbuf = nmb; rx_id++; if (unlikely(rx_id == rxq->nb_rx_desc)) rx_id = 0; /* Prefetch next mbuf */ - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); /* When next RX descriptor is on a cache line boundary, * prefetch the next 4 RX descriptors and next 8 pointers @@ -1555,9 +1555,9 @@ iavf_recv_pkts_flex_rxd(void *rx_queue, */ if ((rx_id & 0x3) == 0) { rte_prefetch0(&rx_ring[rx_id]); - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); } - rxm = rxe; + rxm = rxe.mbuf; dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(nmb)); rxdp->read.hdr_addr = 0; @@ -1611,9 +1611,9 @@ uint16_t iavf_recv_scattered_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct iavf_rx_queue *rxq = rx_queue; - union iavf_rx_flex_desc rxd; - struct rte_mbuf *rxe; + struct ci_rx_queue *rxq = rx_queue; + union ci_rx_flex_desc rxd; + struct ci_rx_entry rxe; struct rte_mbuf *first_seg = rxq->pkt_first_seg; struct rte_mbuf *last_seg = rxq->pkt_last_seg; struct rte_mbuf *nmb, *rxm; @@ -1625,9 +1625,9 @@ iavf_recv_scattered_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, uint64_t pkt_flags; uint64_t ts_ns; - volatile union iavf_rx_desc *rx_ring = rxq->rx_ring; - volatile union iavf_rx_flex_desc *rxdp; - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + volatile union ci_rx_flex_desc *rx_ring = rxq->rx_flex_ring; + volatile union ci_rx_flex_desc *rxdp; + const uint32_t *ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); @@ -1640,7 +1640,7 @@ iavf_recv_scattered_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, } while (nb_rx < nb_pkts) { - rxdp = (volatile union iavf_rx_flex_desc *)&rx_ring[rx_id]; + rxdp = &rx_ring[rx_id]; rx_stat_err0 = rte_le_to_cpu_16(rxdp->wb.status_error0); /* Check the DD bit */ @@ -1660,13 +1660,13 @@ iavf_recv_scattered_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, rxd = *rxdp; nb_hold++; rxe = rxq->sw_ring[rx_id]; - rxq->sw_ring[rx_id] = nmb; + rxq->sw_ring[rx_id].mbuf = nmb; rx_id++; if (rx_id == rxq->nb_rx_desc) rx_id = 0; /* Prefetch next mbuf */ - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); /* When next RX descriptor is on a cache line boundary, * prefetch the next 4 RX descriptors and next 8 pointers @@ -1674,10 +1674,10 @@ iavf_recv_scattered_pkts_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, */ if ((rx_id & 0x3) == 0) { rte_prefetch0(&rx_ring[rx_id]); - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); } - rxm = rxe; + rxm = rxe.mbuf; dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(nmb)); @@ -1788,9 +1788,9 @@ uint16_t iavf_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct iavf_rx_queue *rxq = rx_queue; - union iavf_rx_desc rxd; - struct rte_mbuf *rxe; + struct ci_rx_queue *rxq = rx_queue; + union ci_rx_desc rxd; + struct ci_rx_entry rxe; struct rte_mbuf *first_seg = rxq->pkt_first_seg; struct rte_mbuf *last_seg = rxq->pkt_last_seg; struct rte_mbuf *nmb, *rxm; @@ -1802,9 +1802,9 @@ iavf_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint64_t dma_addr; uint64_t pkt_flags; - volatile union iavf_rx_desc *rx_ring = rxq->rx_ring; - volatile union iavf_rx_desc *rxdp; - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + volatile union ci_rx_desc *rx_ring = rxq->rx_ring; + volatile union ci_rx_desc *rxdp; + const uint32_t *ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; while (nb_rx < nb_pkts) { rxdp = &rx_ring[rx_id]; @@ -1829,13 +1829,13 @@ iavf_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, rxd = *rxdp; nb_hold++; rxe = rxq->sw_ring[rx_id]; - rxq->sw_ring[rx_id] = nmb; + rxq->sw_ring[rx_id].mbuf = nmb; rx_id++; if (rx_id == rxq->nb_rx_desc) rx_id = 0; /* Prefetch next mbuf */ - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); /* When next RX descriptor is on a cache line boundary, * prefetch the next 4 RX descriptors and next 8 pointers @@ -1843,10 +1843,10 @@ iavf_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, */ if ((rx_id & 0x3) == 0) { rte_prefetch0(&rx_ring[rx_id]); - rte_prefetch0(rxq->sw_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id].mbuf); } - rxm = rxe; + rxm = rxe.mbuf; dma_addr = rte_cpu_to_le_64(rte_mbuf_data_iova_default(nmb)); @@ -1945,12 +1945,12 @@ iavf_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, #define IAVF_LOOK_AHEAD 8 static inline int -iavf_rx_scan_hw_ring_flex_rxd(struct iavf_rx_queue *rxq, +iavf_rx_scan_hw_ring_flex_rxd(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - volatile union iavf_rx_flex_desc *rxdp; - struct rte_mbuf **rxep; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t stat_err0; uint16_t pkt_len; @@ -1958,10 +1958,10 @@ iavf_rx_scan_hw_ring_flex_rxd(struct iavf_rx_queue *rxq, int32_t i, j, nb_rx = 0; int32_t nb_staged = 0; uint64_t pkt_flags; - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; uint64_t ts_ns; - rxdp = (volatile union iavf_rx_flex_desc *)&rxq->rx_ring[rxq->rx_tail]; + rxdp = &rxq->rx_flex_ring[rxq->rx_tail]; rxep = &rxq->sw_ring[rxq->rx_tail]; stat_err0 = rte_le_to_cpu_16(rxdp->wb.status_error0); @@ -2020,7 +2020,7 @@ iavf_rx_scan_hw_ring_flex_rxd(struct iavf_rx_queue *rxq, rxq->rx_tail + i * IAVF_LOOK_AHEAD + j); - mb = rxep[j]; + mb = rxep[j].mbuf; pkt_len = (rte_le_to_cpu_16(rxdp[j].wb.pkt_len) & IAVF_RX_FLX_DESC_PKT_LEN_M) - rxq->crc_len; mb->data_len = pkt_len; @@ -2054,11 +2054,11 @@ iavf_rx_scan_hw_ring_flex_rxd(struct iavf_rx_queue *rxq, /* Put up to nb_pkts directly into buffers */ if ((i + j) < nb_pkts) { - rx_pkts[i + j] = rxep[j]; + rx_pkts[i + j] = rxep[j].mbuf; nb_rx++; } else { /* Stage excess pkts received */ - rxq->rx_stage[nb_staged] = rxep[j]; + rxq->rx_stage[nb_staged] = rxep[j].mbuf; nb_staged++; } } @@ -2072,16 +2072,16 @@ iavf_rx_scan_hw_ring_flex_rxd(struct iavf_rx_queue *rxq, /* Clear software ring entries */ for (i = 0; i < (nb_rx + nb_staged); i++) - rxq->sw_ring[rxq->rx_tail + i] = NULL; + rxq->sw_ring[rxq->rx_tail + i].mbuf = NULL; return nb_rx; } static inline int -iavf_rx_scan_hw_ring(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) +iavf_rx_scan_hw_ring(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **rxep; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t pkt_len; uint64_t qword1; @@ -2090,7 +2090,7 @@ iavf_rx_scan_hw_ring(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint1 int32_t i, j, nb_rx = 0; int32_t nb_staged = 0; uint64_t pkt_flags; - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; rxdp = &rxq->rx_ring[rxq->rx_tail]; rxep = &rxq->sw_ring[rxq->rx_tail]; @@ -2146,7 +2146,7 @@ iavf_rx_scan_hw_ring(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint1 IAVF_DUMP_RX_DESC(rxq, &rxdp[j], rxq->rx_tail + i * IAVF_LOOK_AHEAD + j); - mb = rxep[j]; + mb = rxep[j].mbuf; qword1 = rte_le_to_cpu_64 (rxdp[j].wb.qword1.status_error_len); pkt_len = ((qword1 & IAVF_RXD_QW1_LENGTH_PBUF_MASK) >> @@ -2172,10 +2172,10 @@ iavf_rx_scan_hw_ring(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint1 /* Put up to nb_pkts directly into buffers */ if ((i + j) < nb_pkts) { - rx_pkts[i + j] = rxep[j]; + rx_pkts[i + j] = rxep[j].mbuf; nb_rx++; } else { /* Stage excess pkts received */ - rxq->rx_stage[nb_staged] = rxep[j]; + rxq->rx_stage[nb_staged] = rxep[j].mbuf; nb_staged++; } } @@ -2189,13 +2189,13 @@ iavf_rx_scan_hw_ring(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint1 /* Clear software ring entries */ for (i = 0; i < (nb_rx + nb_staged); i++) - rxq->sw_ring[rxq->rx_tail + i] = NULL; + rxq->sw_ring[rxq->rx_tail + i].mbuf = NULL; return nb_rx; } static inline uint16_t -iavf_rx_fill_from_stage(struct iavf_rx_queue *rxq, +iavf_rx_fill_from_stage(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { @@ -2214,10 +2214,10 @@ iavf_rx_fill_from_stage(struct iavf_rx_queue *rxq, } static inline int -iavf_rx_alloc_bufs(struct iavf_rx_queue *rxq) +iavf_rx_alloc_bufs(struct ci_rx_queue *rxq) { - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **rxep; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep; struct rte_mbuf *mb; uint16_t alloc_idx, i; uint64_t dma_addr; @@ -2238,9 +2238,9 @@ iavf_rx_alloc_bufs(struct iavf_rx_queue *rxq) for (i = 0; i < rxq->rx_free_thresh; i++) { if (likely(i < (rxq->rx_free_thresh - 1))) /* Prefetch next mbuf */ - rte_prefetch0(rxep[i + 1]); + rte_prefetch0(rxep[i + 1].mbuf); - mb = rxep[i]; + mb = rxep[i].mbuf; rte_mbuf_refcnt_set(mb, 1); mb->next = NULL; mb->data_off = RTE_PKTMBUF_HEADROOM; @@ -2266,7 +2266,7 @@ iavf_rx_alloc_bufs(struct iavf_rx_queue *rxq) static inline uint16_t rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct iavf_rx_queue *rxq = (struct iavf_rx_queue *)rx_queue; + struct ci_rx_queue *rxq = (struct ci_rx_queue *)rx_queue; uint16_t nb_rx = 0; if (!nb_pkts) @@ -2294,11 +2294,11 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rxq->rx_tail = (uint16_t)(rxq->rx_tail - (nb_rx + nb_staged)); for (i = 0, j = rxq->rx_tail; i < nb_rx; i++, j++) { - rxq->sw_ring[j] = rx_pkts[i]; + rxq->sw_ring[j].mbuf = rx_pkts[i]; rx_pkts[i] = NULL; } for (i = 0, j = rxq->rx_tail + nb_rx; i < nb_staged; i++, j++) { - rxq->sw_ring[j] = rxq->rx_stage[i]; + rxq->sw_ring[j].mbuf = rxq->rx_stage[i]; rx_pkts[i] = NULL; } @@ -3825,13 +3825,13 @@ static uint16_t iavf_recv_pkts_no_poll(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; enum iavf_rx_burst_type rx_burst_type; - if (!rxq->vsi || rxq->vsi->adapter->no_poll) + if (!rxq->iavf_vsi || rxq->iavf_vsi->adapter->no_poll) return 0; - rx_burst_type = rxq->vsi->adapter->rx_burst_type; + rx_burst_type = rxq->iavf_vsi->adapter->rx_burst_type; return iavf_rx_pkt_burst_ops[rx_burst_type].pkt_burst(rx_queue, rx_pkts, nb_pkts); @@ -3947,7 +3947,7 @@ iavf_set_rx_function(struct rte_eth_dev *dev) enum iavf_rx_burst_type rx_burst_type; int no_poll_on_link_down = adapter->devargs.no_poll_on_link_down; int i; - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; bool use_flex = true; for (i = 0; i < dev->data->nb_rx_queues; i++) { @@ -4361,7 +4361,7 @@ void iavf_dev_rxq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, struct rte_eth_rxq_info *qinfo) { - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; rxq = dev->data->rx_queues[queue_id]; @@ -4395,8 +4395,8 @@ uint32_t iavf_dev_rxq_count(void *rx_queue) { #define IAVF_RXQ_SCAN_INTERVAL 4 - volatile union iavf_rx_desc *rxdp; - struct iavf_rx_queue *rxq; + volatile union ci_rx_desc *rxdp; + struct ci_rx_queue *rxq; uint16_t desc = 0; rxq = rx_queue; @@ -4423,7 +4423,7 @@ iavf_dev_rxq_count(void *rx_queue) int iavf_dev_rx_desc_status(void *rx_queue, uint16_t offset) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; volatile uint64_t *status; uint64_t mask; uint32_t desc; diff --git a/drivers/net/intel/iavf/iavf_rxtx.h b/drivers/net/intel/iavf/iavf_rxtx.h index 258103e222..8abcccf8c2 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.h +++ b/drivers/net/intel/iavf/iavf_rxtx.h @@ -5,6 +5,11 @@ #ifndef _IAVF_RXTX_H_ #define _IAVF_RXTX_H_ +/* IAVF does not support 16-byte descriptors */ +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC +#undef RTE_NET_INTEL_USE_16BYTE_DESC +#endif + #include "../common/rx.h" #include "../common/tx.h" @@ -17,7 +22,7 @@ #define IAVF_RING_BASE_ALIGN 128 /* used for Rx Bulk Allocate */ -#define IAVF_RX_MAX_BURST 32 +#define IAVF_RX_MAX_BURST CI_RX_MAX_BURST /* Max data buffer size must be 16K - 128 bytes */ #define IAVF_RX_MAX_DATA_BUF_SIZE (16 * 1024 - 128) @@ -123,63 +128,12 @@ extern uint64_t iavf_timestamp_dynflag; extern int iavf_timestamp_dynfield_offset; extern int rte_pmd_iavf_tx_lldp_dynfield_offset; -/** - * Rx Flex Descriptors - * These descriptors are used instead of the legacy version descriptors - */ -union iavf_32b_rx_flex_desc { - struct { - __le64 pkt_addr; /* Packet buffer address */ - __le64 hdr_addr; /* Header buffer address */ - /* bit 0 of hdr_addr is DD bit */ - __le64 rsvd1; - __le64 rsvd2; - } read; - struct { - /* Qword 0 */ - u8 rxdid; /* descriptor builder profile ID */ - u8 mir_id_umb_cast; /* mirror=[5:0], umb=[7:6] */ - __le16 ptype_flex_flags0; /* ptype=[9:0], ff0=[15:10] */ - __le16 pkt_len; /* [15:14] are reserved */ - __le16 hdr_len_sph_flex_flags1; /* header=[10:0] */ - /* sph=[11:11] */ - /* ff1/ext=[15:12] */ - - /* Qword 1 */ - __le16 status_error0; - __le16 l2tag1; - __le16 flex_meta0; - __le16 flex_meta1; - - /* Qword 2 */ - __le16 status_error1; - u8 flex_flags2; - u8 time_stamp_low; - __le16 l2tag2_1st; - __le16 l2tag2_2nd; - - /* Qword 3 */ - __le16 flex_meta2; - __le16 flex_meta3; - union { - struct { - __le16 flex_meta4; - __le16 flex_meta5; - } flex; - __le32 ts_high; - } flex_ts; - } wb; /* writeback */ -}; - -#define iavf_rx_desc iavf_32byte_rx_desc -#define iavf_rx_flex_desc iavf_32b_rx_flex_desc - -typedef void (*iavf_rxd_to_pkt_fields_t)(struct iavf_rx_queue *rxq, +typedef void (*iavf_rxd_to_pkt_fields_t)(struct ci_rx_queue *rxq, struct rte_mbuf *mb, - volatile union iavf_rx_flex_desc *rxdp); + volatile union ci_rx_flex_desc *rxdp); struct iavf_rxq_ops { - void (*release_mbufs)(struct iavf_rx_queue *rxq); + void (*release_mbufs)(struct ci_rx_queue *rxq); }; struct iavf_txq_ops { @@ -192,59 +146,6 @@ struct iavf_rx_queue_stats { struct iavf_ipsec_crypto_stats ipsec_crypto; }; -/* Structure associated with each Rx queue. */ -struct iavf_rx_queue { - struct rte_mempool *mp; /* mbuf pool to populate Rx ring */ - const struct rte_memzone *mz; /* memzone for Rx ring */ - volatile union iavf_rx_desc *rx_ring; /* Rx ring virtual address */ - uint64_t rx_ring_phys_addr; /* Rx ring DMA address */ - struct rte_mbuf **sw_ring; /* address of SW ring */ - uint16_t nb_rx_desc; /* ring length */ - uint16_t rx_tail; /* current value of tail */ - volatile uint8_t *qrx_tail; /* register address of tail */ - uint16_t rx_free_thresh; /* max free RX desc to hold */ - uint16_t nb_rx_hold; /* number of held free RX desc */ - struct rte_mbuf *pkt_first_seg; /* first segment of current packet */ - struct rte_mbuf *pkt_last_seg; /* last segment of current packet */ - struct rte_mbuf fake_mbuf; /* dummy mbuf */ - uint8_t rxdid; - uint8_t rel_mbufs_type; - - /* used for VPMD */ - uint16_t rxrearm_nb; /* number of remaining to be re-armed */ - uint16_t rxrearm_start; /* the idx we start the re-arming from */ - uint64_t mbuf_initializer; /* value to init mbufs */ - - /* for rx bulk */ - uint16_t rx_nb_avail; /* number of staged packets ready */ - uint16_t rx_next_avail; /* index of next staged packets */ - uint16_t rx_free_trigger; /* triggers rx buffer allocation */ - struct rte_mbuf *rx_stage[IAVF_RX_MAX_BURST * 2]; /* store mbuf */ - - uint16_t port_id; /* device port ID */ - uint8_t crc_len; /* 0 if CRC stripped, 4 otherwise */ - uint8_t fdir_enabled; /* 0 if FDIR disabled, 1 when enabled */ - uint16_t queue_id; /* Rx queue index */ - uint16_t rx_buf_len; /* The packet buffer size */ - uint16_t rx_hdr_len; /* The header buffer size */ - uint16_t max_pkt_len; /* Maximum packet length */ - struct iavf_vsi *vsi; /**< the VSI this queue belongs to */ - - bool q_set; /* if rx queue has been configured */ - bool rx_deferred_start; /* don't start this queue in dev start */ - const struct iavf_rxq_ops *ops; - uint8_t rx_flags; -#define IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG1 BIT(0) -#define IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2 BIT(1) - uint8_t proto_xtr; /* protocol extraction type */ - uint64_t xtr_ol_flag; - /* flexible descriptor metadata extraction offload flag */ - struct iavf_rx_queue_stats *stats; - uint64_t offloads; - uint64_t phc_time; - uint64_t hw_time_update; -}; - /* Offload features */ union iavf_tx_offload { uint64_t data; @@ -662,7 +563,7 @@ uint16_t iavf_xmit_pkts_vec_avx2_offload(void *tx_queue, struct rte_mbuf **tx_pk int iavf_get_monitor_addr(void *rx_queue, struct rte_power_monitor_cond *pmc); int iavf_rx_vec_dev_check(struct rte_eth_dev *dev); int iavf_tx_vec_dev_check(struct rte_eth_dev *dev); -int iavf_rxq_vec_setup(struct iavf_rx_queue *rxq); +int iavf_rxq_vec_setup(struct ci_rx_queue *rxq); int iavf_txq_vec_setup(struct ci_tx_queue *txq); uint16_t iavf_recv_pkts_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); @@ -702,16 +603,16 @@ uint8_t iavf_proto_xtr_type_to_rxdid(uint8_t xtr_type); void iavf_set_default_ptype_table(struct rte_eth_dev *dev); void iavf_tx_queue_release_mbufs_avx512(struct ci_tx_queue *txq); -void iavf_rx_queue_release_mbufs_sse(struct iavf_rx_queue *rxq); +void iavf_rx_queue_release_mbufs_sse(struct ci_rx_queue *rxq); void iavf_tx_queue_release_mbufs_sse(struct ci_tx_queue *txq); -void iavf_rx_queue_release_mbufs_neon(struct iavf_rx_queue *rxq); +void iavf_rx_queue_release_mbufs_neon(struct ci_rx_queue *rxq); static inline -void iavf_dump_rx_descriptor(struct iavf_rx_queue *rxq, +void iavf_dump_rx_descriptor(struct ci_rx_queue *rxq, const volatile void *desc, uint16_t rx_id) { - const volatile union iavf_32byte_rx_desc *rx_desc = desc; + const volatile union ci_rx_desc *rx_desc = desc; printf("Queue %d Rx_desc %d: QW0: 0x%016"PRIx64" QW1: 0x%016"PRIx64 " QW2: 0x%016"PRIx64" QW3: 0x%016"PRIx64"\n", rxq->queue_id, @@ -757,7 +658,7 @@ void iavf_dump_tx_descriptor(const struct ci_tx_queue *txq, #define FDIR_PROC_ENABLE_PER_QUEUE(ad, on) do { \ int i; \ for (i = 0; i < (ad)->dev_data->nb_rx_queues; i++) { \ - struct iavf_rx_queue *rxq = (ad)->dev_data->rx_queues[i]; \ + struct ci_rx_queue *rxq = (ad)->dev_data->rx_queues[i]; \ if (!rxq) \ continue; \ rxq->fdir_enabled = on; \ diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c index b4fe77a98b..b0f36cb515 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c @@ -7,7 +7,7 @@ #include static __rte_always_inline void -iavf_rxq_rearm(struct iavf_rx_queue *rxq) +iavf_rxq_rearm(struct ci_rx_queue *rxq) { iavf_rxq_rearm_common(rxq, false); } @@ -15,19 +15,16 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) #define PKTLEN_SHIFT 10 static __rte_always_inline uint16_t -_iavf_recv_raw_pkts_vec_avx2(struct iavf_rx_queue *rxq, +_iavf_recv_raw_pkts_vec_avx2(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { - /* const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; */ - const uint32_t *type_table = rxq->vsi->adapter->ptype_tbl; - + const uint32_t *type_table = rxq->iavf_vsi->adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - /* struct iavf_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; */ - struct rte_mbuf **sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union iavf_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_desc *rxdp = &rxq->rx_ring[rxq->rx_tail]; const int avx_aligned = ((rxq->rx_tail & 1) == 0); rte_prefetch0(rxdp); @@ -485,20 +482,19 @@ flex_rxd_to_fdir_flags_vec_avx2(const __m256i fdir_id0_7) } static __rte_always_inline uint16_t -_iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, +_iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { - struct iavf_adapter *adapter = rxq->vsi->adapter; + struct iavf_adapter *adapter = rxq->iavf_vsi->adapter; uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; const uint32_t *type_table = adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct rte_mbuf **sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union iavf_rx_flex_desc *rxdp = - (volatile union iavf_rx_flex_desc *)rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_flex_desc *rxdp = rxq->rx_flex_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -1461,7 +1457,7 @@ static __rte_always_inline uint16_t iavf_recv_scattered_burst_vec_avx2(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool offload) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -1550,7 +1546,7 @@ iavf_recv_scattered_burst_vec_avx2_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool offload) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c index 6eac24baf5..bbba564329 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c @@ -27,26 +27,26 @@ #define IAVF_RX_TS_OFFLOAD static __rte_always_inline void -iavf_rxq_rearm(struct iavf_rx_queue *rxq) +iavf_rxq_rearm(struct ci_rx_queue *rxq) { iavf_rxq_rearm_common(rxq, true); } #define IAVF_RX_LEN_MASK 0x80808080 static __rte_always_inline uint16_t -_iavf_recv_raw_pkts_vec_avx512(struct iavf_rx_queue *rxq, +_iavf_recv_raw_pkts_vec_avx512(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { #ifdef IAVF_RX_PTYPE_OFFLOAD - const uint32_t *type_table = rxq->vsi->adapter->ptype_tbl; + const uint32_t *type_table = rxq->iavf_vsi->adapter->ptype_tbl; #endif const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct rte_mbuf **sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union iavf_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_desc *rxdp = rxq->rx_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -577,13 +577,13 @@ flex_rxd_to_fdir_flags_vec_avx512(const __m256i fdir_id0_7) } static __rte_always_inline uint16_t -_iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, +_iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet, bool offload) { - struct iavf_adapter *adapter = rxq->vsi->adapter; + struct iavf_adapter *adapter = rxq->iavf_vsi->adapter; uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; #ifdef IAVF_RX_PTYPE_OFFLOAD const uint32_t *type_table = adapter->ptype_tbl; @@ -591,9 +591,8 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, const __m256i mbuf_init = _mm256_set_epi64x(0, 0, 0, rxq->mbuf_initializer); - struct rte_mbuf **sw_ring = &rxq->sw_ring[rxq->rx_tail]; - volatile union iavf_rx_flex_desc *rxdp = - (volatile union iavf_rx_flex_desc *)rxq->rx_ring + rxq->rx_tail; + struct ci_rx_entry *sw_ring = &rxq->sw_ring[rxq->rx_tail]; + volatile union ci_rx_flex_desc *rxdp = rxq->rx_flex_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -1642,7 +1641,7 @@ static __rte_always_inline uint16_t iavf_recv_scattered_burst_vec_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, bool offload) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ @@ -1718,7 +1717,7 @@ iavf_recv_scattered_burst_vec_avx512_flex_rxd(void *rx_queue, uint16_t nb_pkts, bool offload) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; /* get some new buffers */ diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h index 8c31334570..90a9ac95eb 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h @@ -8,7 +8,6 @@ #include #include -#include "../common/rx.h" #include "iavf.h" #include "iavf_rxtx.h" @@ -21,7 +20,7 @@ iavf_tx_desc_done(struct ci_tx_queue *txq, uint16_t idx) } static inline void -_iavf_rx_queue_release_mbufs_vec(struct iavf_rx_queue *rxq) +_iavf_rx_queue_release_mbufs_vec(struct ci_rx_queue *rxq) { const unsigned int mask = rxq->nb_rx_desc - 1; unsigned int i; @@ -32,15 +31,15 @@ _iavf_rx_queue_release_mbufs_vec(struct iavf_rx_queue *rxq) /* free all mbufs that are valid in the ring */ if (rxq->rxrearm_nb == 0) { for (i = 0; i < rxq->nb_rx_desc; i++) { - if (rxq->sw_ring[i]) - rte_pktmbuf_free_seg(rxq->sw_ring[i]); + if (rxq->sw_ring[i].mbuf) + rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); } } else { for (i = rxq->rx_tail; i != rxq->rxrearm_start; i = (i + 1) & mask) { - if (rxq->sw_ring[i]) - rte_pktmbuf_free_seg(rxq->sw_ring[i]); + if (rxq->sw_ring[i].mbuf) + rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); } } @@ -51,7 +50,7 @@ _iavf_rx_queue_release_mbufs_vec(struct iavf_rx_queue *rxq) } static inline int -iavf_rx_vec_queue_default(struct iavf_rx_queue *rxq) +iavf_rx_vec_queue_default(struct ci_rx_queue *rxq) { if (!rxq) return -1; @@ -117,7 +116,7 @@ static inline int iavf_rx_vec_dev_check_default(struct rte_eth_dev *dev) { int i; - struct iavf_rx_queue *rxq; + struct ci_rx_queue *rxq; int ret; int result = 0; @@ -240,12 +239,12 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, #ifdef RTE_ARCH_X86 static __rte_always_inline void -iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) +iavf_rxq_rearm_common(struct ci_rx_queue *rxq, __rte_unused bool avx512) { int i; uint16_t rx_id; - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **rxp = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; rxdp = rxq->rx_ring + rxq->rxrearm_start; @@ -259,7 +258,7 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) dma_addr0 = _mm_setzero_si128(); for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { - rxp[i] = &rxq->fake_mbuf; + rxp[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } @@ -277,8 +276,8 @@ iavf_rxq_rearm_common(struct iavf_rx_queue *rxq, __rte_unused bool avx512) for (i = 0; i < IAVF_VPMD_RXQ_REARM_THRESH; i += 2, rxp += 2) { __m128i vaddr0, vaddr1; - mb0 = rxp[0]; - mb1 = rxp[1]; + mb0 = rxp[0].mbuf; + mb1 = rxp[1].mbuf; /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c b/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c index 86f3a7839d..562e574aab 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c @@ -15,12 +15,12 @@ #include "iavf_rxtx_vec_common.h" static inline void -iavf_rxq_rearm(struct iavf_rx_queue *rxq) +iavf_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **rxep = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; uint64x2_t dma_addr0, dma_addr1; uint64x2_t zero = vdupq_n_u64(0); @@ -35,7 +35,7 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) if (rxq->rxrearm_nb + IAVF_VPMD_RXQ_REARM_THRESH >= rxq->nb_rx_desc) { for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { - rxep[i] = &rxq->fake_mbuf; + rxep[i].mbuf = &rxq->fake_mbuf; vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i].read), zero); } } @@ -46,8 +46,8 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) /* Initialize the mbufs in vector, process 2 mbufs in one loop */ for (i = 0; i < IAVF_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - mb0 = rxep[0]; - mb1 = rxep[1]; + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; dma_addr0 = vdupq_n_u64(paddr); @@ -75,7 +75,7 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) } static inline void -desc_to_olflags_v(struct iavf_rx_queue *rxq, volatile union iavf_rx_desc *rxdp, +desc_to_olflags_v(struct ci_rx_queue *rxq, volatile union ci_rx_desc *rxdp, uint64x2_t descs[4], struct rte_mbuf **rx_pkts) { RTE_SET_USED(rxdp); @@ -193,17 +193,17 @@ desc_to_ptype_v(uint64x2_t descs[4], struct rte_mbuf **__rte_restrict rx_pkts, * - floor align nb_pkts to a IAVF_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct iavf_rx_queue *__rte_restrict rxq, +_recv_raw_pkts_vec(struct ci_rx_queue *__rte_restrict rxq, struct rte_mbuf **__rte_restrict rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { RTE_SET_USED(split_packet); - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **sw_ring; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; - uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + uint32_t *ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; /* mask to shuffle from desc. to mbuf */ uint8x16_t shuf_msk = { @@ -283,8 +283,8 @@ _recv_raw_pkts_vec(struct iavf_rx_queue *__rte_restrict rxq, descs[0] = vld1q_lane_u64(RTE_CAST_PTR(uint64_t *, rxdp), descs[0], 0); /* B.1 load 4 mbuf point */ - mbp1 = vld1q_u64((uint64_t *)&sw_ring[pos]); - mbp2 = vld1q_u64((uint64_t *)&sw_ring[pos + 2]); + mbp1 = vld1q_u64((uint64_t *)&sw_ring[pos].mbuf); + mbp2 = vld1q_u64((uint64_t *)&sw_ring[pos + 2].mbuf); /* B.2 copy 4 mbuf point into rx_pkts */ vst1q_u64((uint64_t *)&rx_pkts[pos], mbp1); @@ -394,13 +394,13 @@ iavf_recv_pkts_vec(void *__rte_restrict rx_queue, } void __rte_cold -iavf_rx_queue_release_mbufs_neon(struct iavf_rx_queue *rxq) +iavf_rx_queue_release_mbufs_neon(struct ci_rx_queue *rxq) { _iavf_rx_queue_release_mbufs_vec(rxq); } int __rte_cold -iavf_rxq_vec_setup(struct iavf_rx_queue *rxq) +iavf_rxq_vec_setup(struct ci_rx_queue *rxq) { rxq->rel_mbufs_type = IAVF_REL_MBUFS_NEON_VEC; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c index 0633a0c33d..a30ba87a3e 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c @@ -13,13 +13,13 @@ #include static inline void -iavf_rxq_rearm(struct iavf_rx_queue *rxq) +iavf_rxq_rearm(struct ci_rx_queue *rxq) { int i; uint16_t rx_id; - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **rxp = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; struct rte_mbuf *mb0, *mb1; __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM); @@ -33,7 +33,7 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) if (rxq->rxrearm_nb + rxq->rx_free_thresh >= rxq->nb_rx_desc) { dma_addr0 = _mm_setzero_si128(); for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { - rxp[i] = &rxq->fake_mbuf; + rxp[i].mbuf = &rxq->fake_mbuf; _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), dma_addr0); } @@ -47,8 +47,8 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) for (i = 0; i < rxq->rx_free_thresh; i += 2, rxp += 2) { __m128i vaddr0, vaddr1; - mb0 = rxp[0]; - mb1 = rxp[1]; + mb0 = rxp[0].mbuf; + mb1 = rxp[1].mbuf; /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != @@ -88,7 +88,7 @@ iavf_rxq_rearm(struct iavf_rx_queue *rxq) } static inline void -desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], +desc_to_olflags_v(struct ci_rx_queue *rxq, __m128i descs[4], struct rte_mbuf **rx_pkts) { const __m128i mbuf_init = _mm_set_epi64x(0, rxq->mbuf_initializer); @@ -205,7 +205,7 @@ flex_rxd_to_fdir_flags_vec(const __m128i fdir_id0_3) } static inline void -flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], __m128i descs_bh[4], +flex_desc_to_olflags_v(struct ci_rx_queue *rxq, __m128i descs[4], __m128i descs_bh[4], struct rte_mbuf **rx_pkts) { const __m128i mbuf_init = _mm_set_epi64x(0, rxq->mbuf_initializer); @@ -456,16 +456,16 @@ flex_desc_to_ptype_v(__m128i descs[4], struct rte_mbuf **rx_pkts, * - floor align nb_pkts to a IAVF_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, +_recv_raw_pkts_vec(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - volatile union iavf_rx_desc *rxdp; - struct rte_mbuf **sw_ring; + volatile union ci_rx_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; uint64_t var; __m128i shuf_msk; - const uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + const uint32_t *ptype_tbl = rxq->iavf_vsi->adapter->ptype_tbl; __m128i crc_adjust = _mm_set_epi16( 0, 0, 0, /* ignore non-length fields */ @@ -561,7 +561,7 @@ _recv_raw_pkts_vec(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, #endif /* B.1 load 2 (64 bit) or 4 (32 bit) mbuf points */ - mbp1 = _mm_loadu_si128((__m128i *)&sw_ring[pos]); + mbp1 = _mm_loadu_si128((__m128i *)&sw_ring[pos].mbuf); /* Read desc statuses backwards to avoid race condition */ /* A.1 load desc[3] */ descs[3] = _mm_loadu_si128(RTE_CAST_PTR(const __m128i *, rxdp + 3)); @@ -704,16 +704,16 @@ _recv_raw_pkts_vec(struct iavf_rx_queue *rxq, struct rte_mbuf **rx_pkts, * - floor align nb_pkts to a IAVF_VPMD_DESCS_PER_LOOP power-of-two */ static inline uint16_t -_recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, +_recv_raw_pkts_vec_flex_rxd(struct ci_rx_queue *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, uint8_t *split_packet) { - volatile union iavf_rx_flex_desc *rxdp; - struct rte_mbuf **sw_ring; + volatile union ci_rx_flex_desc *rxdp; + struct ci_rx_entry *sw_ring; uint16_t nb_pkts_recd; int pos; uint64_t var; - struct iavf_adapter *adapter = rxq->vsi->adapter; + struct iavf_adapter *adapter = rxq->iavf_vsi->adapter; uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; const uint32_t *ptype_tbl = adapter->ptype_tbl; __m128i crc_adjust = _mm_set_epi16 @@ -767,7 +767,7 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, /* Just the act of getting into the function from the application is * going to cost about 7 cycles */ - rxdp = (volatile union iavf_rx_flex_desc *)rxq->rx_ring + rxq->rx_tail; + rxdp = rxq->rx_flex_ring + rxq->rx_tail; rte_prefetch0(rxdp); @@ -840,7 +840,7 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, #endif /* B.1 load 2 (64 bit) or 4 (32 bit) mbuf points */ - mbp1 = _mm_loadu_si128((__m128i *)&sw_ring[pos]); + mbp1 = _mm_loadu_si128((__m128i *)&sw_ring[pos].mbuf); /* Read desc statuses backwards to avoid race condition */ /* A.1 load desc[3] */ descs[3] = _mm_loadu_si128(RTE_CAST_PTR(const __m128i *, rxdp + 3)); @@ -1182,7 +1182,7 @@ static uint16_t iavf_recv_scattered_burst_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; unsigned int i = 0; @@ -1251,7 +1251,7 @@ iavf_recv_scattered_burst_vec_flex_rxd(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { - struct iavf_rx_queue *rxq = rx_queue; + struct ci_rx_queue *rxq = rx_queue; uint8_t split_flags[IAVF_VPMD_RX_BURST] = {0}; unsigned int i = 0; @@ -1424,7 +1424,7 @@ iavf_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, } void __rte_cold -iavf_rx_queue_release_mbufs_sse(struct iavf_rx_queue *rxq) +iavf_rx_queue_release_mbufs_sse(struct ci_rx_queue *rxq) { _iavf_rx_queue_release_mbufs_vec(rxq); } @@ -1437,7 +1437,7 @@ iavf_txq_vec_setup(struct ci_tx_queue *txq) } int __rte_cold -iavf_rxq_vec_setup(struct iavf_rx_queue *rxq) +iavf_rxq_vec_setup(struct ci_rx_queue *rxq) { rxq->rel_mbufs_type = IAVF_REL_MBUFS_SSE_VEC; rxq->mbuf_initializer = ci_rxq_mbuf_initializer(rxq->port_id); diff --git a/drivers/net/intel/iavf/iavf_vchnl.c b/drivers/net/intel/iavf/iavf_vchnl.c index 2302d2bcf1..b1b7a5bf94 100644 --- a/drivers/net/intel/iavf/iavf_vchnl.c +++ b/drivers/net/intel/iavf/iavf_vchnl.c @@ -1218,7 +1218,7 @@ int iavf_configure_queues(struct iavf_adapter *adapter, uint16_t num_queue_pairs, uint16_t index) { - struct iavf_rx_queue **rxq = (struct iavf_rx_queue **)adapter->dev_data->rx_queues; + struct ci_rx_queue **rxq = (struct ci_rx_queue **)adapter->dev_data->rx_queues; struct ci_tx_queue **txq = (struct ci_tx_queue **)adapter->dev_data->tx_queues; struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(adapter); struct virtchnl_vsi_queue_config_info *vc_config; @@ -2244,9 +2244,9 @@ iavf_get_ptp_cap(struct iavf_adapter *adapter) } int -iavf_get_phc_time(struct iavf_rx_queue *rxq) +iavf_get_phc_time(struct ci_rx_queue *rxq) { - struct iavf_adapter *adapter = rxq->vsi->adapter; + struct iavf_adapter *adapter = rxq->iavf_vsi->adapter; struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(adapter); struct virtchnl_phc_time phc_time; struct iavf_cmd_info args; From patchwork Mon Jun 9 15:37:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154224 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E5E09468B7; Mon, 9 Jun 2025 17:41:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 733BF42D66; Mon, 9 Jun 2025 17:38:33 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id B4170427DE for ; Mon, 9 Jun 2025 17:38:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483511; x=1781019511; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=lKQRtHxitU4S3gpwapDJvXLKpDpxSKNH7s6XXQm0gwA=; b=ngka0ycLIbRwZ3WOXgtcifR4BNYgkIEhzXkz0oWfslCHAoR7ZwxFaQ9h EEt3IDwOkYOGD2hF/TUFGLcXgd0YqIYlPAHJKjR+b/WFuu/ITLj3pFU3C YQpAk7rtgqNAfXRMMNMTa6Q4fKT0BLA9WDaQpJlNJodYYklaKCwb4xydf LcrNikS9Asn7gRD/tzNI6aqa/+fHrrMCgKlEc7a8nAPYDtek9Z2MnQWyi cP3wTLsWYhRONbFyZxj9Mw77PJBo4OY437G5ppmuNO5QUurdYKpCbD0Ab T1WJsopb/JVmh663jrT8fodAw8QqbvYfJ2h9H6CLr50A8PAmlo2r2rcKW A==; X-CSE-ConnectionGUID: yLaKqdIrSdaPtoCxGG4qJA== X-CSE-MsgGUID: YXHJmk80RlajzefGrKiNtg== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012235" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012235" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:30 -0700 X-CSE-ConnectionGUID: gle6w3vhTryoqT/pS+LLsQ== X-CSE-MsgGUID: 9QGRTI4zSUKq1gMMQU7VNw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419752" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:29 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson Subject: [PATCH v6 27/33] net/intel: generalize vectorized Rx rearm Date: Mon, 9 Jun 2025 16:37:25 +0100 Message-ID: <149c5f76a865c556af2c4c9640fdb83d497065be.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is certain amount of duplication between various drivers when it comes to Rx ring rearm. This patch takes implementation from ice driver as a base because it has support for no IOVA in mbuf as well as all vector implementations, and moves them to a common file. While we're at it, also make sure to use common definitions for things like burst size, rearm threshold, and descriptors per loop, which is currently defined separately in each driver. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/common/rx.h | 4 + drivers/net/intel/common/rx_vec_x86.h | 315 ++++++++++++++++++++ drivers/net/intel/ice/ice_rxtx.h | 12 +- drivers/net/intel/ice/ice_rxtx_common_avx.h | 233 --------------- drivers/net/intel/ice/ice_rxtx_vec_avx2.c | 5 +- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 5 +- drivers/net/intel/ice/ice_rxtx_vec_sse.c | 77 +---- 7 files changed, 334 insertions(+), 317 deletions(-) create mode 100644 drivers/net/intel/common/rx_vec_x86.h delete mode 100644 drivers/net/intel/ice/ice_rxtx_common_avx.h diff --git a/drivers/net/intel/common/rx.h b/drivers/net/intel/common/rx.h index 3e3fea76a7..b9ba2dcc98 100644 --- a/drivers/net/intel/common/rx.h +++ b/drivers/net/intel/common/rx.h @@ -15,6 +15,10 @@ #define CI_RX_MAX_BURST 32 #define CI_RX_MAX_NSEG 2 +#define CI_VPMD_RX_BURST 32 +#define CI_VPMD_DESCS_PER_LOOP 4 +#define CI_VPMD_DESCS_PER_LOOP_WIDE 8 +#define CI_VPMD_RX_REARM_THRESH CI_VPMD_RX_BURST struct ci_rx_queue; diff --git a/drivers/net/intel/common/rx_vec_x86.h b/drivers/net/intel/common/rx_vec_x86.h new file mode 100644 index 0000000000..4ad8066630 --- /dev/null +++ b/drivers/net/intel/common/rx_vec_x86.h @@ -0,0 +1,315 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Intel Corporation + */ + +#ifndef _COMMON_INTEL_RX_VEC_X86_H_ +#define _COMMON_INTEL_RX_VEC_X86_H_ + +#include + +#include +#include + +#include "rx.h" + +enum ci_rx_vec_level { + CI_RX_VEC_LEVEL_SSE = 0, + CI_RX_VEC_LEVEL_AVX2, + CI_RX_VEC_LEVEL_AVX512, +}; + +static inline int +_ci_rxq_rearm_get_bufs(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + volatile union ci_rx_desc *rxdp; + int i; + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + if (rte_mempool_get_bulk(rxq->mp, (void **)rxp, rearm_thresh) < 0) { + if (rxq->rxrearm_nb + rearm_thresh >= rxq->nb_rx_desc) { + const __m128i zero = _mm_setzero_si128(); + + for (i = 0; i < CI_VPMD_DESCS_PER_LOOP; i++) { + rxp[i].mbuf = &rxq->fake_mbuf; + _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i]), zero); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += rearm_thresh; + return -1; + } + return 0; +} + +/* + * SSE code path can handle both 16-byte and 32-byte descriptors with one code + * path, as we only ever write 16 bytes at a time. + */ +static __rte_always_inline void +_ci_rxq_rearm_sse(struct ci_rx_queue *rxq) +{ + const __m128i hdroom = _mm_set1_epi64x(RTE_PKTMBUF_HEADROOM); + const __m128i zero = _mm_setzero_si128(); + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + int i; + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < rearm_thresh; i += 2, rxp += 2, rxdp += 2) { + struct rte_mbuf *mb0 = rxp[0].mbuf; + struct rte_mbuf *mb1 = rxp[1].mbuf; + +#if RTE_IOVA_IN_MBUF + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); +#endif + __m128i addr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + __m128i addr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + + /* add headroom to address values */ + addr0 = _mm_add_epi64(addr0, hdroom); + addr1 = _mm_add_epi64(addr1, hdroom); + +#if RTE_IOVA_IN_MBUF + /* move IOVA to Packet Buffer Address, erase Header Buffer Address */ + addr0 = _mm_unpackhi_epi64(addr0, zero); + addr0 = _mm_unpackhi_epi64(addr1, zero); +#else + /* erase Header Buffer Address */ + addr0 = _mm_unpacklo_epi64(addr0, zero); + addr1 = _mm_unpacklo_epi64(addr1, zero); +#endif + + /* flush desc with pa dma_addr */ + _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[0]), addr0); + _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[1]), addr1); + } +} + +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC +#ifdef __AVX2__ +/* AVX2 version for 16-byte descriptors, handles 4 buffers at a time */ +static __rte_always_inline void +_ci_rxq_rearm_avx2(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + const __m256i hdroom = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); + const __m256i zero = _mm256_setzero_si256(); + volatile union ci_rx_desc *rxdp; + int i; + + RTE_BUILD_BUG_ON(sizeof(union ci_rx_desc) != 16); + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + /* Initialize the mbufs in vector, process 4 mbufs in one loop */ + for (i = 0; i < rearm_thresh; i += 4, rxp += 4, rxdp += 4) { + struct rte_mbuf *mb0 = rxp[0].mbuf; + struct rte_mbuf *mb1 = rxp[1].mbuf; + struct rte_mbuf *mb2 = rxp[2].mbuf; + struct rte_mbuf *mb3 = rxp[3].mbuf; + +#if RTE_IOVA_IN_MBUF + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); +#endif + const __m128i vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + const __m128i vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + const __m128i vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); + const __m128i vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 + * into the high lanes. Similarly for 2 & 3 + */ + const __m256i vaddr0_256 = _mm256_castsi128_si256(vaddr0); + const __m256i vaddr2_256 = _mm256_castsi128_si256(vaddr2); + + __m256i addr0_1 = _mm256_inserti128_si256(vaddr0_256, vaddr1, 1); + __m256i addr2_3 = _mm256_inserti128_si256(vaddr2_256, vaddr3, 1); + + /* add headroom to address values */ + addr0_1 = _mm256_add_epi64(addr0_1, hdroom); + addr0_1 = _mm256_add_epi64(addr0_1, hdroom); + +#if RTE_IOVA_IN_MBUF + /* extract IOVA addr into Packet Buffer Address, erase Header Buffer Address */ + addr0_1 = _mm256_unpackhi_epi64(addr0_1, zero); + addr2_3 = _mm256_unpackhi_epi64(addr2_3, zero); +#else + /* erase Header Buffer Address */ + addr0_1 = _mm256_unpacklo_epi64(addr0_1, zero); + addr2_3 = _mm256_unpacklo_epi64(addr2_3, zero); +#endif + + /* flush desc with pa dma_addr */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp[0]), addr0_1); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp[2]), addr2_3); + } +} +#endif /* __AVX2__ */ + +#ifdef __AVX512VL__ +/* AVX512 version for 16-byte descriptors, handles 8 buffers at a time */ +static __rte_always_inline void +_ci_rxq_rearm_avx512(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + const __m512i hdroom = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); + const __m512i zero = _mm512_setzero_si512(); + volatile union ci_rx_desc *rxdp; + int i; + + RTE_BUILD_BUG_ON(sizeof(union ci_rx_desc) != 16); + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + /* Initialize the mbufs in vector, process 8 mbufs in one loop */ + for (i = 0; i < rearm_thresh; i += 8, rxp += 8, rxdp += 8) { + struct rte_mbuf *mb0 = rxp[0].mbuf; + struct rte_mbuf *mb1 = rxp[1].mbuf; + struct rte_mbuf *mb2 = rxp[2].mbuf; + struct rte_mbuf *mb3 = rxp[3].mbuf; + struct rte_mbuf *mb4 = rxp[4].mbuf; + struct rte_mbuf *mb5 = rxp[5].mbuf; + struct rte_mbuf *mb6 = rxp[6].mbuf; + struct rte_mbuf *mb7 = rxp[7].mbuf; + +#if RTE_IOVA_IN_MBUF + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); +#endif + const __m128i vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + const __m128i vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + const __m128i vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); + const __m128i vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + const __m128i vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); + const __m128i vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); + const __m128i vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); + const __m128i vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 + * into the high lanes. Similarly for 2 & 3, and so on. + */ + const __m256i addr0_256 = _mm256_castsi128_si256(vaddr0); + const __m256i addr2_256 = _mm256_castsi128_si256(vaddr2); + const __m256i addr4_256 = _mm256_castsi128_si256(vaddr4); + const __m256i addr6_256 = _mm256_castsi128_si256(vaddr6); + + const __m256i addr0_1 = _mm256_inserti128_si256(addr0_256, vaddr1, 1); + const __m256i addr2_3 = _mm256_inserti128_si256(addr2_256, vaddr3, 1); + const __m256i addr4_5 = _mm256_inserti128_si256(addr4_256, vaddr5, 1); + const __m256i addr6_7 = _mm256_inserti128_si256(addr6_256, vaddr7, 1); + + /** + * merge 0_1 & 2_3, by casting 0_1 to 512-bit and inserting 2_3 + * into the high lanes. Similarly for 4_5 & 6_7, and so on. + */ + const __m512i addr0_1_512 = _mm512_castsi256_si512(addr0_1); + const __m512i addr4_5_512 = _mm512_castsi256_si512(addr4_5); + + __m512i addr0_3 = _mm512_inserti64x4(addr0_1_512, addr2_3, 1); + __m512i addr4_7 = _mm512_inserti64x4(addr4_5_512, addr6_7, 1); + + /* add headroom to address values */ + addr0_3 = _mm512_add_epi64(addr0_3, hdroom); + addr4_7 = _mm512_add_epi64(addr4_7, hdroom); + +#if RTE_IOVA_IN_MBUF + /* extract IOVA addr into Packet Buffer Address, erase Header Buffer Address */ + addr0_3 = _mm512_unpackhi_epi64(addr0_3, zero); + addr4_7 = _mm512_unpackhi_epi64(addr4_7, zero); +#else + /* erase Header Buffer Address */ + addr0_3 = _mm512_unpacklo_epi64(addr0_3, zero); + addr4_7 = _mm512_unpacklo_epi64(addr4_7, zero); +#endif + + /* flush desc with pa dma_addr */ + _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp[0]), addr0_3); + _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp[4]), addr4_7); + } +} +#endif /* __AVX512VL__ */ +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ + +/** + * Rearm the RX queue with new buffers. + * + * This function is inlined, so the last parameter will be constant-propagated + * if specified at compile time, and thus all unnecessary branching will be + * eliminated. + * + * @param rxq + * Pointer to the RX queue structure. + * @param vec_level + * The vectorization level to use for rearming. + */ +static __rte_always_inline void +ci_rxq_rearm(struct ci_rx_queue *rxq, const enum ci_rx_vec_level vec_level) +{ + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + uint16_t rx_id; + + /* Pull 'n' more MBUFs into the software ring */ + if (_ci_rxq_rearm_get_bufs(rxq) < 0) + return; + +#ifdef RTE_NET_INTEL_USE_16BYTE_DESC + switch (vec_level) { + case CI_RX_VEC_LEVEL_AVX512: +#ifdef __AVX512VL__ + _ci_rxq_rearm_avx512(rxq); + break; +#else + /* fall back to AVX2 */ + /* fall through */ +#endif + case CI_RX_VEC_LEVEL_AVX2: +#ifdef __AVX2__ + _ci_rxq_rearm_avx2(rxq); + break; +#else + /* fall back to SSE */ + /* fall through */ +#endif + case CI_RX_VEC_LEVEL_SSE: + _ci_rxq_rearm_sse(rxq, desc_len); + break; + } +#else + /* for 32-byte descriptors only support SSE */ + switch (vec_level) { + case CI_RX_VEC_LEVEL_AVX512: + case CI_RX_VEC_LEVEL_AVX2: + case CI_RX_VEC_LEVEL_SSE: + _ci_rxq_rearm_sse(rxq); + break; + } +#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ + + rxq->rxrearm_start += rearm_thresh; + if (rxq->rxrearm_start >= rxq->nb_rx_desc) + rxq->rxrearm_start = 0; + + rxq->rxrearm_nb -= rearm_thresh; + + rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? + (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); + + /* Update the tail pointer on the NIC */ + rte_write32_wc(rte_cpu_to_le_32(rx_id), rxq->qrx_tail); +} + +#endif /* _COMMON_INTEL_RX_VEC_X86_H_ */ diff --git a/drivers/net/intel/ice/ice_rxtx.h b/drivers/net/intel/ice/ice_rxtx.h index 62f98579f5..aa81859ec0 100644 --- a/drivers/net/intel/ice/ice_rxtx.h +++ b/drivers/net/intel/ice/ice_rxtx.h @@ -28,12 +28,12 @@ #define ICE_TD_CMD ICE_TX_DESC_CMD_EOP -#define ICE_VPMD_RX_BURST 32 -#define ICE_VPMD_TX_BURST 32 -#define ICE_VPMD_RXQ_REARM_THRESH 64 -#define ICE_TX_MAX_FREE_BUF_SZ 64 -#define ICE_VPMD_DESCS_PER_LOOP 4 -#define ICE_VPMD_DESCS_PER_LOOP_WIDE 8 +#define ICE_VPMD_RX_BURST CI_VPMD_RX_BURST +#define ICE_VPMD_TX_BURST 32 +#define ICE_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH +#define ICE_TX_MAX_FREE_BUF_SZ 64 +#define ICE_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP +#define ICE_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE #define ICE_FDIR_PKT_LEN 512 diff --git a/drivers/net/intel/ice/ice_rxtx_common_avx.h b/drivers/net/intel/ice/ice_rxtx_common_avx.h deleted file mode 100644 index 7c65e7ed4d..0000000000 --- a/drivers/net/intel/ice/ice_rxtx_common_avx.h +++ /dev/null @@ -1,233 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2019 Intel Corporation - */ - -#ifndef _ICE_RXTX_COMMON_AVX_H_ -#define _ICE_RXTX_COMMON_AVX_H_ - -#include "ice_rxtx.h" - -#ifdef __AVX2__ -static __rte_always_inline void -ice_rxq_rearm_common(struct ci_rx_queue *rxq, __rte_unused bool avx512) -{ - int i; - uint16_t rx_id; - volatile union ci_rx_flex_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - - rxdp = rxq->rx_flex_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - ICE_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + ICE_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - __m128i dma_addr0; - - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < ICE_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - ICE_VPMD_RXQ_REARM_THRESH; - return; - } - -#ifndef RTE_NET_INTEL_USE_16BYTE_DESC - struct rte_mbuf *mb0, *mb1; - __m128i dma_addr0, dma_addr1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - -#if RTE_IOVA_IN_MBUF - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); -#else - /* convert va to dma_addr hdr/data */ - dma_addr0 = _mm_unpacklo_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpacklo_epi64(vaddr1, vaddr1); -#endif - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } -#else -#ifdef __AVX512VL__ - if (avx512) { - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - struct rte_mbuf *mb4, *mb5, *mb6, *mb7; - __m512i dma_addr0_3, dma_addr4_7; - __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 8 mbufs in one loop */ - for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; - i += 8, rxep += 8, rxdp += 8) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m128i vaddr4, vaddr5, vaddr6, vaddr7; - __m256i vaddr0_1, vaddr2_3; - __m256i vaddr4_5, vaddr6_7; - __m512i vaddr0_3, vaddr4_7; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - mb2 = rxep[2].mbuf; - mb3 = rxep[3].mbuf; - mb4 = rxep[4].mbuf; - mb5 = rxep[5].mbuf; - mb6 = rxep[6].mbuf; - mb7 = rxep[7].mbuf; - -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); - vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); - vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); - vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3, and so on. - */ - vaddr0_1 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), - vaddr1, 1); - vaddr2_3 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), - vaddr3, 1); - vaddr4_5 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr4), - vaddr5, 1); - vaddr6_7 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr6), - vaddr7, 1); - vaddr0_3 = - _mm512_inserti64x4(_mm512_castsi256_si512(vaddr0_1), - vaddr2_3, 1); - vaddr4_7 = - _mm512_inserti64x4(_mm512_castsi256_si512(vaddr4_5), - vaddr6_7, 1); - -#if RTE_IOVA_IN_MBUF - /* convert pa to dma_addr hdr/data */ - dma_addr0_3 = _mm512_unpackhi_epi64(vaddr0_3, vaddr0_3); - dma_addr4_7 = _mm512_unpackhi_epi64(vaddr4_7, vaddr4_7); -#else - /* convert va to dma_addr hdr/data */ - dma_addr0_3 = _mm512_unpacklo_epi64(vaddr0_3, vaddr0_3); - dma_addr4_7 = _mm512_unpacklo_epi64(vaddr4_7, vaddr4_7); -#endif - - /* add headroom to pa values */ - dma_addr0_3 = _mm512_add_epi64(dma_addr0_3, hdr_room); - dma_addr4_7 = _mm512_add_epi64(dma_addr4_7, hdr_room); - - /* flush desc with pa dma_addr */ - _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp->read), dma_addr0_3); - _mm512_store_si512(RTE_CAST_PTR(__m512i *, &(rxdp + 4)->read), dma_addr4_7); - } - } else -#endif /* __AVX512VL__ */ - { - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - __m256i dma_addr0_1, dma_addr2_3; - __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; - i += 4, rxep += 4, rxdp += 4) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m256i vaddr0_1, vaddr2_3; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - mb2 = rxep[2].mbuf; - mb3 = rxep[3].mbuf; - -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3 - */ - vaddr0_1 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), - vaddr1, 1); - vaddr2_3 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), - vaddr3, 1); - -#if RTE_IOVA_IN_MBUF - /* convert pa to dma_addr hdr/data */ - dma_addr0_1 = _mm256_unpackhi_epi64(vaddr0_1, vaddr0_1); - dma_addr2_3 = _mm256_unpackhi_epi64(vaddr2_3, vaddr2_3); -#else - /* convert va to dma_addr hdr/data */ - dma_addr0_1 = _mm256_unpacklo_epi64(vaddr0_1, vaddr0_1); - dma_addr2_3 = _mm256_unpacklo_epi64(vaddr2_3, vaddr2_3); -#endif - - /* add headroom to pa values */ - dma_addr0_1 = _mm256_add_epi64(dma_addr0_1, hdr_room); - dma_addr2_3 = _mm256_add_epi64(dma_addr2_3, hdr_room); - - /* flush desc with pa dma_addr */ - _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp->read), dma_addr0_1); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, &(rxdp + 2)->read), dma_addr2_3); - } - } - -#endif - - rxq->rxrearm_start += ICE_VPMD_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= ICE_VPMD_RXQ_REARM_THRESH; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); -} -#endif /* __AVX2__ */ - -#endif /* _ICE_RXTX_COMMON_AVX_H_ */ diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c index 5b1a13dd22..b952b8dddc 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx2.c @@ -3,14 +3,15 @@ */ #include "ice_rxtx_vec_common.h" -#include "ice_rxtx_common_avx.h" + +#include "../common/rx_vec_x86.h" #include static __rte_always_inline void ice_rxq_rearm(struct ci_rx_queue *rxq) { - ice_rxq_rearm_common(rxq, false); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_AVX2); } static __rte_always_inline __m256i diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c index b943caf0f0..7c6fe82072 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c @@ -3,14 +3,15 @@ */ #include "ice_rxtx_vec_common.h" -#include "ice_rxtx_common_avx.h" + +#include "../common/rx_vec_x86.h" #include static __rte_always_inline void ice_rxq_rearm(struct ci_rx_queue *rxq) { - ice_rxq_rearm_common(rxq, true); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_AVX512); } static inline __m256i diff --git a/drivers/net/intel/ice/ice_rxtx_vec_sse.c b/drivers/net/intel/ice/ice_rxtx_vec_sse.c index cae2188279..d818b3b728 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_sse.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_sse.c @@ -4,6 +4,8 @@ #include "ice_rxtx_vec_common.h" +#include "../common/rx_vec_x86.h" + #include static inline __m128i @@ -28,80 +30,7 @@ ice_flex_rxd_to_fdir_flags_vec(const __m128i fdir_id0_3) static inline void ice_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ci_rx_flex_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - __m128i dma_addr0, dma_addr1; - - rxdp = rxq->rx_flex_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - ICE_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + ICE_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < ICE_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - ICE_VPMD_RXQ_REARM_THRESH; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < ICE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - -#if RTE_IOVA_IN_MBUF - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); -#else - /* convert va to dma_addr hdr/data */ - dma_addr0 = _mm_unpacklo_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpacklo_epi64(vaddr1, vaddr1); -#endif - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += ICE_VPMD_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= ICE_VPMD_RXQ_REARM_THRESH; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_SSE); } static inline void From patchwork Mon Jun 9 15:37:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154225 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5E777468B7; Mon, 9 Jun 2025 17:41:32 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EE1364281D; Mon, 9 Jun 2025 17:38:35 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 9F5D442D7B for ; Mon, 9 Jun 2025 17:38:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483513; x=1781019513; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=uZKZPKDe/3hpQUNMJZavEvPWgopdpPJK88Eeupw1Vto=; b=Vugho3EsqV1fKnSmeS5kOy8Vga5OQlmnXZYoB8WpXAT3fU2BHyNCMeMw 7DZyw7VRVTyWLyQvuuoclthvEY/d8ezm5Gzmczk7c+x0t4uMIKgRks7jU Hg3k/qmWBbsAbw0h0e/gfT+PkqPIc+jt6GT7bjqD08IrYO+zVexlnreQr 0JyOPbLQXBiabNw8qFhpIWS9uAGroG0f5LFxwhDKhpIcJTWjrllReK9Tu Pw8pQJBPkbO6YrridFrZnDv4LRlZ4ANE3Sb7Z+tUxWr9kdAYUPJvKE+tF xiyKT54kVj0UD9RNuBuj7vKRRF5wVe53OP8Xj0b51KxbeqFsopyuNRosj Q==; X-CSE-ConnectionGUID: hEkGrIS5TiGP3kzknxXWEA== X-CSE-MsgGUID: R2nLQUo9SRuGYMiHQtBM5g== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012238" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012238" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:32 -0700 X-CSE-ConnectionGUID: 6gIp4drwTRGox8tdc3CPHQ== X-CSE-MsgGUID: kRiPd1MNSeWF2ga6iraEAA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419767" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:30 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Ian Stokes Subject: [PATCH v6 28/33] net/i40e: use common Rx rearm code Date: Mon, 9 Jun 2025 16:37:26 +0100 Message-ID: <0c8097ad41731d4f4c03c023291564a3d17535ae.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The i40e driver has an implementation of vectorized mbuf rearm code that is identical to the one in the common code, so just use that. In addition, the i40e has implementations of Rx queue rearm for Neon and AltiVec instruction sets, so create common headers for each of the instruction sets, and use that in respective i40e code. While we're at it, also make sure to use common definitions for things like burst size, rearm threshold, and descriptors per loop, which is currently defined separately in each driver. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/common/rx_vec_arm.h | 105 +++++++++ drivers/net/intel/common/rx_vec_ppc.h | 121 ++++++++++ drivers/net/intel/i40e/i40e_rxtx.h | 8 +- drivers/net/intel/i40e/i40e_rxtx_common_avx.h | 215 ------------------ .../net/intel/i40e/i40e_rxtx_vec_altivec.c | 83 +------ drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c | 5 +- drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c | 5 +- drivers/net/intel/i40e/i40e_rxtx_vec_neon.c | 59 +---- drivers/net/intel/i40e/i40e_rxtx_vec_sse.c | 70 +----- 9 files changed, 245 insertions(+), 426 deletions(-) create mode 100644 drivers/net/intel/common/rx_vec_arm.h create mode 100644 drivers/net/intel/common/rx_vec_ppc.h delete mode 100644 drivers/net/intel/i40e/i40e_rxtx_common_avx.h diff --git a/drivers/net/intel/common/rx_vec_arm.h b/drivers/net/intel/common/rx_vec_arm.h new file mode 100644 index 0000000000..2e48d4b6c0 --- /dev/null +++ b/drivers/net/intel/common/rx_vec_arm.h @@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Intel Corporation + */ + +#ifndef _COMMON_INTEL_RX_VEC_ARM_H_ +#define _COMMON_INTEL_RX_VEC_ARM_H_ + +#include + +#include +#include +#include + +#include "rx.h" + +static inline int +_ci_rxq_rearm_get_bufs(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + volatile union ci_rx_desc *rxdp; + int i; + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + if (rte_mempool_get_bulk(rxq->mp, (void **)rxp, rearm_thresh) < 0) { + if (rxq->rxrearm_nb + rearm_thresh >= rxq->nb_rx_desc) { + uint64x2_t zero = vdupq_n_u64(0); + + for (i = 0; i < CI_VPMD_DESCS_PER_LOOP; i++) { + rxp[i].mbuf = &rxq->fake_mbuf; + vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i]), zero); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += rearm_thresh; + return -1; + } + return 0; +} + +static __rte_always_inline void +_ci_rxq_rearm_neon(struct ci_rx_queue *rxq) +{ + const uint64x2_t zero = vdupq_n_u64(0); + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + volatile union ci_rx_desc *rxdp; + int i; + + const uint8x8_t mbuf_init = vld1_u8((uint8_t *)&rxq->mbuf_initializer); + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < rearm_thresh; i += 2, rxp += 2, rxdp += 2) { + struct rte_mbuf *mb0 = rxp[0].mbuf; + struct rte_mbuf *mb1 = rxp[1].mbuf; + + /* + * Flush mbuf with pkt template. + * Data to be rearmed is 6 bytes long. + */ + vst1_u8((uint8_t *)&mb0->rearm_data, mbuf_init); + vst1_u8((uint8_t *)&mb1->rearm_data, mbuf_init); +#if RTE_IOVA_IN_MBUF + const uint64_t addr0 = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; + const uint64_t addr1 = mb1->buf_iova + RTE_PKTMBUF_HEADROOM; +#else + const uint64_t addr0 = (uintptr_t)RTE_PTR_ADD(mb0->buf_addr, RTE_PKTMBUF_HEADROOM); + const uint64_t addr1 = (uintptr_t)RTE_PTR_ADD(mb1->buf_addr, RTE_PKTMBUF_HEADROOM); +#endif + uint64x2_t dma_addr0 = vsetq_lane_u64(addr0, zero, 0); + uint64x2_t dma_addr1 = vsetq_lane_u64(addr1, zero, 0); + /* flush desc with pa dma_addr */ + vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[0]), dma_addr0); + vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[1]), dma_addr1); + } +} + +static __rte_always_inline void +ci_rxq_rearm(struct ci_rx_queue *rxq) +{ + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + uint16_t rx_id; + + /* Pull 'n' more MBUFs into the software ring */ + if (_ci_rxq_rearm_get_bufs(rxq) < 0) + return; + + _ci_rxq_rearm_neon(rxq); + + rxq->rxrearm_start += rearm_thresh; + if (rxq->rxrearm_start >= rxq->nb_rx_desc) + rxq->rxrearm_start = 0; + + rxq->rxrearm_nb -= rearm_thresh; + + rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? + (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); + + /* Update the tail pointer on the NIC */ + rte_write32_wc(rte_cpu_to_le_32(rx_id), rxq->qrx_tail); +} + +#endif /* _COMMON_INTEL_RX_VEC_ARM_H_ */ diff --git a/drivers/net/intel/common/rx_vec_ppc.h b/drivers/net/intel/common/rx_vec_ppc.h new file mode 100644 index 0000000000..97affc34c2 --- /dev/null +++ b/drivers/net/intel/common/rx_vec_ppc.h @@ -0,0 +1,121 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Intel Corporation + */ + +#ifndef _COMMON_INTEL_RX_VEC_PPC_H_ +#define _COMMON_INTEL_RX_VEC_PPC_H_ + +#include + +#include +#include +#include + +#include "rx.h" + +static inline int +_ci_rxq_rearm_get_bufs(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + volatile union ci_rx_desc *rxdp; + int i; + + rxdp = &rxq->rx_ring[rxq->rxrearm_start]; + + if (rte_mempool_get_bulk(rxq->mp, (void **)rxp, rearm_thresh) < 0) { + if (rxq->rxrearm_nb + rearm_thresh >= rxq->nb_rx_desc) { + __vector unsigned long dma_addr0 = (__vector unsigned long){}; + + for (i = 0; i < CI_VPMD_DESCS_PER_LOOP; i++) { + rxp[i].mbuf = &rxq->fake_mbuf; + vec_st(dma_addr0, 0, + RTE_CAST_PTR(__vector unsigned long *, &rxdp[i])); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += rearm_thresh; + return -1; + } + return 0; +} + +/* + * SSE code path can handle both 16-byte and 32-byte descriptors with one code + * path, as we only ever write 16 bytes at a time. + */ +static __rte_always_inline void +_ci_rxq_rearm_altivec(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + __vector unsigned long hdroom = + (__vector unsigned long){RTE_PKTMBUF_HEADROOM, RTE_PKTMBUF_HEADROOM}; + int i; + + volatile union ci_rx_desc *rxdp = rxq->rx_ring + rxq->rxrearm_start; + + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < rearm_thresh; i += 2, rxep += 2) { + __vector unsigned long vaddr0, vaddr1; + struct rte_mbuf *mb0 = rxep[0].mbuf; + struct rte_mbuf *mb1 = rxep[1].mbuf; + + /* Flush mbuf with pkt template. + * Data to be rearmed is 6 bytes long. + * Though, RX will overwrite ol_flags that are coming next + * anyway. So overwrite whole 8 bytes with one load: + * 6 bytes of rearm_data plus first 2 bytes of ol_flags. + */ + *(uint64_t *)&mb0->rearm_data = rxq->mbuf_initializer; + *(uint64_t *)&mb1->rearm_data = rxq->mbuf_initializer; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + vaddr0 = vec_ld(0, (__vector unsigned long *)&mb0->buf_addr); + vaddr1 = vec_ld(0, (__vector unsigned long *)&mb1->buf_addr); + +#if RTE_IOVA_IN_MBUF + /* convert pa to dma_addr hdr/data */ + vaddr0 = vec_mergel(vaddr0, vaddr0); + vaddr1 = vec_mergel(vaddr1, vaddr1); +#else + /* convert va to dma_addr hdr/data */ + vaddr0 = vec_mergeh(vaddr0, vaddr0); + vaddr1 = vec_mergeh(vaddr1, vaddr1); +#endif + + /* add headroom to pa values */ + vaddr0 = vec_add(vaddr0, hdroom); + vaddr1 = vec_add(vaddr1, hdroom); + + /* flush desc with pa dma_addr */ + vec_st(vaddr0, 0, RTE_CAST_PTR(__vector unsigned long *, rxdp++)); + vec_st(vaddr1, 0, RTE_CAST_PTR(__vector unsigned long *, rxdp++)); + } +} + +static __rte_always_inline void +ci_rxq_rearm(struct ci_rx_queue *rxq) +{ + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + uint16_t rx_id; + + /* Pull 'n' more MBUFs into the software ring */ + if (_ci_rxq_rearm_get_bufs(rxq) < 0) + return; + + _ci_rxq_rearm_altivec(rxq); + + rxq->rxrearm_start += rearm_thresh; + if (rxq->rxrearm_start >= rxq->nb_rx_desc) + rxq->rxrearm_start = 0; + + rxq->rxrearm_nb -= rearm_thresh; + + rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? + (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); + + /* Update the tail pointer on the NIC */ + rte_write32_wc(rte_cpu_to_le_32(rx_id), rxq->qrx_tail); +} + +#endif /* _COMMON_INTEL_RX_VEC_ARM_H_ */ diff --git a/drivers/net/intel/i40e/i40e_rxtx.h b/drivers/net/intel/i40e/i40e_rxtx.h index 05c41d473e..984532c507 100644 --- a/drivers/net/intel/i40e/i40e_rxtx.h +++ b/drivers/net/intel/i40e/i40e_rxtx.h @@ -11,11 +11,11 @@ #define I40E_RX_MAX_BURST CI_RX_MAX_BURST #define I40E_TX_MAX_BURST 32 -#define I40E_VPMD_RX_BURST 32 -#define I40E_VPMD_RXQ_REARM_THRESH 32 +#define I40E_VPMD_RX_BURST CI_VPMD_RX_BURST +#define I40E_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH #define I40E_TX_MAX_FREE_BUF_SZ 64 -#define I40E_VPMD_DESCS_PER_LOOP 4 -#define I40E_VPMD_DESCS_PER_LOOP_WIDE 8 +#define I40E_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP +#define I40E_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE #define I40E_RXBUF_SZ_1024 1024 #define I40E_RXBUF_SZ_2048 2048 diff --git a/drivers/net/intel/i40e/i40e_rxtx_common_avx.h b/drivers/net/intel/i40e/i40e_rxtx_common_avx.h deleted file mode 100644 index 97cf5226f6..0000000000 --- a/drivers/net/intel/i40e/i40e_rxtx_common_avx.h +++ /dev/null @@ -1,215 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2010-2015 Intel Corporation - */ - -#ifndef _I40E_RXTX_COMMON_AVX_H_ -#define _I40E_RXTX_COMMON_AVX_H_ -#include -#include -#include - -#include "i40e_ethdev.h" -#include "i40e_rxtx.h" - -#ifdef __AVX2__ -static __rte_always_inline void -i40e_rxq_rearm_common(struct ci_rx_queue *rxq, __rte_unused bool avx512) -{ - int i; - uint16_t rx_id; - volatile union ci_rx_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - I40E_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - __m128i dma_addr0; - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - I40E_VPMD_RXQ_REARM_THRESH; - return; - } - -#ifndef RTE_NET_INTEL_USE_16BYTE_DESC - struct rte_mbuf *mb0, *mb1; - __m128i dma_addr0, dma_addr1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } -#else -#ifdef __AVX512VL__ - if (avx512) { - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - struct rte_mbuf *mb4, *mb5, *mb6, *mb7; - __m512i dma_addr0_3, dma_addr4_7; - __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 8 mbufs in one loop */ - for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; - i += 8, rxep += 8, rxdp += 8) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m128i vaddr4, vaddr5, vaddr6, vaddr7; - __m256i vaddr0_1, vaddr2_3; - __m256i vaddr4_5, vaddr6_7; - __m512i vaddr0_3, vaddr4_7; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - mb2 = rxep[2].mbuf; - mb3 = rxep[3].mbuf; - mb4 = rxep[4].mbuf; - mb5 = rxep[5].mbuf; - mb6 = rxep[6].mbuf; - mb7 = rxep[7].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); - vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); - vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); - vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3, and so on. - */ - vaddr0_1 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), - vaddr1, 1); - vaddr2_3 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), - vaddr3, 1); - vaddr4_5 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr4), - vaddr5, 1); - vaddr6_7 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr6), - vaddr7, 1); - vaddr0_3 = - _mm512_inserti64x4(_mm512_castsi256_si512(vaddr0_1), - vaddr2_3, 1); - vaddr4_7 = - _mm512_inserti64x4(_mm512_castsi256_si512(vaddr4_5), - vaddr6_7, 1); - - /* convert pa to dma_addr hdr/data */ - dma_addr0_3 = _mm512_unpackhi_epi64(vaddr0_3, vaddr0_3); - dma_addr4_7 = _mm512_unpackhi_epi64(vaddr4_7, vaddr4_7); - - /* add headroom to pa values */ - dma_addr0_3 = _mm512_add_epi64(dma_addr0_3, hdr_room); - dma_addr4_7 = _mm512_add_epi64(dma_addr4_7, hdr_room); - - /* flush desc with pa dma_addr */ - _mm512_store_si512(RTE_CAST_PTR(__m512i *, - &rxdp->read), dma_addr0_3); - _mm512_store_si512(RTE_CAST_PTR(__m512i *, - &(rxdp + 4)->read), dma_addr4_7); - } - } else -#endif /* __AVX512VL__*/ - { - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - __m256i dma_addr0_1, dma_addr2_3; - __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; - i += 4, rxep += 4, rxdp += 4) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m256i vaddr0_1, vaddr2_3; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - mb2 = rxep[2].mbuf; - mb3 = rxep[3].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3 - */ - vaddr0_1 = _mm256_inserti128_si256 - (_mm256_castsi128_si256(vaddr0), vaddr1, 1); - vaddr2_3 = _mm256_inserti128_si256 - (_mm256_castsi128_si256(vaddr2), vaddr3, 1); - - /* convert pa to dma_addr hdr/data */ - dma_addr0_1 = _mm256_unpackhi_epi64(vaddr0_1, vaddr0_1); - dma_addr2_3 = _mm256_unpackhi_epi64(vaddr2_3, vaddr2_3); - - /* add headroom to pa values */ - dma_addr0_1 = _mm256_add_epi64(dma_addr0_1, hdr_room); - dma_addr2_3 = _mm256_add_epi64(dma_addr2_3, hdr_room); - - /* flush desc with pa dma_addr */ - _mm256_store_si256(RTE_CAST_PTR(__m256i *, - &rxdp->read), dma_addr0_1); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, - &(rxdp + 2)->read), dma_addr2_3); - } - } - -#endif - - rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; - rx_id = rxq->rxrearm_start - 1; - - if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { - rxq->rxrearm_start = 0; - rx_id = rxq->nb_rx_desc - 1; - } - - rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; - - /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); -} -#endif /* __AVX2__*/ - -#endif /*_I40E_RXTX_COMMON_AVX_H_*/ diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c b/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c index a914ef20f4..8a4a1a77bf 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_altivec.c @@ -13,91 +13,14 @@ #include "i40e_rxtx.h" #include "i40e_rxtx_vec_common.h" +#include "../common/rx_vec_ppc.h" + #include static inline void i40e_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ci_rx_desc *rxdp; - - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - - __vector unsigned long hdr_room = (__vector unsigned long){ - RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM}; - __vector unsigned long dma_addr0, dma_addr1; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - I40E_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - dma_addr0 = (__vector unsigned long){}; - for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - vec_st(dma_addr0, 0, - RTE_CAST_PTR(__vector unsigned long *, &rxdp[i].read)); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - I40E_VPMD_RXQ_REARM_THRESH; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __vector unsigned long vaddr0, vaddr1; - uintptr_t p0, p1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - /* Flush mbuf with pkt template. - * Data to be rearmed is 6 bytes long. - * Though, RX will overwrite ol_flags that are coming next - * anyway. So overwrite whole 8 bytes with one load: - * 6 bytes of rearm_data plus first 2 bytes of ol_flags. - */ - p0 = (uintptr_t)&mb0->rearm_data; - *(uint64_t *)p0 = rxq->mbuf_initializer; - p1 = (uintptr_t)&mb1->rearm_data; - *(uint64_t *)p1 = rxq->mbuf_initializer; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - vaddr0 = vec_ld(0, (__vector unsigned long *)&mb0->buf_addr); - vaddr1 = vec_ld(0, (__vector unsigned long *)&mb1->buf_addr); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = vec_mergel(vaddr0, vaddr0); - dma_addr1 = vec_mergel(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = vec_add(dma_addr0, hdr_room); - dma_addr1 = vec_add(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - vec_st(dma_addr0, 0, RTE_CAST_PTR(__vector unsigned long *, &rxdp++->read)); - vec_st(dma_addr1, 0, RTE_CAST_PTR(__vector unsigned long *, &rxdp++->read)); - } - - rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; - rx_id = rxq->rxrearm_start - 1; - - if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { - rxq->rxrearm_start = 0; - rx_id = rxq->nb_rx_desc - 1; - } - - rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; - - /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ci_rxq_rearm(rxq); } static inline void diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c index fee2a6e670..aeb2756e7a 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx2.c @@ -11,14 +11,15 @@ #include "i40e_ethdev.h" #include "i40e_rxtx.h" #include "i40e_rxtx_vec_common.h" -#include "i40e_rxtx_common_avx.h" + +#include "../common/rx_vec_x86.h" #include static __rte_always_inline void i40e_rxq_rearm(struct ci_rx_queue *rxq) { - i40e_rxq_rearm_common(rxq, false); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_AVX2); } #ifndef RTE_NET_INTEL_USE_16BYTE_DESC diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c index e609b7c411..571987d27a 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c @@ -11,14 +11,15 @@ #include "i40e_ethdev.h" #include "i40e_rxtx.h" #include "i40e_rxtx_vec_common.h" -#include "i40e_rxtx_common_avx.h" + +#include "../common/rx_vec_x86.h" #include static __rte_always_inline void i40e_rxq_rearm(struct ci_rx_queue *rxq) { - i40e_rxq_rearm_common(rxq, true); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_AVX512); } #ifndef RTE_NET_INTEL_USE_16BYTE_DESC diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c index 02ba03c290..64ffb2f6df 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_neon.c @@ -16,65 +16,12 @@ #include "i40e_rxtx.h" #include "i40e_rxtx_vec_common.h" +#include "../common/rx_vec_arm.h" + static inline void i40e_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ci_rx_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - uint64x2_t dma_addr0, dma_addr1; - uint64x2_t zero = vdupq_n_u64(0); - uint64_t paddr; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (unlikely(rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - I40E_VPMD_RXQ_REARM_THRESH) < 0)) { - if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i].read), zero); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - I40E_VPMD_RXQ_REARM_THRESH; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr0 = vdupq_n_u64(paddr); - - /* flush desc with pa dma_addr */ - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr0); - - paddr = mb1->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr1 = vdupq_n_u64(paddr); - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; - rx_id = rxq->rxrearm_start - 1; - - if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { - rxq->rxrearm_start = 0; - rx_id = rxq->nb_rx_desc - 1; - } - - rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; - - rte_io_wmb(); - /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); + ci_rxq_rearm(rxq); } #ifndef RTE_NET_INTEL_USE_16BYTE_DESC diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c index 6bafd96797..15cf07e548 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_sse.c @@ -12,78 +12,14 @@ #include "i40e_rxtx.h" #include "i40e_rxtx_vec_common.h" +#include "../common/rx_vec_x86.h" + #include static inline void i40e_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ci_rx_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - __m128i dma_addr0, dma_addr1; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - I40E_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + I40E_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < I40E_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - I40E_VPMD_RXQ_REARM_THRESH; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < I40E_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += I40E_VPMD_RXQ_REARM_THRESH; - rx_id = rxq->rxrearm_start - 1; - - if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { - rxq->rxrearm_start = 0; - rx_id = rxq->nb_rx_desc - 1; - } - - rxq->rxrearm_nb -= I40E_VPMD_RXQ_REARM_THRESH; - - /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_SSE); } #ifndef RTE_NET_INTEL_USE_16BYTE_DESC From patchwork Mon Jun 9 15:37:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154226 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A57A3468B7; Mon, 9 Jun 2025 17:41:38 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E304E42DBB; Mon, 9 Jun 2025 17:38:36 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 7DF3A42D7D for ; Mon, 9 Jun 2025 17:38:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483515; x=1781019515; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xJIy/i/S0i0eiLOJP83kw2lyp2VyIbcI6S2MADPUzPI=; b=YUGfRif05FlU/8lAx8JJqX1mZKY2Y9BCUAtT889sU1/7FxqF+ZFgU/Bm xzuaS61IloCf7xOvaNIu+xf7fay6lu9pn4d0W21nhwLZMJE9o8Y6aS7QV wKLHaK+aGW32HvDzCgN+skZ7lfOlxYU22hz89ZcHwbSnUahfQ7avigL8Y GDUDPBTJtdY4IsH5ULaly7VGIaTjPKM/gd8c+yCPM0zxDU5ajGi3g/4sh y+bMDRwaKyWWlO8Oyp3Mq8EzXIOqhX6B3embbMMGgFQsPKEsWAFLhqhPL 2Gd7braazEJSHZnNic3WJuk8D6fyHFDpWJgiXgkR3V/1YmrmbaBCAfhB0 w==; X-CSE-ConnectionGUID: bE+Qw1kqREGKhvfAU0nePw== X-CSE-MsgGUID: Vbve6MHvT9mt1HagVA5UlQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012240" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012240" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:34 -0700 X-CSE-ConnectionGUID: Q0L0XkbaQLW35FI1eQR/FA== X-CSE-MsgGUID: P6kM6Ha/T5uWDK0l26xgFA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419775" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:32 -0700 From: Anatoly Burakov To: dev@dpdk.org, Vladimir Medvedkin , Ian Stokes Cc: bruce.richardson@intel.com Subject: [PATCH v6 29/33] net/iavf: use common Rx rearm code Date: Mon, 9 Jun 2025 16:37:27 +0100 Message-ID: <2b8c99e42adc1738e5993cd5705c08e21be2c185.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The iavf driver has implementations of vectorized mbuf rearm code that is identical to the ones in the common code, so just use those. While we're at it, also make sure to use common definitions for things like burst size, rearm threshold, and descriptors per loop, which is currently defined separately in each driver. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/iavf/iavf_rxtx.h | 8 +- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 3 +- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 3 +- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 76 ------------------- drivers/net/intel/iavf/iavf_rxtx_vec_neon.c | 58 +------------- drivers/net/intel/iavf/iavf_rxtx_vec_sse.c | 72 +----------------- 6 files changed, 13 insertions(+), 207 deletions(-) diff --git a/drivers/net/intel/iavf/iavf_rxtx.h b/drivers/net/intel/iavf/iavf_rxtx.h index 8abcccf8c2..1641bfe59a 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.h +++ b/drivers/net/intel/iavf/iavf_rxtx.h @@ -28,11 +28,11 @@ #define IAVF_RX_MAX_DATA_BUF_SIZE (16 * 1024 - 128) /* used for Vector PMD */ -#define IAVF_VPMD_RX_BURST 32 +#define IAVF_VPMD_RX_BURST CI_VPMD_RX_BURST #define IAVF_VPMD_TX_BURST 32 -#define IAVF_VPMD_RXQ_REARM_THRESH 32 -#define IAVF_VPMD_DESCS_PER_LOOP 4 -#define IAVF_VPMD_DESCS_PER_LOOP_WIDE 8 +#define IAVF_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH +#define IAVF_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP +#define IAVF_VPMD_DESCS_PER_LOOP_WIDE CI_VPMD_DESCS_PER_LOOP_WIDE #define IAVF_VPMD_TX_MAX_FREE_BUF 64 #define IAVF_TX_NO_VECTOR_FLAGS ( \ diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c index b0f36cb515..73a6ae5c41 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c @@ -2,6 +2,7 @@ * Copyright(c) 2019 Intel Corporation */ +#include "../common/rx_vec_x86.h" #include "iavf_rxtx_vec_common.h" #include @@ -9,7 +10,7 @@ static __rte_always_inline void iavf_rxq_rearm(struct ci_rx_queue *rxq) { - iavf_rxq_rearm_common(rxq, false); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_AVX2); } #define PKTLEN_SHIFT 10 diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c index bbba564329..9a029ecbe0 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c @@ -2,6 +2,7 @@ * Copyright(c) 2020 Intel Corporation */ +#include "../common/rx_vec_x86.h" #include "iavf_rxtx_vec_common.h" #include @@ -29,7 +30,7 @@ static __rte_always_inline void iavf_rxq_rearm(struct ci_rx_queue *rxq) { - iavf_rxq_rearm_common(rxq, true); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_AVX512); } #define IAVF_RX_LEN_MASK 0x80808080 diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h index 90a9ac95eb..9b14fc7d12 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h @@ -236,80 +236,4 @@ iavf_txd_enable_offload(__rte_unused struct rte_mbuf *tx_pkt, *txd_hi |= ((uint64_t)td_cmd) << IAVF_TXD_QW1_CMD_SHIFT; } - -#ifdef RTE_ARCH_X86 -static __rte_always_inline void -iavf_rxq_rearm_common(struct ci_rx_queue *rxq, __rte_unused bool avx512) -{ - int i; - uint16_t rx_id; - volatile union ci_rx_desc *rxdp; - struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxp, - IAVF_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + IAVF_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - __m128i dma_addr0; - - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { - rxp[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - IAVF_VPMD_RXQ_REARM_THRESH; - return; - } - - struct rte_mbuf *mb0, *mb1; - __m128i dma_addr0, dma_addr1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < IAVF_VPMD_RXQ_REARM_THRESH; i += 2, rxp += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxp[0].mbuf; - mb1 = rxp[1].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += IAVF_VPMD_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= IAVF_VPMD_RXQ_REARM_THRESH; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - IAVF_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); -} -#endif - #endif diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c b/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c index 562e574aab..4ed4e9b336 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_neon.c @@ -14,64 +14,12 @@ #include "iavf_rxtx.h" #include "iavf_rxtx_vec_common.h" +#include "../common/rx_vec_arm.h" + static inline void iavf_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ci_rx_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - uint64x2_t dma_addr0, dma_addr1; - uint64x2_t zero = vdupq_n_u64(0); - uint64_t paddr; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (unlikely(rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - IAVF_VPMD_RXQ_REARM_THRESH) < 0)) { - if (rxq->rxrearm_nb + IAVF_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i].read), zero); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - IAVF_VPMD_RXQ_REARM_THRESH; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < IAVF_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr0 = vdupq_n_u64(paddr); - - /* flush desc with pa dma_addr */ - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr0); - - paddr = mb1->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr1 = vdupq_n_u64(paddr); - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += IAVF_VPMD_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= IAVF_VPMD_RXQ_REARM_THRESH; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - rte_io_wmb(); - /* Update the tail pointer on the NIC */ - IAVF_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); + ci_rxq_rearm(rxq); } static inline void diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c index a30ba87a3e..e4c1f9fc7b 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c @@ -9,82 +9,14 @@ #include "iavf.h" #include "iavf_rxtx.h" #include "iavf_rxtx_vec_common.h" +#include "../common/rx_vec_x86.h" #include static inline void iavf_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - - volatile union ci_rx_desc *rxdp; - struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - __m128i dma_addr0, dma_addr1; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, (void *)rxp, - rxq->rx_free_thresh) < 0) { - if (rxq->rxrearm_nb + rxq->rx_free_thresh >= rxq->nb_rx_desc) { - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < IAVF_VPMD_DESCS_PER_LOOP; i++) { - rxp[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - rxq->rx_free_thresh; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < rxq->rx_free_thresh; i += 2, rxp += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxp[0].mbuf; - mb1 = rxp[1].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += rxq->rx_free_thresh; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= rxq->rx_free_thresh; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - PMD_RX_LOG(DEBUG, "port_id=%u queue_id=%u rx_tail=%u " - "rearm_start=%u rearm_nb=%u", - rxq->port_id, rxq->queue_id, - rx_id, rxq->rxrearm_start, rxq->rxrearm_nb); - - /* Update the tail pointer on the NIC */ - IAVF_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_SSE); } static inline void From patchwork Mon Jun 9 15:37:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154227 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C639B468B7; Mon, 9 Jun 2025 17:41:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3D0C242C24; Mon, 9 Jun 2025 17:38:39 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 11AEA42D90 for ; Mon, 9 Jun 2025 17:38:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483516; x=1781019516; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=y3jUKY8BbwBa2c9bAJINyL6F+vSrvox/M2ihJ+nb6Kg=; b=Y4A+THcbDWEIBM4Fungg5b0p21vQnnl/GM/KCIocHK/4fujMiS7Sh3Zo efDlNLwWVG2ob2kcl85mNijzxS5KZZbSSrgk0utKNRJGrtsAQfhrT4nsC Sqz1xLylqHPklbRxNyEvV6ZZQb9Hbsdwd9w8E3PVO1O16Qa9fAley0Dgr 5SXWgdmiqQA1mmoIRFWqcq/AZvrcNUiFetKAg9gGAmXqFKNHR7EVgSm3Y 2iUITnYxsO/acearmeLFO5cWK1ezYuOfDeF9Mi+v2W//YJcPgl/6f7IIP A6SaQrh1ZWdY6rnfIluxfTRVPgnSLgv2LSF5R0NiRSUR5znWVRBgukudn w==; X-CSE-ConnectionGUID: zmnePAc/SeyBr08nFEo8Hg== X-CSE-MsgGUID: KvdRmOS1RWq5yQ2F9cBKig== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012243" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012243" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:36 -0700 X-CSE-ConnectionGUID: T0knbFmTTqu05eUI9zE9pg== X-CSE-MsgGUID: nr8zXDtLSs6O7ywIeI4i7w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419784" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:34 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Vladimir Medvedkin Subject: [PATCH v6 30/33] net/ixgbe: use common Rx rearm code Date: Mon, 9 Jun 2025 16:37:28 +0100 Message-ID: <06a9e669ace513b9942e317d1b7f632d015a6662.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The ixgbe driver has implementations of vectorized mbuf rearm code that is identical to the ones in the common code, so just use those. Since ixgbe Rx descriptors are always 16-byte wide, force using 16-byte definitions in the common headers with a define flag. While we're at it, also make sure to use common definitions for things like burst size, rearm threshold, and descriptors per loop, which is currently defined separately in each driver. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/common/rx_vec_x86.h | 2 +- drivers/net/intel/ixgbe/ixgbe_rxtx.h | 15 +++- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c | 67 +--------------- drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c | 76 +------------------ 4 files changed, 21 insertions(+), 139 deletions(-) diff --git a/drivers/net/intel/common/rx_vec_x86.h b/drivers/net/intel/common/rx_vec_x86.h index 4ad8066630..ecab8b30a6 100644 --- a/drivers/net/intel/common/rx_vec_x86.h +++ b/drivers/net/intel/common/rx_vec_x86.h @@ -285,7 +285,7 @@ ci_rxq_rearm(struct ci_rx_queue *rxq, const enum ci_rx_vec_level vec_level) /* fall through */ #endif case CI_RX_VEC_LEVEL_SSE: - _ci_rxq_rearm_sse(rxq, desc_len); + _ci_rxq_rearm_sse(rxq); break; } #else diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx.h b/drivers/net/intel/ixgbe/ixgbe_rxtx.h index aad7ee81ee..7950e56ee4 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx.h +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx.h @@ -7,6 +7,15 @@ #include "ixgbe_type.h" +/* + * For IXGBE, descriptor size is always 16 bytes, so in order to have all + * vectorized and common code building correctly and with proper offsets, force + * the common parts to consider IXGBE descriptors to be 16-bytes in size. + */ +#ifndef RTE_NET_INTEL_USE_16BYTE_DESC +#define RTE_NET_INTEL_USE_16BYTE_DESC +#endif + #include "../common/rx.h" #include "../common/tx.h" @@ -36,10 +45,10 @@ #define IXGBE_RX_MAX_BURST CI_RX_MAX_BURST #define IXGBE_TX_MAX_FREE_BUF_SZ 64 -#define IXGBE_VPMD_DESCS_PER_LOOP 4 +#define IXGBE_VPMD_DESCS_PER_LOOP CI_VPMD_DESCS_PER_LOOP -#define IXGBE_VPMD_RXQ_REARM_THRESH 32 -#define IXGBE_VPMD_RX_BURST IXGBE_VPMD_RXQ_REARM_THRESH +#define IXGBE_VPMD_RXQ_REARM_THRESH CI_VPMD_RX_REARM_THRESH +#define IXGBE_VPMD_RX_BURST CI_VPMD_RX_BURST #define RX_RING_SZ ((IXGBE_MAX_RING_DESC + IXGBE_RX_MAX_BURST) * \ sizeof(union ixgbe_adv_rx_desc)) diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c index ce492f2ff1..54f52aa9d7 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_neon.c @@ -11,72 +11,13 @@ #include "ixgbe_rxtx.h" #include "ixgbe_rxtx_vec_common.h" +#include "../common/rx_vec_arm.h" + static inline void ixgbe_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ixgbe_adv_rx_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - uint64x2_t dma_addr0, dma_addr1; - uint64x2_t zero = vdupq_n_u64(0); - uint64_t paddr; - uint8x8_t p; - - rxdp = rxq->ixgbe_rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (unlikely(rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - IXGBE_VPMD_RXQ_REARM_THRESH) < 0)) { - if (rxq->rxrearm_nb + IXGBE_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - for (i = 0; i < IXGBE_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp[i].read), - zero); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - IXGBE_VPMD_RXQ_REARM_THRESH; - return; - } - - p = vld1_u8((uint8_t *)&rxq->mbuf_initializer); - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < IXGBE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - /* - * Flush mbuf with pkt template. - * Data to be rearmed is 6 bytes long. - */ - vst1_u8((uint8_t *)&mb0->rearm_data, p); - paddr = mb0->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr0 = vsetq_lane_u64(paddr, zero, 0); - /* flush desc with pa dma_addr */ - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr0); - - vst1_u8((uint8_t *)&mb1->rearm_data, p); - paddr = mb1->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr1 = vsetq_lane_u64(paddr, zero, 0); - vst1q_u64(RTE_CAST_PTR(uint64_t *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += IXGBE_VPMD_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= IXGBE_VPMD_RXQ_REARM_THRESH; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + RTE_BUILD_BUG_ON(sizeof(union ci_rx_desc) != sizeof(union ixgbe_adv_rx_desc)); + ci_rxq_rearm(rxq); } static inline void diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c index f977489b95..dca3a20ca0 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_sse.c @@ -10,83 +10,15 @@ #include "ixgbe_rxtx.h" #include "ixgbe_rxtx_vec_common.h" +#include "../common/rx_vec_x86.h" + #include static inline void ixgbe_rxq_rearm(struct ci_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ixgbe_adv_rx_desc *rxdp; - struct ci_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - struct rte_mbuf *mb0, *mb1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - __m128i dma_addr0, dma_addr1; - - const __m128i hba_msk = _mm_set_epi64x(0, UINT64_MAX); - - rxdp = rxq->ixgbe_rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - IXGBE_VPMD_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + IXGBE_VPMD_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < IXGBE_VPMD_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[i].read), - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - IXGBE_VPMD_RXQ_REARM_THRESH; - return; - } - - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < IXGBE_VPMD_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&(mb0->buf_addr)); - vaddr1 = _mm_loadu_si128((__m128i *)&(mb1->buf_addr)); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* set Header Buffer Address to zero */ - dma_addr0 = _mm_and_si128(dma_addr0, hba_msk); - dma_addr1 = _mm_and_si128(dma_addr1, hba_msk); - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp++->read), dma_addr1); - } - - rxq->rxrearm_start += IXGBE_VPMD_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= IXGBE_VPMD_RXQ_REARM_THRESH; - - rx_id = (uint16_t) ((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); + RTE_BUILD_BUG_ON(sizeof(union ci_rx_desc) != sizeof(union ixgbe_adv_rx_desc)); + ci_rxq_rearm(rxq, CI_RX_VEC_LEVEL_SSE); } #ifdef RTE_LIB_SECURITY From patchwork Mon Jun 9 15:37:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154228 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2300E468B7; Mon, 9 Jun 2025 17:41:53 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 714F042DC5; Mon, 9 Jun 2025 17:38:40 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 771F042DA7 for ; Mon, 9 Jun 2025 17:38:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483518; x=1781019518; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=Mrjb4bpdbK6RYAD48PYmYg2Q9bjapqxRHZXwOwtzjeo=; b=f1/AEjJA1cZ9b5XASfaVhsTj4lbTIU6Rj1wDtS9JmS9Xnk79nTIMb4zw 79VlgFL2hkU1RKkmdoB/NcS+aO8VNfcclGTEP55beH0p70uQlxO6FOveD DYkzFbdurXLHk6Yyu7WdLN7KoMYJEBVSJdOMJOGDEcYu13XlGHMTINGK0 QvQ2P2R3TTmJfz8hFmQgFHjM0a1Bwd1uHMPve+v6L4FRSA9zfyG8+Kyw9 /IfOFNwqg4xVOhTMK4BuKN5XIQKK7hKSE0ryfHveYlRUltm+v0HvMiblj wGTKFb/gyY2UULm/I4/XzOlWZYQ4baOBrnx6h3RUURXtTKkggC+0PEChC A==; X-CSE-ConnectionGUID: u5y8JzJNQ4anpVSeM/YUmg== X-CSE-MsgGUID: VOD6S7GvQCWczkWH5R4ffQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012244" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012244" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:37 -0700 X-CSE-ConnectionGUID: t55zwdmgQM2GBsUAiQQbqQ== X-CSE-MsgGUID: xtEOuPuuSdm/olJvDmXE7g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419788" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:36 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson Subject: [PATCH v6 31/33] net/intel: support wider x86 vectors for Rx rearm Date: Mon, 9 Jun 2025 16:37:29 +0100 Message-ID: <1b4d678259cb18f3a688770b7a82a60268999700.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, for 32-byte descriptor format, only SSE instruction set is supported. Add implementation for AVX2 and AVX512 instruction sets. Since we are using Rx descriptor definitions from common code, we can just use the generic descriptor definition, as we only ever write the first 16 bytes of it, and the layout is always the same for that part. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- drivers/net/intel/common/rx_vec_x86.h | 368 ++++++++++++++------------ 1 file changed, 201 insertions(+), 167 deletions(-) diff --git a/drivers/net/intel/common/rx_vec_x86.h b/drivers/net/intel/common/rx_vec_x86.h index ecab8b30a6..160b91b5aa 100644 --- a/drivers/net/intel/common/rx_vec_x86.h +++ b/drivers/net/intel/common/rx_vec_x86.h @@ -43,206 +43,251 @@ _ci_rxq_rearm_get_bufs(struct ci_rx_queue *rxq) return 0; } -/* - * SSE code path can handle both 16-byte and 32-byte descriptors with one code - * path, as we only ever write 16 bytes at a time. +/** + * Reformat data from mbuf to descriptor for one RX descriptor, using SSE instruction set. + * + * @param mhdr pointer to first 16 bytes of mbuf header + * @return 16-byte register in descriptor format. */ -static __rte_always_inline void -_ci_rxq_rearm_sse(struct ci_rx_queue *rxq) +static __rte_always_inline __m128i +_ci_rxq_rearm_desc_sse(const __m128i *mhdr) { const __m128i hdroom = _mm_set1_epi64x(RTE_PKTMBUF_HEADROOM); const __m128i zero = _mm_setzero_si128(); + + /* add headroom to address values */ + __m128i reg = _mm_add_epi64(*mhdr, hdroom); + +#if RTE_IOVA_IN_MBUF + /* expect buf_addr (low 64 bit) and buf_iova (high 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + /* move IOVA to Packet Buffer Address, erase Header Buffer Address */ + reg = _mm_unpackhi_epi64(reg, zero); +#else + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_addr) != 0); + /* erase Header Buffer Address */ + reg = _mm_unpacklo_epi64(reg, zero); +#endif + return reg; +} + +static __rte_always_inline void +_ci_rxq_rearm_sse(struct ci_rx_queue *rxq) +{ const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + /* SSE writes 16-bytes regardless of descriptor size */ + const uint8_t desc_per_reg = 1; + const uint8_t desc_per_iter = desc_per_reg * 2; volatile union ci_rx_desc *rxdp; int i; rxdp = &rxq->rx_ring[rxq->rxrearm_start]; /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < rearm_thresh; i += 2, rxp += 2, rxdp += 2) { - struct rte_mbuf *mb0 = rxp[0].mbuf; - struct rte_mbuf *mb1 = rxp[1].mbuf; - -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - __m128i addr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - __m128i addr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - - /* add headroom to address values */ - addr0 = _mm_add_epi64(addr0, hdroom); - addr1 = _mm_add_epi64(addr1, hdroom); - -#if RTE_IOVA_IN_MBUF - /* move IOVA to Packet Buffer Address, erase Header Buffer Address */ - addr0 = _mm_unpackhi_epi64(addr0, zero); - addr0 = _mm_unpackhi_epi64(addr1, zero); -#else - /* erase Header Buffer Address */ - addr0 = _mm_unpacklo_epi64(addr0, zero); - addr1 = _mm_unpacklo_epi64(addr1, zero); -#endif - - /* flush desc with pa dma_addr */ - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[0]), addr0); - _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[1]), addr1); + for (i = 0; i < rearm_thresh; + i += desc_per_iter, + rxp += desc_per_iter, + rxdp += desc_per_iter) { + const __m128i reg0 = _ci_rxq_rearm_desc_sse( + RTE_CAST_PTR(const __m128i *, rxp[0].mbuf)); + const __m128i reg1 = _ci_rxq_rearm_desc_sse( + RTE_CAST_PTR(const __m128i *, rxp[1].mbuf)); + + /* flush descriptors */ + _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[0]), reg0); + _mm_store_si128(RTE_CAST_PTR(__m128i *, &rxdp[desc_per_reg]), reg1); } } -#ifdef RTE_NET_INTEL_USE_16BYTE_DESC #ifdef __AVX2__ -/* AVX2 version for 16-byte descriptors, handles 4 buffers at a time */ -static __rte_always_inline void -_ci_rxq_rearm_avx2(struct ci_rx_queue *rxq) +/** + * Reformat data from mbuf to descriptor for one RX descriptor, using AVX2 instruction set. + * + * Note that for 32-byte descriptors, the second parameter must be zeroed out. + * + * @param mhdr0 pointer to first 16-bytes of 1st mbuf header. + * @param mhdr1 pointer to first 16-bytes of 2nd mbuf header. + * + * @return 32-byte register with two 16-byte descriptors in it. + */ +static __rte_always_inline __m256i +_ci_rxq_rearm_desc_avx2(const __m128i *mhdr0, const __m128i *mhdr1) { - struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; - const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; - const __m256i hdroom = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); + const __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); const __m256i zero = _mm256_setzero_si256(); + + /* merge by casting 0 to 256-bit and inserting 1 into the high lanes */ + __m256i reg = _mm256_inserti128_si256(_mm256_castsi128_si256(*mhdr0), *mhdr1, 1); + + /* add headroom to address values */ + reg = _mm256_add_epi64(reg, hdr_room); + +#if RTE_IOVA_IN_MBUF + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + /* extract IOVA addr into Packet Buffer Address, erase Header Buffer Address */ + reg = _mm256_unpackhi_epi64(reg, zero); +#else + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_addr) != 0); + /* erase Header Buffer Address */ + reg = _mm256_unpacklo_epi64(reg, zero); +#endif + return reg; +} + +static __rte_always_inline void +_ci_rxq_rearm_avx2(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + /* how many descriptors can fit into a register */ + const uint8_t desc_per_reg = sizeof(__m256i) / sizeof(union ci_rx_desc); + /* how many descriptors can fit into one loop iteration */ + const uint8_t desc_per_iter = desc_per_reg * 2; volatile union ci_rx_desc *rxdp; int i; - RTE_BUILD_BUG_ON(sizeof(union ci_rx_desc) != 16); - rxdp = &rxq->rx_ring[rxq->rxrearm_start]; - /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < rearm_thresh; i += 4, rxp += 4, rxdp += 4) { - struct rte_mbuf *mb0 = rxp[0].mbuf; - struct rte_mbuf *mb1 = rxp[1].mbuf; - struct rte_mbuf *mb2 = rxp[2].mbuf; - struct rte_mbuf *mb3 = rxp[3].mbuf; + /* Initialize the mbufs in vector, process 2 or 4 mbufs in one loop */ + for (i = 0; i < rearm_thresh; + i += desc_per_iter, + rxp += desc_per_iter, + rxdp += desc_per_iter) { + __m256i reg0, reg1; -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - const __m128i vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - const __m128i vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - const __m128i vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - const __m128i vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + if (desc_per_iter == 2) { + /* no need to call AVX2 version as we only need two descriptors */ + reg0 = _mm256_castsi128_si256( + _ci_rxq_rearm_desc_sse( + RTE_CAST_PTR(const __m128i *, &rxp[0].mbuf))); + reg1 = _mm256_castsi128_si256( + _ci_rxq_rearm_desc_sse( + RTE_CAST_PTR(const __m128i *, &rxp[1].mbuf))); + } else { + /* 16 byte descriptor times four */ + reg0 = _ci_rxq_rearm_desc_avx2( + RTE_CAST_PTR(const __m128i *, &rxp[0].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[1].mbuf)); + reg1 = _ci_rxq_rearm_desc_avx2( + RTE_CAST_PTR(const __m128i *, &rxp[2].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[3].mbuf)); + } - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3 - */ - const __m256i vaddr0_256 = _mm256_castsi128_si256(vaddr0); - const __m256i vaddr2_256 = _mm256_castsi128_si256(vaddr2); - - __m256i addr0_1 = _mm256_inserti128_si256(vaddr0_256, vaddr1, 1); - __m256i addr2_3 = _mm256_inserti128_si256(vaddr2_256, vaddr3, 1); - - /* add headroom to address values */ - addr0_1 = _mm256_add_epi64(addr0_1, hdroom); - addr0_1 = _mm256_add_epi64(addr0_1, hdroom); - -#if RTE_IOVA_IN_MBUF - /* extract IOVA addr into Packet Buffer Address, erase Header Buffer Address */ - addr0_1 = _mm256_unpackhi_epi64(addr0_1, zero); - addr2_3 = _mm256_unpackhi_epi64(addr2_3, zero); -#else - /* erase Header Buffer Address */ - addr0_1 = _mm256_unpacklo_epi64(addr0_1, zero); - addr2_3 = _mm256_unpacklo_epi64(addr2_3, zero); -#endif - - /* flush desc with pa dma_addr */ - _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp[0]), addr0_1); - _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp[2]), addr2_3); + /* flush descriptors */ + _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp[0]), reg0); + _mm256_store_si256(RTE_CAST_PTR(__m256i *, &rxdp[desc_per_reg]), reg1); } } #endif /* __AVX2__ */ #ifdef __AVX512VL__ -/* AVX512 version for 16-byte descriptors, handles 8 buffers at a time */ -static __rte_always_inline void -_ci_rxq_rearm_avx512(struct ci_rx_queue *rxq) +/** + * Reformat data from mbuf to descriptor for one RX descriptor, using AVX512 instruction set. + * + * Note that for 32-byte descriptors, every second parameter must be zeroed out. + * + * @param mhdr0 pointer to first 16-bytes of 1st mbuf header. + * @param mhdr1 pointer to first 16-bytes of 2nd mbuf header. + * @param mhdr2 pointer to first 16-bytes of 3rd mbuf header. + * @param mhdr3 pointer to first 16-bytes of 4th mbuf header. + * + * @return 64-byte register with four 16-byte descriptors in it. + */ +static __rte_always_inline __m512i +_ci_rxq_rearm_desc_avx512(const __m128i *mhdr0, const __m128i *mhdr1, + const __m128i *mhdr2, const __m128i *mhdr3) { - struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; - const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; - const __m512i hdroom = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); const __m512i zero = _mm512_setzero_si512(); + const __m512i hdroom = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 into the high + * lanes. Similarly for 2 & 3. + */ + const __m256i vaddr0_1 = _mm256_inserti128_si256(_mm256_castsi128_si256(*mhdr0), *mhdr1, 1); + const __m256i vaddr2_3 = _mm256_inserti128_si256(_mm256_castsi128_si256(*mhdr2), *mhdr3, 1); + /* + * merge 0+1 & 2+3, by casting 0+1 to 512-bit and inserting 2+3 into the + * high lanes. + */ + __m512i reg = _mm512_inserti64x4(_mm512_castsi256_si512(vaddr0_1), vaddr2_3, 1); + + /* add headroom to address values */ + reg = _mm512_add_epi64(reg, hdroom); + +#if RTE_IOVA_IN_MBUF + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + /* extract IOVA addr into Packet Buffer Address, erase Header Buffer Address */ + reg = _mm512_unpackhi_epi64(reg, zero); +#else + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_addr) != 0); + /* erase Header Buffer Address */ + reg = _mm512_unpacklo_epi64(reg, zero); +#endif + return reg; +} + +static __rte_always_inline void +_ci_rxq_rearm_avx512(struct ci_rx_queue *rxq) +{ + struct ci_rx_entry *rxp = &rxq->sw_ring[rxq->rxrearm_start]; + const uint16_t rearm_thresh = CI_VPMD_RX_REARM_THRESH; + /* how many descriptors can fit into a register */ + const uint8_t desc_per_reg = sizeof(__m512i) / sizeof(union ci_rx_desc); + /* how many descriptors can fit into one loop iteration */ + const uint8_t desc_per_iter = desc_per_reg * 2; volatile union ci_rx_desc *rxdp; int i; - RTE_BUILD_BUG_ON(sizeof(union ci_rx_desc) != 16); - rxdp = &rxq->rx_ring[rxq->rxrearm_start]; - /* Initialize the mbufs in vector, process 8 mbufs in one loop */ - for (i = 0; i < rearm_thresh; i += 8, rxp += 8, rxdp += 8) { - struct rte_mbuf *mb0 = rxp[0].mbuf; - struct rte_mbuf *mb1 = rxp[1].mbuf; - struct rte_mbuf *mb2 = rxp[2].mbuf; - struct rte_mbuf *mb3 = rxp[3].mbuf; - struct rte_mbuf *mb4 = rxp[4].mbuf; - struct rte_mbuf *mb5 = rxp[5].mbuf; - struct rte_mbuf *mb6 = rxp[6].mbuf; - struct rte_mbuf *mb7 = rxp[7].mbuf; + /* Initialize the mbufs in vector, process 4 or 8 mbufs in one loop */ + for (i = 0; i < rearm_thresh; + i += desc_per_iter, + rxp += desc_per_iter, + rxdp += desc_per_iter) { + __m512i reg0, reg1; -#if RTE_IOVA_IN_MBUF - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); -#endif - const __m128i vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - const __m128i vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - const __m128i vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - const __m128i vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - const __m128i vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); - const __m128i vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); - const __m128i vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); - const __m128i vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); + if (desc_per_iter == 4) { + /* 16-byte descriptor, 16 byte zero, times four */ + const __m128i zero = _mm_setzero_si128(); - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3, and so on. - */ - const __m256i addr0_256 = _mm256_castsi128_si256(vaddr0); - const __m256i addr2_256 = _mm256_castsi128_si256(vaddr2); - const __m256i addr4_256 = _mm256_castsi128_si256(vaddr4); - const __m256i addr6_256 = _mm256_castsi128_si256(vaddr6); - - const __m256i addr0_1 = _mm256_inserti128_si256(addr0_256, vaddr1, 1); - const __m256i addr2_3 = _mm256_inserti128_si256(addr2_256, vaddr3, 1); - const __m256i addr4_5 = _mm256_inserti128_si256(addr4_256, vaddr5, 1); - const __m256i addr6_7 = _mm256_inserti128_si256(addr6_256, vaddr7, 1); - - /** - * merge 0_1 & 2_3, by casting 0_1 to 512-bit and inserting 2_3 - * into the high lanes. Similarly for 4_5 & 6_7, and so on. - */ - const __m512i addr0_1_512 = _mm512_castsi256_si512(addr0_1); - const __m512i addr4_5_512 = _mm512_castsi256_si512(addr4_5); - - __m512i addr0_3 = _mm512_inserti64x4(addr0_1_512, addr2_3, 1); - __m512i addr4_7 = _mm512_inserti64x4(addr4_5_512, addr6_7, 1); - - /* add headroom to address values */ - addr0_3 = _mm512_add_epi64(addr0_3, hdroom); - addr4_7 = _mm512_add_epi64(addr4_7, hdroom); - -#if RTE_IOVA_IN_MBUF - /* extract IOVA addr into Packet Buffer Address, erase Header Buffer Address */ - addr0_3 = _mm512_unpackhi_epi64(addr0_3, zero); - addr4_7 = _mm512_unpackhi_epi64(addr4_7, zero); -#else - /* erase Header Buffer Address */ - addr0_3 = _mm512_unpacklo_epi64(addr0_3, zero); - addr4_7 = _mm512_unpacklo_epi64(addr4_7, zero); -#endif + reg0 = _ci_rxq_rearm_desc_avx512( + RTE_CAST_PTR(const __m128i *, &rxp[0].mbuf), + &zero, + RTE_CAST_PTR(const __m128i *, &rxp[1].mbuf), + &zero); + reg1 = _ci_rxq_rearm_desc_avx512( + RTE_CAST_PTR(const __m128i *, &rxp[2].mbuf), + &zero, + RTE_CAST_PTR(const __m128i *, &rxp[3].mbuf), + &zero); + } else { + /* 16-byte descriptor times eight */ + reg0 = _ci_rxq_rearm_desc_avx512( + RTE_CAST_PTR(const __m128i *, &rxp[0].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[1].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[2].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[3].mbuf)); + reg1 = _ci_rxq_rearm_desc_avx512( + RTE_CAST_PTR(const __m128i *, &rxp[4].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[5].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[6].mbuf), + RTE_CAST_PTR(const __m128i *, &rxp[7].mbuf)); + } /* flush desc with pa dma_addr */ - _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp[0]), addr0_3); - _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp[4]), addr4_7); + _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp[0]), reg0); + _mm512_store_si512(RTE_CAST_PTR(__m512i *, &rxdp[desc_per_reg]), reg1); } } #endif /* __AVX512VL__ */ -#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ /** * Rearm the RX queue with new buffers. @@ -266,7 +311,6 @@ ci_rxq_rearm(struct ci_rx_queue *rxq, const enum ci_rx_vec_level vec_level) if (_ci_rxq_rearm_get_bufs(rxq) < 0) return; -#ifdef RTE_NET_INTEL_USE_16BYTE_DESC switch (vec_level) { case CI_RX_VEC_LEVEL_AVX512: #ifdef __AVX512VL__ @@ -288,16 +332,6 @@ ci_rxq_rearm(struct ci_rx_queue *rxq, const enum ci_rx_vec_level vec_level) _ci_rxq_rearm_sse(rxq); break; } -#else - /* for 32-byte descriptors only support SSE */ - switch (vec_level) { - case CI_RX_VEC_LEVEL_AVX512: - case CI_RX_VEC_LEVEL_AVX2: - case CI_RX_VEC_LEVEL_SSE: - _ci_rxq_rearm_sse(rxq); - break; - } -#endif /* RTE_NET_INTEL_USE_16BYTE_DESC */ rxq->rxrearm_start += rearm_thresh; if (rxq->rxrearm_start >= rxq->nb_rx_desc) From patchwork Mon Jun 9 15:37:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154229 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CC005468B7; Mon, 9 Jun 2025 17:42:01 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 15FF042DDE; Mon, 9 Jun 2025 17:38:42 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id 078274270A for ; Mon, 9 Jun 2025 17:38:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483519; x=1781019519; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=rlQQpIrJMZYiI3cXw8YFGz8n2T28PgvdiR2UMl/B1TU=; b=j7leUksg/0sHQuO6Bm+/n2QBUe4SSA7DUS5/SDCqXzAxUaDnK/Tt6f2q beebSq9WbAx38F7ybdjP1P0g7HaxnDjsMDI5texARHJX5dvbGXoJI0d9g SS7kPvu6wiwrRbfiiRCl0SrbbzHGixx0EVBf2owy/ws0jOj5+cdD2Eguf WS3CkjinCJbZj5ULwp3NT8KrDelGZCQc0TIangeQZq1VqKngfmBrRtxcn r4wYCR6YR9uvVykOmyf4bjJ8ljlxtR/uX0IdqnJ6yoobJlnsC+B9TCiKB pKQFYAMsij0Tk/ux+MP4ooN9YVyrQpGZQm1cMPRBQoqBsPDSprA1tAmTf A==; X-CSE-ConnectionGUID: JOW4lO28QheA38Wo/SjiXg== X-CSE-MsgGUID: wzXmQ8XvTt26QG0mecVNQQ== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012245" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012245" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:39 -0700 X-CSE-ConnectionGUID: CMZauvF/Ru+wKAri9Fo+Pw== X-CSE-MsgGUID: pC33HSZEQEOdkxh3Bv0RMA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419796" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:37 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Ian Stokes , Vladimir Medvedkin Subject: [PATCH v6 32/33] net/intel: add common Rx mbuf recycle Date: Mon, 9 Jun 2025 16:37:30 +0100 Message-ID: X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, there are duplicate implementations of Rx mbuf recycle in some drivers, specifically ixgbe and i40e. Move them into a common header. While we're at it, also support no-IOVA-in-mbuf case. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v5: - Renamed paddr to iova drivers/net/intel/common/recycle_mbufs.h | 68 +++++++++++++++++++ .../i40e/i40e_recycle_mbufs_vec_common.c | 37 +--------- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.c | 35 +--------- 3 files changed, 74 insertions(+), 66 deletions(-) create mode 100644 drivers/net/intel/common/recycle_mbufs.h diff --git a/drivers/net/intel/common/recycle_mbufs.h b/drivers/net/intel/common/recycle_mbufs.h new file mode 100644 index 0000000000..1aea611c80 --- /dev/null +++ b/drivers/net/intel/common/recycle_mbufs.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2025 Intel Corporation + */ + + #ifndef _COMMON_INTEL_RECYCLE_MBUFS_H_ +#define _COMMON_INTEL_RECYCLE_MBUFS_H_ + +#include +#include + +#include +#include +#include + +#include "rx.h" +#include "tx.h" + +/** + * Recycle mbufs for Rx queue. + * + * @param rxq Rx queue pointer + * @param nb_mbufs number of mbufs to recycle + * @param desc_len length of Rx descriptor + */ +static __rte_always_inline void +ci_rx_recycle_mbufs(struct ci_rx_queue *rxq, const uint16_t nb_mbufs) +{ + struct ci_rx_entry *rxep; + volatile union ci_rx_desc *rxdp; + uint16_t rx_id; + uint16_t i; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + for (i = 0; i < nb_mbufs; i++) { + struct rte_mbuf *mb = rxep[i].mbuf; + +#if RTE_IOVA_IN_MBUF + const uint64_t iova = mb->buf_iova + RTE_PKTMBUF_HEADROOM; + const uint64_t dma_addr = rte_cpu_to_le_64(iova); +#else + const uint64_t vaddr = (uintptr_t)mb->buf_addr + RTE_PKTMBUF_HEADROOM; + const uint64_t dma_addr = rte_cpu_to_le_64(vaddr); +#endif + + rxdp[i].read.hdr_addr = 0; + rxdp[i].read.pkt_addr = dma_addr; + } + + /* Update the descriptor initializer index */ + rxq->rxrearm_start += nb_mbufs; + rx_id = rxq->rxrearm_start - 1; + + if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { + rxq->rxrearm_start = 0; + rx_id = rxq->nb_rx_desc - 1; + } + + rxq->rxrearm_nb -= nb_mbufs; + + rte_io_wmb(); + + /* Update the tail pointer on the NIC */ + rte_write32_wc_relaxed(rte_cpu_to_le_32(rx_id), rxq->qrx_tail); +} + +#endif diff --git a/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c b/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c index 20d9fd7b22..0b036faea9 100644 --- a/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c +++ b/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c @@ -10,43 +10,12 @@ #include "i40e_ethdev.h" #include "i40e_rxtx.h" +#include "../common/recycle_mbufs.h" + void i40e_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) { - struct ci_rx_queue *rxq = rx_queue; - struct ci_rx_entry *rxep; - volatile union ci_rx_desc *rxdp; - uint16_t rx_id; - uint64_t paddr; - uint64_t dma_addr; - uint16_t i; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - rxep = &rxq->sw_ring[rxq->rxrearm_start]; - - for (i = 0; i < nb_mbufs; i++) { - /* Initialize rxdp descs. */ - paddr = (rxep[i].mbuf)->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr = rte_cpu_to_le_64(paddr); - /* flush desc with pa dma_addr */ - rxdp[i].read.hdr_addr = 0; - rxdp[i].read.pkt_addr = dma_addr; - } - - /* Update the descriptor initializer index */ - rxq->rxrearm_start += nb_mbufs; - rx_id = rxq->rxrearm_start - 1; - - if (unlikely(rxq->rxrearm_start >= rxq->nb_rx_desc)) { - rxq->rxrearm_start = 0; - rx_id = rxq->nb_rx_desc - 1; - } - - rxq->rxrearm_nb -= nb_mbufs; - - rte_io_wmb(); - /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE_RELAXED(rxq->qrx_tail, rx_id); + ci_rx_recycle_mbufs(rx_queue, nb_mbufs); } uint16_t diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c index 5f231b9012..486dae4178 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c @@ -10,6 +10,8 @@ #include "ixgbe_rxtx.h" #include "ixgbe_rxtx_vec_common.h" +#include "../common/recycle_mbufs.h" + void __rte_cold ixgbe_tx_free_swring_vec(struct ci_tx_queue *txq) { @@ -173,38 +175,7 @@ ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, void ixgbe_recycle_rx_descriptors_refill_vec(void *rx_queue, uint16_t nb_mbufs) { - struct ci_rx_queue *rxq = rx_queue; - struct ci_rx_entry *rxep; - volatile union ixgbe_adv_rx_desc *rxdp; - uint16_t rx_id; - uint64_t paddr; - uint64_t dma_addr; - uint16_t i; - - rxdp = rxq->ixgbe_rx_ring + rxq->rxrearm_start; - rxep = &rxq->sw_ring[rxq->rxrearm_start]; - - for (i = 0; i < nb_mbufs; i++) { - /* Initialize rxdp descs. */ - paddr = (rxep[i].mbuf)->buf_iova + RTE_PKTMBUF_HEADROOM; - dma_addr = rte_cpu_to_le_64(paddr); - /* Flush descriptors with pa dma_addr */ - rxdp[i].read.hdr_addr = 0; - rxdp[i].read.pkt_addr = dma_addr; - } - - /* Update the descriptor initializer index */ - rxq->rxrearm_start += nb_mbufs; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= nb_mbufs; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ci_rx_recycle_mbufs(rx_queue, nb_mbufs); } uint16_t From patchwork Mon Jun 9 15:37:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Burakov, Anatoly" X-Patchwork-Id: 154230 X-Patchwork-Delegate: bruce.richardson@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A136468B7; Mon, 9 Jun 2025 17:42:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4E73F42DE9; Mon, 9 Jun 2025 17:38:43 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by mails.dpdk.org (Postfix) with ESMTP id C9B7D41140 for ; Mon, 9 Jun 2025 17:38:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1749483521; x=1781019521; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=XJyRj2alMDn+zNBHiZFEEswt18XCJB41yALuTp2ytWI=; b=BGN2QhPUQWyB2ZdmlU/5yRsHr3yY7uunifbuVMLgChow92Pqh60SzzJz Ss8rkcKLxliyukt6mtG7sZexPmLMbZSgrPLK+azzWN+x1wl+8cxcRhGG9 29aW4p9YFVWnRm88IW8PCeKNpgsKUvLPsTTVMxPd9UT3QIAXM5XcvR9md kLbQ2dv4H2dbewsDNMkKCzncKjcuG4HV4RltmT3ikIxIm5tmlA7DvjGwm ZM6P3h+U9WKXK/X81w2RP9viRrcV9egjlYwmdrveOeKVTHrPIuZpbwz3A siRUd3ldFl4tsnKBnA7LTwgASwrFA8cQnHYEWMhsCQnTJDJNEn5isq3qk Q==; X-CSE-ConnectionGUID: k9of6qIzRiCSpCfLDMHRzg== X-CSE-MsgGUID: efvGgzazTuO+JmkuMyIYDw== X-IronPort-AV: E=McAfee;i="6800,10657,11459"; a="69012247" X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="69012247" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jun 2025 08:38:41 -0700 X-CSE-ConnectionGUID: GXdNHXOyT2ikubytK8d4dw== X-CSE-MsgGUID: YEI0mEofTCSkhCE8LZw+wQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,222,1744095600"; d="scan'208";a="151419811" Received: from silpixa00401119.ir.intel.com ([10.55.129.167]) by fmviesa005.fm.intel.com with ESMTP; 09 Jun 2025 08:38:39 -0700 From: Anatoly Burakov To: dev@dpdk.org, Bruce Richardson , Ian Stokes , Vladimir Medvedkin Subject: [PATCH v6 33/33] net/intel: add common Tx mbuf recycle Date: Mon, 9 Jun 2025 16:37:31 +0100 Message-ID: <3754cb667620b2c419dff4f70c625d3e8b55be1c.1749483382.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Currently, there are duplicate implementations of Tx mbuf recycle in some drivers, specifically ixgbe and i40e. Move them into a common header. Signed-off-by: Anatoly Burakov Acked-by: Bruce Richardson --- Notes: v3 -> v4: - Use the common desc_done function to check for DD bit status - Add a desc_done implementation for ixgbe drivers/net/intel/common/recycle_mbufs.h | 105 ++++++++++++++++++ .../i40e/i40e_recycle_mbufs_vec_common.c | 90 +-------------- .../net/intel/ixgbe/ixgbe_rxtx_vec_common.c | 88 +-------------- 3 files changed, 109 insertions(+), 174 deletions(-) diff --git a/drivers/net/intel/common/recycle_mbufs.h b/drivers/net/intel/common/recycle_mbufs.h index 1aea611c80..fbe09eb5d0 100644 --- a/drivers/net/intel/common/recycle_mbufs.h +++ b/drivers/net/intel/common/recycle_mbufs.h @@ -65,4 +65,109 @@ ci_rx_recycle_mbufs(struct ci_rx_queue *rxq, const uint16_t nb_mbufs) rte_write32_wc_relaxed(rte_cpu_to_le_32(rx_id), rxq->qrx_tail); } +/** + * Recycle buffers on Tx. + * + * @param txq Tx queue pointer + * @param desc_done function to check if the Tx descriptor is done + * @param recycle_rxq_info recycling mbuf information + * + * @return how many buffers were recycled + */ +static __rte_always_inline uint16_t +ci_tx_recycle_mbufs(struct ci_tx_queue *txq, ci_desc_done_fn desc_done, + struct rte_eth_recycle_rxq_info *recycle_rxq_info) +{ + struct ci_tx_entry *txep; + struct rte_mbuf **rxep; + int i, n; + uint16_t nb_recycle_mbufs; + uint16_t avail = 0; + uint16_t mbuf_ring_size = recycle_rxq_info->mbuf_ring_size; + uint16_t mask = recycle_rxq_info->mbuf_ring_size - 1; + uint16_t refill_requirement = recycle_rxq_info->refill_requirement; + uint16_t refill_head = *recycle_rxq_info->refill_head; + uint16_t receive_tail = *recycle_rxq_info->receive_tail; + + /* Get available recycling Rx buffers. */ + avail = (mbuf_ring_size - (refill_head - receive_tail)) & mask; + + /* Check Tx free thresh and Rx available space. */ + if (txq->nb_tx_free > txq->tx_free_thresh || avail <= txq->tx_rs_thresh) + return 0; + + if (!desc_done(txq, txq->tx_next_dd)) { + /* If the Tx descriptor is not done, we can not recycle + * buffers. + */ + return 0; + } + + n = txq->tx_rs_thresh; + nb_recycle_mbufs = n; + + /* Mbufs recycle mode can only support no ring buffer wrapping around. + * Two case for this: + * + * case 1: The refill head of Rx buffer ring needs to be aligned with + * mbuf ring size. In this case, the number of Tx freeing buffers + * should be equal to refill_requirement. + * + * case 2: The refill head of Rx ring buffer does not need to be aligned + * with mbuf ring size. In this case, the update of refill head can not + * exceed the Rx mbuf ring size. + */ + if ((refill_requirement && refill_requirement != n) || + (!refill_requirement && (refill_head + n > mbuf_ring_size))) + return 0; + + /* First buffer to free from S/W ring is at index + * tx_next_dd - (tx_rs_thresh-1). + */ + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; + rxep = recycle_rxq_info->mbuf_ring; + rxep += refill_head; + + /* is fast-free enabled in offloads? */ + if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { + /* Avoid txq containing buffers from unexpected mempool. */ + if (unlikely(recycle_rxq_info->mp + != txep[0].mbuf->pool)) + return 0; + + /* Directly put mbufs from Tx to Rx. */ + for (i = 0; i < n; i++) + rxep[i] = txep[i].mbuf; + } else { + for (i = 0; i < n; i++) { + rxep[i] = rte_pktmbuf_prefree_seg(txep[i].mbuf); + + /* If Tx buffers are not the last reference or from + * unexpected mempool, previous copied buffers are + * considered as invalid. + */ + if (unlikely(rxep[i] == NULL || + recycle_rxq_info->mp != txep[i].mbuf->pool)) + nb_recycle_mbufs = 0; + } + /* If Tx buffers are not the last reference or + * from unexpected mempool, all recycled buffers + * are put into mempool. + */ + if (nb_recycle_mbufs == 0) + for (i = 0; i < n; i++) { + if (rxep[i] != NULL) + rte_mempool_put(rxep[i]->pool, rxep[i]); + } + } + + /* Update counters for Tx. */ + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); + if (txq->tx_next_dd >= txq->nb_tx_desc) + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); + + return nb_recycle_mbufs; +} + #endif diff --git a/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c b/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c index 0b036faea9..5faaff28c4 100644 --- a/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c +++ b/drivers/net/intel/i40e/i40e_recycle_mbufs_vec_common.c @@ -10,6 +10,8 @@ #include "i40e_ethdev.h" #include "i40e_rxtx.h" +#include "i40e_rxtx_vec_common.h" + #include "../common/recycle_mbufs.h" void @@ -23,92 +25,6 @@ i40e_recycle_tx_mbufs_reuse_vec(void *tx_queue, struct rte_eth_recycle_rxq_info *recycle_rxq_info) { struct ci_tx_queue *txq = tx_queue; - struct ci_tx_entry *txep; - struct rte_mbuf **rxep; - int i, n; - uint16_t nb_recycle_mbufs; - uint16_t avail = 0; - uint16_t mbuf_ring_size = recycle_rxq_info->mbuf_ring_size; - uint16_t mask = recycle_rxq_info->mbuf_ring_size - 1; - uint16_t refill_requirement = recycle_rxq_info->refill_requirement; - uint16_t refill_head = *recycle_rxq_info->refill_head; - uint16_t receive_tail = *recycle_rxq_info->receive_tail; - /* Get available recycling Rx buffers. */ - avail = (mbuf_ring_size - (refill_head - receive_tail)) & mask; - - /* Check Tx free thresh and Rx available space. */ - if (txq->nb_tx_free > txq->tx_free_thresh || avail <= txq->tx_rs_thresh) - return 0; - - /* check DD bits on threshold descriptor */ - if ((txq->i40e_tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & - rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != - rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) - return 0; - - n = txq->tx_rs_thresh; - nb_recycle_mbufs = n; - - /* Mbufs recycle mode can only support no ring buffer wrapping around. - * Two case for this: - * - * case 1: The refill head of Rx buffer ring needs to be aligned with - * mbuf ring size. In this case, the number of Tx freeing buffers - * should be equal to refill_requirement. - * - * case 2: The refill head of Rx ring buffer does not need to be aligned - * with mbuf ring size. In this case, the update of refill head can not - * exceed the Rx mbuf ring size. - */ - if ((refill_requirement && refill_requirement != n) || - (!refill_requirement && (refill_head + n > mbuf_ring_size))) - return 0; - - /* First buffer to free from S/W ring is at index - * tx_next_dd - (tx_rs_thresh-1). - */ - txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; - rxep = recycle_rxq_info->mbuf_ring; - rxep += refill_head; - - if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { - /* Avoid txq contains buffers from unexpected mempool. */ - if (unlikely(recycle_rxq_info->mp - != txep[0].mbuf->pool)) - return 0; - - /* Directly put mbufs from Tx to Rx. */ - for (i = 0; i < n; i++) - rxep[i] = txep[i].mbuf; - } else { - for (i = 0; i < n; i++) { - rxep[i] = rte_pktmbuf_prefree_seg(txep[i].mbuf); - - /* If Tx buffers are not the last reference or from - * unexpected mempool, previous copied buffers are - * considered as invalid. - */ - if (unlikely(rxep[i] == NULL || - recycle_rxq_info->mp != txep[i].mbuf->pool)) - nb_recycle_mbufs = 0; - } - /* If Tx buffers are not the last reference or - * from unexpected mempool, all recycled buffers - * are put into mempool. - */ - if (nb_recycle_mbufs == 0) - for (i = 0; i < n; i++) { - if (rxep[i] != NULL) - rte_mempool_put(rxep[i]->pool, rxep[i]); - } - } - - /* Update counters for Tx. */ - txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); - txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); - if (txq->tx_next_dd >= txq->nb_tx_desc) - txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); - - return nb_recycle_mbufs; + return ci_tx_recycle_mbufs(txq, i40e_tx_desc_done, recycle_rxq_info); } diff --git a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c index 486dae4178..94fbde1de2 100644 --- a/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c +++ b/drivers/net/intel/ixgbe/ixgbe_rxtx_vec_common.c @@ -182,91 +182,5 @@ uint16_t ixgbe_recycle_tx_mbufs_reuse_vec(void *tx_queue, struct rte_eth_recycle_rxq_info *recycle_rxq_info) { - struct ci_tx_queue *txq = tx_queue; - struct ci_tx_entry *txep; - struct rte_mbuf **rxep; - int i, n; - uint16_t nb_recycle_mbufs; - uint16_t avail = 0; - uint16_t mbuf_ring_size = recycle_rxq_info->mbuf_ring_size; - uint16_t mask = recycle_rxq_info->mbuf_ring_size - 1; - uint16_t refill_requirement = recycle_rxq_info->refill_requirement; - uint16_t refill_head = *recycle_rxq_info->refill_head; - uint16_t receive_tail = *recycle_rxq_info->receive_tail; - - /* Get available recycling Rx buffers. */ - avail = (mbuf_ring_size - (refill_head - receive_tail)) & mask; - - /* Check Tx free thresh and Rx available space. */ - if (txq->nb_tx_free > txq->tx_free_thresh || avail <= txq->tx_rs_thresh) - return 0; - - /* check DD bits on threshold descriptor */ - if (!ixgbe_tx_desc_done(txq, txq->tx_next_dd)) - return 0; - - n = txq->tx_rs_thresh; - nb_recycle_mbufs = n; - - /* Mbufs recycle can only support no ring buffer wrapping around. - * Two case for this: - * - * case 1: The refill head of Rx buffer ring needs to be aligned with - * buffer ring size. In this case, the number of Tx freeing buffers - * should be equal to refill_requirement. - * - * case 2: The refill head of Rx ring buffer does not need to be aligned - * with buffer ring size. In this case, the update of refill head can not - * exceed the Rx buffer ring size. - */ - if ((refill_requirement && refill_requirement != n) || - (!refill_requirement && (refill_head + n > mbuf_ring_size))) - return 0; - - /* First buffer to free from S/W ring is at index - * tx_next_dd - (tx_rs_thresh-1). - */ - txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; - rxep = recycle_rxq_info->mbuf_ring; - rxep += refill_head; - - if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { - /* Avoid txq contains buffers from unexpected mempool. */ - if (unlikely(recycle_rxq_info->mp - != txep[0].mbuf->pool)) - return 0; - - /* Directly put mbufs from Tx to Rx. */ - for (i = 0; i < n; i++) - rxep[i] = txep[i].mbuf; - } else { - for (i = 0; i < n; i++) { - rxep[i] = rte_pktmbuf_prefree_seg(txep[i].mbuf); - - /* If Tx buffers are not the last reference or from - * unexpected mempool, previous copied buffers are - * considered as invalid. - */ - if (unlikely(rxep[i] == NULL || - recycle_rxq_info->mp != txep[i].mbuf->pool)) - nb_recycle_mbufs = 0; - } - /* If Tx buffers are not the last reference or - * from unexpected mempool, all recycled buffers - * are put into mempool. - */ - if (nb_recycle_mbufs == 0) - for (i = 0; i < n; i++) { - if (rxep[i] != NULL) - rte_mempool_put(rxep[i]->pool, rxep[i]); - } - } - - /* Update counters for Tx. */ - txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); - txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); - if (txq->tx_next_dd >= txq->nb_tx_desc) - txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); - - return nb_recycle_mbufs; + return ci_tx_recycle_mbufs(tx_queue, ixgbe_tx_desc_done, recycle_rxq_info); }