From patchwork Wed Jun 14 01:49:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhichao Zeng X-Patchwork-Id: 128643 X-Patchwork-Delegate: qi.z.zhang@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 930D142CB0; Wed, 14 Jun 2023 03:43:43 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7484C42670; Wed, 14 Jun 2023 03:43:41 +0200 (CEST) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 5021642670 for ; Wed, 14 Jun 2023 03:43:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686707019; x=1718243019; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j9TK15it5jsqiKzlYeSwaip6aYX/1Ck8TrA7/4Dq9DA=; b=Z5GScAbVTIITFBbhEUUzXS+mOWDwaOtWPZxv4uTg5CpgovuyR6E+tK9w 5k0k+1LQWHHavoQG7Xdt6WIh2iGnGC5jU6fMOvZ1o4AoLEgibV9vlmJp3 FZd85l/OQxw6zURR53Dqw1A6GTvvx/Tqgu63o4+n06zlpm0Lmch0Bec75 zH4ts/S+P782iHvHEkTLiQVUusfD63O4pJHGc3z/J7BOR/iJuUXR4u0ye glTUjnxdrGYw7dw3JQUjQ4mZvo83f0j2A7CmFrmCxIse+Iiq1GVEjRFwM G+mII7mZGOLitd196l+sRpxfyN19qriut8Li/emiJHaXuo4IMCbVg2tET A==; X-IronPort-AV: E=McAfee;i="6600,9927,10740"; a="422095142" X-IronPort-AV: E=Sophos;i="6.00,241,1681196400"; d="scan'208";a="422095142" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2023 18:43:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10740"; a="741655338" X-IronPort-AV: E=Sophos;i="6.00,241,1681196400"; d="scan'208";a="741655338" Received: from unknown (HELO zhichao-dpdk..) ([10.239.252.103]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2023 18:43:35 -0700 From: Zhichao Zeng To: dev@dpdk.org Cc: qi.z.zhang@intel.com, yaqi.tang@intel.com, Zhichao Zeng , Jingjing Wu , Beilei Xing , Bruce Richardson , Konstantin Ananyev Subject: [PATCH v5 1/3] net/iavf: support Rx timestamp offload on AVX512 Date: Wed, 14 Jun 2023 09:49:46 +0800 Message-Id: <20230614014948.3495063-2-zhichaox.zeng@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230614014948.3495063-1-zhichaox.zeng@intel.com> References: <20230526095055.2855121-1-zhichaox.zeng@intel.com> <20230614014948.3495063-1-zhichaox.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch enables Rx timestamp offload on the AVX512 data path. It significantly reduces the performance drop when RTE_ETH_RX_OFFLOAD_TIMESTAMP is enabled. Signed-off-by: Zhichao Zeng --- v5: fix CI errors --- v4: rework avx2 patch based on offload path --- v3: logging with driver dedicated macro --- v2: fix compile warning --- drivers/net/iavf/iavf_rxtx.h | 3 +- drivers/net/iavf/iavf_rxtx_vec_avx512.c | 208 +++++++++++++++++++++++- drivers/net/iavf/iavf_rxtx_vec_common.h | 3 - 3 files changed, 205 insertions(+), 9 deletions(-) diff --git a/drivers/net/iavf/iavf_rxtx.h b/drivers/net/iavf/iavf_rxtx.h index 547b68f441..0345a6a51d 100644 --- a/drivers/net/iavf/iavf_rxtx.h +++ b/drivers/net/iavf/iavf_rxtx.h @@ -47,7 +47,8 @@ RTE_ETH_RX_OFFLOAD_CHECKSUM | \ RTE_ETH_RX_OFFLOAD_SCTP_CKSUM | \ RTE_ETH_RX_OFFLOAD_VLAN | \ - RTE_ETH_RX_OFFLOAD_RSS_HASH) + RTE_ETH_RX_OFFLOAD_RSS_HASH | \ + RTE_ETH_RX_OFFLOAD_TIMESTAMP) /** * According to the vlan capabilities returned by the driver and FW, the vlan tci diff --git a/drivers/net/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/iavf/iavf_rxtx_vec_avx512.c index bd2788121b..3e66df5341 100644 --- a/drivers/net/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/iavf/iavf_rxtx_vec_avx512.c @@ -16,18 +16,20 @@ /****************************************************************************** * If user knows a specific offload is not enabled by APP, * the macro can be commented to save the effort of fast path. - * Currently below 2 features are supported in RX path, + * Currently below 6 features are supported in RX path, * 1, checksum offload * 2, VLAN/QINQ stripping * 3, RSS hash * 4, packet type analysis * 5, flow director ID report + * 6, timestamp offload ******************************************************************************/ #define IAVF_RX_CSUM_OFFLOAD #define IAVF_RX_VLAN_OFFLOAD #define IAVF_RX_RSS_OFFLOAD #define IAVF_RX_PTYPE_OFFLOAD #define IAVF_RX_FDIR_OFFLOAD +#define IAVF_RX_TS_OFFLOAD static __rte_always_inline void iavf_rxq_rearm(struct iavf_rx_queue *rxq) @@ -587,9 +589,9 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, bool offload) { struct iavf_adapter *adapter = rxq->vsi->adapter; - +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; - +#endif #ifdef IAVF_RX_PTYPE_OFFLOAD const uint32_t *type_table = adapter->ptype_tbl; #endif @@ -618,6 +620,25 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, rte_cpu_to_le_32(1 << IAVF_RX_FLEX_DESC_STATUS0_DD_S))) return 0; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC +#ifdef IAVF_RX_TS_OFFLOAD + uint8_t inflection_point = 0; + bool is_tsinit = false; + __m256i hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, (uint32_t)rxq->phc_time); + + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); + + if (unlikely(sw_cur_time - rxq->hw_time_update > 4)) { + hw_low_last = _mm256_setzero_si256(); + is_tsinit = 1; + } else { + hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, (uint32_t)rxq->phc_time); + } + } +#endif +#endif + /* constants used in processing loop */ const __m512i crc_adjust = _mm512_set_epi32 @@ -1081,12 +1102,13 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, #ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC if (offload) { -#ifdef IAVF_RX_RSS_OFFLOAD +#if defined(IAVF_RX_RSS_OFFLOAD) || defined(IAVF_RX_TS_OFFLOAD) /** * needs to load 2nd 16B of each desc for RSS hash parsing, * will cause performance drop to get into this context. */ if (offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH || + offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP || rxq->rx_flags & IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2) { /* load bottom half of every 32B desc */ const __m128i raw_desc_bh7 = @@ -1138,6 +1160,7 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, (_mm256_castsi128_si256(raw_desc_bh0), raw_desc_bh1, 1); +#ifdef IAVF_RX_RSS_OFFLOAD if (offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH) { /** * to shift the 32b RSS hash value to the @@ -1278,7 +1301,125 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, mb0_1 = _mm256_or_si256 (mb0_1, vlan_tci0_1); } - } /* if() on RSS hash parsing */ +#endif /* IAVF_RX_RSS_OFFLOAD */ + +#ifdef IAVF_RX_TS_OFFLOAD + if (offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + uint32_t mask = 0xFFFFFFFF; + __m256i ts; + __m256i ts_low = _mm256_setzero_si256(); + __m256i ts_low1; + __m256i ts_low2; + __m256i max_ret; + __m256i cmp_ret; + uint8_t ret = 0; + uint8_t shift = 8; + __m256i ts_desp_mask = _mm256_set_epi32(mask, 0, 0, 0, mask, 0, 0, 0); + __m256i cmp_mask = _mm256_set1_epi32(mask); + __m256i ts_permute_mask = _mm256_set_epi32(7, 3, 6, 2, 5, 1, 4, 0); + + ts = _mm256_and_si256(raw_desc_bh0_1, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, _mm256_srli_si256(ts, 3 * 4)); + ts = _mm256_and_si256(raw_desc_bh2_3, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, _mm256_srli_si256(ts, 2 * 4)); + ts = _mm256_and_si256(raw_desc_bh4_5, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, _mm256_srli_si256(ts, 4)); + ts = _mm256_and_si256(raw_desc_bh6_7, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, ts); + + ts_low1 = _mm256_permutevar8x32_epi32(ts_low, ts_permute_mask); + ts_low2 = _mm256_permutevar8x32_epi32(ts_low1, + _mm256_set_epi32(6, 5, 4, 3, 2, 1, 0, 7)); + ts_low2 = _mm256_and_si256(ts_low2, + _mm256_set_epi32(mask, mask, mask, mask, mask, mask, mask, 0)); + ts_low2 = _mm256_or_si256(ts_low2, hw_low_last); + hw_low_last = _mm256_and_si256(ts_low1, + _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, mask)); + + *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 0); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 1], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 1); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 2], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 2); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 3], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 3); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 4], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 4); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 5], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 5); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 6], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 6); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 7], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 7); + + if (unlikely(is_tsinit)) { + uint32_t in_timestamp; + + if (iavf_get_phc_time(rxq)) + PMD_DRV_LOG(ERR, "get physical time failed"); + in_timestamp = *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset, uint32_t *); + rxq->phc_time = iavf_tstamp_convert_32b_64b(rxq->phc_time, in_timestamp); + } + + *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 1], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 2], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 3], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 4], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 5], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 6], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 7], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + + max_ret = _mm256_max_epu32(ts_low2, ts_low1); + cmp_ret = _mm256_andnot_si256(_mm256_cmpeq_epi32(max_ret, ts_low1), cmp_mask); + + if (_mm256_testz_si256(cmp_ret, cmp_mask)) { + inflection_point = 0; + } else { + inflection_point = 1; + while (shift > 1) { + shift = shift >> 1; + __m256i mask_low = _mm256_setzero_si256(); + __m256i mask_high = _mm256_setzero_si256(); + switch (shift) { + case 4: + mask_low = _mm256_set_epi32(0, 0, 0, 0, mask, mask, mask, mask); + mask_high = _mm256_set_epi32(mask, mask, mask, mask, 0, 0, 0, 0); + break; + case 2: + mask_low = _mm256_srli_si256(cmp_mask, 2 * 4); + mask_high = _mm256_slli_si256(cmp_mask, 2 * 4); + break; + case 1: + mask_low = _mm256_srli_si256(cmp_mask, 1 * 4); + mask_high = _mm256_slli_si256(cmp_mask, 1 * 4); + break; + } + ret = _mm256_testz_si256(cmp_ret, mask_low); + if (ret) { + ret = _mm256_testz_si256(cmp_ret, mask_high); + inflection_point += ret ? 0 : shift; + cmp_mask = mask_high; + } else { + cmp_mask = mask_low; + } + } + } + mbuf_flags = _mm256_or_si256(mbuf_flags, + _mm256_set1_epi32(iavf_timestamp_dynflag)); + } +#endif /* IAVF_RX_TS_OFFLOAD */ + } /* if() on RSS hash or RX timestamp parsing */ #endif } #endif @@ -1411,10 +1552,67 @@ _iavf_recv_raw_pkts_vec_avx512_flex_rxd(struct iavf_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC +#ifdef IAVF_RX_TS_OFFLOAD + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + inflection_point = (inflection_point <= burst) ? inflection_point : 0; + switch (inflection_point) { + case 1: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 2: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 1], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 3: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 2], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 4: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 3], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 5: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 4], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 6: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 5], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 7: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 6], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 8: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 7], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + rxq->phc_time += (uint64_t)1 << 32; + /* fallthrough */ + case 0: + break; + default: + PMD_DRV_LOG(ERR, "invalid inflection point for rx timestamp"); + break; + } + + rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); + } +#endif +#endif if (burst != IAVF_DESCS_PER_LOOP_AVX) break; } +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC +#ifdef IAVF_RX_TS_OFFLOAD + if (received > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) + rxq->phc_time = *RTE_MBUF_DYNFIELD(rx_pkts[received - 1], + iavf_timestamp_dynfield_offset, rte_mbuf_timestamp_t *); +#endif +#endif + /* update tail pointers */ rxq->rx_tail += received; rxq->rx_tail &= (rxq->nb_rx_desc - 1); diff --git a/drivers/net/iavf/iavf_rxtx_vec_common.h b/drivers/net/iavf/iavf_rxtx_vec_common.h index cc38f70ce2..ddb13ce8c3 100644 --- a/drivers/net/iavf/iavf_rxtx_vec_common.h +++ b/drivers/net/iavf/iavf_rxtx_vec_common.h @@ -231,9 +231,6 @@ iavf_rx_vec_queue_default(struct iavf_rx_queue *rxq) if (rxq->proto_xtr != IAVF_PROTO_XTR_NONE) return -1; - if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) - return -1; - if (rxq->offloads & IAVF_RX_VECTOR_OFFLOAD) return IAVF_VECTOR_OFFLOAD_PATH; From patchwork Wed Jun 14 01:49:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhichao Zeng X-Patchwork-Id: 128644 X-Patchwork-Delegate: qi.z.zhang@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0139342CB0; Wed, 14 Jun 2023 03:43:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CED4542D13; Wed, 14 Jun 2023 03:43:44 +0200 (CEST) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id C328942D12 for ; Wed, 14 Jun 2023 03:43:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686707023; x=1718243023; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZCzO2KGB91rV/SHqzgDxr3ws5Bm2+WRGHowPHCBQ5D8=; b=nMhLoDKADUWdqWZ37x1rdzs9zLLVw0wW8nJGbja0RkNmS9QFgpCWRhf5 fV/0CWzMnUd1EShG84vwBZvdcRVn7pgzCh0S+7bUWq3O2SaBb6hHfShsh jVZAi4JYNg84+LnSBqLKPMMw3aSh0BU7ryusX9bvyAu4PfcxY+DHAVpXU RPrVJto1luybNcbJeFR3+5hBiBqci/qml6NyOyA3lCR4aaJKRKEd9SJul GOhiU6uaEXVVjm05oGW2jHtH5au8CKYF95oGMqR8EJmL2mKWofmUAvxGS +0MTFBSYOnfMTHg/9A9jq0cxylRhfqDc4lyyRXjF/Onp9Fnr+YTmLmBnR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10740"; a="422095161" X-IronPort-AV: E=Sophos;i="6.00,241,1681196400"; d="scan'208";a="422095161" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2023 18:43:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10740"; a="741655346" X-IronPort-AV: E=Sophos;i="6.00,241,1681196400"; d="scan'208";a="741655346" Received: from unknown (HELO zhichao-dpdk..) ([10.239.252.103]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2023 18:43:40 -0700 From: Zhichao Zeng To: dev@dpdk.org Cc: qi.z.zhang@intel.com, yaqi.tang@intel.com, Zhichao Zeng , Bruce Richardson , Konstantin Ananyev , Jingjing Wu , Beilei Xing Subject: [PATCH v5 2/3] net/iavf: support Rx timestamp offload on AVX2 Date: Wed, 14 Jun 2023 09:49:47 +0800 Message-Id: <20230614014948.3495063-3-zhichaox.zeng@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230614014948.3495063-1-zhichaox.zeng@intel.com> References: <20230526095055.2855121-1-zhichaox.zeng@intel.com> <20230614014948.3495063-1-zhichaox.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch enables Rx timestamp offload on the AVX2 data path. It significantly reduces the performance drop when RTE_ETH_RX_OFFLOAD_TIMESTAMP is enabled. --- v5: fix CI errors --- v4: rework avx2 patch based on offload path --- v3: logging with driver dedicated macro --- v2: fix compile warning Signed-off-by: Zhichao Zeng --- drivers/net/iavf/iavf_rxtx_vec_avx2.c | 191 +++++++++++++++++++++++++- 1 file changed, 187 insertions(+), 4 deletions(-) diff --git a/drivers/net/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/iavf/iavf_rxtx_vec_avx2.c index c7f8b6ef71..c10f24036e 100644 --- a/drivers/net/iavf/iavf_rxtx_vec_avx2.c +++ b/drivers/net/iavf/iavf_rxtx_vec_avx2.c @@ -532,7 +532,9 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, struct iavf_adapter *adapter = rxq->vsi->adapter; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; +#endif const uint32_t *type_table = adapter->ptype_tbl; const __m256i mbuf_init = _mm256_set_epi64x(0, 0, @@ -558,6 +560,21 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, if (!(rxdp->wb.status_error0 & rte_cpu_to_le_32(1 << IAVF_RX_FLEX_DESC_STATUS0_DD_S))) return 0; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC + bool is_tsinit = false; + uint8_t inflection_point = 0; + __m256i hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, rxq->phc_time); + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); + + if (unlikely(sw_cur_time - rxq->hw_time_update > 4)) { + hw_low_last = _mm256_setzero_si256(); + is_tsinit = 1; + } else { + hw_low_last = _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, rxq->phc_time); + } + } +#endif /* constants used in processing loop */ const __m256i crc_adjust = @@ -967,10 +984,11 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, if (offload) { #ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC /** - * needs to load 2nd 16B of each desc for RSS hash parsing, + * needs to load 2nd 16B of each desc, * will cause performance drop to get into this context. */ if (offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH || + offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP || rxq->rx_flags & IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2) { /* load bottom half of every 32B desc */ const __m128i raw_desc_bh7 = @@ -1053,7 +1071,7 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, mb4_5 = _mm256_or_si256(mb4_5, rss_hash4_5); mb2_3 = _mm256_or_si256(mb2_3, rss_hash2_3); mb0_1 = _mm256_or_si256(mb0_1, rss_hash0_1); - } + } /* if() on RSS hash parsing */ if (rxq->rx_flags & IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2) { /* merge the status/error-1 bits into one register */ @@ -1132,8 +1150,121 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, mb4_5 = _mm256_or_si256(mb4_5, vlan_tci4_5); mb2_3 = _mm256_or_si256(mb2_3, vlan_tci2_3); mb0_1 = _mm256_or_si256(mb0_1, vlan_tci0_1); - } - } /* if() on RSS hash parsing */ + } /* if() on Vlan parsing */ + + if (offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + uint32_t mask = 0xFFFFFFFF; + __m256i ts; + __m256i ts_low = _mm256_setzero_si256(); + __m256i ts_low1; + __m256i ts_low2; + __m256i max_ret; + __m256i cmp_ret; + uint8_t ret = 0; + uint8_t shift = 8; + __m256i ts_desp_mask = _mm256_set_epi32(mask, 0, 0, 0, mask, 0, 0, 0); + __m256i cmp_mask = _mm256_set1_epi32(mask); + __m256i ts_permute_mask = _mm256_set_epi32(7, 3, 6, 2, 5, 1, 4, 0); + + ts = _mm256_and_si256(raw_desc_bh0_1, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, _mm256_srli_si256(ts, 3 * 4)); + ts = _mm256_and_si256(raw_desc_bh2_3, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, _mm256_srli_si256(ts, 2 * 4)); + ts = _mm256_and_si256(raw_desc_bh4_5, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, _mm256_srli_si256(ts, 4)); + ts = _mm256_and_si256(raw_desc_bh6_7, ts_desp_mask); + ts_low = _mm256_or_si256(ts_low, ts); + + ts_low1 = _mm256_permutevar8x32_epi32(ts_low, ts_permute_mask); + ts_low2 = _mm256_permutevar8x32_epi32(ts_low1, + _mm256_set_epi32(6, 5, 4, 3, 2, 1, 0, 7)); + ts_low2 = _mm256_and_si256(ts_low2, + _mm256_set_epi32(mask, mask, mask, mask, mask, mask, mask, 0)); + ts_low2 = _mm256_or_si256(ts_low2, hw_low_last); + hw_low_last = _mm256_and_si256(ts_low1, + _mm256_set_epi32(0, 0, 0, 0, 0, 0, 0, mask)); + + *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 0); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 1], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 1); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 2], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 2); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 3], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 3); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 4], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 4); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 5], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 5); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 6], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 6); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 7], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm256_extract_epi32(ts_low1, 7); + + if (unlikely(is_tsinit)) { + uint32_t in_timestamp; + if (iavf_get_phc_time(rxq)) + PMD_DRV_LOG(ERR, "get physical time failed"); + in_timestamp = *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset, uint32_t *); + rxq->phc_time = iavf_tstamp_convert_32b_64b(rxq->phc_time, in_timestamp); + } + + *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 1], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 2], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 3], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 4], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 5], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 6], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[i + 7], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + + max_ret = _mm256_max_epu32(ts_low2, ts_low1); + cmp_ret = _mm256_andnot_si256(_mm256_cmpeq_epi32(max_ret, ts_low1), cmp_mask); + + if (_mm256_testz_si256(cmp_ret, cmp_mask)) { + inflection_point = 0; + } else { + inflection_point = 1; + while (shift > 1) { + shift = shift >> 1; + __m256i mask_low = _mm256_setzero_si256(); + __m256i mask_high = _mm256_setzero_si256(); + switch (shift) { + case 4: + mask_low = _mm256_set_epi32(0, 0, 0, 0, mask, mask, mask, mask); + mask_high = _mm256_set_epi32(mask, mask, mask, mask, 0, 0, 0, 0); + break; + case 2: + mask_low = _mm256_srli_si256(cmp_mask, 2 * 4); + mask_high = _mm256_slli_si256(cmp_mask, 2 * 4); + break; + case 1: + mask_low = _mm256_srli_si256(cmp_mask, 1 * 4); + mask_high = _mm256_slli_si256(cmp_mask, 1 * 4); + break; + } + ret = _mm256_testz_si256(cmp_ret, mask_low); + if (ret) { + ret = _mm256_testz_si256(cmp_ret, mask_high); + inflection_point += ret ? 0 : shift; + cmp_mask = mask_high; + } else { + cmp_mask = mask_low; + } + } + } + mbuf_flags = _mm256_or_si256(mbuf_flags, _mm256_set1_epi32(iavf_timestamp_dynflag)); + } /* if() on Timestamp parsing */ + } #endif } @@ -1265,10 +1396,62 @@ _iavf_recv_raw_pkts_vec_avx2_flex_rxd(struct iavf_rx_queue *rxq, (_mm_cvtsi128_si64 (_mm256_castsi256_si128(status0_7))); received += burst; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + inflection_point = (inflection_point <= burst) ? inflection_point : 0; + switch (inflection_point) { + case 1: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 0], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 2: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 1], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 3: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 2], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 4: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 3], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 5: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 4], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 6: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 5], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 7: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 6], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 8: + *RTE_MBUF_DYNFIELD(rx_pkts[i + 7], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + rxq->phc_time += (uint64_t)1 << 32; + /* fallthrough */ + case 0: + break; + default: + PMD_DRV_LOG(ERR, "invalid inflection point for rx timestamp"); + break; + } + + rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); + } +#endif if (burst != IAVF_DESCS_PER_LOOP_AVX) break; } +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC + if (received > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) + rxq->phc_time = *RTE_MBUF_DYNFIELD(rx_pkts[received - 1], iavf_timestamp_dynfield_offset, rte_mbuf_timestamp_t *); +#endif + /* update tail pointers */ rxq->rx_tail += received; rxq->rx_tail &= (rxq->nb_rx_desc - 1); From patchwork Wed Jun 14 01:49:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhichao Zeng X-Patchwork-Id: 128645 X-Patchwork-Delegate: qi.z.zhang@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9A8FC42CB0; Wed, 14 Jun 2023 03:44:01 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E58EA42D20; Wed, 14 Jun 2023 03:43:48 +0200 (CEST) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id B86C942BFE for ; Wed, 14 Jun 2023 03:43:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686707027; x=1718243027; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=daRBQ+j+AFXhWFVWEi6bIPVN7YgaBoHEvRT8cZwJ0Ug=; b=c1MADWUtRN232eC0ePqMemGi97F0rdX/OXxjLyI9028Rd3uzQ+78dj0c v6wNoDdqyqH62f/eHo+lsKRU6OdGi01Qns1OiiAlbt+ADlhyy1PQOd0iW jdTZjvE4tIq/iqSHgvjcuiIr8MDegMmPYF/hFmMdxtjmeVk4ZYn0B2k2w /N30kuK3tDNX10KP1n32300r2CXAeVvYSn1oLwTu3NOpa35kGt1YIT8mU 1/1jmY4EAGEFdoUVwuv/UcmVw6bPm0OG6zWOPqaiTwBEbw7NfRKz77lhh Zl+RggmCOYDxkvaCZp0fGT1nJZjn5gVRKKJg38k87+yInlnMh618XcmBS Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10740"; a="422095177" X-IronPort-AV: E=Sophos;i="6.00,241,1681196400"; d="scan'208";a="422095177" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2023 18:43:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10740"; a="741655352" X-IronPort-AV: E=Sophos;i="6.00,241,1681196400"; d="scan'208";a="741655352" Received: from unknown (HELO zhichao-dpdk..) ([10.239.252.103]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2023 18:43:44 -0700 From: Zhichao Zeng To: dev@dpdk.org Cc: qi.z.zhang@intel.com, yaqi.tang@intel.com, Zhichao Zeng , Bruce Richardson , Konstantin Ananyev , Jingjing Wu , Beilei Xing Subject: [PATCH v5 3/3] net/iavf: support Rx timestamp offload on SSE Date: Wed, 14 Jun 2023 09:49:48 +0800 Message-Id: <20230614014948.3495063-4-zhichaox.zeng@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230614014948.3495063-1-zhichaox.zeng@intel.com> References: <20230526095055.2855121-1-zhichaox.zeng@intel.com> <20230614014948.3495063-1-zhichaox.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch enables Rx timestamp offload on the SSE data path. It significantly reduces the performance drop when RTE_ETH_RX_OFFLOAD_TIMESTAMP is enabled. Tested-by: Yaqi Tang --- v5: fix CI errors --- v4: rework avx2 patch based on offload path --- v3: logging with driver dedicated macro --- v2: fix compile warning and timestamp error Signed-off-by: Zhichao Zeng --- drivers/net/iavf/iavf_rxtx_vec_sse.c | 160 ++++++++++++++++++++++++++- 1 file changed, 157 insertions(+), 3 deletions(-) diff --git a/drivers/net/iavf/iavf_rxtx_vec_sse.c b/drivers/net/iavf/iavf_rxtx_vec_sse.c index 3f30be01aa..892bfa4cf3 100644 --- a/drivers/net/iavf/iavf_rxtx_vec_sse.c +++ b/drivers/net/iavf/iavf_rxtx_vec_sse.c @@ -392,6 +392,11 @@ flex_desc_to_olflags_v(struct iavf_rx_queue *rxq, __m128i descs[4], _mm_extract_epi32(fdir_id0_3, 3); } /* if() on fdir_enabled */ +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) + flags = _mm_or_si128(flags, _mm_set1_epi32(iavf_timestamp_dynflag)); +#endif + /** * At this point, we have the 4 sets of flags in the low 16-bits * of each 32-bit value in flags. @@ -723,7 +728,9 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, int pos; uint64_t var; struct iavf_adapter *adapter = rxq->vsi->adapter; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC uint64_t offloads = adapter->dev_data->dev_conf.rxmode.offloads; +#endif const uint32_t *ptype_tbl = adapter->ptype_tbl; __m128i crc_adjust = _mm_set_epi16 (0, 0, 0, /* ignore non-length fields */ @@ -793,6 +800,24 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, rte_cpu_to_le_32(1 << IAVF_RX_FLEX_DESC_STATUS0_DD_S))) return 0; +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC + uint8_t inflection_point = 0; + bool is_tsinit = false; + __m128i hw_low_last = _mm_set_epi32(0, 0, 0, (uint32_t)rxq->phc_time); + + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + uint64_t sw_cur_time = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); + + if (unlikely(sw_cur_time - rxq->hw_time_update > 4)) { + hw_low_last = _mm_setzero_si128(); + is_tsinit = 1; + } else { + hw_low_last = _mm_set_epi32(0, 0, 0, (uint32_t)rxq->phc_time); + } + } + +#endif + /** * Compile-time verify the shuffle mask * NOTE: some field positions already verified above, but duplicated @@ -825,7 +850,7 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, rxdp += IAVF_VPMD_DESCS_PER_LOOP) { __m128i descs[IAVF_VPMD_DESCS_PER_LOOP]; #ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC - __m128i descs_bh[IAVF_VPMD_DESCS_PER_LOOP]; + __m128i descs_bh[IAVF_VPMD_DESCS_PER_LOOP] = {_mm_setzero_si128()}; #endif __m128i pkt_mb0, pkt_mb1, pkt_mb2, pkt_mb3; __m128i staterr, sterr_tmp1, sterr_tmp2; @@ -895,10 +920,11 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, #ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC /** - * needs to load 2nd 16B of each desc for RSS hash parsing, + * needs to load 2nd 16B of each desc, * will cause performance drop to get into this context. */ if (offloads & RTE_ETH_RX_OFFLOAD_RSS_HASH || + offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP || rxq->rx_flags & IAVF_RX_FLAGS_VLAN_TAG_LOC_L2TAG2_2) { /* load bottom half of every 32B desc */ descs_bh[3] = _mm_load_si128 @@ -964,7 +990,94 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, pkt_mb2 = _mm_or_si128(pkt_mb2, vlan_tci2); pkt_mb1 = _mm_or_si128(pkt_mb1, vlan_tci1); pkt_mb0 = _mm_or_si128(pkt_mb0, vlan_tci0); - } + } /* if() on Vlan parsing */ + + if (offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + uint32_t mask = 0xFFFFFFFF; + __m128i ts; + __m128i ts_low = _mm_setzero_si128(); + __m128i ts_low1; + __m128i max_ret; + __m128i cmp_ret; + uint8_t ret = 0; + uint8_t shift = 4; + __m128i ts_desp_mask = _mm_set_epi32(mask, 0, 0, 0); + __m128i cmp_mask = _mm_set1_epi32(mask); + + ts = _mm_and_si128(descs_bh[0], ts_desp_mask); + ts_low = _mm_or_si128(ts_low, _mm_srli_si128(ts, 3 * 4)); + ts = _mm_and_si128(descs_bh[1], ts_desp_mask); + ts_low = _mm_or_si128(ts_low, _mm_srli_si128(ts, 2 * 4)); + ts = _mm_and_si128(descs_bh[2], ts_desp_mask); + ts_low = _mm_or_si128(ts_low, _mm_srli_si128(ts, 1 * 4)); + ts = _mm_and_si128(descs_bh[3], ts_desp_mask); + ts_low = _mm_or_si128(ts_low, ts); + + ts_low1 = _mm_slli_si128(ts_low, 4); + ts_low1 = _mm_and_si128(ts_low, _mm_set_epi32(mask, mask, mask, 0)); + ts_low1 = _mm_or_si128(ts_low1, hw_low_last); + hw_low_last = _mm_and_si128(ts_low, _mm_set_epi32(0, 0, 0, mask)); + + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 0], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm_extract_epi32(ts_low, 0); + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 1], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm_extract_epi32(ts_low, 1); + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 2], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm_extract_epi32(ts_low, 2); + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 3], + iavf_timestamp_dynfield_offset, uint32_t *) = _mm_extract_epi32(ts_low, 3); + + if (unlikely(is_tsinit)) { + uint32_t in_timestamp; + + if (iavf_get_phc_time(rxq)) + PMD_DRV_LOG(ERR, "get physical time failed"); + in_timestamp = *RTE_MBUF_DYNFIELD(rx_pkts[pos + 0], + iavf_timestamp_dynfield_offset, uint32_t *); + rxq->phc_time = iavf_tstamp_convert_32b_64b(rxq->phc_time, in_timestamp); + } + + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 0], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 1], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 2], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 3], + iavf_timestamp_dynfield_offset + 4, uint32_t *) = (uint32_t)(rxq->phc_time >> 32); + + max_ret = _mm_max_epu32(ts_low, ts_low1); + cmp_ret = _mm_andnot_si128(_mm_cmpeq_epi32(max_ret, ts_low), cmp_mask); + + if (_mm_testz_si128(cmp_ret, cmp_mask)) { + inflection_point = 0; + } else { + inflection_point = 1; + while (shift > 1) { + shift = shift >> 1; + __m128i mask_low = _mm_setzero_si128(); + __m128i mask_high = _mm_setzero_si128(); + switch (shift) { + case 2: + mask_low = _mm_set_epi32(0, 0, mask, mask); + mask_high = _mm_set_epi32(mask, mask, 0, 0); + break; + case 1: + mask_low = _mm_srli_si128(cmp_mask, 4); + mask_high = _mm_slli_si128(cmp_mask, 4); + break; + } + ret = _mm_testz_si128(cmp_ret, mask_low); + if (ret) { + ret = _mm_testz_si128(cmp_ret, mask_high); + inflection_point += ret ? 0 : shift; + cmp_mask = mask_high; + } else { + cmp_mask = mask_low; + } + } + } + } /* if() on Timestamp parsing */ flex_desc_to_olflags_v(rxq, descs, descs_bh, &rx_pkts[pos]); #else @@ -1011,10 +1124,51 @@ _recv_raw_pkts_vec_flex_rxd(struct iavf_rx_queue *rxq, /* C.4 calc available number of desc */ var = __builtin_popcountll(_mm_cvtsi128_si64(staterr)); nb_pkts_recd += var; + +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC + if (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP) { + inflection_point = (inflection_point <= var) ? inflection_point : 0; + switch (inflection_point) { + case 1: + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 0], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 2: + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 1], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 3: + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 2], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + /* fallthrough */ + case 4: + *RTE_MBUF_DYNFIELD(rx_pkts[pos + 3], + iavf_timestamp_dynfield_offset + 4, uint32_t *) += 1; + rxq->phc_time += (uint64_t)1 << 32; + /* fallthrough */ + case 0: + break; + default: + PMD_DRV_LOG(ERR, "invalid inflection point for rx timestamp"); + break; + } + + rxq->hw_time_update = rte_get_timer_cycles() / (rte_get_timer_hz() / 1000); + } +#endif + if (likely(var != IAVF_VPMD_DESCS_PER_LOOP)) break; } +#ifndef RTE_LIBRTE_IAVF_16BYTE_RX_DESC +#ifdef IAVF_RX_TS_OFFLOAD + if (nb_pkts_recd > 0 && (rxq->offloads & RTE_ETH_RX_OFFLOAD_TIMESTAMP)) + rxq->phc_time = *RTE_MBUF_DYNFIELD(rx_pkts[nb_pkts_recd - 1], + iavf_timestamp_dynfield_offset, uint32_t *); +#endif +#endif + /* Update our internal tail pointer */ rxq->rx_tail = (uint16_t)(rxq->rx_tail + nb_pkts_recd); rxq->rx_tail = (uint16_t)(rxq->rx_tail & (rxq->nb_rx_desc - 1));