From patchwork Tue Jan 15 17:38:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongseok Koh X-Patchwork-Id: 49850 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 84FC52BE5; Tue, 15 Jan 2019 18:39:06 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id E71FC2BE5 for ; Tue, 15 Jan 2019 18:39:04 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from yskoh@mellanox.com) with ESMTPS (AES256-SHA encrypted); 15 Jan 2019 19:39:04 +0200 Received: from scfae-sc-2.mti.labs.mlnx (scfae-sc-2.mti.labs.mlnx [10.101.0.96]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0FHd1L3025106; Tue, 15 Jan 2019 19:39:02 +0200 From: Yongseok Koh To: shahafs@mellanox.com Cc: dev@dpdk.org, erezf@mellanox.com, stable@dpdk.org Date: Tue, 15 Jan 2019 09:38:58 -0800 Message-Id: <20190115173859.43152-1-yskoh@mellanox.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH 1/2] net/mlx5: fix Rx packet padding X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Rx packet padding is supposed to be set by an environment variable - MLX5_PMD_ENABLE_PADDING, but it has been missing for some time by mistake. Rather than using such a variable, a PMD parameter (rxq_pkt_pad_en) is added instead. Fixes: a1366b1a2be3 ("net/mlx5: add reference counter on DPDK Rx queues") Cc: stable@dpdk.org Signed-off-by: Yongseok Koh Reviewed-by: Erez Ferber --- doc/guides/nics/mlx5.rst | 27 +++++++++++++-------------- drivers/net/mlx5/mlx5.c | 18 +++++++++++++++--- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 5ddca44eed..3f168b1613 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -233,20 +233,6 @@ Environment variables enabled and most useful when ``CONFIG_RTE_EAL_PMD_PATH`` is also set, since ``LD_LIBRARY_PATH`` has no effect in this case. -- ``MLX5_PMD_ENABLE_PADDING`` - - Enables HW packet padding in PCI bus transactions. - - When packet size is cache aligned and CRC stripping is enabled, 4 fewer - bytes are written to the PCI bus. Enabling padding makes such packets - aligned again. - - In cases where PCI bandwidth is the bottleneck, padding can improve - performance by 10%. - - This is disabled by default since this can also decrease performance for - unaligned packet sizes. - - ``MLX5_SHUT_UP_BF`` Configures HW Tx doorbell register as IO-mapped. @@ -301,6 +287,19 @@ Run-time configuration - CPU having 128B cacheline with ConnectX-5 and Bluefield. +- ``rxq_pkt_pad_en`` parameter [int] + + A nonzero value enables padding Rx packet to the size of cacheline on PCI + transaction. This feature would waste PCI bandwidth but could improve + performance by avoiding partial cacheline write which may cause costly + read-modify-copy in memory transaction on some architectures. Disabled by + default. + + Supported on: + + - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefield. + - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5, ConnectX-6 and Bluefield. + - ``mprq_en`` parameter [int] A nonzero value enables configuring Multi-Packet Rx queues. Rx queue is diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index a84e1afa09..741bc7fc07 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -54,6 +54,9 @@ /* Device parameter to enable RX completion entry padding to 128B. */ #define MLX5_RXQ_CQE_PAD_EN "rxq_cqe_pad_en" +/* Device parameter to enable padding Rx packet to cacheline size. */ +#define MLX5_RXQ_PKT_PAD_EN "rxq_pkt_pad_en" + /* Device parameter to enable Multi-Packet Rx queue. */ #define MLX5_RX_MPRQ_EN "mprq_en" @@ -486,6 +489,8 @@ mlx5_args_check(const char *key, const char *val, void *opaque) config->cqe_comp = !!tmp; } else if (strcmp(MLX5_RXQ_CQE_PAD_EN, key) == 0) { config->cqe_pad = !!tmp; + } else if (strcmp(MLX5_RXQ_PKT_PAD_EN, key) == 0) { + config->hw_padding = !!tmp; } else if (strcmp(MLX5_RX_MPRQ_EN, key) == 0) { config->mprq.enabled = !!tmp; } else if (strcmp(MLX5_RX_MPRQ_LOG_STRIDE_NUM, key) == 0) { @@ -541,6 +546,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) const char **params = (const char *[]){ MLX5_RXQ_CQE_COMP_EN, MLX5_RXQ_CQE_PAD_EN, + MLX5_RXQ_PKT_PAD_EN, MLX5_RX_MPRQ_EN, MLX5_RX_MPRQ_LOG_STRIDE_NUM, MLX5_RX_MPRQ_MAX_MEMCPY_LEN, @@ -735,6 +741,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; + unsigned int hw_padding = 0; unsigned int mps; unsigned int cqe_comp; unsigned int cqe_pad = 0; @@ -1060,10 +1067,14 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); #ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!attr.rx_pad_end_addr_align; + hw_padding = !!attr.rx_pad_end_addr_align; #endif - DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", - (config.hw_padding ? "" : "not ")); + if (config.hw_padding && !hw_padding) { + DRV_LOG(DEBUG, "Rx end alignment padding isn't supported"); + config.hw_padding = 0; + } else if (config.hw_padding) { + DRV_LOG(DEBUG, "Rx end alignment padding is enabled"); + } config.tso = (attr.tso_caps.max_tso > 0 && (attr.tso_caps.supported_qpts & (1 << IBV_QPT_RAW_PACKET))); @@ -1440,6 +1451,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); /* Default configuration. */ dev_config = (struct mlx5_dev_config){ + .hw_padding = 0, .mps = MLX5_ARG_UNSET, .tx_vec_en = 1, .rx_vec_en = 1, From patchwork Tue Jan 15 17:38:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yongseok Koh X-Patchwork-Id: 49851 X-Patchwork-Delegate: shahafs@mellanox.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2574D2C15; Tue, 15 Jan 2019 18:39:11 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 80FE6316B for ; Tue, 15 Jan 2019 18:39:09 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from yskoh@mellanox.com) with ESMTPS (AES256-SHA encrypted); 15 Jan 2019 19:39:06 +0200 Received: from scfae-sc-2.mti.labs.mlnx (scfae-sc-2.mti.labs.mlnx [10.101.0.96]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0FHd1L4025106; Tue, 15 Jan 2019 19:39:04 +0200 From: Yongseok Koh To: shahafs@mellanox.com Cc: dev@dpdk.org, erezf@mellanox.com, stable@dpdk.org Date: Tue, 15 Jan 2019 09:38:59 -0800 Message-Id: <20190115173859.43152-2-yskoh@mellanox.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190115173859.43152-1-yskoh@mellanox.com> References: <20190115173859.43152-1-yskoh@mellanox.com> Subject: [dpdk-dev] [PATCH 2/2] net/mlx5: fix deprecated library API for Rx packet padding X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In rdma-core library IBV_WQ_FLAG_RX_END_PADDING is renamed to IBV_WQ_FLAGS_PCI_WRITE_END_PADDING. Way to query the capability is also changed. Cc: stable@dpdk.org Signed-off-by: Yongseok Koh Reviewed-by: Erez Ferber --- drivers/net/mlx5/Makefile | 5 +++++ drivers/net/mlx5/meson.build | 2 ++ drivers/net/mlx5/mlx5.c | 5 ++++- drivers/net/mlx5/mlx5_rxq.c | 7 +++++-- 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 688678ac84..9a7da18196 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -121,6 +121,11 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh enum IBV_FLOW_SPEC_MPLS \ $(AUTOCONF_OUTPUT) $Q sh -- '$<' '$@' \ + HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING \ + infiniband/verbs.h \ + enum IBV_WQ_FLAGS_PCI_WRITE_END_PADDING \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ HAVE_IBV_WQ_FLAG_RX_END_PADDING \ infiniband/verbs.h \ enum IBV_WQ_FLAG_RX_END_PADDING \ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index 6413cefd06..4540c455b7 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -103,6 +103,8 @@ if build 'mlx5dv_create_flow_action_packet_reformat' ], [ 'HAVE_IBV_DEVICE_MPLS_SUPPORT', 'infiniband/verbs.h', 'IBV_FLOW_SPEC_MPLS' ], + [ 'HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING', 'infiniband/verbs.h', + 'IBV_WQ_FLAGS_PCI_WRITE_END_PADDING' ], [ 'HAVE_IBV_WQ_FLAG_RX_END_PADDING', 'infiniband/verbs.h', 'IBV_WQ_FLAG_RX_END_PADDING' ], [ 'HAVE_IBV_DEVX_OBJ', 'infiniband/mlx5dv.h', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 741bc7fc07..a913a5955f 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1066,8 +1066,11 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, IBV_RAW_PACKET_CAP_SCATTER_FCS); DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); -#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING +#if defined(HAVE_IBV_WQ_FLAG_RX_END_PADDING) hw_padding = !!attr.rx_pad_end_addr_align; +#elif defined(HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING) + hw_padding = !!(attr.device_cap_flags_ex & + IBV_DEVICE_PCI_WRITE_END_PADDING); #endif if (config.hw_padding && !hw_padding) { DRV_LOG(DEBUG, "Rx end alignment padding isn't supported"); diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 34c8338723..10b6ce0c12 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -881,12 +881,15 @@ mlx5_rxq_ibv_new(struct rte_eth_dev *dev, uint16_t idx) attr.wq.ibv.create_flags |= IBV_WQ_FLAGS_SCATTER_FCS; attr.wq.ibv.comp_mask |= IBV_WQ_INIT_ATTR_FLAGS; } -#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING if (config->hw_padding) { +#if defined(HAVE_IBV_WQ_FLAG_RX_END_PADDING) attr.wq.ibv.create_flags |= IBV_WQ_FLAG_RX_END_PADDING; attr.wq.ibv.comp_mask |= IBV_WQ_INIT_ATTR_FLAGS; - } +#elif defined(HAVE_IBV_WQ_FLAGS_PCI_WRITE_END_PADDING) + attr.wq.ibv.create_flags |= IBV_WQ_FLAGS_PCI_WRITE_END_PADDING; + attr.wq.ibv.comp_mask |= IBV_WQ_INIT_ATTR_FLAGS; #endif + } #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT attr.wq.mlx5 = (struct mlx5dv_wq_init_attr){ .comp_mask = 0,