From patchwork Wed Jul 4 17:27:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42256 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CA5341BF24; Wed, 4 Jul 2018 19:27:53 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id C885E1BF1A for ; Wed, 4 Jul 2018 19:27:51 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id v3-v6so3403348wmh.0 for ; Wed, 04 Jul 2018 10:27:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=QLWEKxuqhF2sx7AXNTonfLHxyzB3mWcORADPDgYW6JM=; b=xSkBaLjU3b9OA/oCTQM2BFlmozyK7xObXzag2tK0wddW11h0x/2dsph5nLGJBSP6BE ZPPT15RDcL3pcckATkksPLfeGiKYchSMnBT4CxgeO4GS4P9e9IQwPoW4igH1JEWKZShF +r83c8qKUZsd+CNDGss2Y8Ita1Z1hOnLkYSJ/79oRPAJqMXDlUFGx3+cbfdwtcm2VC/Z NIKlVyq1L/uyvZsmdDYoMCssNcMfZ3u02qtyTEAzrCbxanJ6ZRjNP98ryL625vmi3ElN uziCUZ2DLMajhf5LSuicnSbil7s4WKPPMojBBHvLeI31Y3CoDYeLTLr6IVczZfE/igZx xvsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=QLWEKxuqhF2sx7AXNTonfLHxyzB3mWcORADPDgYW6JM=; b=U7yMj/BsKq+PD84zMI25ipgmXKXwszfzDF2hMFDircb+I88F6emUtRVPmjTSjA6PvA uOpva1rZ5h2V6T8f3w6wJFhC18H6BJZAPX7oCi3EKHFxoRSUrJKPgZtzIhIgH5WeogV8 scUyQgm68ESOu+rVFBxtlPGCD7CtZ5+3oLVVEnyQp9ZPUpM0E8ZiLQUjBhZMBkHiG3JL T/AAtvpP4mN4t9DHWfreNNlzXF0dNBWt2PgFYvQ5mV3GArOErMmZPBZKoDcFB21x35Ai H0BUkjtq+5ruacvFrnLBrDWv2oLlL6zbmRujK9HyeJkRM3MP/OejI8ec3Vnale033J2E OCLQ== X-Gm-Message-State: APt69E3M2a8IAwXyyo4Ki8c+K4HBob5tm07eyxhsNYyIedUzqkyi+yfv EnaXbNifV4rQHbgxAuf2UTKSpw== X-Google-Smtp-Source: AAOMgpfbccluRQFqWzceWnnuVXk3vMsu+VcmNnYFpBt8R6DcFtQWle0Uh2qGOhfIvJtYDz//u2BT/A== X-Received: by 2002:adf:d142:: with SMTP id b2-v6mr2195518wri.17.1530725271579; Wed, 04 Jul 2018 10:27:51 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id i12-v6sm4464769wrn.48.2018.07.04.10.27.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:27:50 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:34 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-2-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 01/10] net/mlx5: rename confusing object in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" There are several attribute objects in this function: - IB device attributes (struct ibv_device_attr_ex device_attr). - Direct Verbs attributes (struct mlx5dv_context attrs_out). - Port attributes (struct ibv_port_attr). - IB device attributes again (struct ibv_device_attr_ex device_attr_ex). "attrs_out" is both odd and initialized using a nonstandard syntax. Rename it "dv_attr" for consistency. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Fixed ctx -> attr_ctx in mlx5_pci_probe(). --- drivers/net/mlx5/mlx5.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d081bdd05..22cbce8d5 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,6 +654,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; struct ibv_context *attr_ctx = NULL; struct ibv_device_attr_ex device_attr; @@ -670,7 +671,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; int i; - struct mlx5dv_context attrs_out = {0}; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif @@ -736,21 +736,21 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif /* * Multi-packet send is supported by ConnectX-4 Lx PF as well * as all ConnectX-5 devices. */ #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS; #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - attrs_out.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; + dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &attrs_out); - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { - if (attrs_out.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { + mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { + if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); mps = MLX5_MPW_ENHANCED; } else { @@ -762,14 +762,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mps = MLX5_MPW_DISABLED; } #ifdef HAVE_IBV_MLX5_MOD_SWP - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_SWP) - swp = attrs_out.sw_parsing_caps.sw_parsing_offloads; + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) + swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = - attrs_out.striding_rq_caps; + dv_attr.striding_rq_caps; DRV_LOG(DEBUG, "\tmin_single_stride_log_num_of_bytes: %d", mprq_caps.min_single_stride_log_num_of_bytes); @@ -794,15 +794,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } #endif if (RTE_CACHE_LINE_SIZE == 128 && - !(attrs_out.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) + !(dv_attr.flags & MLX5DV_CONTEXT_FLAGS_CQE_128B_COMP)) cqe_comp = 0; else cqe_comp = 1; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT - if (attrs_out.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { - tunnel_en = ((attrs_out.tunnel_offloads_caps & + if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { + tunnel_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_VXLAN) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_GRE)); } DRV_LOG(DEBUG, "tunnel offloading is %ssupported", @@ -812,9 +812,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT - mpls_en = ((attrs_out.tunnel_offloads_caps & + mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && - (attrs_out.tunnel_offloads_caps & + (dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_UDP)); DRV_LOG(DEBUG, "MPLS over GRE/UDP tunnel offloading is %ssupported", mpls_en ? "" : "not "); From patchwork Wed Jul 4 17:27:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42257 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6622D1BF30; Wed, 4 Jul 2018 19:27:55 +0200 (CEST) Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53]) by dpdk.org (Postfix) with ESMTP id CEC1F1BF27 for ; Wed, 4 Jul 2018 19:27:53 +0200 (CEST) Received: by mail-wm0-f53.google.com with SMTP id s12-v6so7142793wmc.1 for ; Wed, 04 Jul 2018 10:27:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=LZ7BxY/4owWBpMHTwS+IFzwvWnorKYfJ62epOi5lr6Y=; b=DGA2jEZWDggaVnYO8WKgcLLq766NiYudcElzs8rZ52aF4G3VNjslx+2va75Qr6KY/z o+zxvJtAAdkm53g9WI35rqvtPZioT2e8E/dTiJK49zqeycDFBZMsRo2rjb+lXREBuKY2 SArLMa1wCKMWAQWIZEgxruC3+qMvjSSRooPd6MPe9Kfk1qDwBx0+iPJkES8AiQrUGMEh ZSFAURXyAnlFUledXGstzvAqIsV5Jx2AguVYiH4sx7Yx2cDyFdMRy1BYd/36wj07G7s6 082+eCtY0PaNnNOWT3QSkN4TmInjMLgKgELo+jkZ+3YteqIcDzbxBh5vW4L2YEUMa8Uj qAoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=LZ7BxY/4owWBpMHTwS+IFzwvWnorKYfJ62epOi5lr6Y=; b=F/r7aS3byYpW6nWTqBGyr8ZduebGm43flHRCe75t5N5e/CK3A/9w/Mbr0ivHxMBZkJ lm+jkz2TwN8gzXPg87zZbaAUyzpTfHOjsZk8gqgv2EexOyT5wqLiAfvIbvx5J93qpf84 9uQc08oyBQkeV3d9X2+dfY2T58C888GoZlGG1t9S4UmWMw81C32ZncXVR3aHRHu1hM// p1pv/CMbfp81QcllfJuF4Wx0j/JkjYLjFWvCTJ0wQNiMuLzBatwQsPqRSkfKR8U5lIAA Lgai2nwiBUlMaKma/Wi/VP+/jo4WAIeIWScdtnHFUKpELbTlllA90unFsMLXEVLbqthb 9nug== X-Gm-Message-State: APt69E1Nc9zsvcYD69XvXz8tNpVR0Spvd6/7R8JR+/xYfXiZp4eMg2Gw tLBGZqnclQUog5R9V4VfBtQ+3U94 X-Google-Smtp-Source: AAOMgpf3+w8CxSfO8FZzaQIaXP4TAfYUp6Fn3PNQd+JhEnVVtD0ArsfSmIwnFlHG9lXS3KG/01BkQQ== X-Received: by 2002:a1c:c91a:: with SMTP id f26-v6mr2273416wmb.140.1530725273508; Wed, 04 Jul 2018 10:27:53 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id x10-v6sm4125032wrn.25.2018.07.04.10.27.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:27:52 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:36 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-3-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 02/10] net/mlx5: remove redundant objects in probe code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch gets rid of redundant calls to open the device and query its attributes in order to simplify the code. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v2 changes: - Minor indent fix on existing code. --- drivers/net/mlx5/mlx5.c | 64 +++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 22cbce8d5..4e7f29f5b 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -654,10 +654,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **list = NULL; struct ibv_device *ibv_dev; + struct ibv_context *ctx = NULL; + struct ibv_device_attr_ex attr; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; int err = 0; - struct ibv_context *attr_ctx = NULL; - struct ibv_device_attr_ex device_attr; unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; @@ -714,12 +714,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || (pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - attr_ctx = mlx5_glue->open_device(list[i]); + ctx = mlx5_glue->open_device(list[i]); rte_errno = errno; err = rte_errno; break; } - if (attr_ctx == NULL) { + if (ctx == NULL) { switch (err) { case 0: DRV_LOG(ERR, @@ -748,7 +748,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; #endif - mlx5_glue->dv_query_device(attr_ctx, &dv_attr); + mlx5_glue->dv_query_device(ctx, &dv_attr); if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED) { if (dv_attr.flags & MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW) { DRV_LOG(DEBUG, "enhanced MPW is supported"); @@ -822,23 +822,20 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif - err = mlx5_glue->query_device_ex(attr_ctx, NULL, &device_attr); + err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", - device_attr.orig_attr.phys_port_cnt); - for (i = 0; i < device_attr.orig_attr.phys_port_cnt; i++) { + DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); + for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { char name[RTE_ETH_NAME_MAX_LEN]; int len; uint32_t port = i + 1; /* ports are indexed from one */ - struct ibv_context *ctx = NULL; struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct priv *priv = NULL; struct rte_eth_dev *eth_dev = NULL; - struct ibv_device_attr_ex device_attr_ex; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -865,7 +862,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, len = snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (device_attr.orig_attr.phys_port_cnt > 1) + if (attr.orig_attr.phys_port_cnt > 1) snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); @@ -907,7 +904,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, continue; } DRV_LOG(DEBUG, "using port %u", port); - ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) + ctx = mlx5_glue->open_device(ibv_dev); if (ctx == NULL) { err = ENODEV; goto port_error; @@ -949,7 +947,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); - priv->device_attr = device_attr; + priv->device_attr = attr; priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; @@ -960,17 +958,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); goto port_error; } - err = mlx5_glue->query_device_ex(ctx, NULL, &device_attr_ex); - if (err) { - DRV_LOG(ERR, "ibv_query_device_ex() failed"); - goto port_error; - } - config.hw_csum = !!(device_attr_ex.device_cap_flags_ex & + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); DRV_LOG(DEBUG, "checksum offloading is %ssupported", (config.hw_csum ? "" : "not ")); #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!(device_attr.max_counter_sets); + config.flow_counter_en = !!attr.max_counter_sets; mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", @@ -978,7 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, cs_desc.attributes); #endif config.ind_table_max_size = - device_attr_ex.rss_caps.max_rwq_indirection_table_size; + attr.rss_caps.max_rwq_indirection_table_size; /* Remove this check once DPDK supports larger/variable * indirection tables. */ if (config.ind_table_max_size > @@ -986,29 +979,28 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", config.ind_table_max_size); - config.hw_vlan_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_vlan_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); DRV_LOG(DEBUG, "VLAN stripping is %ssupported", (config.hw_vlan_strip ? "" : "not ")); - config.hw_fcs_strip = !!(device_attr_ex.raw_packet_caps & + config.hw_fcs_strip = !!(attr.raw_packet_caps & IBV_RAW_PACKET_CAP_SCATTER_FCS); DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", (config.hw_fcs_strip ? "" : "not ")); #ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!device_attr_ex.rx_pad_end_addr_align; + config.hw_padding = !!attr.rx_pad_end_addr_align; #endif DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", (config.hw_padding ? "" : "not ")); config.vf = vf; - config.tso = ((device_attr_ex.tso_caps.max_tso > 0) && - (device_attr_ex.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); if (config.tso) - config.tso_max_payload_sz = - device_attr_ex.tso_caps.max_tso; + config.tso_max_payload_sz = attr.tso_caps.max_tso; if (config.mps && !mps) { DRV_LOG(ERR, "multi-packet send not supported on this device" @@ -1170,14 +1162,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); + /* + * Each eth_dev instance is assigned its own Verbs context, + * since this one is consumed, let the next iteration open + * another. + */ + ctx = NULL; continue; port_error: if (priv) rte_free(priv); if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); - if (ctx) - claim_zero(mlx5_glue->close_device(ctx)); if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) rte_eth_dev_release_port(eth_dev); break; @@ -1189,8 +1185,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * way to enumerate the registered ethdevs to free the previous ones. */ error: - if (attr_ctx) - claim_zero(mlx5_glue->close_device(attr_ctx)); + if (ctx) + claim_zero(mlx5_glue->close_device(ctx)); if (list) mlx5_glue->free_device_list(list); if (err) { From patchwork Wed Jul 4 17:27:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42258 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DBB6B1BF45; Wed, 4 Jul 2018 19:27:56 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id BF8471BF3C for ; Wed, 4 Jul 2018 19:27:55 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id v16-v6so7127118wmv.5 for ; Wed, 04 Jul 2018 10:27:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=9hrYhb06M81OLEec/FAJOkvBG9oAKlUR/iA+Ybmj35w=; b=WrMQitCpAji05eFy+pIiF+E7paWvl02IHRO5v1+Rg6i4tlq+X7INxLCMgAP4fifiLa qfCUpFvBp3YECqtyieNMwD4/4tb39N5OyuvfEzIsqfSUz3xHmjkfLUkzo62s7vQ92lvQ f+tlO8SO4ylwwG3zesKOSTIyOh3283ZWsX7M/UNMA0v/iObg9aRkYx1UvUr3Vfkp5g+Z 8VDGgknaUjufDu0GWyto3COzxM5bS81kIIIMJaVJMoG9bAW09MJXgulsEifqoPtIWx+F EVmIA+/pnBNmKgVVkcrlM+94z+1dImmBFtPSbc2e1TjvASLmeC2T91qDnI7WL9TRaJEV UFGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=9hrYhb06M81OLEec/FAJOkvBG9oAKlUR/iA+Ybmj35w=; b=p75KQACeW6ocRs9Tb3cJtU1KHMB7GZabETHufuqCe6LL6hfQa7T+rCjBM1mX6dhNbv Eo3Egu0jOy4x9/XWIRVOaGPBQYw2HhjiNsNSwauuN5JPGTbKdBjP4VisirWPkhXyW5aI Bu3uAU2pJJUY66yBPDspUbyz5M9y77Vr1Q7g5oXleqJYcwIFE4NsnkRuFZ5Vds+agRW7 +T4Ktf5Z8tQKbjatKYkqsm83iASSbhKI3ihkWKT7MkOoKO7nQw4mURc8jPoI4sdSzfuX WRgbMjDqhY77QJ1iCsMVuPcozOY4WauNK1pU2UVU9CdHwgwaBFlSl0lvepSNhYy3aPnk KXfA== X-Gm-Message-State: APt69E0lpm6ufNl2QtgiiPh92CkwGDcxlzOXYW3GFGCuzxUChz3cJiwS C3tMqFXZ/4Q8ey1FS2mCa6EnLw== X-Google-Smtp-Source: AAOMgpddC49WdFS+QIqxN+MgihItj/FTkmslxRv+3rsOX5Q6YMUcEnIqXUwv217AE6YktFi9KAnFhg== X-Received: by 2002:adf:ee43:: with SMTP id w3-v6mr2365608wro.63.1530725275416; Wed, 04 Jul 2018 10:27:55 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id u7-v6sm9053847wmd.3.2018.07.04.10.27.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:27:54 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:38 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Xueming Li Message-ID: <20180704172322.22571-4-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 03/10] net/mlx5: drop useless support for several Verbs ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Unlike mlx4 from which this capability was inherited, mlx5 devices expose exactly one Verbs port per PCI bus address. Each physical port gets assigned its own bus address with a single Verbs port. While harmless, this code requires an extra loop that would get in the way of subsequent refactoring. No functional impact. Signed-off-by: Adrien Mazarguil Cc: Shahaf Shuler Cc: Xueming Li --- v3 changes: This patch was not present in prior revisions. As discussed [1], it was added after finally deciding to remove this support. [1] https://mails.dpdk.org/archives/dev/2018-June/105661.html --- drivers/net/mlx5/mlx5.c | 96 +++++++++++++------------------------ drivers/net/mlx5/mlx5.h | 1 - drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_txq.c | 2 +- 4 files changed, 34 insertions(+), 67 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 4e7f29f5b..717d8b268 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -652,11 +652,13 @@ static int mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) { - struct ibv_device **list = NULL; - struct ibv_device *ibv_dev; + struct ibv_device **list; struct ibv_context *ctx = NULL; struct ibv_device_attr_ex attr; + struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct rte_eth_dev *eth_dev = NULL; + struct priv *priv = NULL; int err = 0; unsigned int vf = 0; unsigned int mps; @@ -719,6 +721,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; break; } + mlx5_glue->free_device_list(list); if (ctx == NULL) { switch (err) { case 0: @@ -733,7 +736,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } goto error; } - ibv_dev = list[i]; DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; @@ -827,15 +829,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DEBUG("ibv_query_device_ex() failed"); goto error; } - DRV_LOG(INFO, "%u port(s) detected", attr.orig_attr.phys_port_cnt); - for (i = 0; i < attr.orig_attr.phys_port_cnt; i++) { + { char name[RTE_ETH_NAME_MAX_LEN]; - int len; - uint32_t port = i + 1; /* ports are indexed from one */ struct ibv_port_attr port_attr; - struct ibv_pd *pd = NULL; - struct priv *priv = NULL; - struct rte_eth_dev *eth_dev = NULL; struct ether_addr mac; struct mlx5_dev_config config = { .cqe_comp = cqe_comp, @@ -859,11 +855,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - len = snprintf(name, sizeof(name), PCI_PRI_FMT, + snprintf(name, sizeof(name), PCI_PRI_FMT, pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - if (attr.orig_attr.phys_port_cnt > 1) - snprintf(name + len, sizeof(name), " port %u", i); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -901,31 +895,22 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); rte_eth_dev_probing_finish(eth_dev); - continue; - } - DRV_LOG(DEBUG, "using port %u", port); - if (!ctx) - ctx = mlx5_glue->open_device(ibv_dev); - if (ctx == NULL) { - err = ENODEV; - goto port_error; + claim_zero(mlx5_glue->close_device(ctx)); + return 0; } /* Check port status. */ - err = mlx5_glue->query_port(ctx, port, &port_attr); + err = mlx5_glue->query_port(ctx, 1, &port_attr); if (err) { DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto port_error; + goto error; } if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, - "port %d is not configured in Ethernet mode", - port); + DRV_LOG(ERR, "port is not configured in Ethernet mode"); err = EINVAL; - goto port_error; + goto error; } if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port %d is not active: \"%s\" (%d)", - port, + DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", mlx5_glue->port_state_str(port_attr.state), port_attr.state); /* Allocate protection domain. */ @@ -933,7 +918,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (pd == NULL) { DRV_LOG(ERR, "PD allocation failure"); err = ENOMEM; - goto port_error; + goto error; } /* from rte_ethdev.c */ priv = rte_zmalloc("ethdev private structure", @@ -942,13 +927,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (priv == NULL) { DRV_LOG(ERR, "priv allocation failure"); err = ENOMEM; - goto port_error; + goto error; } priv->ctx = ctx; strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); priv->device_attr = attr; - priv->port = port; priv->pd = pd; priv->mtu = ETHER_MTU; err = mlx5_args(&config, pci_dev->device.devargs); @@ -956,7 +940,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", strerror(rte_errno)); - goto port_error; + goto error; } config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); @@ -1006,7 +990,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, "multi-packet send not supported on this device" " (" MLX5_TXQ_MPW_EN ")"); err = ENOTSUP; - goto port_error; + goto error; } DRV_LOG(INFO, "%s MPS is %s", config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", @@ -1038,7 +1022,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (eth_dev == NULL) { DRV_LOG(ERR, "can not allocate rte ethdev"); err = ENOMEM; - goto port_error; + goto error; } eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; @@ -1049,7 +1033,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = mlx5_uar_init_primary(eth_dev); if (err) { err = rte_errno; - goto port_error; + goto error; } /* Configure the first MAC address by default. */ if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { @@ -1058,7 +1042,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, " loaded? (errno: %s)", eth_dev->data->port_id, strerror(rte_errno)); err = ENODEV; - goto port_error; + goto error; } DRV_LOG(INFO, "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", @@ -1082,7 +1066,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = mlx5_get_mtu(eth_dev, &priv->mtu); if (err) { err = rte_errno; - goto port_error; + goto error; } DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, priv->mtu); @@ -1131,7 +1115,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "port %u drop queue allocation failed: %s", eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; - goto port_error; + goto error; } /* Supported Verbs flow priority number detection. */ if (verb_priorities == 0) @@ -1140,7 +1124,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", eth_dev->data->port_id, verb_priorities); err = ENOTSUP; - goto port_error; + goto error; } priv->config.max_verbs_prio = verb_priorities; /* @@ -1154,7 +1138,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->device->numa_node); if (err) { err = rte_errno; - goto port_error; + goto error; } /* Add device to memory callback list. */ rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); @@ -1162,33 +1146,17 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - /* - * Each eth_dev instance is assigned its own Verbs context, - * since this one is consumed, let the next iteration open - * another. - */ - ctx = NULL; - continue; -port_error: - if (priv) - rte_free(priv); - if (pd) - claim_zero(mlx5_glue->dealloc_pd(pd)); - if (eth_dev && rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_eth_dev_release_port(eth_dev); - break; + return 0; } - /* - * XXX if something went wrong in the loop above, there is a resource - * leak (ctx, pd, priv, dpdk ethdev) but we can do nothing about it as - * long as the dpdk does not provide a way to deallocate a ethdev and a - * way to enumerate the registered ethdevs to free the previous ones. - */ error: + if (priv) + rte_free(priv); + if (pd) + claim_zero(mlx5_glue->dealloc_pd(pd)); + if (eth_dev) + rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (list) - mlx5_glue->free_device_list(list); if (err) { rte_errno = err; return -rte_errno; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 997b04a33..f55ff4a21 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -159,7 +159,6 @@ struct priv { unsigned int vlan_filter_n; /* Number of configured VLAN filters. */ /* Device properties. */ uint16_t mtu; /* Configured MTU. */ - uint8_t port; /* Physical port number. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index ebe5cb6e3..819f5baad 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -166,7 +166,7 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) if (dev_port == dev_port_prev) goto try_dev_id; dev_port_prev = dev_port; - if (dev_port == (priv->port - 1u)) + if (dev_port == 0) strlcpy(match, name, sizeof(match)); } closedir(dir); diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 669b91319..5057561ae 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -434,7 +434,7 @@ mlx5_txq_ibv_new(struct rte_eth_dev *dev, uint16_t idx) /* Move the QP to this state. */ .qp_state = IBV_QPS_INIT, /* Primary port number. */ - .port_num = priv->port + .port_num = 1, }; ret = mlx5_glue->modify_qp(tmpl.qp, &attr.mod, (IBV_QP_STATE | IBV_QP_PORT)); From patchwork Wed Jul 4 17:27:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42259 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 51BB61BF59; Wed, 4 Jul 2018 19:27:59 +0200 (CEST) Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id A75D01BF23 for ; Wed, 4 Jul 2018 19:27:57 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id v25-v6so7147830wmc.0 for ; Wed, 04 Jul 2018 10:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=fT1UOIK/XCbuOKTx0zT7+xbKpOhTb8rdQQ55imjh3+B6D6OvOSQJEwmFxzCcDBmRNQ V5b1w9W8hcRUvktSGaKhAdUGBjPy9+0dmdPozZ7LAB/Ovvz6sKkQA8GG0wRkK24x0tdn 0B7eeGia4LbQmSVgfwh7ag9cXpfcv1iMYnMFmrnvn0T7wgDDzKeN6JCW3WNRBEJnFPhe uziq6p1IkkVh8/kKqKEV/HdIdgRSUHYs82pHyV9R/xPiQ6oUPVekeHo8Mn76pShRfdgs S1E/OaGc+xiP0DDDPsOtSV1YbB3m+pkgcqYtKvFmEVZ8TtgvAwuZXUr9sNKYKaG6eMC7 kzBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=WeDqRqvmdzsXRg7MOVj4HafcpqYPtVEfIa/BL915r50=; b=eN+Fwl7rc+aSjpOFSwoBCOOzDvIFZ1jp7rOPo+ptSYWT3tdPUizoID6TQvqoeZMNLP 3PvWDZWi8MnBahujyQR0Z5AAZCIfOKtD9zoEUlmnPVPcb1GGFYHVv8WJAJLvFofAf4dw Pihvvk4DD2QcdVMQLNirncfRuuQeIeNHCd61QWL+e7ehtwpu+tZ+iIPN1j7FqChLov30 zmnGrnGXQCwP36at58c+LGiXyX0QbuIVYl/I10N3rDJMes4zn9VrW6zsHSUOUpSnio5r xCaO8qGmbq2NcR04bB9BLp35FRwdIixO8SaL6+DrrgTwv3qsLOfviFX6a59zmfiWyE6z O3RA== X-Gm-Message-State: APt69E3rFH9SacD9VL05YwPWiXrbD3Y6crCKD2S/SZpxTVNcfMs95+2l 5L0T9V+9/aVTto+Hu7ivolKO2q2M X-Google-Smtp-Source: AAOMgpfJxWMqh29FOP/w51VTzEAHsdcOY7S69KLmGccmKMnFp2QhYF4SIivMTGL5flgp1GtoHCZkUg== X-Received: by 2002:a1c:8893:: with SMTP id k141-v6mr2203134wmd.36.1530725277297; Wed, 04 Jul 2018 10:27:57 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id v15-v6sm3008635wmc.16.2018.07.04.10.27.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:27:56 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:40 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-5-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 04/10] net/mlx5: split PCI from generic probing code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" All the generic probing code needs is an IB device. While this device is currently supplied by a PCI lookup, other methods will be added soon. This patch divides the original function, which has become huge over time, as follows: 1. PCI-specific (mlx5_pci_probe()). 2. Verbs device (mlx5_dev_spawn()). (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Shahaf Shuler --- v3 changes: - Moved VF device check within mlx5_pci_probe() after identifying the device instead of before that. - Merged mlx5_dev_spawn_one() with mlx5_dev_spawn() since there is no need anymore for an intermediate function to iterate over Verbs ports. v2 changes: - Fixed device naming. A port suffix is now appended only if several IB ports happen to be detected. - Added separate message to distinguish missing kernel drivers from other initialization errors, as it was confusing. --- drivers/net/mlx5/mlx5.c | 195 ++++++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 86 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 717d8b268..8916d4684 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "mlx5.h" #include "mlx5_utils.h" @@ -635,32 +636,31 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) } /** - * DPDK callback to register a PCI device. - * - * This function creates an Ethernet device for each port of a given - * PCI device. + * Spawn an Ethernet device from Verbs information. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). - * @param[in] pci_dev - * PCI device information. + * @param dpdk_dev + * Backing DPDK device. + * @param ibv_dev + * Verbs device. + * @param vf + * If nonzero, enable VF-specific features. * * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. + * A valid Ethernet device object on success, NULL otherwise and rte_errno + * is set. */ -static int -mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +static struct rte_eth_dev * +mlx5_dev_spawn(struct rte_device *dpdk_dev, + struct ibv_device *ibv_dev, + int vf) { - struct ibv_device **list; - struct ibv_context *ctx = NULL; + struct ibv_context *ctx; struct ibv_device_attr_ex attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; - unsigned int vf = 0; unsigned int mps; unsigned int cqe_comp; unsigned int tunnel_en = 0; @@ -672,71 +672,18 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int mprq_max_stride_size_n = 0; unsigned int mprq_min_stride_num_n = 0; unsigned int mprq_max_stride_num_n = 0; - int i; #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); - assert(pci_drv == &mlx5_driver); - list = mlx5_glue->get_device_list(&i); - if (list == NULL) { - assert(errno); - err = errno; - if (errno == ENOSYS) - DRV_LOG(ERR, - "cannot list devices, is ib_uverbs loaded?"); - goto error; - } - assert(i >= 0); - /* - * For each listed device, check related sysfs entry against - * the provided PCI ID. - */ - while (i != 0) { - struct rte_pci_addr pci_addr; - - --i; - DRV_LOG(DEBUG, "checking device \"%s\"", list[i]->name); - if (mlx5_ibv_device_to_pci_addr(list[i], &pci_addr)) - continue; - if ((pci_dev->addr.domain != pci_addr.domain) || - (pci_dev->addr.bus != pci_addr.bus) || - (pci_dev->addr.devid != pci_addr.devid) || - (pci_dev->addr.function != pci_addr.function)) - continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", - list[i]->name); - vf = ((pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) || - (pci_dev->id.device_id == - PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)); - ctx = mlx5_glue->open_device(list[i]); - rte_errno = errno; - err = rte_errno; - break; - } - mlx5_glue->free_device_list(list); - if (ctx == NULL) { - switch (err) { - case 0: - DRV_LOG(ERR, - "cannot access device, is mlx5_ib loaded?"); - err = ENODEV; - break; - case EINVAL: - DRV_LOG(ERR, - "cannot use device, are drivers up to date?"); - break; - } - goto error; + errno = 0; + ctx = mlx5_glue->open_device(ibv_dev); + if (!ctx) { + rte_errno = errno ? errno : ENODEV; + return NULL; } - DRV_LOG(DEBUG, "device opened"); #ifdef HAVE_IBV_MLX5_MOD_SWP dv_attr.comp_mask |= MLX5DV_CONTEXT_MASK_SWP; #endif @@ -855,9 +802,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, }, }; - snprintf(name, sizeof(name), PCI_PRI_FMT, - pci_dev->addr.domain, pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -866,7 +811,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, err = rte_errno; goto error; } - eth_dev->device = &pci_dev->device; + eth_dev->device = dpdk_dev; eth_dev->dev_ops = &mlx5_dev_sec_ops; err = mlx5_uar_init_secondary(eth_dev); if (err) { @@ -894,9 +839,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_select_rx_function(eth_dev); eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); - rte_eth_dev_probing_finish(eth_dev); claim_zero(mlx5_glue->close_device(ctx)); - return 0; + return eth_dev; } /* Check port status. */ err = mlx5_glue->query_port(ctx, 1, &port_attr); @@ -935,7 +879,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; - err = mlx5_args(&config, pci_dev->device.devargs); + err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; DRV_LOG(ERR, "failed to process device arguments: %s", @@ -1027,8 +971,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; - eth_dev->device = &pci_dev->device; - rte_eth_copy_pci_info(eth_dev, pci_dev); + eth_dev->device = dpdk_dev; eth_dev->device->driver = &mlx5_driver.driver; err = mlx5_uar_init_primary(eth_dev); if (err) { @@ -1146,7 +1089,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv, mem_event_cb); rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); rte_eth_dev_probing_finish(eth_dev); - return 0; + return eth_dev; } error: if (priv) @@ -1157,11 +1100,91 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, rte_eth_dev_release_port(eth_dev); if (ctx) claim_zero(mlx5_glue->close_device(ctx)); - if (err) { - rte_errno = err; + assert(err > 0); + rte_errno = err; + return NULL; +} + +/** + * DPDK callback to register a PCI device. + * + * This function spawns an Ethernet device out of a given PCI device. + * + * @param[in] pci_drv + * PCI driver structure (mlx5_driver). + * @param[in] pci_dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct ibv_device **ibv_list; + struct rte_eth_dev *eth_dev = NULL; + int vf; + int ret; + + assert(pci_drv == &mlx5_driver); + errno = 0; + ibv_list = mlx5_glue->get_device_list(&ret); + if (!ibv_list) { + rte_errno = errno ? errno : ENOSYS; + DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } - return 0; + while (ret-- > 0) { + struct rte_pci_addr pci_addr; + + DRV_LOG(DEBUG, "checking device \"%s\"", ibv_list[ret]->name); + if (mlx5_ibv_device_to_pci_addr(ibv_list[ret], &pci_addr)) + continue; + if (pci_dev->addr.domain != pci_addr.domain || + pci_dev->addr.bus != pci_addr.bus || + pci_dev->addr.devid != pci_addr.devid || + pci_dev->addr.function != pci_addr.function) + continue; + DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + ibv_list[ret]->name); + break; + } + switch (pci_dev->id.device_id) { + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF: + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF: + vf = 1; + break; + default: + vf = 0; + } + if (ret >= 0) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + mlx5_glue->free_device_list(ibv_list); + if (!ret) { + DRV_LOG(WARNING, + "no Verbs device matches PCI device " PCI_PRI_FMT "," + " are kernel drivers loaded?", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + rte_errno = ENOENT; + ret = -rte_errno; + } else if (!eth_dev) { + DRV_LOG(ERR, + "probe of PCI device " PCI_PRI_FMT " aborted after" + " encountering an error: %s", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function, + strerror(rte_errno)); + ret = -rte_errno; + } else { + rte_eth_copy_pci_info(eth_dev, pci_dev); + rte_eth_dev_probing_finish(eth_dev); + ret = 0; + } + return ret; } static const struct rte_pci_id mlx5_pci_id_map[] = { From patchwork Wed Jul 4 17:27:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42260 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D19871BF63; Wed, 4 Jul 2018 19:28:00 +0200 (CEST) Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by dpdk.org (Postfix) with ESMTP id D28C61BF2F for ; Wed, 4 Jul 2018 19:27:59 +0200 (CEST) Received: by mail-wm0-f54.google.com with SMTP id l15-v6so186425wmc.1 for ; Wed, 04 Jul 2018 10:27:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=DmREd05I1V0ABxeZRbzLBJH+y7fOYJDLHEFO/UKDYhU=; b=jKT4tU4RGGk3vqHFNn3usiFLcXmU92L8ft6s9WmCcTF2xcx9Gm3LsARu52QfnR6BpN 3YqmekaCj56Ld88OS7HYvOcmUOG8J/ONDv5THBZp3udQcs9uDDIMSbSrkbcMOG8e4wZz 9Ei91CsTsq5f+HmM+1TgcsT6aO+YuXM7BKSbaeo5H1pD8bs+B8mj3YWRc3T8Bdn1NW+d nvKNxtymmbdazF4s0vB5sWnLI0zGk7Wiccm2EEQtRxkZEygj555P2SO6jZo6aY35Wbd2 ZsnjD9eXCWZ7oHiHE0g5YBHaU8ALe1KjQoET8eQ7GAniTI9Td3v2Civmm2DfIAUlyDb2 qgjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DmREd05I1V0ABxeZRbzLBJH+y7fOYJDLHEFO/UKDYhU=; b=Lb/HAtEDQA6On2Vs/yAzISquOtyBzWnBmkdCkpYXduB9JQRnCsi/9nA3fb+bXa/Wmh OYvLnuHi+XAPAePY5qEeEj42T1pM594sLJyNzAUaFjlPZWfidv76ik4ccd8fNbbjU9nC 1700cRVel2D/RO6VLIYHuhANNvDtnhH2J4kTpCoo/MAcjdfzBaqoiXUmNiRjkqmpjDBb R4aXk7O9bd5Dd5ZYPaP++GOHEijZ54eWS6RRHqi5wYPNsXtuoGeS43oLDPvpJ68KFgEV f+JoJjYkCkwxFMTFT5E0Jl2FGPMEDuDDIvYzDaQjtrCkVAnOBfunaVOh4IGKp6K/kXQg 2Dpw== X-Gm-Message-State: APt69E2YCNXZylW4urnDv3X2oydpfFKpQsD+qpSD3x+sHUCHHWe3Fd7p /QXEdGGTx06p/6ZHmMqGxfBpYA== X-Google-Smtp-Source: AAOMgpdo8UmfZlj1Iede1rETrdGk5Prf829tzda02z1LRkQIaNkRWpBkrzzorWesxTPSnK5T6ieh7A== X-Received: by 2002:a1c:3a8f:: with SMTP id h137-v6mr2332841wma.41.1530725279284; Wed, 04 Jul 2018 10:27:59 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id r123-v6sm7622271wmb.21.2018.07.04.10.27.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:27:58 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:42 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-6-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 05/10] net/mlx5: re-indent generic probing function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since commit "net/mlx5: drop useless support for several Verbs ports" removed an inner loop, mlx5_dev_spawn() is left with an unnecessary indent level. This patch eliminates a block, moves its local variables to function scope, and re-indents its contents (diff best viewed with --ignore-all-space). No functional impact. Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming(Steven) Li --- v3 changes: - Reworded commit log since original patch was modified. This patch is also much shorter as a consequence. --- drivers/net/mlx5/mlx5.c | 578 +++++++++++++++++++++---------------------- 1 file changed, 282 insertions(+), 296 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 8916d4684..1054bf6d0 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -656,8 +656,25 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, { struct ibv_context *ctx; struct ibv_device_attr_ex attr; + struct ibv_port_attr port_attr; struct ibv_pd *pd = NULL; struct mlx5dv_context dv_attr = { .comp_mask = 0 }; + struct mlx5_dev_config config = { + .vf = !!vf, + .tx_vec_en = 1, + .rx_vec_en = 1, + .mpw_hdr_dseg = 0, + .txq_inline = MLX5_ARG_UNSET, + .txqs_inline = MLX5_ARG_UNSET, + .inline_max_packet_sz = MLX5_ARG_UNSET, + .vf_nl_en = 1, + .mprq = { + .enabled = 0, + .stride_num_n = MLX5_MPRQ_STRIDE_NUM_N, + .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, + .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, + }, + }; struct rte_eth_dev *eth_dev = NULL; struct priv *priv = NULL; int err = 0; @@ -675,6 +692,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, #ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT struct ibv_counter_set_description cs_desc = { .counter_type = 0 }; #endif + struct ether_addr mac; + char name[RTE_ETH_NAME_MAX_LEN]; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -710,11 +729,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(DEBUG, "MPW isn't supported"); mps = MLX5_MPW_DISABLED; } + config.mps = mps; #ifdef HAVE_IBV_MLX5_MOD_SWP if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_SWP) swp = dv_attr.sw_parsing_caps.sw_parsing_offloads; DRV_LOG(DEBUG, "SWP support: %u", swp); #endif + config.swp = !!swp; #ifdef HAVE_IBV_DEVICE_STRIDING_RQ_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { struct mlx5dv_striding_rq_caps mprq_caps = @@ -740,6 +761,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, mprq_caps.min_single_wqe_log_num_of_strides; mprq_max_stride_num_n = mprq_caps.max_single_wqe_log_num_of_strides; + config.mprq.stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); } #endif if (RTE_CACHE_LINE_SIZE == 128 && @@ -747,6 +770,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, cqe_comp = 0; else cqe_comp = 1; + config.cqe_comp = cqe_comp; #ifdef HAVE_IBV_DEVICE_TUNNEL_SUPPORT if (dv_attr.comp_mask & MLX5DV_CONTEXT_MASK_TUNNEL_OFFLOADS) { tunnel_en = ((dv_attr.tunnel_offloads_caps & @@ -760,6 +784,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "tunnel offloading disabled due to old OFED/rdma-core version"); #endif + config.tunnel_en = tunnel_en; #ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT mpls_en = ((dv_attr.tunnel_offloads_caps & MLX5DV_RAW_PACKET_CAP_TUNNELED_OFFLOAD_CW_MPLS_OVER_GRE) && @@ -771,326 +796,287 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DRV_LOG(WARNING, "MPLS over GRE/UDP tunnel offloading disabled due to" " old OFED/rdma-core version or firmware configuration"); #endif + config.mpls_en = mpls_en; err = mlx5_glue->query_device_ex(ctx, NULL, &attr); if (err) { DEBUG("ibv_query_device_ex() failed"); goto error; } - { - char name[RTE_ETH_NAME_MAX_LEN]; - struct ibv_port_attr port_attr; - struct ether_addr mac; - struct mlx5_dev_config config = { - .cqe_comp = cqe_comp, - .mps = mps, - .tunnel_en = tunnel_en, - .mpls_en = mpls_en, - .tx_vec_en = 1, - .rx_vec_en = 1, - .mpw_hdr_dseg = 0, - .txq_inline = MLX5_ARG_UNSET, - .txqs_inline = MLX5_ARG_UNSET, - .inline_max_packet_sz = MLX5_ARG_UNSET, - .vf_nl_en = 1, - .swp = !!swp, - .mprq = { - .enabled = 0, /* Disabled by default. */ - .stride_num_n = RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n), - .max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN, - .min_rxqs_num = MLX5_MPRQ_MIN_RXQS, - }, - }; - - rte_strlcpy(name, dpdk_dev->name, sizeof(name)); - if (rte_eal_process_type() == RTE_PROC_SECONDARY) { - eth_dev = rte_eth_dev_attach_secondary(name); - if (eth_dev == NULL) { - DRV_LOG(ERR, "can not attach rte ethdev"); - rte_errno = ENOMEM; - err = rte_errno; - goto error; - } - eth_dev->device = dpdk_dev; - eth_dev->dev_ops = &mlx5_dev_sec_ops; - err = mlx5_uar_init_secondary(eth_dev); - if (err) { - err = rte_errno; - goto error; - } - /* Receive command fd from primary process */ - err = mlx5_socket_connect(eth_dev); - if (err < 0) { - err = rte_errno; - goto error; - } - /* Remap UAR for Tx queues. */ - err = mlx5_tx_uar_remap(eth_dev, err); - if (err) { - err = rte_errno; - goto error; - } - /* - * Ethdev pointer is still required as input since - * the primary device is not accessible from the - * secondary process. - */ - eth_dev->rx_pkt_burst = - mlx5_select_rx_function(eth_dev); - eth_dev->tx_pkt_burst = - mlx5_select_tx_function(eth_dev); - claim_zero(mlx5_glue->close_device(ctx)); - return eth_dev; - } - /* Check port status. */ - err = mlx5_glue->query_port(ctx, 1, &port_attr); - if (err) { - DRV_LOG(ERR, "port query failed: %s", strerror(err)); - goto error; - } - if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { - DRV_LOG(ERR, "port is not configured in Ethernet mode"); - err = EINVAL; - goto error; - } - if (port_attr.state != IBV_PORT_ACTIVE) - DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", - mlx5_glue->port_state_str(port_attr.state), - port_attr.state); - /* Allocate protection domain. */ - pd = mlx5_glue->alloc_pd(ctx); - if (pd == NULL) { - DRV_LOG(ERR, "PD allocation failure"); - err = ENOMEM; - goto error; - } - /* from rte_ethdev.c */ - priv = rte_zmalloc("ethdev private structure", - sizeof(*priv), - RTE_CACHE_LINE_SIZE); - if (priv == NULL) { - DRV_LOG(ERR, "priv allocation failure"); - err = ENOMEM; - goto error; - } - priv->ctx = ctx; - strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, - sizeof(priv->ibdev_path)); - priv->device_attr = attr; - priv->pd = pd; - priv->mtu = ETHER_MTU; - err = mlx5_args(&config, dpdk_dev->devargs); - if (err) { - err = rte_errno; - DRV_LOG(ERR, "failed to process device arguments: %s", - strerror(rte_errno)); - goto error; - } - config.hw_csum = !!(attr.device_cap_flags_ex & - IBV_DEVICE_RAW_IP_CSUM); - DRV_LOG(DEBUG, "checksum offloading is %ssupported", - (config.hw_csum ? "" : "not ")); -#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT - config.flow_counter_en = !!attr.max_counter_sets; - mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); - DRV_LOG(DEBUG, - "counter type = %d, num of cs = %ld, attributes = %d", - cs_desc.counter_type, cs_desc.num_of_cs, - cs_desc.attributes); -#endif - config.ind_table_max_size = - attr.rss_caps.max_rwq_indirection_table_size; - /* Remove this check once DPDK supports larger/variable - * indirection tables. */ - if (config.ind_table_max_size > - (unsigned int)ETH_RSS_RETA_SIZE_512) - config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; - DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", - config.ind_table_max_size); - config.hw_vlan_strip = !!(attr.raw_packet_caps & - IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); - DRV_LOG(DEBUG, "VLAN stripping is %ssupported", - (config.hw_vlan_strip ? "" : "not ")); - - config.hw_fcs_strip = !!(attr.raw_packet_caps & - IBV_RAW_PACKET_CAP_SCATTER_FCS); - DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", - (config.hw_fcs_strip ? "" : "not ")); - -#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING - config.hw_padding = !!attr.rx_pad_end_addr_align; -#endif - DRV_LOG(DEBUG, - "hardware Rx end alignment padding is %ssupported", - (config.hw_padding ? "" : "not ")); - config.vf = vf; - config.tso = (attr.tso_caps.max_tso > 0 && - (attr.tso_caps.supported_qpts & - (1 << IBV_QPT_RAW_PACKET))); - if (config.tso) - config.tso_max_payload_sz = attr.tso_caps.max_tso; - if (config.mps && !mps) { - DRV_LOG(ERR, - "multi-packet send not supported on this device" - " (" MLX5_TXQ_MPW_EN ")"); - err = ENOTSUP; - goto error; - } - DRV_LOG(INFO, "%s MPS is %s", - config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", - config.mps != MLX5_MPW_DISABLED ? "enabled" : - "disabled"); - if (config.cqe_comp && !cqe_comp) { - DRV_LOG(WARNING, "Rx CQE compression isn't supported"); - config.cqe_comp = 0; - } - if (config.mprq.enabled && mprq) { - if (config.mprq.stride_num_n > mprq_max_stride_num_n || - config.mprq.stride_num_n < mprq_min_stride_num_n) { - config.mprq.stride_num_n = - RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, - mprq_min_stride_num_n); - DRV_LOG(WARNING, - "the number of strides" - " for Multi-Packet RQ is out of range," - " setting default value (%u)", - 1 << config.mprq.stride_num_n); - } - config.mprq.min_stride_size_n = mprq_min_stride_size_n; - config.mprq.max_stride_size_n = mprq_max_stride_size_n; - } else if (config.mprq.enabled && !mprq) { - DRV_LOG(WARNING, "Multi-Packet RQ isn't supported"); - config.mprq.enabled = 0; - } - eth_dev = rte_eth_dev_allocate(name); + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + if (rte_eal_process_type() == RTE_PROC_SECONDARY) { + eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { - DRV_LOG(ERR, "can not allocate rte ethdev"); - err = ENOMEM; + DRV_LOG(ERR, "can not attach rte ethdev"); + rte_errno = ENOMEM; + err = rte_errno; goto error; } - eth_dev->data->dev_private = priv; - priv->dev_data = eth_dev->data; - eth_dev->data->mac_addrs = priv->mac; eth_dev->device = dpdk_dev; - eth_dev->device->driver = &mlx5_driver.driver; - err = mlx5_uar_init_primary(eth_dev); + eth_dev->dev_ops = &mlx5_dev_sec_ops; + err = mlx5_uar_init_secondary(eth_dev); if (err) { err = rte_errno; goto error; } - /* Configure the first MAC address by default. */ - if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { - DRV_LOG(ERR, - "port %u cannot get MAC address, is mlx5_en" - " loaded? (errno: %s)", - eth_dev->data->port_id, strerror(rte_errno)); - err = ENODEV; - goto error; - } - DRV_LOG(INFO, - "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", - eth_dev->data->port_id, - mac.addr_bytes[0], mac.addr_bytes[1], - mac.addr_bytes[2], mac.addr_bytes[3], - mac.addr_bytes[4], mac.addr_bytes[5]); -#ifndef NDEBUG - { - char ifname[IF_NAMESIZE]; - - if (mlx5_get_ifname(eth_dev, &ifname) == 0) - DRV_LOG(DEBUG, "port %u ifname is \"%s\"", - eth_dev->data->port_id, ifname); - else - DRV_LOG(DEBUG, "port %u ifname is unknown", - eth_dev->data->port_id); - } -#endif - /* Get actual MTU if possible. */ - err = mlx5_get_mtu(eth_dev, &priv->mtu); - if (err) { + /* Receive command fd from primary process */ + err = mlx5_socket_connect(eth_dev); + if (err < 0) { err = rte_errno; goto error; } - DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, - priv->mtu); - /* - * Initialize burst functions to prevent crashes before link-up. - */ - eth_dev->rx_pkt_burst = removed_rx_burst; - eth_dev->tx_pkt_burst = removed_tx_burst; - eth_dev->dev_ops = &mlx5_dev_ops; - /* Register MAC address. */ - claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; - mlx5_nl_mac_addr_sync(eth_dev); - } - TAILQ_INIT(&priv->flows); - TAILQ_INIT(&priv->ctrl_flows); - /* Hint libmlx5 to use PMD allocator for data plane resources */ - struct mlx5dv_ctx_allocators alctr = { - .alloc = &mlx5_alloc_verbs_buf, - .free = &mlx5_free_verbs_buf, - .data = priv, - }; - mlx5_glue->dv_set_context_attr(ctx, - MLX5DV_CTX_ATTR_BUF_ALLOCATORS, - (void *)((uintptr_t)&alctr)); - /* Bring Ethernet device up. */ - DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", - eth_dev->data->port_id); - mlx5_set_link_up(eth_dev); - /* - * Even though the interrupt handler is not installed yet, - * interrupts will still trigger on the asyn_fd from - * Verbs context returned by ibv_open_device(). - */ - mlx5_link_update(eth_dev, 0); - /* Store device configuration on private structure. */ - priv->config = config; - /* Create drop queue. */ - err = mlx5_flow_create_drop_queue(eth_dev); + /* Remap UAR for Tx queues. */ + err = mlx5_tx_uar_remap(eth_dev, err); if (err) { - DRV_LOG(ERR, "port %u drop queue allocation failed: %s", - eth_dev->data->port_id, strerror(rte_errno)); err = rte_errno; goto error; } - /* Supported Verbs flow priority number detection. */ - if (verb_priorities == 0) - verb_priorities = mlx5_get_max_verbs_prio(eth_dev); - if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { - DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", - eth_dev->data->port_id, verb_priorities); - err = ENOTSUP; - goto error; - } - priv->config.max_verbs_prio = verb_priorities; /* - * Once the device is added to the list of memory event - * callback, its global MR cache table cannot be expanded - * on the fly because of deadlock. If it overflows, lookup - * should be done by searching MR list linearly, which is slow. + * Ethdev pointer is still required as input since + * the primary device is not accessible from the + * secondary process. */ - err = mlx5_mr_btree_init(&priv->mr.cache, - MLX5_MR_BTREE_CACHE_N * 2, - eth_dev->device->numa_node); - if (err) { - err = rte_errno; - goto error; - } - /* Add device to memory callback list. */ - rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); - LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, - priv, mem_event_cb); - rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); - rte_eth_dev_probing_finish(eth_dev); + eth_dev->rx_pkt_burst = mlx5_select_rx_function(eth_dev); + eth_dev->tx_pkt_burst = mlx5_select_tx_function(eth_dev); + claim_zero(mlx5_glue->close_device(ctx)); return eth_dev; } + /* Check port status. */ + err = mlx5_glue->query_port(ctx, 1, &port_attr); + if (err) { + DRV_LOG(ERR, "port query failed: %s", strerror(err)); + goto error; + } + if (port_attr.link_layer != IBV_LINK_LAYER_ETHERNET) { + DRV_LOG(ERR, "port is not configured in Ethernet mode"); + err = EINVAL; + goto error; + } + if (port_attr.state != IBV_PORT_ACTIVE) + DRV_LOG(DEBUG, "port is not active: \"%s\" (%d)", + mlx5_glue->port_state_str(port_attr.state), + port_attr.state); + /* Allocate protection domain. */ + pd = mlx5_glue->alloc_pd(ctx); + if (pd == NULL) { + DRV_LOG(ERR, "PD allocation failure"); + err = ENOMEM; + goto error; + } + priv = rte_zmalloc("ethdev private structure", + sizeof(*priv), + RTE_CACHE_LINE_SIZE); + if (priv == NULL) { + DRV_LOG(ERR, "priv allocation failure"); + err = ENOMEM; + goto error; + } + priv->ctx = ctx; + strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, + sizeof(priv->ibdev_path)); + priv->device_attr = attr; + priv->pd = pd; + priv->mtu = ETHER_MTU; + err = mlx5_args(&config, dpdk_dev->devargs); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + goto error; + } + config.hw_csum = !!(attr.device_cap_flags_ex & IBV_DEVICE_RAW_IP_CSUM); + DRV_LOG(DEBUG, "checksum offloading is %ssupported", + (config.hw_csum ? "" : "not ")); +#ifdef HAVE_IBV_DEVICE_COUNTERS_SET_SUPPORT + config.flow_counter_en = !!attr.max_counter_sets; + mlx5_glue->describe_counter_set(ctx, 0, &cs_desc); + DRV_LOG(DEBUG, "counter type = %d, num of cs = %ld, attributes = %d", + cs_desc.counter_type, cs_desc.num_of_cs, + cs_desc.attributes); +#endif + config.ind_table_max_size = + attr.rss_caps.max_rwq_indirection_table_size; + /* + * Remove this check once DPDK supports larger/variable + * indirection tables. + */ + if (config.ind_table_max_size > (unsigned int)ETH_RSS_RETA_SIZE_512) + config.ind_table_max_size = ETH_RSS_RETA_SIZE_512; + DRV_LOG(DEBUG, "maximum Rx indirection table size is %u", + config.ind_table_max_size); + config.hw_vlan_strip = !!(attr.raw_packet_caps & + IBV_RAW_PACKET_CAP_CVLAN_STRIPPING); + DRV_LOG(DEBUG, "VLAN stripping is %ssupported", + (config.hw_vlan_strip ? "" : "not ")); + config.hw_fcs_strip = !!(attr.raw_packet_caps & + IBV_RAW_PACKET_CAP_SCATTER_FCS); + DRV_LOG(DEBUG, "FCS stripping configuration is %ssupported", + (config.hw_fcs_strip ? "" : "not ")); +#ifdef HAVE_IBV_WQ_FLAG_RX_END_PADDING + config.hw_padding = !!attr.rx_pad_end_addr_align; +#endif + DRV_LOG(DEBUG, "hardware Rx end alignment padding is %ssupported", + (config.hw_padding ? "" : "not ")); + config.tso = (attr.tso_caps.max_tso > 0 && + (attr.tso_caps.supported_qpts & + (1 << IBV_QPT_RAW_PACKET))); + if (config.tso) + config.tso_max_payload_sz = attr.tso_caps.max_tso; + if (config.mps && !mps) { + DRV_LOG(ERR, + "multi-packet send not supported on this device" + " (" MLX5_TXQ_MPW_EN ")"); + err = ENOTSUP; + goto error; + } + DRV_LOG(INFO, "%sMPS is %s", + config.mps == MLX5_MPW_ENHANCED ? "enhanced " : "", + config.mps != MLX5_MPW_DISABLED ? "enabled" : "disabled"); + if (config.cqe_comp && !cqe_comp) { + DRV_LOG(WARNING, "Rx CQE compression isn't supported"); + config.cqe_comp = 0; + } + if (config.mprq.enabled && mprq) { + if (config.mprq.stride_num_n > mprq_max_stride_num_n || + config.mprq.stride_num_n < mprq_min_stride_num_n) { + config.mprq.stride_num_n = + RTE_MAX(MLX5_MPRQ_STRIDE_NUM_N, + mprq_min_stride_num_n); + DRV_LOG(WARNING, + "the number of strides" + " for Multi-Packet RQ is out of range," + " setting default value (%u)", + 1 << config.mprq.stride_num_n); + } + config.mprq.min_stride_size_n = mprq_min_stride_size_n; + config.mprq.max_stride_size_n = mprq_max_stride_size_n; + } else if (config.mprq.enabled && !mprq) { + DRV_LOG(WARNING, "Multi-Packet RQ isn't supported"); + config.mprq.enabled = 0; + } + eth_dev = rte_eth_dev_allocate(name); + if (eth_dev == NULL) { + DRV_LOG(ERR, "can not allocate rte ethdev"); + err = ENOMEM; + goto error; + } + eth_dev->data->dev_private = priv; + priv->dev_data = eth_dev->data; + eth_dev->data->mac_addrs = priv->mac; + eth_dev->device = dpdk_dev; + eth_dev->device->driver = &mlx5_driver.driver; + err = mlx5_uar_init_primary(eth_dev); + if (err) { + err = rte_errno; + goto error; + } + /* Configure the first MAC address by default. */ + if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) { + DRV_LOG(ERR, + "port %u cannot get MAC address, is mlx5_en" + " loaded? (errno: %s)", + eth_dev->data->port_id, strerror(rte_errno)); + err = ENODEV; + goto error; + } + DRV_LOG(INFO, + "port %u MAC address is %02x:%02x:%02x:%02x:%02x:%02x", + eth_dev->data->port_id, + mac.addr_bytes[0], mac.addr_bytes[1], + mac.addr_bytes[2], mac.addr_bytes[3], + mac.addr_bytes[4], mac.addr_bytes[5]); +#ifndef NDEBUG + { + char ifname[IF_NAMESIZE]; + + if (mlx5_get_ifname(eth_dev, &ifname) == 0) + DRV_LOG(DEBUG, "port %u ifname is \"%s\"", + eth_dev->data->port_id, ifname); + else + DRV_LOG(DEBUG, "port %u ifname is unknown", + eth_dev->data->port_id); + } +#endif + /* Get actual MTU if possible. */ + err = mlx5_get_mtu(eth_dev, &priv->mtu); + if (err) { + err = rte_errno; + goto error; + } + DRV_LOG(DEBUG, "port %u MTU is %u", eth_dev->data->port_id, + priv->mtu); + /* Initialize burst functions to prevent crashes before link-up. */ + eth_dev->rx_pkt_burst = removed_rx_burst; + eth_dev->tx_pkt_burst = removed_tx_burst; + eth_dev->dev_ops = &mlx5_dev_ops; + /* Register MAC address. */ + claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); + priv->nl_socket = -1; + priv->nl_sn = 0; + if (vf && config.vf_nl_en) { + priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); + if (priv->nl_socket < 0) + priv->nl_socket = -1; + mlx5_nl_mac_addr_sync(eth_dev); + } + TAILQ_INIT(&priv->flows); + TAILQ_INIT(&priv->ctrl_flows); + /* Hint libmlx5 to use PMD allocator for data plane resources */ + struct mlx5dv_ctx_allocators alctr = { + .alloc = &mlx5_alloc_verbs_buf, + .free = &mlx5_free_verbs_buf, + .data = priv, + }; + mlx5_glue->dv_set_context_attr(ctx, MLX5DV_CTX_ATTR_BUF_ALLOCATORS, + (void *)((uintptr_t)&alctr)); + /* Bring Ethernet device up. */ + DRV_LOG(DEBUG, "port %u forcing Ethernet interface up", + eth_dev->data->port_id); + mlx5_set_link_up(eth_dev); + /* + * Even though the interrupt handler is not installed yet, + * interrupts will still trigger on the asyn_fd from + * Verbs context returned by ibv_open_device(). + */ + mlx5_link_update(eth_dev, 0); + /* Store device configuration on private structure. */ + priv->config = config; + /* Create drop queue. */ + err = mlx5_flow_create_drop_queue(eth_dev); + if (err) { + DRV_LOG(ERR, "port %u drop queue allocation failed: %s", + eth_dev->data->port_id, strerror(rte_errno)); + err = rte_errno; + goto error; + } + /* Supported Verbs flow priority number detection. */ + if (verb_priorities == 0) + verb_priorities = mlx5_get_max_verbs_prio(eth_dev); + if (verb_priorities < MLX5_VERBS_FLOW_PRIO_8) { + DRV_LOG(ERR, "port %u wrong Verbs flow priorities: %u", + eth_dev->data->port_id, verb_priorities); + err = ENOTSUP; + goto error; + } + priv->config.max_verbs_prio = verb_priorities; + /* + * Once the device is added to the list of memory event + * callback, its global MR cache table cannot be expanded + * on the fly because of deadlock. If it overflows, lookup + * should be done by searching MR list linearly, which is slow. + */ + err = mlx5_mr_btree_init(&priv->mr.cache, + MLX5_MR_BTREE_CACHE_N * 2, + eth_dev->device->numa_node); + if (err) { + err = rte_errno; + goto error; + } + /* Add device to memory callback list. */ + rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); + LIST_INSERT_HEAD(&mlx5_shared_data->mem_event_cb_list, + priv, mem_event_cb); + rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); + return eth_dev; error: if (priv) rte_free(priv); From patchwork Wed Jul 4 17:27:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42261 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 637F91BF51; Wed, 4 Jul 2018 19:28:04 +0200 (CEST) Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by dpdk.org (Postfix) with ESMTP id F2FAB1BF67 for ; Wed, 4 Jul 2018 19:28:01 +0200 (CEST) Received: by mail-wm0-f66.google.com with SMTP id w16-v6so7092160wmc.2 for ; Wed, 04 Jul 2018 10:28:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=EM3d5ieHBtd5+GEdF7A4rgOgY3EDx79w9cdZs81cSjg=; b=N4FHoykg6jluw82u/08YKMDht6GuAxnRcor7lNwWXo/kr6848y6/9AujX7jF3dPX+C +AVwAmgYy43zJcjlEPdrWonxdNbTnJVcJTaf+5ek391fvvVKhLToHORhTHXsI2OjCK/F iemjt1hAMsraJ17OhQHDMg5OYgv3/i+8WlN7YzQKKonA28LGS2tUjkmtL/+lEoqgkuG4 SAsLDHZ2FjOPLk5U/I6OCf0DNw9ntDBttvxRYgiJX2bTw197W9X1wjLsSl38lhomEq3h G1BUkT/SlttIVkqDjSVKNLbHSB6yCn+AS+H6qsVjK3Ix/qmHEL1PLExPtvZTA6T4t12F 0Z3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=EM3d5ieHBtd5+GEdF7A4rgOgY3EDx79w9cdZs81cSjg=; b=cLFjoXyUUAWwg0j489gq9tZfu2sOVSBbQAnK/CbAvMNfWVoYpkXLYE3pTev2cyEkPn pnriKClwYyv2Mcai6UjBB24Sr4Zi8wCAQXzSE5b13aY0nJ/M+ogA4c0/MPrknzSOorbM Ud+AMeoX50SvYmNEV0ToDhOT/Q39VdTWv5HbS1rwtifdju+dIi7KogaYt7d9/O+Jhf2o 7WDgWB2G1oN8oKaw9aYSktwSsh/JK6zA+wgmS6mlg/Y7aL4Wyj5CvPofkubLcOVUKN7/ iigRXR5pMk+qhcC0twRQJI5dTeIOUE0yQv31xf2EiqN5okek0S4RzAHks6+IY5TXCZ2J 6Ejg== X-Gm-Message-State: APt69E0nslX5lIpJpbttB646C/xS6d2YQEgQEvmdr33MkHpEIgYSa8iI yp0al2G+cQe7Hnn5EV5MlW/Qlw== X-Google-Smtp-Source: AAOMgpdyxyGbiZxlskaG6TvGTBtXMNR6axqytosFZWuktjN8ZzZvYzTPJywBxFnWtE2Xb4tThuNx4w== X-Received: by 2002:a1c:8a04:: with SMTP id m4-v6mr2088134wmd.137.1530725281217; Wed, 04 Jul 2018 10:28:01 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 127-v6sm5938960wmd.18.2018.07.04.10.28.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:28:00 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:44 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Xueming Li Message-ID: <20180704172322.22571-7-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 06/10] net/mlx5: add port representor awareness X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The current PCI probing method is not aware of Verbs port representors, which appear as standard Verbs devices bound to the same PCI address and cannot be distinguished. Problem is that more often than not, the wrong Verbs device is used, resulting in unexpected traffic. This patch makes the driver discard representors to only use the master device. If unable to identify it (e.g. kernel drivers not recent enough), either: - There is only one matching device which isn't identified as a representor, in that case use it. - Otherwise log an error and do not probe the device. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li Cc: Xueming Li --- v3 changes: - Replaced all heuristics (including mlx5_cmp_ibv_name()) with Netlink queries to associate IB devices with network interfaces. - Reworded commit log. v2 changes: - Fixed digit detection in mlx5_cmp_ibv_name() so that "foo1" and "foo10" are compared on the integer conversion of "1" against "10" instead of "" and "0". --- drivers/net/mlx5/Makefile | 30 ++++ drivers/net/mlx5/mlx5.c | 109 +++++++++++++-- drivers/net/mlx5/mlx5.h | 16 ++- drivers/net/mlx5/mlx5_nl.c | 297 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 428 insertions(+), 24 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 955861a41..745752e23 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -152,6 +152,36 @@ mlx5_autoconf.h.new: $(RTE_SDK)/buildtools/auto-config-h.sh infiniband/verbs.h \ enum IBV_FLOW_SPEC_ACTION_COUNT \ $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_CMD_GET \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_CMD_GET \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_CMD_PORT_GET \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_CMD_PORT_GET \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_DEV_INDEX \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_DEV_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_DEV_NAME \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_DEV_NAME \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_PORT_INDEX \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_PORT_INDEX \ + $(AUTOCONF_OUTPUT) + $Q sh -- '$<' '$@' \ + HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX \ + linux/rdma_netlink.h \ + enum RDMA_NLDEV_ATTR_NDEV_INDEX \ + $(AUTOCONF_OUTPUT) # Create mlx5_autoconf.h or update it in case it differs from the new one. diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 1054bf6d0..d06ba9886 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -13,6 +13,7 @@ #include #include #include +#include #include /* Verbs header. */ @@ -274,8 +275,10 @@ mlx5_dev_close(struct rte_eth_dev *dev) mlx5_socket_uninit(dev); if (priv->config.vf) mlx5_nl_mac_addr_flush(dev); - if (priv->nl_socket >= 0) - close(priv->nl_socket); + if (priv->nl_socket_route >= 0) + close(priv->nl_socket_route); + if (priv->nl_socket_rdma >= 0) + close(priv->nl_socket_rdma); ret = mlx5_hrxq_ibv_verify(dev); if (ret) DRV_LOG(WARNING, "port %u some hash Rx queue still remain", @@ -876,6 +879,10 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->device_attr = attr; priv->pd = pd; priv->mtu = ETHER_MTU; + /* Some internal functions rely on Netlink sockets, open them now. */ + priv->nl_socket_rdma = mlx5_nl_init(0, NETLINK_RDMA); + priv->nl_socket_route = mlx5_nl_init(RTMGRP_LINK, NETLINK_ROUTE); + priv->nl_sn = 0; err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -1010,14 +1017,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, eth_dev->dev_ops = &mlx5_dev_ops; /* Register MAC address. */ claim_zero(mlx5_mac_addr_add(eth_dev, &mac, 0, 0)); - priv->nl_socket = -1; - priv->nl_sn = 0; - if (vf && config.vf_nl_en) { - priv->nl_socket = mlx5_nl_init(RTMGRP_LINK); - if (priv->nl_socket < 0) - priv->nl_socket = -1; + if (vf && config.vf_nl_en) mlx5_nl_mac_addr_sync(eth_dev); - } TAILQ_INIT(&priv->flows); TAILQ_INIT(&priv->ctrl_flows); /* Hint libmlx5 to use PMD allocator for data plane resources */ @@ -1078,8 +1079,13 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, rte_rwlock_write_unlock(&mlx5_shared_data->mem_event_rwlock); return eth_dev; error: - if (priv) + if (priv) { + if (priv->nl_socket_route >= 0) + close(priv->nl_socket_route); + if (priv->nl_socket_rdma >= 0) + close(priv->nl_socket_rdma); rte_free(priv); + } if (pd) claim_zero(mlx5_glue->dealloc_pd(pd)); if (eth_dev) @@ -1110,6 +1116,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, { struct ibv_device **ibv_list; struct rte_eth_dev *eth_dev = NULL; + unsigned int n = 0; int vf; int ret; @@ -1121,6 +1128,9 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?"); return -rte_errno; } + + struct ibv_device *ibv_match[ret + 1]; + while (ret-- > 0) { struct rte_pci_addr pci_addr; @@ -1132,10 +1142,81 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid != pci_addr.devid || pci_dev->addr.function != pci_addr.function) continue; - DRV_LOG(INFO, "PCI information matches, using device \"%s\"", + DRV_LOG(INFO, "PCI information matches for device \"%s\"", ibv_list[ret]->name); + ibv_match[n++] = ibv_list[ret]; + } + ibv_match[n] = NULL; + + unsigned int ifindex[n]; + struct mlx5_switch_info info[n]; + int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; + int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; + unsigned int i; + + /* + * The existence of several matching entries (n > 1) means port + * representors have been instantiated. No existing Verbs call nor + * /sys entries can tell them apart, this can only be done through + * Netlink calls assuming kernel drivers are recent enough to + * support them. + * + * In the event of identification failure through Netlink, either: + * + * 1. No device matches (n == 0), complain and bail out. + * 2. A single IB device matches (n == 1) and is not a representor, + * assume no switch support. + * 3. Otherwise no safe assumptions can be made; complain louder and + * bail out. + */ + for (i = 0; i != n; ++i) { + if (nl_rdma < 0) + ifindex[i] = 0; + else + ifindex[i] = mlx5_nl_ifindex(nl_rdma, + ibv_match[i]->name); + if (nl_route < 0 || + !ifindex[i] || + mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { + ifindex[i] = 0; + memset(&info[i], 0, sizeof(info[i])); + continue; + } + } + if (nl_rdma >= 0) + close(nl_rdma); + if (nl_route >= 0) + close(nl_route); + /* Look for master device. */ + for (i = 0; i != n; ++i) { + if (!info[i].master) + continue; + /* Make it the first entry. */ + if (i == 0) + break; + ibv_match[n] = ibv_match[0]; + ibv_match[0] = ibv_match[i]; + ibv_match[n] = NULL; break; } + if (n && i == n) { + if (n == 1 && !info[0].representor) { + /* Case #2. */ + DRV_LOG(INFO, "no switch support detected"); + } else if (n == 1) { + /* Case #3. */ + DRV_LOG(ERR, + "device looks like a port representor, this is" + " not supported yet"); + n = 0; + } else { + /* Case #3. */ + DRV_LOG(ERR, + "unable to tell which of the matching devices" + " is the master (lack of kernel support?)"); + n = 0; + } + } switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1146,10 +1227,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, default: vf = 0; } - if (ret >= 0) - eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_list[ret], vf); + if (n) + eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); mlx5_glue->free_device_list(ibv_list); - if (!ret) { + if (!n) { DRV_LOG(WARNING, "no Verbs device matches PCI device " PCI_PRI_FMT "," " are kernel drivers loaded?", diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index f55ff4a21..704046270 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -53,6 +53,14 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX5BF = 0xa2d2, }; +/** Switch information returned by mlx5_nl_switch_info(). */ +struct mlx5_switch_info { + uint32_t master:1; /**< Master device. */ + uint32_t representor:1; /**< Representor device. */ + int32_t port_name; /**< Representor port name. */ + uint64_t switch_id; /**< Switch identifier. */ +}; + LIST_HEAD(mlx5_dev_list, priv); /* Shared memory between primary and secondary processes. */ @@ -195,7 +203,8 @@ struct priv { struct mlx5_dev_config config; /* Device configuration. */ struct mlx5_verbs_alloc_ctx verbs_alloc_ctx; /* Context for Verbs allocator. */ - int nl_socket; /* Netlink socket. */ + int nl_socket_rdma; /* Netlink socket (NETLINK_RDMA). */ + int nl_socket_route; /* Netlink socket (NETLINK_ROUTE). */ uint32_t nl_sn; /* Netlink message sequence number. */ }; @@ -342,7 +351,7 @@ int mlx5_socket_connect(struct rte_eth_dev *priv); /* mlx5_nl.c */ -int mlx5_nl_init(uint32_t nlgroups); +int mlx5_nl_init(uint32_t nlgroups, int protocol); int mlx5_nl_mac_addr_add(struct rte_eth_dev *dev, struct ether_addr *mac, uint32_t index); int mlx5_nl_mac_addr_remove(struct rte_eth_dev *dev, struct ether_addr *mac, @@ -351,5 +360,8 @@ void mlx5_nl_mac_addr_sync(struct rte_eth_dev *dev); void mlx5_nl_mac_addr_flush(struct rte_eth_dev *dev); int mlx5_nl_promisc(struct rte_eth_dev *dev, int enable); int mlx5_nl_allmulti(struct rte_eth_dev *dev, int enable); +unsigned int mlx5_nl_ifindex(int nl, const char *name); +int mlx5_nl_switch_info(int nl, unsigned int ifindex, + struct mlx5_switch_info *info); #endif /* RTE_PMD_MLX5_H_ */ diff --git a/drivers/net/mlx5/mlx5_nl.c b/drivers/net/mlx5/mlx5_nl.c index dca85835a..58ef2f4f0 100644 --- a/drivers/net/mlx5/mlx5_nl.c +++ b/drivers/net/mlx5/mlx5_nl.c @@ -3,10 +3,21 @@ * Copyright 2018 Mellanox Technologies, Ltd */ +#include +#include #include #include +#include +#include +#include +#include +#include +#include +#include #include +#include + #include "mlx5.h" #include "mlx5_utils.h" @@ -27,6 +38,29 @@ ((struct rtattr *)(((char *)(r)) + NLMSG_ALIGN(sizeof(struct ndmsg)))) #endif +/* + * The following definitions are normally found in rdma/rdma_netlink.h, + * however they are so recent that most systems do not expose them yet. + */ +#ifndef HAVE_RDMA_NLDEV_CMD_GET +#define RDMA_NLDEV_CMD_GET 1 +#endif +#ifndef HAVE_RDMA_NLDEV_CMD_PORT_GET +#define RDMA_NLDEV_CMD_PORT_GET 5 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_DEV_INDEX +#define RDMA_NLDEV_ATTR_DEV_INDEX 1 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_DEV_NAME +#define RDMA_NLDEV_ATTR_DEV_NAME 2 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX +#define RDMA_NLDEV_ATTR_PORT_INDEX 3 +#endif +#ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX +#define RDMA_NLDEV_ATTR_NDEV_INDEX 50 +#endif + /* Add/remove MAC address through Netlink */ struct mlx5_nl_mac_addr { struct ether_addr (*mac)[]; @@ -34,18 +68,27 @@ struct mlx5_nl_mac_addr { int mac_n; /**< Number of addresses in the array. */ }; +/** Data structure used by mlx5_nl_ifindex_cb(). */ +struct mlx5_nl_ifindex_data { + const char *name; /**< IB device name (in). */ + uint32_t ibindex; /**< IB device index (out). */ + uint32_t ifindex; /**< Network interface index (out). */ +}; + /** * Opens a Netlink socket. * * @param nl_groups * Netlink group value (e.g. RTMGRP_LINK). + * @param protocol + * Netlink protocol (e.g. NETLINK_ROUTE, NETLINK_RDMA). * * @return * A file descriptor on success, a negative errno value otherwise and * rte_errno is set. */ int -mlx5_nl_init(uint32_t nl_groups) +mlx5_nl_init(uint32_t nl_groups, int protocol) { int fd; int sndbuf_size = MLX5_SEND_BUF_SIZE; @@ -56,7 +99,7 @@ mlx5_nl_init(uint32_t nl_groups) }; int ret; - fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE); + fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, protocol); if (fd == -1) { rte_errno = errno; return -rte_errno; @@ -334,9 +377,9 @@ mlx5_nl_mac_addr_list(struct rte_eth_dev *dev, struct ether_addr (*mac)[], int ret; uint32_t sn = priv->nl_sn++; - if (priv->nl_socket == -1) + if (priv->nl_socket_route == -1) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; ret = mlx5_nl_request(fd, &req.hdr, sn, &req.ifm, sizeof(struct ifinfomsg)); if (ret < 0) @@ -398,9 +441,9 @@ mlx5_nl_mac_addr_modify(struct rte_eth_dev *dev, struct ether_addr *mac, int ret; uint32_t sn = priv->nl_sn++; - if (priv->nl_socket == -1) + if (priv->nl_socket_route == -1) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; memcpy(RTA_DATA(&req.rta), mac, ETHER_ADDR_LEN); req.hdr.nlmsg_len = NLMSG_ALIGN(req.hdr.nlmsg_len) + RTA_ALIGN(req.rta.rta_len); @@ -569,9 +612,9 @@ mlx5_nl_device_flags(struct rte_eth_dev *dev, uint32_t flags, int enable) int ret; assert(!(flags & ~(IFF_PROMISC | IFF_ALLMULTI))); - if (priv->nl_socket < 0) + if (priv->nl_socket_route < 0) return 0; - fd = priv->nl_socket; + fd = priv->nl_socket_route; ret = mlx5_nl_send(fd, &req.hdr, priv->nl_sn++); if (ret < 0) return ret; @@ -625,3 +668,241 @@ mlx5_nl_allmulti(struct rte_eth_dev *dev, int enable) strerror(rte_errno)); return ret; } + +/** + * Process network interface information from Netlink message. + * + * @param nh + * Pointer to Netlink message header. + * @param arg + * Opaque data pointer for this callback. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_nl_ifindex_cb(struct nlmsghdr *nh, void *arg) +{ + struct mlx5_nl_ifindex_data *data = arg; + size_t off = NLMSG_HDRLEN; + uint32_t ibindex = 0; + uint32_t ifindex = 0; + int found = 0; + + if (nh->nlmsg_type != + RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_GET) && + nh->nlmsg_type != + RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET)) + goto error; + while (off < nh->nlmsg_len) { + struct nlattr *na = (void *)((uintptr_t)nh + off); + void *payload = (void *)((uintptr_t)na + NLA_HDRLEN); + + if (na->nla_len > nh->nlmsg_len - off) + goto error; + switch (na->nla_type) { + case RDMA_NLDEV_ATTR_DEV_INDEX: + ibindex = *(uint32_t *)payload; + break; + case RDMA_NLDEV_ATTR_DEV_NAME: + if (!strcmp(payload, data->name)) + found = 1; + break; + case RDMA_NLDEV_ATTR_NDEV_INDEX: + ifindex = *(uint32_t *)payload; + break; + default: + break; + } + off += NLA_ALIGN(na->nla_len); + } + if (found) { + data->ibindex = ibindex; + data->ifindex = ifindex; + } + return 0; +error: + rte_errno = EINVAL; + return -rte_errno; +} + +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name) +{ + static const uint32_t pindex = 1; + uint32_t seq = random(); + struct mlx5_nl_ifindex_data data = { + .name = name, + .ibindex = 0, /* Determined during first pass. */ + .ifindex = 0, /* Determined during second pass. */ + }; + union { + struct nlmsghdr nh; + uint8_t buf[NLMSG_HDRLEN + + NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; + } req = { + .nh = { + .nlmsg_len = NLMSG_LENGTH(0), + .nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, + RDMA_NLDEV_CMD_GET), + .nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK | NLM_F_DUMP, + }, + }; + struct nlattr *na; + int ret; + + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret < 0) + return 0; + ret = mlx5_nl_recv(nl, seq, mlx5_nl_ifindex_cb, &data); + if (ret < 0) + return 0; + if (!data.ibindex) + goto error; + ++seq; + req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, + RDMA_NLDEV_CMD_PORT_GET); + req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; + req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); + na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); + na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; + memcpy((void *)((uintptr_t)na + NLA_HDRLEN), + &data.ibindex, sizeof(data.ibindex)); + na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); + na->nla_len = NLA_HDRLEN + sizeof(pindex); + na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; + memcpy((void *)((uintptr_t)na + NLA_HDRLEN), + &pindex, sizeof(pindex)); + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret < 0) + return 0; + ret = mlx5_nl_recv(nl, seq, mlx5_nl_ifindex_cb, &data); + if (ret < 0) + return 0; + if (!data.ifindex) + goto error; + return data.ifindex; +error: + rte_errno = ENODEV; + return 0; +} + +/** + * Process switch information from Netlink message. + * + * @param nh + * Pointer to Netlink message header. + * @param arg + * Opaque data pointer for this callback. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_nl_switch_info_cb(struct nlmsghdr *nh, void *arg) +{ + struct mlx5_switch_info info = { + .master = 0, + .representor = 0, + .port_name = 0, + .switch_id = 0, + }; + size_t off = NLMSG_LENGTH(sizeof(struct ifinfomsg)); + bool port_name_set = false; + bool switch_id_set = false; + + if (nh->nlmsg_type != RTM_NEWLINK) + goto error; + while (off < nh->nlmsg_len) { + struct rtattr *ra = (void *)((uintptr_t)nh + off); + void *payload = RTA_DATA(ra); + char *end; + unsigned int i; + + if (ra->rta_len > nh->nlmsg_len - off) + goto error; + switch (ra->rta_type) { + case IFLA_PHYS_PORT_NAME: + errno = 0; + info.port_name = strtol(payload, &end, 0); + if (errno || + (size_t)(end - (char *)payload) != strlen(payload)) + goto error; + port_name_set = true; + break; + case IFLA_PHYS_SWITCH_ID: + info.switch_id = 0; + for (i = 0; i < RTA_PAYLOAD(ra); ++i) { + info.switch_id <<= 8; + info.switch_id |= ((uint8_t *)payload)[i]; + } + switch_id_set = true; + break; + } + off += RTA_ALIGN(ra->rta_len); + } + info.master = switch_id_set && !port_name_set; + info.representor = switch_id_set && port_name_set; + memcpy(arg, &info, sizeof(info)); + return 0; +error: + rte_errno = EINVAL; + return -rte_errno; +} + +/** + * Get switch information associated with network interface. + * + * @param nl + * Netlink socket of the ROUTE kind (NETLINK_ROUTE). + * @param ifindex + * Network interface index. + * @param[out] info + * Switch information object, populated in case of success. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_nl_switch_info(int nl, unsigned int ifindex, struct mlx5_switch_info *info) +{ + uint32_t seq = random(); + struct { + struct nlmsghdr nh; + struct ifinfomsg info; + } req = { + .nh = { + .nlmsg_len = NLMSG_LENGTH(sizeof(req.info)), + .nlmsg_type = RTM_GETLINK, + .nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK, + }, + .info = { + .ifi_family = AF_UNSPEC, + .ifi_index = ifindex, + }, + }; + int ret; + + ret = mlx5_nl_send(nl, &req.nh, seq); + if (ret >= 0) + ret = mlx5_nl_recv(nl, seq, mlx5_nl_switch_info_cb, info); + return ret; +} From patchwork Wed Jul 4 17:27:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42262 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D67BD1BF50; Wed, 4 Jul 2018 19:28:06 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id B00051BF0C for ; Wed, 4 Jul 2018 19:28:03 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id n17-v6so7036247wmh.2 for ; Wed, 04 Jul 2018 10:28:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=N95PkMgu99LfIpaeEjG9ouWkkGK6wGD46IB10BWioUI=; b=MW9kvf6Qv/cHWJK061oVfFHq1ugeEtHyxCSwH/ThtdQHBtbbJ3ZY1IaNkyz8r+x+ZU NJNIMt+EmPrt2bStOs/5MZIXv1/CadnvM+til7MTDikfQ8dvnm3wtAfI2+3x+rTD3Q3H KFM6pIB1agprnPwbINOyltR7tL2s8/ELrQNiI4sBK/kiyOJ5iJdV1D4GHmyj3E6gr2qC 0QlyO4GwhLAgv1qGt0wAXbR/EP4OMOz0gb0E4iS2GZiD2pmlWyCbDmcU+7wM+1wf63Yt Vq0OAQVqsoCfjwmZHFDnBVL5BEHVrtho8VsY2nDWcX9yo7qFwyii3pDebzap1Pu5T6+B cHIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=N95PkMgu99LfIpaeEjG9ouWkkGK6wGD46IB10BWioUI=; b=eHePxdavMcWfzf62q7U+EcVWBiXM5wixv+QJWWok3QY1as1gJrIoPCmV397Xr+ik32 3aJ/rGXj7/XwFuB5HAUCr5jsNVyvrBG5F15F8aCbQmoXC7vCnv4i8O0qsRM1mWT+rGjX XtECDHkRfQjFIqwnRUmCWexvw9jDWCRryLjaBu/w0ehMk/Jss06esOLdzXLJII/g/wRa F6zxN7mzHT0vY5Er9CNBqo/y2sXihmTXdhqZ2u7zsE4l/9pNm7FHB36XA6U3DEV1QiGJ ORv8BdBCDEnJ4dIV+HgHQifeXvEY+NtKz1cdL67cTw9b4h5avkrRStTIvWtb5/UFStWU dOzg== X-Gm-Message-State: APt69E1FWW3w8nHAjELB6XYUswUH/uBYDDs6EN9D/A/f7n0hfW5+Ki90 FmQrb2mYuCu7uHavibpnfdwL/Hii X-Google-Smtp-Source: AAOMgpfCpLFISJb/uItmHN5Ijti82dwLsLhFmOJ1hQQy+jqZrF2f3WAaKqtRAe7xi9gU+hfPbW++qg== X-Received: by 2002:adf:9025:: with SMTP id h34-v6mr2358768wrh.123.1530725283185; Wed, 04 Jul 2018 10:28:03 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id f190-v6sm5965226wmd.0.2018.07.04.10.28.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:28:02 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:46 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org, Nelio Laranjeiro , Xueming Li Message-ID: <20180704172322.22571-8-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 07/10] net/mlx5: probe all port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Probe existing port representors in addition to their master device and associate them automatically. To avoid collision between Ethernet devices, they are named as follows: - "{DBDF}" for master/switch devices. - "{DBDF}_representor_{rep}" with "rep" starting from 0 for port representors. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Signed-off-by: Nelio Laranjeiro Reviewed-by: Xueming Li Cc: Xueming Li Cc: Shahaf Shuler --- v3 changes: - Nelio introduced mlx5_dev_to_port_id() to prevent the master device from releasing a domain ID while representors are still bound. It is now released by the last device closed. - Reverted to original naming convention as requested by Xueming and Shahaf; "net_" prefix and "_0" suffix were dropped. - mlx5_dev_spawn() (previously mlx5_dev_spawn_one()) now decides on its own whether underlying device is a representor. - Devices can now be probed in any order and not necessarily all at once; representors can exist without a master device. - mlx5_pci_probe() iterates on the list of devices directly instead of relying on an intermediate function (previously mlx5_dev_spawn()). - mlx5_get_ifname() was rewritten to rely on mlx5_nl_ifindex() when faced with a representor. - Since it is not necessarily present, master device is now dynamically retrieved in mlx5_dev_infos_get(). v2 changes: - Added representor information to dev_infos_get(). DPDK port ID of master device is now stored in the private structure to retrieve it conveniently. - Master device is assigned dummy representor ID value -1 to better distinguish from the the first actual representor reported by dev_infos_get() as those are indexed from 0. - Added RTE_ETH_DEV_REPRESENTOR device flag. --- drivers/net/mlx5/mlx5.c | 109 +++++++++++++++++++++-------- drivers/net/mlx5/mlx5.h | 12 +++- drivers/net/mlx5/mlx5_ethdev.c | 135 ++++++++++++++++++++++++++++++++---- drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_stats.c | 6 +- 5 files changed, 215 insertions(+), 49 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d06ba9886..52bfc5c63 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -307,6 +307,9 @@ mlx5_dev_close(struct rte_eth_dev *dev) if (ret) DRV_LOG(WARNING, "port %u some flows still remain", dev->data->port_id); + i = mlx5_dev_to_port_id(dev->device, NULL, 0); + if (i == 1 && priv->domain_id != RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); memset(priv, 0, sizeof(*priv)); } @@ -647,6 +650,8 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * Verbs device. * @param vf * If nonzero, enable VF-specific features. + * @param[in] switch_info + * Switch properties of Ethernet device. * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno @@ -655,7 +660,8 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) static struct rte_eth_dev * mlx5_dev_spawn(struct rte_device *dpdk_dev, struct ibv_device *ibv_dev, - int vf) + int vf, + const struct mlx5_switch_info *switch_info) { struct ibv_context *ctx; struct ibv_device_attr_ex attr; @@ -697,6 +703,9 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, #endif struct ether_addr mac; char name[RTE_ETH_NAME_MAX_LEN]; + uint16_t port_id; + int own_domain_id = 0; + unsigned int i; /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); @@ -805,7 +814,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, DEBUG("ibv_query_device_ex() failed"); goto error; } - rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + if (!switch_info->representor) + rte_strlcpy(name, dpdk_dev->name, sizeof(name)); + else + snprintf(name, sizeof(name), "%s_representor_%u", + dpdk_dev->name, switch_info->port_name); + DRV_LOG(DEBUG, "naming Ethernet device \"%s\"", name); if (rte_eal_process_type() == RTE_PROC_SECONDARY) { eth_dev = rte_eth_dev_attach_secondary(name); if (eth_dev == NULL) { @@ -874,6 +888,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, goto error; } priv->ctx = ctx; + strncpy(priv->ibdev_name, priv->ctx->device->name, + sizeof(priv->ibdev_name)); strncpy(priv->ibdev_path, priv->ctx->device->ibdev_path, sizeof(priv->ibdev_path)); priv->device_attr = attr; @@ -883,6 +899,32 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, priv->nl_socket_rdma = mlx5_nl_init(0, NETLINK_RDMA); priv->nl_socket_route = mlx5_nl_init(RTMGRP_LINK, NETLINK_ROUTE); priv->nl_sn = 0; + priv->representor = !!switch_info->representor; + priv->domain_id = RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID; + priv->representor_id = + switch_info->representor ? switch_info->port_name : -1; + /* + * Look for sibling devices in order to reuse their switch domain, + * otherwise allocate one. + */ + i = mlx5_dev_to_port_id(dpdk_dev, &port_id, 1); + if (i > 0) { + const struct priv *opriv = + rte_eth_devices[port_id].data->dev_private; + + if (opriv) + priv->domain_id = opriv->domain_id; + } + if (priv->domain_id == RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID) { + err = rte_eth_switch_domain_alloc(&priv->domain_id); + if (err) { + err = rte_errno; + DRV_LOG(ERR, "unable to allocate switch domain: %s", + strerror(rte_errno)); + goto error; + } + own_domain_id = 1; + } err = mlx5_args(&config, dpdk_dev->devargs); if (err) { err = rte_errno; @@ -966,6 +1008,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err = ENOMEM; goto error; } + if (priv->representor) + eth_dev->data->dev_flags |= RTE_ETH_DEV_REPRESENTOR; eth_dev->data->dev_private = priv; priv->dev_data = eth_dev->data; eth_dev->data->mac_addrs = priv->mac; @@ -1084,6 +1128,8 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, close(priv->nl_socket_route); if (priv->nl_socket_rdma >= 0) close(priv->nl_socket_rdma); + if (own_domain_id) + claim_zero(rte_eth_switch_domain_free(priv->domain_id)); rte_free(priv); } if (pd) @@ -1100,7 +1146,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, /** * DPDK callback to register a PCI device. * - * This function spawns an Ethernet device out of a given PCI device. + * This function spawns Ethernet devices out of a given PCI device. * * @param[in] pci_drv * PCI driver structure (mlx5_driver). @@ -1115,7 +1161,6 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) { struct ibv_device **ibv_list; - struct rte_eth_dev *eth_dev = NULL; unsigned int n = 0; int vf; int ret; @@ -1150,9 +1195,11 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int ifindex[n]; struct mlx5_switch_info info[n]; + struct rte_eth_dev *eth_list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; + unsigned int u; /* * The existence of several matching entries (n > 1) means port @@ -1187,28 +1234,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_rdma); if (nl_route >= 0) close(nl_route); - /* Look for master device. */ - for (i = 0; i != n; ++i) { - if (!info[i].master) - continue; - /* Make it the first entry. */ - if (i == 0) - break; - ibv_match[n] = ibv_match[0]; - ibv_match[0] = ibv_match[i]; - ibv_match[n] = NULL; - break; - } - if (n && i == n) { - if (n == 1 && !info[0].representor) { + /* Count unidentified devices. */ + for (u = 0, i = 0; i != n; ++i) + if (!info[i].master && !info[i].representor) + ++u; + if (u) { + if (n == 1 && u == 1) { /* Case #2. */ DRV_LOG(INFO, "no switch support detected"); - } else if (n == 1) { - /* Case #3. */ - DRV_LOG(ERR, - "device looks like a port representor, this is" - " not supported yet"); - n = 0; } else { /* Case #3. */ DRV_LOG(ERR, @@ -1227,8 +1260,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, default: vf = 0; } - if (n) - eth_dev = mlx5_dev_spawn(&pci_dev->device, ibv_match[0], vf); + for (i = 0; i != n; ++i) { + uint32_t restore; + + eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], + vf, &info[i]); + if (!eth_list[i]) + break; + restore = eth_list[i]->data->dev_flags; + rte_eth_copy_pci_info(eth_list[i], pci_dev); + /* Restore non-PCI flags cleared by the above call. */ + eth_list[i]->data->dev_flags |= restore; + rte_eth_dev_probing_finish(eth_list[i]); + } mlx5_glue->free_device_list(ibv_list); if (!n) { DRV_LOG(WARNING, @@ -1238,7 +1282,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid, pci_dev->addr.function); rte_errno = ENOENT; ret = -rte_errno; - } else if (!eth_dev) { + } else if (i != n) { DRV_LOG(ERR, "probe of PCI device " PCI_PRI_FMT " aborted after" " encountering an error: %s", @@ -1246,9 +1290,16 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, pci_dev->addr.devid, pci_dev->addr.function, strerror(rte_errno)); ret = -rte_errno; + /* Roll back. */ + while (i--) { + mlx5_dev_close(eth_list[i]); + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + rte_free(eth_list[i]->data->dev_private); + claim_zero(rte_eth_dev_release_port(eth_list[i])); + } + /* Restore original error. */ + rte_errno = -ret; } else { - rte_eth_copy_pci_info(eth_dev, pci_dev); - rte_eth_dev_probing_finish(eth_dev); ret = 0; } return ret; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 704046270..cc01310e0 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -159,6 +159,7 @@ struct priv { struct ibv_context *ctx; /* Verbs context. */ struct ibv_device_attr_ex device_attr; /* Device properties. */ struct ibv_pd *pd; /* Protection Domain. */ + char ibdev_name[IBV_SYSFS_NAME_MAX]; /* IB device name. */ char ibdev_path[IBV_SYSFS_PATH_MAX]; /* IB device path for secondary */ struct ether_addr mac[MLX5_MAX_MAC_ADDRESSES]; /* MAC addresses. */ BITFIELD_DECLARE(mac_own, uint64_t, MLX5_MAX_MAC_ADDRESSES); @@ -168,6 +169,9 @@ struct priv { /* Device properties. */ uint16_t mtu; /* Configured MTU. */ unsigned int isolated:1; /* Whether isolated mode is enabled. */ + unsigned int representor:1; /* Device is a port representor. */ + uint16_t domain_id; /* Switch domain identifier. */ + int32_t representor_id; /* Port representor identifier. */ /* RX/TX queues. */ unsigned int rxqs_n; /* RX queues array size. */ unsigned int txqs_n; /* TX queues array size. */ @@ -217,9 +221,12 @@ int mlx5_getenv_int(const char *); /* mlx5_ethdev.c */ +int mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]); int mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]); int mlx5_ifindex(const struct rte_eth_dev *dev); -int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr); +int mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master); int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu); int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags); @@ -244,6 +251,9 @@ int mlx5_set_link_up(struct rte_eth_dev *dev); int mlx5_is_removed(struct rte_eth_dev *dev); eth_tx_burst_t mlx5_select_tx_function(struct rte_eth_dev *dev); eth_rx_burst_t mlx5_select_rx_function(struct rte_eth_dev *dev); +unsigned int mlx5_dev_to_port_id(const struct rte_device *dev, + uint16_t *port_list, + unsigned int port_list_n); /* mlx5_mac.c */ diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 819f5baad..390e602c1 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -27,6 +27,7 @@ #include #include +#include #include #include #include @@ -93,7 +94,7 @@ struct ethtool_link_settings { #endif /** - * Get interface name from private structure. + * Get master interface name from private structure. * * @param[in] dev * Pointer to Ethernet device. @@ -104,7 +105,8 @@ struct ethtool_link_settings { * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +mlx5_get_master_ifname(const struct rte_eth_dev *dev, + char (*ifname)[IF_NAMESIZE]) { struct priv *priv = dev->data->dev_private; DIR *dir; @@ -179,6 +181,39 @@ mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) } /** + * Get interface name from private structure. + * + * This is a port representor-aware version of mlx5_get_master_ifname(). + * + * @param[in] dev + * Pointer to Ethernet device. + * @param[out] ifname + * Interface name output buffer. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_get_ifname(const struct rte_eth_dev *dev, char (*ifname)[IF_NAMESIZE]) +{ + struct priv *priv = dev->data->dev_private; + unsigned int ifindex = + priv->nl_socket_rdma >= 0 ? + mlx5_nl_ifindex(priv->nl_socket_rdma, priv->ibdev_name) : 0; + + if (!ifindex) { + if (!priv->representor) + return mlx5_get_master_ifname(dev, ifname); + rte_errno = ENXIO; + return -rte_errno; + } + if (if_indextoname(ifindex, &(*ifname)[0])) + return 0; + rte_errno = errno; + return -rte_errno; +} + +/** * Get the interface index from device name. * * @param[in] dev @@ -214,12 +249,16 @@ mlx5_ifindex(const struct rte_eth_dev *dev) * Request number to pass to ioctl(). * @param[out] ifr * Interface request structure output buffer. + * @param master + * When device is a port representor, perform request on master device + * instead. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) +mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr, + int master) { int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP); int ret = 0; @@ -228,7 +267,10 @@ mlx5_ifreq(const struct rte_eth_dev *dev, int req, struct ifreq *ifr) rte_errno = errno; return -rte_errno; } - ret = mlx5_get_ifname(dev, &ifr->ifr_name); + if (master) + ret = mlx5_get_master_ifname(dev, &ifr->ifr_name); + else + ret = mlx5_get_ifname(dev, &ifr->ifr_name); if (ret) goto error; ret = ioctl(sock, req, ifr); @@ -258,7 +300,7 @@ int mlx5_get_mtu(struct rte_eth_dev *dev, uint16_t *mtu) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request); + int ret = mlx5_ifreq(dev, SIOCGIFMTU, &request, 0); if (ret) return ret; @@ -282,7 +324,7 @@ mlx5_set_mtu(struct rte_eth_dev *dev, uint16_t mtu) { struct ifreq request = { .ifr_mtu = mtu, }; - return mlx5_ifreq(dev, SIOCSIFMTU, &request); + return mlx5_ifreq(dev, SIOCSIFMTU, &request, 0); } /** @@ -302,13 +344,13 @@ int mlx5_set_flags(struct rte_eth_dev *dev, unsigned int keep, unsigned int flags) { struct ifreq request; - int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request); + int ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &request, 0); if (ret) return ret; request.ifr_flags &= keep; request.ifr_flags |= flags & ~keep; - return mlx5_ifreq(dev, SIOCSIFFLAGS, &request); + return mlx5_ifreq(dev, SIOCSIFFLAGS, &request, 0); } /** @@ -477,6 +519,32 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) info->speed_capa = priv->link_speed_capa; info->flow_type_rss_offloads = ~MLX5_RSS_HF_MASK; mlx5_set_default_params(dev, info); + info->switch_info.name = dev->data->name; + info->switch_info.domain_id = priv->domain_id; + info->switch_info.port_id = priv->representor_id; + if (priv->representor) { + uint16_t port_id[mlx5_dev_to_port_id(dev->device, NULL, 0)]; + unsigned int i = + RTE_MIN(mlx5_dev_to_port_id(dev->device, port_id, + RTE_DIM(port_id)), + RTE_DIM(port_id)); + + while (i--) { + struct priv *opriv = + rte_eth_devices[port_id[i]].data->dev_private; + + if (!opriv || + opriv->representor || + opriv->domain_id != priv->domain_id) + continue; + /* + * Override switch name with that of the master + * device. + */ + info->switch_info.name = opriv->dev_data->name; + break; + } + } } /** @@ -540,7 +608,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, int link_speed = 0; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -550,7 +618,7 @@ mlx5_link_update_unlocked_gset(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&edata; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GSET) failed: %s", @@ -611,7 +679,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, uint64_t sc; int ret; - ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr); + ret = mlx5_ifreq(dev, SIOCGIFFLAGS, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCGIFFLAGS) failed: %s", dev->data->port_id, strerror(rte_errno)); @@ -621,7 +689,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, dev_link.link_status = ((ifr.ifr_flags & IFF_UP) && (ifr.ifr_flags & IFF_RUNNING)); ifr.ifr_data = (void *)&gcmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -638,7 +706,7 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, *ecmd = gcmd; ifr.ifr_data = (void *)ecmd; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(DEBUG, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GLINKSETTINGS)" @@ -801,7 +869,7 @@ mlx5_dev_get_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) int ret; ifr.ifr_data = (void *)ðpause; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_GPAUSEPARAM) failed:" @@ -854,7 +922,7 @@ mlx5_dev_set_flow_ctrl(struct rte_eth_dev *dev, struct rte_eth_fc_conf *fc_conf) ethpause.tx_pause = 1; else ethpause.tx_pause = 0; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 0); if (ret) { DRV_LOG(WARNING, "port %u ioctl(SIOCETHTOOL, ETHTOOL_SPAUSEPARAM)" @@ -1193,3 +1261,40 @@ mlx5_is_removed(struct rte_eth_dev *dev) return 1; return 0; } + +/** + * Get port ID list of mlx5 instances sharing a common device. + * + * @param[in] dev + * Device to look for. + * @param[out] port_list + * Result buffer for collected port IDs. + * @param port_list_n + * Maximum number of entries in result buffer. If 0, @p port_list can be + * NULL. + * + * @return + * Number of matching instances regardless of the @p port_list_n + * parameter, 0 if none were found. + */ +unsigned int +mlx5_dev_to_port_id(const struct rte_device *dev, uint16_t *port_list, + unsigned int port_list_n) +{ + uint16_t id; + unsigned int n = 0; + + RTE_ETH_FOREACH_DEV(id) { + struct rte_eth_dev *ldev = &rte_eth_devices[id]; + + if (!ldev->device || + !ldev->device->driver || + strcmp(ldev->device->driver->name, MLX5_DRIVER_NAME) || + ldev->device != dev) + continue; + if (n < port_list_n) + port_list[n] = id; + n++; + } + return n; +} diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index 672a47619..12ee37f55 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -49,7 +49,7 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[ETHER_ADDR_LEN]) struct ifreq request; int ret; - ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request); + ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request, 0); if (ret) return ret; memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN); diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.c index 875dd1027..91f3d474a 100644 --- a/drivers/net/mlx5/mlx5_stats.c +++ b/drivers/net/mlx5/mlx5_stats.c @@ -146,7 +146,7 @@ mlx5_read_dev_counters(struct rte_eth_dev *dev, uint64_t *stats) et_stats->cmd = ETHTOOL_GSTATS; et_stats->n_stats = xstats_ctrl->stats_n; ifr.ifr_data = (caddr_t)et_stats; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to read statistic values from device", @@ -194,7 +194,7 @@ mlx5_ethtool_get_stats_n(struct rte_eth_dev *dev) { drvinfo.cmd = ETHTOOL_GDRVINFO; ifr.ifr_data = (caddr_t)&drvinfo; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to query number of statistics", dev->data->port_id); @@ -244,7 +244,7 @@ mlx5_xstats_init(struct rte_eth_dev *dev) strings->string_set = ETH_SS_STATS; strings->len = dev_stats_n; ifr.ifr_data = (caddr_t)strings; - ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr); + ret = mlx5_ifreq(dev, SIOCETHTOOL, &ifr, 1); if (ret) { DRV_LOG(WARNING, "port %u unable to get statistic names", dev->data->port_id); From patchwork Wed Jul 4 17:27:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42263 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DD0DB1BF79; Wed, 4 Jul 2018 19:28:11 +0200 (CEST) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id AA5B91BF50 for ; Wed, 4 Jul 2018 19:28:05 +0200 (CEST) Received: by mail-wm0-f65.google.com with SMTP id n17-v6so7036383wmh.2 for ; Wed, 04 Jul 2018 10:28:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=/jMy1nFWflphL4wCACofjdcj1fA/62A4fxNekaSHXhY=; b=HMMfKepLcpFIC4Wws8Du+pGLdX6xx1LzTj7xerIyEgPhTLFyiqKTGfNQmctHdOP7Ag byMRTJ9OlhMi0wKUG1cXu08MsMp4a7FpE99F3m2LdycVkzOvmzDImHYs+BjQ6HHRe/8H R555lCpV2DRLOSKfp/LLeG9obrqPwc2EXzdKo+YwL+vET12HKFU1q7A0MgDEF6YkRJCX eZAmUrr27yh71t5m9Bpfu7YXnb5w253igTzMGKAi7uNZoTBnE8mUwvNq+Uhd/G6qebjd DsweFRbxypc6AUYHuVLUBlwr8wMwNF1FFX2tQhx/KbkUdhuoVFFYqc0ukR0MYQ+uBhck KyZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=/jMy1nFWflphL4wCACofjdcj1fA/62A4fxNekaSHXhY=; b=E3cAOBmRciZbDjydG4uodb9YPrh8dUV2ebisviCpW6mH5cOT08YkC3vJzsqGj3/ph7 heD9codw+wDhA2KI8VKpRaW+L9gnK6vtECcssjCNhhCfLLjcIg0VBuc1ugpKY2FAeBAH VUoY4M6XWhX5eGHBcDCnmqHo5esbEeAkSsAIiC5AMMWyJwgdZmYMHDPj3AeA5gV+NbCW XjSLAT4apoIlff3PyBh4+8enQIawg5BXMsYZdPWQ6Ypbtcs0IQwCqzK7U+YgQku9hl65 lOhwKOxmw1e4U/QRn9B1/jcT5vmkxsJsc7BoqZ9/SW+mpUzGZ8/ygISPoy9l1xJ28cot UXTg== X-Gm-Message-State: APt69E3rS6O/8js/4uIVmjJnpf0Y/xQYDCKtSPDlywXtLh7vyM1zKzBq AH3cUrG4DXZc+VfrkmJayEPO1A== X-Google-Smtp-Source: AAOMgpcPHVEmj8ytodEAa1mFE78fZmmZkLO5Er5vwyyADojvTUwR4C4EkEP1/vPCw3X/JOdg5rqKVQ== X-Received: by 2002:adf:9aeb:: with SMTP id a98-v6mr2169144wrc.110.1530725285099; Wed, 04 Jul 2018 10:28:05 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id i4-v6sm7013151wmf.4.2018.07.04.10.28.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:28:04 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:48 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-9-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 08/10] net/mlx5: probe port representors in natural order X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Port representors are probed in whatever unspecified order ibv_get_device_list() returns them. This is counterintuitive to users since DPDK port IDs assignment almost never follows the same sequence as representor IDs. Additionally, the master device does not necessarily inherit the lowest DPDK port ID. Signed-off-by: Adrien Mazarguil --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 95 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 52bfc5c63..05dd3bbe7 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1143,6 +1143,52 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, return NULL; } +/** Data associated with devices to spawn. */ +struct mlx5_dev_spawn_data { + unsigned int ifindex; /**< Network interface index. */ + struct mlx5_switch_info info; /**< Switch information. */ + struct ibv_device *ibv_dev; /**< Associated IB device. */ + struct rte_eth_dev *eth_dev; /**< Associated Ethernet device. */ +}; + +/** + * Comparison callback to sort device data. + * + * This is meant to be used with qsort(). + * + * @param a[in] + * Pointer to pointer to first data object. + * @param b[in] + * Pointer to pointer to second data object. + * + * @return + * 0 if both objects are equal, less than 0 if the first argument is less + * than the second, greater than 0 otherwise. + */ +static int +mlx5_dev_spawn_data_cmp(const void *a, const void *b) +{ + const struct mlx5_switch_info *si_a = + &((const struct mlx5_dev_spawn_data *)a)->info; + const struct mlx5_switch_info *si_b = + &((const struct mlx5_dev_spawn_data *)b)->info; + int ret; + + /* Master device first. */ + ret = si_b->master - si_a->master; + if (ret) + return ret; + /* Then representor devices. */ + ret = si_b->representor - si_a->representor; + if (ret) + return ret; + /* Unidentified devices come last in no specific order. */ + if (!si_a->representor) + return 0; + /* Order representors by name. */ + return si_a->port_name - si_b->port_name; +} + /** * DPDK callback to register a PCI device. * @@ -1193,9 +1239,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } ibv_match[n] = NULL; - unsigned int ifindex[n]; - struct mlx5_switch_info info[n]; - struct rte_eth_dev *eth_list[n]; + struct mlx5_dev_spawn_data list[n]; int nl_route = n ? mlx5_nl_init(0, NETLINK_ROUTE) : -1; int nl_rdma = n ? mlx5_nl_init(0, NETLINK_RDMA) : -1; unsigned int i; @@ -1217,16 +1261,19 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * bail out. */ for (i = 0; i != n; ++i) { + list[i].ibv_dev = ibv_match[i]; + list[i].eth_dev = NULL; if (nl_rdma < 0) - ifindex[i] = 0; + list[i].ifindex = 0; else - ifindex[i] = mlx5_nl_ifindex(nl_rdma, - ibv_match[i]->name); + list[i].ifindex = mlx5_nl_ifindex + (nl_rdma, list[i].ibv_dev->name); if (nl_route < 0 || - !ifindex[i] || - mlx5_nl_switch_info(nl_route, ifindex[i], &info[i])) { - ifindex[i] = 0; - memset(&info[i], 0, sizeof(info[i])); + !list[i].ifindex || + mlx5_nl_switch_info(nl_route, list[i].ifindex, + &list[i].info)) { + list[i].ifindex = 0; + memset(&list[i].info, 0, sizeof(list[i].info)); continue; } } @@ -1236,7 +1283,7 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, close(nl_route); /* Count unidentified devices. */ for (u = 0, i = 0; i != n; ++i) - if (!info[i].master && !info[i].representor) + if (!list[i].info.master && !list[i].info.representor) ++u; if (u) { if (n == 1 && u == 1) { @@ -1250,6 +1297,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, n = 0; } } + /* + * Sort list to probe devices in natural order for users convenience + * (i.e. master first, then representors from lowest to highest ID). + */ + if (n) + qsort(list, n, sizeof(*list), mlx5_dev_spawn_data_cmp); switch (pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: @@ -1263,15 +1316,15 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, for (i = 0; i != n; ++i) { uint32_t restore; - eth_list[i] = mlx5_dev_spawn(&pci_dev->device, ibv_match[i], - vf, &info[i]); - if (!eth_list[i]) + list[i].eth_dev = mlx5_dev_spawn + (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); + if (!list[i].eth_dev) break; - restore = eth_list[i]->data->dev_flags; - rte_eth_copy_pci_info(eth_list[i], pci_dev); + restore = list[i].eth_dev->data->dev_flags; + rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ - eth_list[i]->data->dev_flags |= restore; - rte_eth_dev_probing_finish(eth_list[i]); + list[i].eth_dev->data->dev_flags |= restore; + rte_eth_dev_probing_finish(list[i].eth_dev); } mlx5_glue->free_device_list(ibv_list); if (!n) { @@ -1292,10 +1345,10 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { - mlx5_dev_close(eth_list[i]); + mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) - rte_free(eth_list[i]->data->dev_private); - claim_zero(rte_eth_dev_release_port(eth_list[i])); + rte_free(list[i].eth_dev->data->dev_private); + claim_zero(rte_eth_dev_release_port(list[i].eth_dev)); } /* Restore original error. */ rte_errno = -ret; From patchwork Wed Jul 4 17:27:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42264 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 450E41BF47; Wed, 4 Jul 2018 19:28:18 +0200 (CEST) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by dpdk.org (Postfix) with ESMTP id 3C1B91BF75 for ; Wed, 4 Jul 2018 19:28:07 +0200 (CEST) Received: by mail-wm0-f68.google.com with SMTP id v3-v6so3404591wmh.0 for ; Wed, 04 Jul 2018 10:28:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=F6UKdF8chVk02b+gy5+r5VSdXFrrdqIlPgEhvkrtS4E=; b=WL1I6HKr9JFpi5Iuvqt9HRwBaYIHl0zQAmpu6+ig+NugWOqP97R9+U7nKdUgqAeOPp VTOqXjbYbGNWYIK3cSs8Ds4Kv/qbMkwfBLUC6MYttCDQxtYPRFfaPeUZJ6HBJ8u7O3Ag 1iWBtM79l87qV7qlS4hOYL2jOTCj+9ti2fO4XgMRYrwrqhTZ6UHqMr5IkYJk3lzt+Wfq ehdoZ/9AbIbmn9ZqJkX/0OH/HkaBHMGrF1LgFD+ChNidWSUhPJW+VGipYobDOhGgEJBS jEOWQnBFUrFmeYXfLZoC7iXbl+HPGvoSrZsDGnGyG6W5DFlVOm///vi5Sx4lmZorUQbx tohg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=F6UKdF8chVk02b+gy5+r5VSdXFrrdqIlPgEhvkrtS4E=; b=mwHGKPUSK1kRnmER+hEeoZeqSQDChsOeVGw7IotcHEGYtJq3mZWstGTxNuFALxkITa uF3pI4k2rl1ltgZa8WUC2q2Poan9f1ze2w88GNvEPCndczNmXpOm1f/1gVY1Tcm68oBJ uolRy2lZ3uGv2yh+QLmhx9ZVjxp+4x48iMbQepj/oeJnJPrWCFKEXxFZJ4BWACaRVtkN SF5NfU37dq+V5rTfPO0Yz0je8qSFE/DiaEIpl/1mKu5TFSNli3xUOHCO2Vcbjr/kaV9Q ckLTfiYhDrg2vx6EyQRqC9DLNctOpreeCdTuV9XvJWOj98tf+xgcPzYQQucmW/QlFsQ3 50lg== X-Gm-Message-State: APt69E0M9YwQqWy+xOKw89Ho/rVbSGWTPbAprmqTmryZbbic8dAT2r8n 9376jyFw3bi63W54eO6f5xqWfQ== X-Google-Smtp-Source: AAOMgpehrbDCeTCp4mrWaU+ucNv83Tm8LeHvxc/AJUTnpnG3AIVSPaFNHIahSoySsYj2lJzmObo5nQ== X-Received: by 2002:a1c:7501:: with SMTP id o1-v6mr2221648wmc.133.1530725286988; Wed, 04 Jul 2018 10:28:06 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id r10-v6sm6549505wrm.66.2018.07.04.10.28.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:28:06 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:50 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-10-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 09/10] net/mlx5: add parameter for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Prior to this patch, all port representors detected on a given device were probed and Ethernet devices instantiated for each of them. This patch adds support for the standard "representor" parameter, which implies that port representors are not probed by default anymore, except for the list provided through device arguments. (Patch based on prior work from Yuanhan Liu) Signed-off-by: Adrien Mazarguil Reviewed-by: Xueming Li --- v3 changes: - Adapted representor detection to the reworked mlx5_dev_spawn(). v2 changes: - Added error message for when rte_eth_devargs_parse() fails. --- doc/guides/nics/mlx5.rst | 12 ++++++++ doc/guides/prog_guide/poll_mode_drv.rst | 2 ++ drivers/net/mlx5/mlx5.c | 41 ++++++++++++++++++++++++++-- 3 files changed, 52 insertions(+), 3 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 7dd9c1c5e..0d0d21727 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -392,6 +392,18 @@ Run-time configuration Disabled by default. +- ``representor`` parameter [list] + + This parameter can be used to instantiate DPDK Ethernet devices from + existing port (or VF) representors configured on the device. + + It is a standard parameter whose format is described in + :ref:`ethernet_device_standard_device_arguments`. + + For instance, to probe port representors 0 through 2:: + + representor=[0-2] + Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index af82352a0..58d49ba0f 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -365,6 +365,8 @@ Ethernet Device API The Ethernet device API exported by the Ethernet PMDs is described in the *DPDK API Reference*. +.. _ethernet_device_standard_device_arguments: + Ethernet Device Standard Device Arguments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 05dd3bbe7..7e757274a 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -92,6 +92,9 @@ /* Activate Netlink support in VF mode. */ #define MLX5_VF_NL_EN "vf_nl_en" +/* Select port representors to instantiate. */ +#define MLX5_REPRESENTOR "representor" + #ifndef HAVE_IBV_MLX5_MOD_MPW #define MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED (1 << 2) #define MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW (1 << 3) @@ -426,6 +429,9 @@ mlx5_args_check(const char *key, const char *val, void *opaque) struct mlx5_dev_config *config = opaque; unsigned long tmp; + /* No-op, port representors are processed in mlx5_dev_spawn(). */ + if (!strcmp(MLX5_REPRESENTOR, key)) + return 0; errno = 0; tmp = strtoul(val, NULL, 0); if (errno) { @@ -498,6 +504,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_RX_VEC_EN, MLX5_L3_VXLAN_EN, MLX5_VF_NL_EN, + MLX5_REPRESENTOR, NULL, }; struct rte_kvargs *kvlist; @@ -655,7 +662,9 @@ mlx5_uar_init_secondary(struct rte_eth_dev *dev) * * @return * A valid Ethernet device object on success, NULL otherwise and rte_errno - * is set. + * is set. The following error is defined: + * + * EBUSY: device is not supposed to be spawned. */ static struct rte_eth_dev * mlx5_dev_spawn(struct rte_device *dpdk_dev, @@ -707,6 +716,26 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, int own_domain_id = 0; unsigned int i; + /* Determine if this port representor is supposed to be spawned. */ + if (switch_info->representor && dpdk_dev->devargs) { + struct rte_eth_devargs eth_da; + + err = rte_eth_devargs_parse(dpdk_dev->devargs->args, ð_da); + if (err) { + rte_errno = -err; + DRV_LOG(ERR, "failed to process device arguments: %s", + strerror(rte_errno)); + return NULL; + } + for (i = 0; i < eth_da.nb_representor_ports; ++i) + if (eth_da.representor_ports[i] == + (uint16_t)switch_info->port_name) + break; + if (i == eth_da.nb_representor_ports) { + rte_errno = EBUSY; + return NULL; + } + } /* Prepare shared data between primary and secondary process. */ mlx5_prepare_shared_data(); errno = 0; @@ -1318,8 +1347,12 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, list[i].eth_dev = mlx5_dev_spawn (&pci_dev->device, list[i].ibv_dev, vf, &list[i].info); - if (!list[i].eth_dev) - break; + if (!list[i].eth_dev) { + if (rte_errno != EBUSY) + break; + /* Device is disabled, ignore it. */ + continue; + } restore = list[i].eth_dev->data->dev_flags; rte_eth_copy_pci_info(list[i].eth_dev, pci_dev); /* Restore non-PCI flags cleared by the above call. */ @@ -1345,6 +1378,8 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, ret = -rte_errno; /* Roll back. */ while (i--) { + if (!list[i].eth_dev) + continue; mlx5_dev_close(list[i].eth_dev); if (rte_eal_process_type() == RTE_PROC_PRIMARY) rte_free(list[i].eth_dev->data->dev_private); From patchwork Wed Jul 4 17:27:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adrien Mazarguil X-Patchwork-Id: 42265 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2815E1BF8B; Wed, 4 Jul 2018 19:28:22 +0200 (CEST) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by dpdk.org (Postfix) with ESMTP id 0D4451BF35 for ; Wed, 4 Jul 2018 19:28:09 +0200 (CEST) Received: by mail-wm0-f67.google.com with SMTP id v16-v6so7128147wmv.5 for ; Wed, 04 Jul 2018 10:28:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=UYsTeFfmplZ7XjtM2kW7mPPIoZxfncWw0NFhC9lq9dI=; b=vOewRUclPKVC29zZrsvX5SoCf36qtPu1YlCOIimWKiHc9UG4kO0pniOjHvTPFdQnaF gi11Og4lLu8JcRhs8oI9W6y35Q8/WE2BOk9aw0LrGTZlMNaebLIFqBUK9/GICDklNLCb Bru0Pup/1VRLgtZ6IYgrC1xAtz433El/NNJHsLTbHGvjwNhL97yCs02jy5HGla5DWJ1G Usmmtr57lvT3X+fZAQjITTOL7j9RG0itdLaueOO1wSRFnM9Kh7T6/aBIWRfmxFVx3xTn voK0Vt2OiHFV+y7k+7OywpgXHr2i8HwGv5Dvo906DOgnh07d6/m/h1rT7s3+pMlArDxY 4ifQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=UYsTeFfmplZ7XjtM2kW7mPPIoZxfncWw0NFhC9lq9dI=; b=lZ1HSfgtEMcSqCDXC4tGKV9F6zYQheYVikbBLNmnQeppBI9MkG0UsmoIc1uTZ8E3+Z y7Xc4qTuKZCmlTCrLyrZD9eOY8PEfFi4/jx1nooAax9lIzkOcVost1GKUORecsYaNrVY nnkbp8W8uLOd/vHPfPyC2TYDtcByTyL9P++176Wz1Sb/nTCpms5njhn0+jZE5Uu9l7uv zmOqvGysL3iYpqBd7N7266VDDgVce0TN9EJXUT1aBi23nhsMEt1N0UujNGmn11NsfB5s sX0TISPJ6CkXXt5Y7iBJWUcBYvzxs/x7cNvtW9kZIZF/DL/3KR0nvw8mLwavJTOQkB2s IYTA== X-Gm-Message-State: APt69E1FlsH6YcRamV/7KZXxcRPm9OG6UeJTnnhyTkB+hqFWuiMKwUFE u/a3tjwo+r5GQ0V5TKtYCFCpZw== X-Google-Smtp-Source: AAOMgpfbBiLbbC+E9D8U5MtD8bgmU2lYs3OPYdX0wlrk+M1cpsPnv0KwJVXklG/3qy+2bRP7WCwCdg== X-Received: by 2002:a5d:4643:: with SMTP id j3-v6mr2203720wrs.52.1530725288886; Wed, 04 Jul 2018 10:28:08 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id s200-v6sm9391913wmb.44.2018.07.04.10.28.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Jul 2018 10:28:08 -0700 (PDT) Date: Wed, 4 Jul 2018 19:27:51 +0200 From: Adrien Mazarguil To: Shahaf Shuler Cc: dev@dpdk.org Message-ID: <20180704172322.22571-11-adrien.mazarguil@6wind.com> References: <20180614083047.10812-1-adrien.mazarguil@6wind.com> <20180704172322.22571-1-adrien.mazarguil@6wind.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180704172322.22571-1-adrien.mazarguil@6wind.com> X-Mailer: git-send-email 2.11.0 Subject: [dpdk-dev] [PATCH v3 10/10] net/mlx5: support negative identifiers for port representors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch brings support for BlueField representors. Signed-off-by: Adrien Mazarguil Cc: Shahaf Shuler --- v3 changes: - This patch was not present in prior revisions. --- drivers/net/mlx5/mlx5.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 7e757274a..d2031c633 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1305,6 +1305,14 @@ mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, memset(&list[i].info, 0, sizeof(list[i].info)); continue; } + /* + * Port representors not associated with any VFs (e.g. on + * BlueField devices) report -1 as a port identifier. + * Quietly set it to zero since DPDK only supports positive + * values. + */ + if (list[i].info.representor && list[i].info.port_name == -1) + list[i].info.port_name = 0; } if (nl_rdma >= 0) close(nl_rdma);