From patchwork Fri Feb 3 05:21:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiawei Wang X-Patchwork-Id: 122994 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9FBE041BB7; Fri, 3 Feb 2023 06:21:55 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E930A42D0D; Fri, 3 Feb 2023 06:21:47 +0100 (CET) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2043.outbound.protection.outlook.com [40.107.96.43]) by mails.dpdk.org (Postfix) with ESMTP id 109CB41611 for ; Fri, 3 Feb 2023 06:21:47 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KXS/1L4t4sFq784UY/MJykTPBwwV44c7Y7WWKMI4n7mU8fZqhBR2V5cvX8iwXbOsgWb5sxUFe0YIEz8gj2aZ0B3+dI7DH7DRWiSQGZCEf/SGwHDtlv+bTAjS5O3Emc6lPQkAqIExMnPp5zP8y1dtku9EJ48KdBeJv1T2MvFDGuCh8POoE1B9jZKMOaTDiHvuO2LpAu3O4/f72MT9sLHYM1362sCUZP9HJAut18zlUpKtkpBzT+Gzv/ZzeBz69oYeQpxbEVA8Go1D30MvOKO0POsWd+9LHnrJsrncgiHW5BPokRUR6PijmAal+M83x9BSWKNg9CsNpTblal+QYIyqPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pCfX3nPd141Nw54zS79fQO+QwsD0CpMbShWqb4ldNxk=; b=aDjPpM6JJj1wTDSpo9aof4XdS+F2WIikwWJPlk4RuoKVx95UYrqE+0sj8POEc90C+6WWThnzrGWfdE6lVZutwIjcNGQs2n178YUE+dU/P+eJca6O9rsT5lHtOYgb1v253fxM7eIT5zNihrx+6NY3qqgoKDAnrtGY/66oMTx4LOyMJq13GpgbXysN3SxlLrRqPJw3bTUgpFkrZlOZunCFVpU5F96gZAl4KqsmG3MVT6KgwmhS0TtMVIBh7K+wmdo95TT6iO+2qWzixrJ6io8cNMtatM+kkptgkCnuwSCcD5c+Lyz0rlkN9maSu9SYNoSvA5U0MzJd3qROeucliXXMsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pCfX3nPd141Nw54zS79fQO+QwsD0CpMbShWqb4ldNxk=; b=o4EOCKkoIhLazKGJHY4emM8dmCfm5tsBGJBDpVZvfumT+K0qFkPM6WqSRKcph6yeBTw+yGLQe11GFrTEHXHg9TOYgP+AJFVXxR7AI1AEovyFBzkb2ZIntiROMivnJuwlP2oXCVfclmsMQXOWlX+FLa6bHT5zsGhoM1XgQo41aLEszkLekFV7JK0DvN9c3OrJQpFHK3Q6pFtmPNc7syVgbTBYMv1yyET4NpGLGCzBN4ozNWg/vl3eUY9iYKXqmv60ljmpeYKXkqBW2xz/N6cNjddMTLGfWjbPk0KvQU0C5GE9E3OK1NjN9kbcu/NljVORyASYtoEI9zkpk2LnavOAtg== Received: from BN9PR03CA0300.namprd03.prod.outlook.com (2603:10b6:408:f5::35) by MN2PR12MB4334.namprd12.prod.outlook.com (2603:10b6:208:1d1::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.29; Fri, 3 Feb 2023 05:21:45 +0000 Received: from BN8NAM11FT112.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f5:cafe::53) by BN9PR03CA0300.outlook.office365.com (2603:10b6:408:f5::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.27 via Frontend Transport; Fri, 3 Feb 2023 05:21:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN8NAM11FT112.mail.protection.outlook.com (10.13.176.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.28 via Frontend Transport; Fri, 3 Feb 2023 05:21:44 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 2 Feb 2023 21:21:38 -0800 Received: from nvidia.com (10.126.231.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 2 Feb 2023 21:21:36 -0800 From: Jiawei Wang To: , , Matan Azrad CC: , Subject: [PATCH 3/3] drivers: enhance the Tx queue affinity Date: Fri, 3 Feb 2023 07:21:07 +0200 Message-ID: <20230203052107.48293-4-jiaweiw@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230203052107.48293-1-jiaweiw@nvidia.com> References: <20230203052107.48293-1-jiaweiw@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT112:EE_|MN2PR12MB4334:EE_ X-MS-Office365-Filtering-Correlation-Id: 1888cc48-2702-437f-1b4e-08db05a68a96 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iFRm+bsi0zXUMxzWthKFQgVOHNVkTA/9+0ptpl8bIzBFrI8Ja4BVspLC3qEeb4FaU4FIzq8PE9Hv8R3esy0HqiwPRCqfT2XLzTrFyEtkI6bOOTSG01KSIQt6eqwlUseiAZhgUBIfP5+dS7ZwJMiXmK8s/qTZdVIFZYx97z7rp20xToPive5717Z8i64yOvMausMy8SmPs7klXHs2JONwYRcCpGredVWY84op+Z0sKIhColsF3335iSEE2FdUJhScsirsrUfBzM2w45zxKy5HWkAd2rJyUVuqHowLBlgmf49ARl3SUWGMEoSZqSfg4yZ+mQjORXOw1Vtrp2a7cM9qabaqlZ59+TFJQwuzlTluMJmiPGiA5okRjOf2s2tqH/+Flw12X9bD9DYXsFYDPihuqOFZ5ycQPiWjLf11U2ByfLO5K2mvpqE0okrFNvCHSMDdAOw9tVn5Awo7xe1o16+aYN/PfxmEbooolqOtwTtG2Sxs6QHG3e1uzAtxUVtlQ+WLAgmeoIJ9SbUXdl4W1ut+PDPhA5bt0pqBn4+H/jXe0QDz7Y7yv/lsmNtcKbSswmA6pjOb0RcOOThCR7Fs0sXTc4iYKsIDdmOgau5HNKLI4OPgqGNZnpr/ogR9xOJyq1aU4iMZYqKIIc2+d7cLkpLqj7ofBXoakYsG9g3stcsdvzpog4IVCAiQwB6cMxuKn1HCPSW3eYUuz9b0L5ffUZXG4w== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(136003)(376002)(396003)(346002)(39860400002)(451199018)(46966006)(40470700004)(36840700001)(36860700001)(82310400005)(7636003)(316002)(5660300002)(54906003)(6636002)(110136005)(426003)(336012)(2616005)(47076005)(83380400001)(40480700001)(40460700003)(82740400003)(36756003)(55016003)(86362001)(356005)(16526019)(7696005)(186003)(2906002)(6286002)(26005)(478600001)(70206006)(4326008)(70586007)(1076003)(107886003)(6666004)(8676002)(41300700001)(8936002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2023 05:21:44.8829 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1888cc48-2702-437f-1b4e-08db05a68a96 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT112.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4334 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Previous patch supports the tx phy affinity configuration in the Tx queue API, it supports to set the affinity value per Queue. This patch updates TIS creation with tx_phy_affinity value of Tx queue, TIS index 1 goes to hardware port 0, TIS index 2 goes to hardware port 1, and TIS index 0 is reserved for default HWS hash mode. Signed-off-by: Jiawei Wang --- drivers/common/mlx5/mlx5_prm.h | 8 ------- drivers/net/mlx5/mlx5.c | 43 +++++++++++++++------------------- drivers/net/mlx5/mlx5_devx.c | 24 ++++++++++--------- drivers/net/mlx5/mlx5_ethdev.c | 1 + drivers/net/mlx5/mlx5_tx.h | 1 + drivers/net/mlx5/mlx5_txq.c | 8 +++++++ 6 files changed, 42 insertions(+), 43 deletions(-) diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 8bbb800206..ded001d0b2 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -2331,14 +2331,6 @@ struct mlx5_ifc_query_nic_vport_context_in_bits { u8 reserved_at_68[0x18]; }; -/* - * lag_tx_port_affinity: 0 auto-selection, 1 PF1, 2 PF2 vice versa. - * Each TIS binds to one PF by setting lag_tx_port_affinity (>0). - * Once LAG enabled, we create multiple TISs and bind each one to - * different PFs, then TIS[i] gets affinity i+1 and goes to PF i+1. - */ -#define MLX5_IFC_LAG_MAP_TIS_AFFINITY(index, num) ((num) ? \ - (index) % (num) + 1 : 0) struct mlx5_ifc_tisc_bits { u8 strict_lag_tx_port_affinity[0x1]; u8 reserved_at_1[0x3]; diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index b8643cebdd..c75c98b8b0 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1162,9 +1162,9 @@ mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev) static int mlx5_setup_tis(struct mlx5_dev_ctx_shared *sh) { - int i; struct mlx5_devx_lag_context lag_ctx = { 0 }; struct mlx5_devx_tis_attr tis_attr = { 0 }; + int i; tis_attr.transport_domain = sh->td->id; if (sh->bond.n_port) { @@ -1178,35 +1178,30 @@ mlx5_setup_tis(struct mlx5_dev_ctx_shared *sh) DRV_LOG(ERR, "Failed to query lag affinity."); return -1; } - if (sh->lag.affinity_mode == MLX5_LAG_MODE_TIS) { - for (i = 0; i < sh->bond.n_port; i++) { - tis_attr.lag_tx_port_affinity = - MLX5_IFC_LAG_MAP_TIS_AFFINITY(i, - sh->bond.n_port); - sh->tis[i] = mlx5_devx_cmd_create_tis(sh->cdev->ctx, - &tis_attr); - if (!sh->tis[i]) { - DRV_LOG(ERR, "Failed to TIS %d/%d for bonding device" - " %s.", i, sh->bond.n_port, - sh->ibdev_name); - return -1; - } - } + if (sh->lag.affinity_mode == MLX5_LAG_MODE_TIS) DRV_LOG(DEBUG, "LAG number of ports : %d, affinity_1 & 2 : pf%d & %d.\n", sh->bond.n_port, lag_ctx.tx_remap_affinity_1, lag_ctx.tx_remap_affinity_2); - return 0; - } - if (sh->lag.affinity_mode == MLX5_LAG_MODE_HASH) + else if (sh->lag.affinity_mode == MLX5_LAG_MODE_HASH) DRV_LOG(INFO, "Device %s enabled HW hash based LAG.", sh->ibdev_name); } - tis_attr.lag_tx_port_affinity = 0; - sh->tis[0] = mlx5_devx_cmd_create_tis(sh->cdev->ctx, &tis_attr); - if (!sh->tis[0]) { - DRV_LOG(ERR, "Failed to TIS 0 for bonding device" - " %s.", sh->ibdev_name); - return -1; + for (i = 0; i <= sh->bond.n_port; i++) { + /* + * lag_tx_port_affinity: 0 auto-selection, 1 PF1, 2 PF2 vice versa. + * Each TIS binds to one PF by setting lag_tx_port_affinity (> 0). + * Once LAG enabled, we create multiple TISs and bind each one to + * different PFs, then TIS[i+1] gets affinity i+1 and goes to PF i+1. + * TIS[0] is reserved for HW Hash mode. + */ + tis_attr.lag_tx_port_affinity = i; + sh->tis[i] = mlx5_devx_cmd_create_tis(sh->cdev->ctx, &tis_attr); + if (!sh->tis[i]) { + DRV_LOG(ERR, "Failed to create TIS %d/%d for [bonding] device" + " %s.", i, sh->bond.n_port, + sh->ibdev_name); + return -1; + } } return 0; } diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index f6e1943fd7..a3fe0b3b98 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1190,17 +1190,19 @@ static uint32_t mlx5_get_txq_tis_num(struct rte_eth_dev *dev, uint16_t queue_idx) { struct mlx5_priv *priv = dev->data->dev_private; - int tis_idx; - - if (priv->sh->bond.n_port && priv->sh->lag.affinity_mode == - MLX5_LAG_MODE_TIS) { - tis_idx = (priv->lag_affinity_idx + queue_idx) % - priv->sh->bond.n_port; - DRV_LOG(INFO, "port %d txq %d gets affinity %d and maps to PF %d.", - dev->data->port_id, queue_idx, tis_idx + 1, - priv->sh->lag.tx_remap_affinity[tis_idx]); - } else { - tis_idx = 0; + struct mlx5_txq_data *txq_data = (*priv->txqs)[queue_idx]; + int tis_idx = 0; + + if (priv->sh->bond.n_port) { + if (txq_data->tx_phy_affinity) { + tis_idx = txq_data->tx_phy_affinity; + } else if (priv->sh->lag.affinity_mode == MLX5_LAG_MODE_TIS) { + tis_idx = (priv->lag_affinity_idx + queue_idx) % + priv->sh->bond.n_port + 1; + DRV_LOG(INFO, "port %d txq %d gets affinity %d and maps to PF %d.", + dev->data->port_id, queue_idx, tis_idx, + priv->sh->lag.tx_remap_affinity[tis_idx - 1]); + } } MLX5_ASSERT(priv->sh->tis[tis_idx]); return priv->sh->tis[tis_idx]->id; diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 4a85415ff3..354bb4f022 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -352,6 +352,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) info->switch_info.domain_id = priv->domain_id; info->switch_info.port_id = priv->representor_id; info->switch_info.rx_domain = 0; /* No sub Rx domains. */ + info->nb_phy_ports = priv->sh->bond.n_port; if (priv->representor) { uint16_t port_id; diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h index a056be7ca8..674c2aebe5 100644 --- a/drivers/net/mlx5/mlx5_tx.h +++ b/drivers/net/mlx5/mlx5_tx.h @@ -144,6 +144,7 @@ struct mlx5_txq_data { uint16_t inlen_send; /* Ordinary send data inline size. */ uint16_t inlen_empw; /* eMPW max packet size to inline. */ uint16_t inlen_mode; /* Minimal data length to inline. */ + uint8_t tx_phy_affinity; /* TxQ affinity configuration. */ uint32_t qp_num_8s; /* QP number shifted by 8. */ uint64_t offloads; /* Offloads for Tx Queue. */ struct mlx5_mr_ctrl mr_ctrl; /* MR control descriptor. */ diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 5543f2c570..4e53706807 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -392,6 +392,13 @@ mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, container_of(txq, struct mlx5_txq_ctrl, txq); int res; + if (conf->tx_phy_affinity > priv->num_lag_ports) { + rte_errno = EINVAL; + DRV_LOG(ERR, "port %u unable to setup Tx queue index %u" + " affinity is %u exceeds the maximum %u", dev->data->port_id, + idx, conf->tx_phy_affinity, priv->num_lag_ports); + return -rte_errno; + } res = mlx5_tx_queue_pre_setup(dev, idx, &desc); if (res) return res; @@ -1095,6 +1102,7 @@ mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, tmpl->txq.elts_m = desc - 1; tmpl->txq.port_id = dev->data->port_id; tmpl->txq.idx = idx; + tmpl->txq.tx_phy_affinity = conf->tx_phy_affinity; txq_set_params(tmpl); if (txq_adjust_params(tmpl)) goto error;