From patchwork Wed Nov 24 09:40:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 104660 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 449A3A0C52; Wed, 24 Nov 2021 10:41:01 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 74FFD41176; Wed, 24 Nov 2021 10:40:54 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2077.outbound.protection.outlook.com [40.107.244.77]) by mails.dpdk.org (Postfix) with ESMTP id F3EF34116D for ; Wed, 24 Nov 2021 10:40:51 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HHF8iIaxe0WvRo+HXKdZpbzndy/GIqtxCFVxMNmTmPgAq+Txe4daEQHJP6bb1pNWGivVK83Q79n9L/S8lTpdIsMjyOyc6amXgZW9l1xOoAE7ha4tJNwdcMRU7ajKjinbteu41f6QpoVsFAlAefDHLNzgNiC+k+xD2Wc7Tc6mYFub+qk8obe3Kvmlur1M3YdfznrOOkVyEoy2F2DCnAUCYIUm6NGhKBO176kfrpPrJiTaSpgEsbIElOLwfDyDkJfomUhpP3LW34LHoMjJht9DBLb6RPW46iJkez7qF/e0o35p+SbEbmpp8dQxH7ag9lT51TgPqYb0uIQ/8vqFyFQfYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CFRYjF8f8yOEMLARjKK5vQ1Bxgzctnr6vpKtTBIGxDc=; b=b1Bh8ojjV42rHgPY0+aULixVcaxKd9JIn47H8eApeOCtQxS1jXe8BAKvG0wHArb0ttd3+y+jLskYI3f6EvVyQgY7y45I2gRrSL5W66PqF+/lvJbjXhyo0G3sY55JFRoonulTN0/xfPEkVRPA3EYRpbYFfKtmi/KHM9EDskgKosqPlpBrVqJJ1CM9Dd07R/fZLvCK1d/JyxarBgg7G8fnBEslaq1CYJhun809cox+Uk+1ASOTFdoDfly2kFZwSMZoOt7sVtuoIfsVCsqeDZm7mUeIGxu6zpuYbrZP+OxonaFmi6uRzYzsz8zLua0hR9BuDo9oPRHA6e4xp1SRbC0kYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CFRYjF8f8yOEMLARjKK5vQ1Bxgzctnr6vpKtTBIGxDc=; b=a4BeuREzEOdVgd/Eqopv1ngTjX9UAnWkeN9Ed3l127EiyLui3m4caARnt6ot2VW4hEj5okwvfZKip8OvGVjjPeHQ9AV2R38DaQ6NTr6J5/63Yku4MpToUCrdMbFmEiGzjeWz/cJITuajFCLA1QfCJxhVeimfU1Syauz9xzqA2XQZCUd1lQ76dHxYDNoggK4v7jlFJrawGXszBXc0YHjZwMbpdFWfrtPK1k5/X2Oon8AqGL0i3G3NjPM8OP+QktOTo/QJvjwsMRC/Nr/g6SdTUIwrqSDlAl72QNNNzziGs96icO4HRg9qHGMnYXSeJeRq1CbdPUPJUnb20qlaIQ2eYA== Received: from DM5PR10CA0022.namprd10.prod.outlook.com (2603:10b6:4:2::32) by DM6PR12MB4011.namprd12.prod.outlook.com (2603:10b6:5:1c5::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4713.22; Wed, 24 Nov 2021 09:40:49 +0000 Received: from DM6NAM11FT064.eop-nam11.prod.protection.outlook.com (2603:10b6:4:2:cafe::a7) by DM5PR10CA0022.outlook.office365.com (2603:10b6:4:2::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.21 via Frontend Transport; Wed, 24 Nov 2021 09:40:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT064.mail.protection.outlook.com (10.13.172.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4713.20 via Frontend Transport; Wed, 24 Nov 2021 09:40:48 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 24 Nov 2021 09:40:45 +0000 From: Dmitry Kozlyuk To: CC: Raslan Darawsheh , Matan Azrad , Viacheslav Ovsiienko Subject: [PATCH v3 1/2] net/mlx5: fix indirect RSS creation when port is stopped Date: Wed, 24 Nov 2021 11:40:29 +0200 Message-ID: <20211124094030.3360057-2-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211124094030.3360057-1-dkozlyuk@nvidia.com> References: <20211123223159.3324247-1-dkozlyuk@nvidia.com> <20211124094030.3360057-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d7b780dd-8c53-4f9b-b37a-08d9af2e7f27 X-MS-TrafficTypeDiagnostic: DM6PR12MB4011: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3513; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: U+VEiKIP8wSfEHxvk11aF6tsSnMLEhFFn8rQK8xy0m3Y3Kh2kM1FxT0g/Oeh2kYeP5Q72fcTY/S4ziAcez9gz5cdLChGjyGVHXtRke1dWHMxDJrkEVqkujQ6Oo7F2QoNx46U9Soukd7mdVMkGpApT05/k1G814wXjZAsnr+D1N0OpknvrL5L4TSKWEJUxZPtzUMdpy51IBCgScUgDG6DHICH8KMxyU55LUe8oreZHrR5P6wkM9JW+zDGxpBPVZk4DfwxgwQ3dZWA0k1Ei0pZSp2taYBgnCBiQ/dkdcI63fPs96wLLzcEtbYyU8Oi+w1KLq4kVhFC+kezTuyF3ysABW/uTk6nmdnUedUM3macZVeKy26cx+EAtAyHSiD927WKJ/ynjwrzL9otX3rXmG87Fvbk6AFF9NZHu1Nv/OwQ/Xy3J6Chv9V6zm0vEBB17luO6sk9lzbbBsegKSAPufqQZ4EwMCWekEMNAgoPWsUlRGxdrahwcx6i307fGoqsVKpp5YKj8UMb9SaJSA4ohL3H31EDW+1t/Mt8y8TL3yxxFAVedKnNK7LOsNtMdQ5UTNsFNHUrj1HFaaKx6OwzNZKX8xUn/PIDOrlSedfM/qXY7HlVPPymgxGcX+ZT/HvFvc/qVBTottNgKekjvqEHs69ELPsFxwyC6Z5r8Sxcmlle1M7XGbSEoPZOS5AE8zGtd25qVGCYpbu2C0GFXyU8AqnAs5BF3seSYK/IIRbx0o454Hs= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(8936002)(26005)(83380400001)(70586007)(6916009)(7696005)(36860700001)(4326008)(2906002)(70206006)(426003)(1076003)(2616005)(336012)(36756003)(7636003)(6666004)(47076005)(356005)(107886003)(82310400004)(186003)(16526019)(8676002)(316002)(55016003)(86362001)(6286002)(54906003)(508600001)(5660300002)(309714004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Nov 2021 09:40:48.5131 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d7b780dd-8c53-4f9b-b37a-08d9af2e7f27 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT064.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4011 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org mlx5_ind_table_obj_setup() was incrementing RxQ reference counters even when the port was stopped, which prevented RxQ release and triggered an internal assertion. Only increment reference counter when the port is started. Fixes: ec4e11d41d12 ("net/mlx5: preserve indirect actions on restart") Signed-off-by: Dmitry Kozlyuk Reviewed-by: Matan Azrad --- drivers/net/mlx5/mlx5_flow_dv.c | 3 ++- drivers/net/mlx5/mlx5_rx.h | 3 ++- drivers/net/mlx5/mlx5_rxq.c | 41 +++++++++++++++++++++------------ 3 files changed, 30 insertions(+), 17 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index 9d4bd0560c..a8f63e22c4 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -14700,7 +14700,8 @@ __flow_dv_action_rss_setup(struct rte_eth_dev *dev, size_t i; int err; - if (mlx5_ind_table_obj_setup(dev, shared_rss->ind_tbl)) { + if (mlx5_ind_table_obj_setup(dev, shared_rss->ind_tbl, + !!dev->data->dev_started)) { return rte_flow_error_set(error, rte_errno, RTE_FLOW_ERROR_TYPE_UNSPECIFIED, NULL, "cannot setup indirection table"); diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index b19464bb37..283242f530 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -228,7 +228,8 @@ int mlx5_ind_table_obj_release(struct rte_eth_dev *dev, bool standalone, bool deref_rxqs); int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, - struct mlx5_ind_table_obj *ind_tbl); + struct mlx5_ind_table_obj *ind_tbl, + bool ref_qs); int mlx5_ind_table_obj_modify(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl, uint16_t *queues, const uint32_t queues_n, diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 191f1e483f..2dd9490c36 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2263,39 +2263,45 @@ mlx5_ind_table_obj_verify(struct rte_eth_dev *dev) * Pointer to Ethernet device. * @param ind_table * Indirection table to modify. + * @param ref_qs + * Whether to increment RxQ reference counters. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, - struct mlx5_ind_table_obj *ind_tbl) + struct mlx5_ind_table_obj *ind_tbl, + bool ref_qs) { struct mlx5_priv *priv = dev->data->dev_private; uint32_t queues_n = ind_tbl->queues_n; uint16_t *queues = ind_tbl->queues; - unsigned int i, j; + unsigned int i = 0, j; int ret = 0, err; const unsigned int n = rte_is_power_of_2(queues_n) ? log2above(queues_n) : log2above(priv->config.ind_table_max_size); - for (i = 0; i != queues_n; ++i) { - if (mlx5_rxq_ref(dev, queues[i]) == NULL) { - ret = -rte_errno; - goto error; + if (ref_qs) + for (i = 0; i != queues_n; ++i) { + if (mlx5_rxq_ref(dev, queues[i]) == NULL) { + ret = -rte_errno; + goto error; + } } - } ret = priv->obj_ops.ind_table_new(dev, n, ind_tbl); if (ret) goto error; __atomic_fetch_add(&ind_tbl->refcnt, 1, __ATOMIC_RELAXED); return 0; error: - err = rte_errno; - for (j = 0; j < i; j++) - mlx5_rxq_deref(dev, ind_tbl->queues[j]); - rte_errno = err; + if (ref_qs) { + err = rte_errno; + for (j = 0; j < i; j++) + mlx5_rxq_deref(dev, queues[j]); + rte_errno = err; + } DRV_LOG(DEBUG, "Port %u cannot setup indirection table.", dev->data->port_id); return ret; @@ -2312,13 +2318,15 @@ mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, * Number of queues in the array. * @param standalone * Indirection table for Standalone queue. + * @param ref_qs + * Whether to increment RxQ reference counters. * * @return * The Verbs/DevX object initialized, NULL otherwise and rte_errno is set. */ static struct mlx5_ind_table_obj * mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues, - uint32_t queues_n, bool standalone) + uint32_t queues_n, bool standalone, bool ref_qs) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_ind_table_obj *ind_tbl; @@ -2333,7 +2341,7 @@ mlx5_ind_table_obj_new(struct rte_eth_dev *dev, const uint16_t *queues, ind_tbl->queues_n = queues_n; ind_tbl->queues = (uint16_t *)(ind_tbl + 1); memcpy(ind_tbl->queues, queues, queues_n * sizeof(*queues)); - ret = mlx5_ind_table_obj_setup(dev, ind_tbl); + ret = mlx5_ind_table_obj_setup(dev, ind_tbl, ref_qs); if (ret < 0) { mlx5_free(ind_tbl); return NULL; @@ -2537,6 +2545,7 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx, struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_hrxq *hrxq = mlx5_ipool_get(priv->sh->ipool[MLX5_IPOOL_HRXQ], hrxq_idx); + bool dev_started = !!dev->data->dev_started; int ret; if (!hrxq) { @@ -2565,7 +2574,8 @@ mlx5_hrxq_modify(struct rte_eth_dev *dev, uint32_t hrxq_idx, ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n); if (!ind_tbl) ind_tbl = mlx5_ind_table_obj_new(dev, queues, queues_n, - hrxq->standalone); + hrxq->standalone, + dev_started); } if (!ind_tbl) { rte_errno = ENOMEM; @@ -2657,7 +2667,8 @@ __mlx5_hrxq_create(struct rte_eth_dev *dev, ind_tbl = mlx5_ind_table_obj_get(dev, queues, queues_n); if (!ind_tbl) ind_tbl = mlx5_ind_table_obj_new(dev, queues, queues_n, - standalone); + standalone, + !!dev->data->dev_started); if (!ind_tbl) return NULL; hrxq = mlx5_ipool_zmalloc(priv->sh->ipool[MLX5_IPOOL_HRXQ], &hrxq_idx); From patchwork Wed Nov 24 09:40:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 104659 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9883DA0C52; Wed, 24 Nov 2021 10:40:55 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A4584116E; Wed, 24 Nov 2021 10:40:53 +0100 (CET) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2058.outbound.protection.outlook.com [40.107.212.58]) by mails.dpdk.org (Postfix) with ESMTP id 0B0814116E for ; Wed, 24 Nov 2021 10:40:52 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Jpg4EaVHifW9yKfYaZ/8yKZpz/rqYhUJLncTzj9m0KcrzmWaRz08da9sP4ibWMBtV/3/K/vkECMFn6cl5uAfbP3KPOvLpv1+pg8QryjUv1oXoNiuKtUhdol9oaMA6B2S5XaaHVnC7tfHuP+9d7y11/PnS3RJ1EJ+tBLQcEWYhrBx9F9DOL5JRFfzXo7X09glx0FB+VRaUhiLU3DwJITD4AhtjAnEWkuv8fR49EFzh8IWx876yBrQy7tva3LXPJ9F5xUWI4avFvShE6A0WbxPBSpvuIGON81m82yyxgZqRPdL7U85OBYe2B3BL1sxvH4iIjjIqPUGDLd+486IFmDcTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ieeIfmfBuG+S+jx9dmfFEnVZGIBHjqvivJj4BBXFN9E=; b=VrDd7GZwyj6/3FXruX4eL7nZbja01zxeErO5AA8aVQbO0chOvCfOKkzs9BK2Uj7CHVl90pTIZK1MB3wgHDhXWqJa3lVYlQrifUJkc+NfVrJolpv6oIkml01KNNqO06YoQiHQdSWFkGH6aj48z0bP7kxKYzHR5tCyhQfqgWbCAKBhHkkKZGuh0Nicuv2UXJM8qVT4lm7NeD3uNJ2fHucrED/7FeUqKMAB9JQom3BjcoJqR+a06HMLxYUazBet4avG7AfZ5T6GX2Jiekshwpm61O2rAQY1glwgGTyGioOwfqeHsMzSextqe29Dp5cM9V7ioNBIz3E8WwKcrF7e2aBIUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ieeIfmfBuG+S+jx9dmfFEnVZGIBHjqvivJj4BBXFN9E=; b=d21BBRmSZkTVQwDPJCm1LvfFMVRrjqAjxd5CzITVDbavUFPtIYVBLDIUTvM3Irj2CAGIdg7oCtxQIekkeqhrp5ai11y+7KnDXO9eeRJXLx8Udw8I9DbYUhfo59BWGoz4WUgS8iq9aCbmZFzo6J8mrQi8yMwurXcatzzu6Gj88UqZW2vBCQvUMkpIGBCQwfCsmAjb0oMl8f2JI79SBuJIS/Au7sgPN2kcuO163rBsWRBXMhP0OD1ncaAUiR/4vErB/uic6tq54RFSgOtTbha17WsguuGWMY2yN6jA8YV2mQ5Q/vTlOXL0roF5FVlOzsCJ4TkMeg8KsZ9JhlZpauYE/g== Received: from DM5PR10CA0020.namprd10.prod.outlook.com (2603:10b6:4:2::30) by DM5PR12MB4679.namprd12.prod.outlook.com (2603:10b6:4:a2::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4713.22; Wed, 24 Nov 2021 09:40:50 +0000 Received: from DM6NAM11FT064.eop-nam11.prod.protection.outlook.com (2603:10b6:4:2:cafe::5c) by DM5PR10CA0020.outlook.office365.com (2603:10b6:4:2::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.21 via Frontend Transport; Wed, 24 Nov 2021 09:40:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT064.mail.protection.outlook.com (10.13.172.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4713.20 via Frontend Transport; Wed, 24 Nov 2021 09:40:49 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 24 Nov 2021 09:40:47 +0000 From: Dmitry Kozlyuk To: CC: Raslan Darawsheh , Matan Azrad , Viacheslav Ovsiienko Subject: [PATCH v3 2/2] net/mlx5: fix RxQ reference counting for indirect RSS Date: Wed, 24 Nov 2021 11:40:30 +0200 Message-ID: <20211124094030.3360057-3-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211124094030.3360057-1-dkozlyuk@nvidia.com> References: <20211123223159.3324247-1-dkozlyuk@nvidia.com> <20211124094030.3360057-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b8b6d102-c721-4b77-3c48-08d9af2e7fc5 X-MS-TrafficTypeDiagnostic: DM5PR12MB4679: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: F9FUy3TaypHTJiaXQkMyaTQNuQBXLfXbPYGw8f/fBlJ9+8cx7aaYGB8evJQE6sGt9JsHmSb9HvNwOvve+P53YPA46SPyML4rKzGh9PKevXal35q2eYxDubvF7h/iMDxY5hj2KYYxwVp+oEbYY0/gNCVpLB0Zqimw0CyXGU8Gz0FxU00BxdUxM3qjvGOFV72PG0FBTu4FANLzQeTbyef+mvJHirIPlhj0KrnJEbKtkfle5Y67cVzm1IBZ91/iQqCr64Jn/pxCsSHETCzd53JxC0s38Ptz40lS6PRqaH1NgJvb1pEFKXE0NbMWUejV6LNJQKXleeN2Ve/1PfjOvmE070OqDwc6Z3oxnRyxATI6yCPdcpJG2ag/gewV0DZIQWhF+ObgtCKdfFAjqD7/lme8iOohS/SFwiKEh2FPNBwWKCCxMNLtbkqiG5x5XlHRzAR7iubhINJ3ljuS9SFD7L4W+yzU7nGvjYLg2anOps2mAqhQOsKGLlWukPYi6CPNxEBaMG5GTxmqASqa9QV98+kmT1w9uHGOYhSf3GrOTB/8Cw0v/QGfaCEI30vw5oilAVw+ypBNWI5a/PtTT172el/Yb86zcgZUyYglNYFKLC321a57KHdyG89D46Ut3Q486qqk2Gv37XF6MaKZKwnzdtajCRVEa0u3Rqja5IU4vhsxl7PVYHKS8HMquJcmTOmZWjr/+69YHLyzC1cEjcOLXB6uwsC5XHxdWF4Z+bCqD4SHEPA= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(426003)(1076003)(336012)(6916009)(70586007)(16526019)(186003)(107886003)(8676002)(2616005)(36756003)(5660300002)(83380400001)(86362001)(6286002)(316002)(508600001)(55016003)(7696005)(54906003)(8936002)(2906002)(356005)(4326008)(47076005)(70206006)(6666004)(7636003)(26005)(36860700001)(82310400004)(309714004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Nov 2021 09:40:49.5545 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b8b6d102-c721-4b77-3c48-08d9af2e7fc5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT064.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB4679 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org mlx5_ind_table_obj_modify() was not changing the reference counters of neither the new set of RxQs, nor the old set of RxQs. On the other hand, creation of the RSS incremented the RxQ refcnt. If an RxQ was present in both the initial and the modified set, its reference counter was incremented one extra time compared to the queues that were only present in the new set. This prevented releasing said RxQ resources on port stop: flow indirect_action 0 create action_id 1 \ action rss queues 0 1 end / end flow indirect_action 0 update 1 \ action rss queues 2 3 end / end quit ... mlx5_net: mlx5.c:1622: mlx5_dev_close(): port 0 some Rx queue objects still remain mlx5_net: mlx5.c:1626: mlx5_dev_close(): port 0 some Rx queues still remain Increment reference counters for the new set of RxQs and decrement them for the old set of RxQs when needed. Remove explicit referencing of RxQ from mlx5_ind_table_obj_attach() because it reuses mlx5_ind_table_obj_modify() code doing this. Fixes: ec4e11d41d12 ("net/mlx5: preserve indirect actions on restart") Signed-off-by: Dmitry Kozlyuk Reviewed-by: Matan Azrad --- drivers/net/mlx5/mlx5_flow_dv.c | 6 ++++- drivers/net/mlx5/mlx5_rx.h | 3 ++- drivers/net/mlx5/mlx5_rxq.c | 44 ++++++++++++++++++++------------- 3 files changed, 34 insertions(+), 19 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index a8f63e22c4..9979d16f74 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -15073,6 +15073,7 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx, void *queue = NULL; uint16_t *queue_old = NULL; uint32_t queue_size = action_conf->queue_num * sizeof(uint16_t); + bool dev_started = !!dev->data->dev_started; if (!shared_rss) return rte_flow_error_set(error, EINVAL, @@ -15095,7 +15096,10 @@ __flow_dv_action_rss_update(struct rte_eth_dev *dev, uint32_t idx, rte_spinlock_lock(&shared_rss->action_rss_sl); queue_old = shared_rss->ind_tbl->queues; ret = mlx5_ind_table_obj_modify(dev, shared_rss->ind_tbl, - queue, action_conf->queue_num, true); + queue, action_conf->queue_num, + true /* standalone */, + dev_started /* ref_new_qs */, + dev_started /* deref_old_qs */); if (ret) { mlx5_free(queue); ret = rte_flow_error_set(error, rte_errno, diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index 283242f530..f10eee406b 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -233,7 +233,8 @@ int mlx5_ind_table_obj_setup(struct rte_eth_dev *dev, int mlx5_ind_table_obj_modify(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl, uint16_t *queues, const uint32_t queues_n, - bool standalone); + bool standalone, + bool ref_new_qs, bool deref_old_qs); int mlx5_ind_table_obj_attach(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl); int mlx5_ind_table_obj_detach(struct rte_eth_dev *dev, diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 2dd9490c36..dadcd0825d 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2387,6 +2387,10 @@ mlx5_ind_table_obj_check_standalone(struct rte_eth_dev *dev __rte_unused, * Number of queues in the array. * @param standalone * Indirection table for Standalone queue. + * @param ref_new_qs + * Whether to increment new RxQ set reference counters. + * @param deref_old_qs + * Whether to decrement old RxQ set reference counters. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. @@ -2395,10 +2399,10 @@ int mlx5_ind_table_obj_modify(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl, uint16_t *queues, const uint32_t queues_n, - bool standalone) + bool standalone, bool ref_new_qs, bool deref_old_qs) { struct mlx5_priv *priv = dev->data->dev_private; - unsigned int i; + unsigned int i = 0, j; int ret = 0, err; const unsigned int n = rte_is_power_of_2(queues_n) ? log2above(queues_n) : @@ -2408,22 +2412,30 @@ mlx5_ind_table_obj_modify(struct rte_eth_dev *dev, RTE_SET_USED(standalone); if (mlx5_ind_table_obj_check_standalone(dev, ind_tbl) < 0) return -rte_errno; - for (i = 0; i != queues_n; ++i) { - if (!mlx5_rxq_get(dev, queues[i])) { - ret = -rte_errno; - goto error; + if (ref_new_qs) + for (i = 0; i != queues_n; ++i) { + if (!mlx5_rxq_ref(dev, queues[i])) { + ret = -rte_errno; + goto error; + } } - } MLX5_ASSERT(priv->obj_ops.ind_table_modify); ret = priv->obj_ops.ind_table_modify(dev, n, queues, queues_n, ind_tbl); if (ret) goto error; + if (deref_old_qs) + for (i = 0; i < ind_tbl->queues_n; i++) + claim_nonzero(mlx5_rxq_deref(dev, ind_tbl->queues[i])); ind_tbl->queues_n = queues_n; ind_tbl->queues = queues; return 0; error: - err = rte_errno; - rte_errno = err; + if (ref_new_qs) { + err = rte_errno; + for (j = 0; j < i; j++) + mlx5_rxq_deref(dev, queues[j]); + rte_errno = err; + } DRV_LOG(DEBUG, "Port %u cannot setup indirection table.", dev->data->port_id); return ret; @@ -2444,19 +2456,17 @@ int mlx5_ind_table_obj_attach(struct rte_eth_dev *dev, struct mlx5_ind_table_obj *ind_tbl) { - unsigned int i; int ret; ret = mlx5_ind_table_obj_modify(dev, ind_tbl, ind_tbl->queues, - ind_tbl->queues_n, true); - if (ret != 0) { + ind_tbl->queues_n, + true /* standalone */, + true /* ref_new_qs */, + false /* deref_old_qs */); + if (ret != 0) DRV_LOG(ERR, "Port %u could not modify indirect table obj %p", dev->data->port_id, (void *)ind_tbl); - return ret; - } - for (i = 0; i < ind_tbl->queues_n; i++) - mlx5_rxq_ref(dev, ind_tbl->queues[i]); - return 0; + return ret; } /**