From patchwork Tue Jun 27 08:27:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dariusz Sosnowski X-Patchwork-Id: 129017 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8755E42D70; Tue, 27 Jun 2023 10:28:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A58042BC9; Tue, 27 Jun 2023 10:28:29 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2076.outbound.protection.outlook.com [40.107.243.76]) by mails.dpdk.org (Postfix) with ESMTP id C08F642B8B for ; Tue, 27 Jun 2023 10:28:27 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QYR/Hn9p9AOSLkvuR8gZF1btD2ac/t3Ct7QasrOB15SrSOKNCiZF8boSpwgTJSqDNpxHV4uyaKXTIMOUpMYYivSh9NXjjljYyUEAp4Aa7DQHUNsa2BOW82n3uwTF0uI10HbQwapEAnZKP0vjE3CqtARbriMo4+adTPtmTcPWNZXYeSp2vROlzAuQbY45+ix1ceRm0IOkn8jp29zyvf3Qg40qpk2czVW9qFwEnexldxb/Ap1tiqUKeCA0hvCW2u98esbhQmBWUiQAyFmCLh584hX/gpnM+lxUF9ZCWTAZ/mW/26L96jg7mZJpyM61T08ANLuu6kLeboYTUpT3/3cIgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=laTfqg4T7VuFfm9FaLpNkzPNjhiobreoWsSVmC9MpoY=; b=XBCNIuqXgEpU78Imw0zrLGp0klE81G5gpLnO9IDxIla7J/9mQ7NkttiW+39CD26AN5TVQ/EVDip2XxogBXRZ7fu/NhG3N71hYTv2c8fqIeR8xcoixwuUNtHhswfLGKrUOtLUPl0R4gs/uYI7YgtaoRmVb5WhPWFSmuoD9jACFtnPjLsSAQ51mfOOM0MvypIyKVJsfhvTKorLnDGMilr5mwyw/tmxVCkQCUy+sS1MSB9J/QA6oSALmskvsBdz7broVHhoirxDF3yX0FBtZoiyoTMFUBHPRXbZvcpy6dBL7knmceq3yZkqcuorZOjx3JWUE1sV8LS7jjjuWsxmGbRbvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=laTfqg4T7VuFfm9FaLpNkzPNjhiobreoWsSVmC9MpoY=; b=ERo7rcROYmsBvohal2NBP98vh65LMXEum3+rAEsZEMrczKEQf0TkY72a6eJYLIh5xtul6V5ovxh0DlD8mbbez94OEHB3HOWkLkIbQU/KOto9at7p6j5Rng2jUcsXw4vcgQAikxL9Qgn3AU3HkzMmQ9gxHfrv/d5/moaJ41r8rD/C9JWhSjHlWav7f/C/ABBxiTdaom0ocK1lYpPiBy01WMkr7xB0m8EayfSvamqSFVgVSD3EHh7skiyceswGO7kFFW2jG3v2DbCQWLqZoI4GPdUHtxRM+VzjwPwVqDYFLBoqgUfegqerQxxEQOj2QJjB1JJmD3WgwXTYn/nbx5GTow== Received: from DM6PR17CA0011.namprd17.prod.outlook.com (2603:10b6:5:1b3::24) by DM6PR12MB4354.namprd12.prod.outlook.com (2603:10b6:5:28f::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.21; Tue, 27 Jun 2023 08:28:26 +0000 Received: from DM6NAM11FT060.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1b3:cafe::8e) by DM6PR17CA0011.outlook.office365.com (2603:10b6:5:1b3::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.34 via Frontend Transport; Tue, 27 Jun 2023 08:28:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT060.mail.protection.outlook.com (10.13.173.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.47 via Frontend Transport; Tue, 27 Jun 2023 08:28:25 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Tue, 27 Jun 2023 01:28:12 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Tue, 27 Jun 2023 01:28:10 -0700 From: Dariusz Sosnowski To: Matan Azrad , Viacheslav Ovsiienko , Ori Kam , Suanming Mou CC: Subject: [PATCH 1/2] net/mlx5: fix queue used to deallocate counter Date: Tue, 27 Jun 2023 11:27:45 +0300 Message-ID: <20230627082746.2466304-2-dsosnowski@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230627082746.2466304-1-dsosnowski@nvidia.com> References: <20230627082746.2466304-1-dsosnowski@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT060:EE_|DM6PR12MB4354:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b8f7b50-c5ac-48a8-861b-08db76e87a56 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gwY7TFSz8KRm+BsjQFzpl/irnU3rO6F3cuTF8qZhRtBCqIxQG5WY6Sfs3XLI8rTWp7LMsmAzkDDGRtQWBv8NoLln8imsTcnEmuDah6B5pdzllPr2neH2NinO2J4prmqCDJ8KyUQUiu+TJxwI1Hx7Vf2Tw9BdaWx3yni1MlD+7WpU1SulTSYcSFv/duQlNkK5j8zT8NzsDp8xBD/uX2zMiCn10GwXc/+KtpvgIoRbvqICYBeaNPGVjTdQGGtvXde+M4gl0UWyhVlyWagk78FQsCSdG3w4tcGaKhLfDDv0Ki2yB9PKgjt0r2hpDyUIYgyzatbCBRWJUvNXK1Jysa7jN55wY4JbQQoRCtLykRX1PCaLmhSLmPSwrKejfXx/OSS3e1spTjmhgwoMkFHCyzDZo45bmwm8bWn4G4xrhM6B1GuYBaGGGz0UlJXszb62Zq/pyUMjMwxdYnyIY48792SgVSegb2WaQaGgGjz3/sycnFn1bZ5gP6XnHQCM2X6ez1yL3WRN+80kW5Uk0dzU9WfeT50TiT7U7S3vjwhIK9rDJ9vK6woJ5DmsiPbQxStL/o7Kyxs9caAjjBUTx8hlFduE8Z4mrjVJGQfkT03y7EMzzidKNXZf0gCxaVLMbGxE5IfmK3ZVzqnwvFsXHFG+FdVTAq9pMrIBoJRtM52aWbp/9KhVkXKvgn0DXQ0zG5vJK93bOG9lJnuBtGuxSf62tte8nnPVUGf9V5ZGUHPl3/XTHbZerFICeTquIJAFRU1lRe4x X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(136003)(396003)(376002)(39860400002)(451199021)(46966006)(40470700004)(36840700001)(6286002)(110136005)(40480700001)(478600001)(55016003)(7696005)(40460700003)(26005)(1076003)(6666004)(4326008)(82310400005)(5660300002)(2906002)(36756003)(6636002)(70586007)(70206006)(86362001)(8936002)(83380400001)(41300700001)(8676002)(316002)(36860700001)(336012)(426003)(47076005)(186003)(82740400003)(2616005)(16526019)(7636003)(356005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2023 08:28:25.8475 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0b8f7b50-c5ac-48a8-861b-08db76e87a56 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT060.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4354 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When ports are configured to share flow engine resources (configured through rte_flow_configure()), counter objects used during flow rule creation are pulled directly from the shared counter pool. This is done by calling mlx5_hws_cnt_pool_get() with NULL pointer passed as queue pointer. When a flow rule was destroyed on the host port, counter object was not returned to the pool, because mlx5_hws_cnt_pool_put() was always called with a pointer to the currently used queue. This forced placing the counter in the local queue cache. As a result, during the application runtime, some counter objects became inaccessible, since this queue is never fully flushed. This patch fixes that behavior, by changing the flow rule destroying logic. If port is configured to use shared resources, mlx5_hws_cnt_pool_put() is called with NULL pointer passed as queue pointer. Fixes: 13ea6bdcc7ee ("net/mlx5: support counters in cross port shared mode") Cc: viacheslavo@nvidia.com Signed-off-by: Dariusz Sosnowski Acked-by: Ori Kam --- drivers/net/mlx5/mlx5_flow_hw.c | 14 +++++++++----- drivers/net/mlx5/mlx5_hws_cnt.h | 16 ++++++++++++++++ 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c index b5137a822a..a0c7956626 100644 --- a/drivers/net/mlx5/mlx5_flow_hw.c +++ b/drivers/net/mlx5/mlx5_flow_hw.c @@ -2273,6 +2273,7 @@ flow_hw_actions_construct(struct rte_eth_dev *dev, struct mlx5_hrxq *hrxq; uint32_t ct_idx; cnt_id_t cnt_id; + uint32_t *cnt_queue; uint32_t mtr_id; action = &actions[act_data->action_src]; @@ -2429,10 +2430,9 @@ flow_hw_actions_construct(struct rte_eth_dev *dev, break; /* Fall-through. */ case RTE_FLOW_ACTION_TYPE_COUNT: - ret = mlx5_hws_cnt_pool_get(priv->hws_cpool, - (priv->shared_refcnt || - priv->hws_cpool->cfg.host_cpool) ? - NULL : &queue, &cnt_id, age_idx); + /* If the port is engaged in resource sharing, do not use queue cache. */ + cnt_queue = mlx5_hws_cnt_is_pool_shared(priv) ? NULL : &queue; + ret = mlx5_hws_cnt_pool_get(priv->hws_cpool, cnt_queue, &cnt_id, age_idx); if (ret != 0) return ret; ret = mlx5_hws_cnt_pool_get_action_offset @@ -3014,6 +3014,8 @@ flow_hw_age_count_release(struct mlx5_priv *priv, uint32_t queue, struct rte_flow_hw *flow, struct rte_flow_error *error) { + uint32_t *cnt_queue; + if (mlx5_hws_cnt_is_shared(priv->hws_cpool, flow->cnt_id)) { if (flow->age_idx && !mlx5_hws_age_is_indirect(flow->age_idx)) { /* Remove this AGE parameter from indirect counter. */ @@ -3024,8 +3026,10 @@ flow_hw_age_count_release(struct mlx5_priv *priv, uint32_t queue, } return; } + /* If the port is engaged in resource sharing, do not use queue cache. */ + cnt_queue = mlx5_hws_cnt_is_pool_shared(priv) ? NULL : &queue; /* Put the counter first to reduce the race risk in BG thread. */ - mlx5_hws_cnt_pool_put(priv->hws_cpool, &queue, &flow->cnt_id); + mlx5_hws_cnt_pool_put(priv->hws_cpool, cnt_queue, &flow->cnt_id); flow->cnt_id = 0; if (flow->age_idx) { if (mlx5_hws_age_is_indirect(flow->age_idx)) { diff --git a/drivers/net/mlx5/mlx5_hws_cnt.h b/drivers/net/mlx5/mlx5_hws_cnt.h index b4f3db0533..f37a7d6151 100644 --- a/drivers/net/mlx5/mlx5_hws_cnt.h +++ b/drivers/net/mlx5/mlx5_hws_cnt.h @@ -553,6 +553,22 @@ mlx5_hws_cnt_pool_get(struct mlx5_hws_cnt_pool *cpool, uint32_t *queue, return 0; } +/** + * Check if counter pool allocated for HWS is shared between ports. + * + * @param[in] priv + * Pointer to the port private data structure. + * + * @return + * True if counter pools is shared between ports. False otherwise. + */ +static __rte_always_inline bool +mlx5_hws_cnt_is_pool_shared(struct mlx5_priv *priv) +{ + return priv && priv->hws_cpool && + (priv->shared_refcnt || priv->hws_cpool->cfg.host_cpool != NULL); +} + static __rte_always_inline unsigned int mlx5_hws_cnt_pool_get_size(struct mlx5_hws_cnt_pool *cpool) {