From patchwork Sun May 8 14:25:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 110907 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 83D49A0510; Sun, 8 May 2022 16:27:15 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 38FC24282F; Sun, 8 May 2022 16:26:51 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by mails.dpdk.org (Postfix) with ESMTP id CD8254282F for ; Sun, 8 May 2022 16:26:48 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C7QebJ8kQORZGpVqsWG7z3D1S4qEEO3lR2Y1/5PF8CVKrCS7+iYCjVb3D7ZJjZRC8yTZgnycxWiJSxON0idePo/6RVcsftCZcCcwKWcXoR9h+J3fCxjD2xiQnn9D24NlbEjVx2eeX1PTo47+uya33Q5V3EIOn22gIBPw2fym/BaMS56FEmotHI38q2cmwtBBIajvBaCl3IVO7kmdvvYtp74aBWxj6rRT04PfxTMG5+n4v8pvfS5uGcCYmjsAonrVSDAU6ZdIZBZkLella9cS/34fCHK8LJlUoE6k9qfdRFitu32UqJcjHKkQpHI8YnpHZkKZvrAwJYTUQrQpFEsg7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=v5NRRq4xqnWep/hh4/ePXyobRSHXVIphFtdsj9EG11g=; b=ghzXk5+iGAup0MzpO6ruWeDPMa3NYpWZntn7Lq7jSO5u4MaW60m2+C/xS+1C3bkj+o2NS11DYK7NQ++SxT/FXP8ntBJa+VrxNpQfUyGv0eyQFz0W1Ff39QaT/3elh3qm9neDIAEsK9zqr7EbE5yCy3nZHqR1w0ZLzPpqKaPC6mNiSU4+vwKcgQkQh3QH627JiGNRlBqe3JIaAu7BuhB19Pc89hOy/6gJEPB8dLAcSnh4yzyYhxTR64pDpV6wfcl5ha+jw0yaAQVXitxqvu8TSVf2iNDsYMDEQmr/IqRPLm7HJCCKMrEAvHM4IYbYPbG13nwUgO7N7G1qj8RnZUiBOQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=v5NRRq4xqnWep/hh4/ePXyobRSHXVIphFtdsj9EG11g=; b=ocb6gH61RPgI3June3g1W+wbxhb/dvmJ2tzHiUvTtnkKBMerHABDiLmRHZs6meLhsuHEUNSQrUBpGVQXfxyh3B24LOOA2HIA4wt08jdRpmBd1xoiS+/PJnis/Tx2PDR3tbi3wlPT6r9w0AvCLD4y8d/rrzO10UZsRXCt81KNX5Z/5gdretri24fsTealj72GxJm5PUrcTalU1NUyiKv3/2aRhVlrtOxSTdh8yVhtad3udXzMY5TCf/LXJdesUJsDZOjGGFkRagax+6dpvEwgYkDF2cIGNzv/t8L6zYc0OVlTZZNfHb9iIk+jUOsvlUFO378EhAPwTMBp7Y5Me9bHGA== Received: from DM6PR10CA0009.namprd10.prod.outlook.com (2603:10b6:5:60::22) by DS7PR12MB6046.namprd12.prod.outlook.com (2603:10b6:8:85::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.20; Sun, 8 May 2022 14:26:47 +0000 Received: from DM6NAM11FT008.eop-nam11.prod.protection.outlook.com (2603:10b6:5:60:cafe::85) by DM6PR10CA0009.outlook.office365.com (2603:10b6:5:60::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24 via Frontend Transport; Sun, 8 May 2022 14:26:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by DM6NAM11FT008.mail.protection.outlook.com (10.13.172.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5227.15 via Frontend Transport; Sun, 8 May 2022 14:26:47 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Sun, 8 May 2022 14:26:46 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Sun, 8 May 2022 07:26:44 -0700 From: Xueming Li To: , Maxime Coquelin CC: Subject: [PATCH v3 7/7] vdpa/mlx5: make statistics counter persistent Date: Sun, 8 May 2022 17:25:54 +0300 Message-ID: <20220508142554.560354-8-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220508142554.560354-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220508142554.560354-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b833dedb-5950-49f4-c3d4-08da30fec8a1 X-MS-TrafficTypeDiagnostic: DS7PR12MB6046:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iWxJZBQmg3pRsAv8PsCtHfthYXxgZlrAah/37Iu4V5i3CgTu2srm0YP/LPkBrT9lL76dtd4YadSQhK6zvhB3thrKb5xVci0TgANNd0q53n0YvqvevjyNhfgCjG0TxTe8MY9YiBlAB9nrVy26GNHHueaa9BbQC278rIu+oGkcLexfzZWKfyQKR/y/RLegsKF59CqIvUEQauZyjPxsY0p4M3C2ugtOL/8RTP1qQ0hHtsGdVkZeJktsuEwK1muhJZDuqvd6Nht8ZSYJt1WpBH5SBBVpOMwes7UjyiOYEqan0uk5l2W2Qjnul2YCDcg/hPLcZUtjKlT9vtypt8nJIgmYrZ9VTgA7BUmznr8Tkq9lGP8Dim/qEQlncSIRMx96o54qLeIAJCYSpyBhYhzOdpqYPKvY4z/uTPV84N9PVhtpUf8YPrVjH3kMe1YPHAmNt29FSmiFFXPPrmpDsL/+6i8CWLHtoE2dpUMGhsqIINhxm8NWuxkp8sr3o+jJ0MJjX95jUF4KYP/aK+x8XsM9pi9of7oAO/pQwkhZ1ij3/wAn+7zg4uWDr8SeZccIUrIBN+qI6Ytkxm855egB+OXS3PLh5zloASxkpha5kPvg0rJ4+Ze09+2VDqf35HbZEEvWel4feGb1lPWX6DVgHCRxz91z+jnIexWBlU1xur+ZqzsPk7yTH7oQu58xPVQ2hQ1EbJDBd0n16Mz5MqEl8bErwtmiPw== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(70206006)(81166007)(70586007)(36860700001)(2906002)(47076005)(426003)(356005)(8676002)(336012)(40460700003)(83380400001)(4326008)(55016003)(316002)(508600001)(6666004)(110136005)(36756003)(7696005)(2616005)(26005)(1076003)(82310400005)(107886003)(6286002)(86362001)(186003)(16526019)(5660300002)(8936002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2022 14:26:47.0967 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b833dedb-5950-49f4-c3d4-08da30fec8a1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT008.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6046 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In order to speed-up the device suspend and resume, make the statistics counters persistent in reconfiguration until the device gets removed. Signed-off-by: Xueming Li Reviewed-by: Maxime Coquelin --- doc/guides/vdpadevs/mlx5.rst | 6 ++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 19 +++++++---------- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 32 +++++++++++------------------ 4 files changed, 26 insertions(+), 32 deletions(-) diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst index acb791032ad..3ded142311e 100644 --- a/doc/guides/vdpadevs/mlx5.rst +++ b/doc/guides/vdpadevs/mlx5.rst @@ -109,3 +109,9 @@ Upon potential hardware errors, mlx5 PMD try to recover, give up if failed 3 times in 3 seconds, virtq will be put in disable state. User should check log to get error information, or query vdpa statistics counter to know error type and count report. + +Statistics +^^^^^^^^^^ + +The device statistics counter persists in reconfiguration until the device gets +removed. User can reset counters by calling function rte_vdpa_reset_stats(). diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index b1d5487080d..76fa5d4299e 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -388,12 +388,7 @@ mlx5_vdpa_get_stats(struct rte_vdpa_device *vdev, int qid, DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (priv->state == MLX5_VDPA_STATE_PROBED) { - DRV_LOG(ERR, "Device %s was not configured.", - vdev->device->name); - return -ENODATA; - } - if (qid >= (int)priv->nr_virtqs) { + if (qid >= (int)priv->caps.max_num_virtio_queues * 2) { DRV_LOG(ERR, "Too big vring id: %d for device %s.", qid, vdev->device->name); return -E2BIG; @@ -416,12 +411,7 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (priv->state == MLX5_VDPA_STATE_PROBED) { - DRV_LOG(ERR, "Device %s was not configured.", - vdev->device->name); - return -ENODATA; - } - if (qid >= (int)priv->nr_virtqs) { + if (qid >= (int)priv->caps.max_num_virtio_queues * 2) { DRV_LOG(ERR, "Too big vring id: %d for device %s.", qid, vdev->device->name); return -E2BIG; @@ -695,6 +685,11 @@ mlx5_vdpa_release_dev_resources(struct mlx5_vdpa_priv *priv) uint32_t i; mlx5_vdpa_dev_cache_clean(priv); + for (i = 0; i < priv->caps.max_num_virtio_queues * 2; i++) { + if (!priv->virtqs[i].counters) + continue; + claim_zero(mlx5_devx_cmd_destroy(priv->virtqs[i].counters)); + } mlx5_vdpa_event_qp_global_release(priv); mlx5_vdpa_err_event_unset(priv); if (priv->steer.tbl) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 24bafe85b44..e7f3319f896 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -92,6 +92,7 @@ struct mlx5_vdpa_virtq { struct rte_intr_handle *intr_handle; uint64_t err_time[3]; /* RDTSC time of recent errors. */ uint32_t n_retry; + struct mlx5_devx_virtio_q_couners_attr stats; struct mlx5_devx_virtio_q_couners_attr reset; }; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index 0dfeb8fce24..e025be47d27 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -127,14 +127,9 @@ void mlx5_vdpa_virtqs_release(struct mlx5_vdpa_priv *priv) { int i; - struct mlx5_vdpa_virtq *virtq; - for (i = 0; i < priv->nr_virtqs; i++) { - virtq = &priv->virtqs[i]; - mlx5_vdpa_virtq_unset(virtq); - if (virtq->counters) - claim_zero(mlx5_devx_cmd_destroy(virtq->counters)); - } + for (i = 0; i < priv->nr_virtqs; i++) + mlx5_vdpa_virtq_unset(&priv->virtqs[i]); priv->features = 0; priv->nr_virtqs = 0; } @@ -590,7 +585,7 @@ mlx5_vdpa_virtq_stats_get(struct mlx5_vdpa_priv *priv, int qid, struct rte_vdpa_stat *stats, unsigned int n) { struct mlx5_vdpa_virtq *virtq = &priv->virtqs[qid]; - struct mlx5_devx_virtio_q_couners_attr attr = {0}; + struct mlx5_devx_virtio_q_couners_attr *attr = &virtq->stats; int ret; if (!virtq->counters) { @@ -598,7 +593,7 @@ mlx5_vdpa_virtq_stats_get(struct mlx5_vdpa_priv *priv, int qid, "is invalid.", qid); return -EINVAL; } - ret = mlx5_devx_cmd_query_virtio_q_counters(virtq->counters, &attr); + ret = mlx5_devx_cmd_query_virtio_q_counters(virtq->counters, attr); if (ret) { DRV_LOG(ERR, "Failed to read virtq %d stats from HW.", qid); return ret; @@ -608,37 +603,37 @@ mlx5_vdpa_virtq_stats_get(struct mlx5_vdpa_priv *priv, int qid, return ret; stats[MLX5_VDPA_STATS_RECEIVED_DESCRIPTORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_RECEIVED_DESCRIPTORS, - .value = attr.received_desc - virtq->reset.received_desc, + .value = attr->received_desc - virtq->reset.received_desc, }; if (ret == MLX5_VDPA_STATS_COMPLETED_DESCRIPTORS) return ret; stats[MLX5_VDPA_STATS_COMPLETED_DESCRIPTORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_COMPLETED_DESCRIPTORS, - .value = attr.completed_desc - virtq->reset.completed_desc, + .value = attr->completed_desc - virtq->reset.completed_desc, }; if (ret == MLX5_VDPA_STATS_BAD_DESCRIPTOR_ERRORS) return ret; stats[MLX5_VDPA_STATS_BAD_DESCRIPTOR_ERRORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_BAD_DESCRIPTOR_ERRORS, - .value = attr.bad_desc_errors - virtq->reset.bad_desc_errors, + .value = attr->bad_desc_errors - virtq->reset.bad_desc_errors, }; if (ret == MLX5_VDPA_STATS_EXCEED_MAX_CHAIN) return ret; stats[MLX5_VDPA_STATS_EXCEED_MAX_CHAIN] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_EXCEED_MAX_CHAIN, - .value = attr.exceed_max_chain - virtq->reset.exceed_max_chain, + .value = attr->exceed_max_chain - virtq->reset.exceed_max_chain, }; if (ret == MLX5_VDPA_STATS_INVALID_BUFFER) return ret; stats[MLX5_VDPA_STATS_INVALID_BUFFER] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_INVALID_BUFFER, - .value = attr.invalid_buffer - virtq->reset.invalid_buffer, + .value = attr->invalid_buffer - virtq->reset.invalid_buffer, }; if (ret == MLX5_VDPA_STATS_COMPLETION_ERRORS) return ret; stats[MLX5_VDPA_STATS_COMPLETION_ERRORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_COMPLETION_ERRORS, - .value = attr.error_cqes - virtq->reset.error_cqes, + .value = attr->error_cqes - virtq->reset.error_cqes, }; return ret; } @@ -649,11 +644,8 @@ mlx5_vdpa_virtq_stats_reset(struct mlx5_vdpa_priv *priv, int qid) struct mlx5_vdpa_virtq *virtq = &priv->virtqs[qid]; int ret; - if (!virtq->counters) { - DRV_LOG(ERR, "Failed to read virtq %d statistics - virtq " - "is invalid.", qid); - return -EINVAL; - } + if (virtq->counters == NULL) /* VQ not enabled. */ + return 0; ret = mlx5_devx_cmd_query_virtio_q_counters(virtq->counters, &virtq->reset); if (ret)