From patchwork Thu Feb 24 15:51:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 108308 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C638CA034C; Thu, 24 Feb 2022 16:52:08 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AFD9642710; Thu, 24 Feb 2022 16:51:59 +0100 (CET) Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam08on2046.outbound.protection.outlook.com [40.107.100.46]) by mails.dpdk.org (Postfix) with ESMTP id 0EA8A42710 for ; Thu, 24 Feb 2022 16:51:58 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Pm31dkhXwkgPptQ7ToAH+t2wZdwnlF/05kNpLcohpImPlc4WWqqpIeN46SSSquEq9M16NVRPgYCvmQoSXQJfqx4ztZ314xCGlf1IEotE3OY2gcR5FIbE/kmqq+ReYcZ3dn4/gFpW4WBbqEZupwMEjdtMHcmnILzFTfFHckLtkPp040BiBaDmX2tZ35TpF5lAiSi4jsvToWmh/bj5btHuyY45V2I5GWTXn1D7G8AmBhtuTqKy5Ic+pxiAYXE1/MIsn2+ITbv3cQB8q55hKiCIErqFmo3DRGtiuQvUmY3ROxrEzxFLUld47ABy5cwGksGM1ycvC9JZWewWjXJazSYE9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IYCPrawLNUc9scAT0jTmKKrcnbSNfX77Vn64YqoJm8E=; b=lEQ10UEx9XUKV8jbXPGG7QVRv/SHrFmShTLYtjl0YoSmNY8Accwa0FQO+trHWCw1GZgJy3cJ21YE0IxZXBcqc1/xmZ4lUgNwGggwrHn+wwLlmbw3+DglyhNO6BD+BVIURBTu+we/n+IDhitqSLZAyMSkFvEc+arXl8aBjcTEVzY8MZm+aq5xeAHqEJb7Sg0eq4Xukjbw8zW226kLveG0v8QLkaqd26zSTiz/RKRGyybiUugOeJUdFHwXDRldu56CLLTMs90gatbQ010sm/69+lKeIlwfkKqrdebyvt1BKHatPJ1mWQ8taEMmIbg87eZ0KwfGSgLH1d+cgTy7XKjhzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=temperror (sender ip is 12.22.5.238) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=temperror action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IYCPrawLNUc9scAT0jTmKKrcnbSNfX77Vn64YqoJm8E=; b=k9de7KTTURBGhLNfsKoCqlE82RocGVE9sZ88YDpxl4hZMBljCd4cZfaSGfj4ymm8f4LkXGBe3jGot0y7Je3np72PadCd1IWkOpjCNhla2iaxgehwV6lDvRF36VNwo0j3qaLBbTPUQ1zx6gs++gSt9KdY5C1yEjqKqXezj/f/yirvLsWjcqlI817WC/WURAvgj6znDSsiT4xXaJL8htsGFhtdhfafOrnsXx77bElIqn7p1HVF8Mv/unZ8AkQAbzWQ7MueLggtY98mjorzsSyCg7zN7jL1SbhlDpK9XDMN6JnakIYC9D6nkTXIgReb4//V6+MoolRc+eEMJDsXQkzOjQ== Received: from BN1PR12CA0011.namprd12.prod.outlook.com (2603:10b6:408:e1::16) by BYAPR12MB3269.namprd12.prod.outlook.com (2603:10b6:a03:12f::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.21; Thu, 24 Feb 2022 15:51:56 +0000 Received: from BN8NAM11FT026.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e1:cafe::d) by BN1PR12CA0011.outlook.office365.com (2603:10b6:408:e1::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.21 via Frontend Transport; Thu, 24 Feb 2022 15:51:55 +0000 X-MS-Exchange-Authentication-Results: spf=temperror (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=temperror action=none header.from=nvidia.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of nvidia.com: DNS Timeout) Received: from mail.nvidia.com (12.22.5.238) by BN8NAM11FT026.mail.protection.outlook.com (10.13.177.51) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.24 via Frontend Transport; Thu, 24 Feb 2022 15:51:54 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Feb 2022 15:51:49 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Thu, 24 Feb 2022 07:51:46 -0800 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" Subject: [PATCH v2 7/7] vdpa/mlx5: make statistics counter persistent Date: Thu, 24 Feb 2022 23:51:01 +0800 Message-ID: <20220224155101.1991626-8-xuemingl@nvidia.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220224155101.1991626-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220224155101.1991626-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 18bb6cab-5e5d-498d-ff83-08d9f7ad948c X-MS-TrafficTypeDiagnostic: BYAPR12MB3269:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 576fvQAQSsy2ZEG7hIl3A3Uhq5m1zuU7BsuYvClIQJ/N6lAxmaKOiMBsrhNQDxzdZzZvu3UVm9Q8Ia9/5L0pSQ6LWjkquf3BbdYF1Gpcg1nvCX0M9FgveNHCuNjvTr/1OOwHZUyuMrmYA0LeW+L6AdhTP3yg3BHVp3JlJt3VZULRBpNNbsiaUN+9lfDNjnWzM4zNBlSARGH5D9R7ro9nt2Fg/o61PnKcIbEDQb4BW2TDFP0hQ9IAaONW+p1khVX6JhfmlyiitEH0IqP13ehYmoYTLpIvoqiRjSpdVOkPjX78VnQAxLJOvWZ1JEqIp42T9lsVzr31WasuWZ02RxDcXQNX29z3deUrCGY9pLQqJC/UO4VP125zAgTCa3v0/4pSfGNcHu5+aSwiszPraVczViBp2AHDUU4jxbBNurHMEnqu7QKDtOAFDS89ILOITdzhWUPeuxsAKLbeU1mhBdRYobPWk3KQh2VPOiH/0icDZrWfvujCv3il1fJ56liNgomFs5fNK/MZOYlrPxf4uerrQYw48E09veIXK1D0mcCVO7QfMTatZPJVyD8q6gqsmN0YODlTLpAZH3I/rCvc0gN5yIrcd9/5vhq4R6/WC1HhegP3kq7qzmuTbhyN3iN+vX3d1P6QSmJU8l+DDxK9UbCPGi8+C4wImGoz5Fzv3SryEejTGbn5Zpf5hLr089Ljk6qfCklOBtB8pTbi1x4bcMVZkg== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(336012)(16526019)(2616005)(47076005)(36860700001)(6286002)(426003)(86362001)(26005)(1076003)(107886003)(356005)(36756003)(83380400001)(63370400001)(186003)(63350400001)(81166007)(508600001)(8676002)(54906003)(6666004)(6916009)(2906002)(316002)(55016003)(70586007)(4326008)(5660300002)(8936002)(70206006)(7696005)(82310400004)(40460700003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 15:51:54.0182 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 18bb6cab-5e5d-498d-ff83-08d9f7ad948c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT026.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB3269 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org To speed the device suspend and resume time, make counter persitent in reconfiguration until the device gets removed. Signed-off-by: Xueming Li Reviewed-by: Maxime Coquelin --- doc/guides/vdpadevs/mlx5.rst | 6 ++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 19 +++++++---------- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 32 +++++++++++------------------ 4 files changed, 26 insertions(+), 32 deletions(-) diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst index 30f0b62eb41..070208d3952 100644 --- a/doc/guides/vdpadevs/mlx5.rst +++ b/doc/guides/vdpadevs/mlx5.rst @@ -182,3 +182,9 @@ Upon potential hardware errors, mlx5 PMD try to recover, give up if failed 3 times in 3 seconds, virtq will be put in disable state. User should check log to get error information, or query vdpa statistics counter to know error type and count report. + +Statistics +^^^^^^^^^^ + +The device statistics counter persists in reconfiguration until the device gets +removed. User can reset counters by calling function rte_vdpa_reset_stats(). diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 47874c9b1ff..c695f176c9d 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -388,12 +388,7 @@ mlx5_vdpa_get_stats(struct rte_vdpa_device *vdev, int qid, DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (priv->state == MLX5_VDPA_STATE_PROBED) { - DRV_LOG(ERR, "Device %s was not configured.", - vdev->device->name); - return -ENODATA; - } - if (qid >= (int)priv->nr_virtqs) { + if (qid >= (int)priv->caps.max_num_virtio_queues * 2) { DRV_LOG(ERR, "Too big vring id: %d for device %s.", qid, vdev->device->name); return -E2BIG; @@ -416,12 +411,7 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (priv->state == MLX5_VDPA_STATE_PROBED) { - DRV_LOG(ERR, "Device %s was not configured.", - vdev->device->name); - return -ENODATA; - } - if (qid >= (int)priv->nr_virtqs) { + if (qid >= (int)priv->caps.max_num_virtio_queues * 2) { DRV_LOG(ERR, "Too big vring id: %d for device %s.", qid, vdev->device->name); return -E2BIG; @@ -691,6 +681,11 @@ mlx5_vdpa_release_dev_resources(struct mlx5_vdpa_priv *priv) uint32_t i; mlx5_vdpa_dev_cache_clean(priv); + for (i = 0; i < priv->caps.max_num_virtio_queues * 2; i++) { + if (!priv->virtqs[i].counters) + continue; + claim_zero(mlx5_devx_cmd_destroy(priv->virtqs[i].counters)); + } mlx5_vdpa_event_qp_global_release(priv); mlx5_vdpa_err_event_unset(priv); if (priv->steer.tbl) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 24bafe85b44..e7f3319f896 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -92,6 +92,7 @@ struct mlx5_vdpa_virtq { struct rte_intr_handle *intr_handle; uint64_t err_time[3]; /* RDTSC time of recent errors. */ uint32_t n_retry; + struct mlx5_devx_virtio_q_couners_attr stats; struct mlx5_devx_virtio_q_couners_attr reset; }; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index c42846ecb3c..d2c91b25db1 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -127,14 +127,9 @@ void mlx5_vdpa_virtqs_release(struct mlx5_vdpa_priv *priv) { int i; - struct mlx5_vdpa_virtq *virtq; - for (i = 0; i < priv->nr_virtqs; i++) { - virtq = &priv->virtqs[i]; - mlx5_vdpa_virtq_unset(virtq); - if (virtq->counters) - claim_zero(mlx5_devx_cmd_destroy(virtq->counters)); - } + for (i = 0; i < priv->nr_virtqs; i++) + mlx5_vdpa_virtq_unset(&priv->virtqs[i]); priv->features = 0; priv->nr_virtqs = 0; } @@ -590,7 +585,7 @@ mlx5_vdpa_virtq_stats_get(struct mlx5_vdpa_priv *priv, int qid, struct rte_vdpa_stat *stats, unsigned int n) { struct mlx5_vdpa_virtq *virtq = &priv->virtqs[qid]; - struct mlx5_devx_virtio_q_couners_attr attr = {0}; + struct mlx5_devx_virtio_q_couners_attr *attr = &virtq->stats; int ret; if (!virtq->counters) { @@ -598,7 +593,7 @@ mlx5_vdpa_virtq_stats_get(struct mlx5_vdpa_priv *priv, int qid, "is invalid.", qid); return -EINVAL; } - ret = mlx5_devx_cmd_query_virtio_q_counters(virtq->counters, &attr); + ret = mlx5_devx_cmd_query_virtio_q_counters(virtq->counters, attr); if (ret) { DRV_LOG(ERR, "Failed to read virtq %d stats from HW.", qid); return ret; @@ -608,37 +603,37 @@ mlx5_vdpa_virtq_stats_get(struct mlx5_vdpa_priv *priv, int qid, return ret; stats[MLX5_VDPA_STATS_RECEIVED_DESCRIPTORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_RECEIVED_DESCRIPTORS, - .value = attr.received_desc - virtq->reset.received_desc, + .value = attr->received_desc - virtq->reset.received_desc, }; if (ret == MLX5_VDPA_STATS_COMPLETED_DESCRIPTORS) return ret; stats[MLX5_VDPA_STATS_COMPLETED_DESCRIPTORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_COMPLETED_DESCRIPTORS, - .value = attr.completed_desc - virtq->reset.completed_desc, + .value = attr->completed_desc - virtq->reset.completed_desc, }; if (ret == MLX5_VDPA_STATS_BAD_DESCRIPTOR_ERRORS) return ret; stats[MLX5_VDPA_STATS_BAD_DESCRIPTOR_ERRORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_BAD_DESCRIPTOR_ERRORS, - .value = attr.bad_desc_errors - virtq->reset.bad_desc_errors, + .value = attr->bad_desc_errors - virtq->reset.bad_desc_errors, }; if (ret == MLX5_VDPA_STATS_EXCEED_MAX_CHAIN) return ret; stats[MLX5_VDPA_STATS_EXCEED_MAX_CHAIN] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_EXCEED_MAX_CHAIN, - .value = attr.exceed_max_chain - virtq->reset.exceed_max_chain, + .value = attr->exceed_max_chain - virtq->reset.exceed_max_chain, }; if (ret == MLX5_VDPA_STATS_INVALID_BUFFER) return ret; stats[MLX5_VDPA_STATS_INVALID_BUFFER] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_INVALID_BUFFER, - .value = attr.invalid_buffer - virtq->reset.invalid_buffer, + .value = attr->invalid_buffer - virtq->reset.invalid_buffer, }; if (ret == MLX5_VDPA_STATS_COMPLETION_ERRORS) return ret; stats[MLX5_VDPA_STATS_COMPLETION_ERRORS] = (struct rte_vdpa_stat) { .id = MLX5_VDPA_STATS_COMPLETION_ERRORS, - .value = attr.error_cqes - virtq->reset.error_cqes, + .value = attr->error_cqes - virtq->reset.error_cqes, }; return ret; } @@ -649,11 +644,8 @@ mlx5_vdpa_virtq_stats_reset(struct mlx5_vdpa_priv *priv, int qid) struct mlx5_vdpa_virtq *virtq = &priv->virtqs[qid]; int ret; - if (!virtq->counters) { - DRV_LOG(ERR, "Failed to read virtq %d statistics - virtq " - "is invalid.", qid); - return -EINVAL; - } + if (virtq->counters == NULL) /* VQ not enabled. */ + return 0; ret = mlx5_devx_cmd_query_virtio_q_counters(virtq->counters, &virtq->reset); if (ret)