From patchwork Thu Feb 24 14:38:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 108291 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 080F0A034E; Thu, 24 Feb 2022 15:38:50 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9EC7842726; Thu, 24 Feb 2022 15:38:35 +0100 (CET) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2044.outbound.protection.outlook.com [40.107.237.44]) by mails.dpdk.org (Postfix) with ESMTP id 4EB844270D for ; Thu, 24 Feb 2022 15:38:33 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ma27lAF3vo1hZIr2HFtqifiVf76hrBAW5DUvXdRJgfAepD3kx+OqKwtgdy/VHGItMEbO70G29OXQKA3f/NpfqTMqzy0j0ydF+CT7bNS1/NkVCFO+fU/TcpwNa0Jvh5Q+SkswBaWDkykXVpSG/6Ou4LzrW3M6Y9hTyNZ8vbLvoeQi7ypekNvSNLMUcvn6tQjWwi1COfPEQoYX5h0a4VV9N2IBB0aS5fjihkqfuanz9lB0OuGEnESSXUmM6NmMioX6VS5mgVCEXFOIIWEBUjYznAa/uGMvd3pPS/Il6mkha81XLJrytUkCxzCxdSEhr5G9lnFJMaUF0h74GcmsD/M6kA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1XIJVSeYObyHggMrA4p8UzsV4f9rmy+23mKlN8dnoiM=; b=dD79VoQmt22okEleVQXGS4KCJHws26tBm7GDGHzphOT+YJo6gz7Q1j2wGYj/GAio9N/DwAPchjEb56UEKl6tT6ci8FUPgT7mOPn62Z+TJznzS1eAJeVVj0MmWt/vapurjBIb2Xs+7KP6VHvlkvOH4DuUddOiKAFR1USVVp9iOrynZKmKxHjicnItufjcYQ6nn8msdB2JFxgLzDJPCmqiDvpl8dM1Qup5QOjQFJ9t8ja8YBEx8Rm1j+jfvwOLSOrOfj0YGGPuIdtgyA3blriuK3Eb1pLHwj3p3QxZAcv7C8ZUlprdeel/mvPfzrf0rvDL9FAIkCu/cytZvlKBT7uM+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1XIJVSeYObyHggMrA4p8UzsV4f9rmy+23mKlN8dnoiM=; b=eC58Z9RWA6mbcqQu0P4TZmMrpLX6mkWWaB8CsbKac2VZi6kD79d/0L2e7nmVkazoVNRfag01SJbGOOMTbVExrfxbmESEn8R9dpQKqS20WUOKmwoToaMpugWoK/p+YiPCvAQ953nWGesITCM+ryRudAJ8qUZZgcGkg/cyvnqTYMnYmsoZCMKqpqC0PIjFvyznB3AG1mboCQY111xb9ajsRutOg/wQIZ7pyqQIWJP4EfUYUZxJVQjxK7MLTl4UBWTGaNIAaBwqeyYLZsO2mGilkyRWxkE8ba4u7dRYtco865asXvW5DtgJbovy/pjSoUfLbWBRsYixb052AzKRuuaq7w== Received: from MW4PR04CA0333.namprd04.prod.outlook.com (2603:10b6:303:8a::8) by MN2PR12MB3631.namprd12.prod.outlook.com (2603:10b6:208:c2::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.22; Thu, 24 Feb 2022 14:38:29 +0000 Received: from CO1NAM11FT055.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8a:cafe::db) by MW4PR04CA0333.outlook.office365.com (2603:10b6:303:8a::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.23 via Frontend Transport; Thu, 24 Feb 2022 14:38:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT055.mail.protection.outlook.com (10.13.175.129) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Thu, 24 Feb 2022 14:38:28 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Feb 2022 14:38:27 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Thu, 24 Feb 2022 06:38:24 -0800 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" Subject: [PATCH v1 3/7] vdpa/mlx5: no kick handling during shutdown Date: Thu, 24 Feb 2022 22:38:05 +0800 Message-ID: <20220224143809.1977642-4-xuemingl@nvidia.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220224143809.1977642-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220224143809.1977642-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f547d95-5834-4bda-fa45-08d9f7a352c3 X-MS-TrafficTypeDiagnostic: MN2PR12MB3631:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1XMmLgeN6lIZJiZgeZ7uOnnTGOaRk+c0KBf4g8iuCWeOnMlONHdvd1E8BGB1nkuyLgO7Nw9b/ZmO/88ds41qaNPnPjzoM0cR13UOPoIzz0p+AF7hdIwUU2kzZFvaoKqbJNqhVeq15cMDdwb5slby0zfoAvYIFVrB9wkC57NaO3mBB9OcknBuUeWqUCGfy+ZnWdwH70H7TlcEELsCHlCLY5W1r72MTuRPlgp7WL9UDL6cx1dYJv9k06Pucxpqab9/yHvOL1ZZmRmDv2v+aRsQZ90r20aFCp0aTiBPb1V158AUDmoUKkI4GgiYs6fiC3iREUsayguCV2dT0owRhxi9O4waGpoYbOcpDsn9qWHrZNgGaQihCuDgLuzyvXVWkCCpEafZtw8saUxQCynKhNxQ2gHrFYSpTAT1XWxKWL5BuXxrWFib0ZVK6W0nG1NOokrtjEGV62ZaNGkSdZkgNx9UXKucLWjthxARpIg1ajaRn1DVP4Mo3aJ2C19XRXO4bDT6cprMX2OjyIkNm2xJN3ehvR9UoTjjk2uJNc6TbVP6X/Sw0QVKCmBYPv0EuTNJ5BKvkUckJVBCzrDr784qN2MP6zU2b6BIIgB6ZTpbHG1yQdtMF+abZdBL82E1yP09oF9X9cAJwO6As70J9AbAtGkPZXkNsY+cjXJOCLFRd8dTdYRwbhFibjONf0v4B1d9NjN2zTfTgjPq2i63/HftJXOnFfi5uSl6V0DCyBlbY1HV348ggNmQLSowF7VsUxXe4Ido X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(36756003)(1076003)(7696005)(47076005)(426003)(336012)(107886003)(2616005)(16526019)(26005)(186003)(6286002)(8936002)(6666004)(36860700001)(6916009)(5660300002)(508600001)(81166007)(70586007)(70206006)(8676002)(356005)(4326008)(54906003)(83380400001)(2906002)(86362001)(55016003)(40460700003)(82310400004)(316002)(36900700001)(309714004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 14:38:28.9003 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8f547d95-5834-4bda-fa45-08d9f7a352c3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT055.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3631 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When Qemu suspend a VM, hw notifier is un-mmapped while vCPU thread may still active and write notifier through kick socket. PMD kick handler thread tries to install hw notifier through client socket in such case will timeout and slow down device close. This patch skips hw notifier install if VQ or device in middle of shutdown. Signed-off-by: Xueming Li --- drivers/vdpa/mlx5/mlx5_vdpa.c | 17 ++++++++++------- drivers/vdpa/mlx5/mlx5_vdpa.h | 8 +++++++- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 12 +++++++++++- 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 8dfaba791dc..a93a9e78f7f 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -252,13 +252,15 @@ mlx5_vdpa_dev_close(int vid) } mlx5_vdpa_err_event_unset(priv); mlx5_vdpa_cqe_event_unset(priv); - if (priv->configured) + if (priv->state == MLX5_VDPA_STATE_CONFIGURED) { ret |= mlx5_vdpa_lm_log(priv); + priv->state = MLX5_VDPA_STATE_IN_PROGRESS; + } mlx5_vdpa_steer_unset(priv); mlx5_vdpa_virtqs_release(priv); mlx5_vdpa_event_qp_global_release(priv); mlx5_vdpa_mem_dereg(priv); - priv->configured = 0; + priv->state = MLX5_VDPA_STATE_PROBED; priv->vid = 0; /* The mutex may stay locked after event thread cancel - initiate it. */ pthread_mutex_init(&priv->vq_config_lock, NULL); @@ -277,7 +279,8 @@ mlx5_vdpa_dev_config(int vid) DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name); return -EINVAL; } - if (priv->configured && mlx5_vdpa_dev_close(vid)) { + if (priv->state == MLX5_VDPA_STATE_CONFIGURED && + mlx5_vdpa_dev_close(vid)) { DRV_LOG(ERR, "Failed to reconfigure vid %d.", vid); return -1; } @@ -291,7 +294,7 @@ mlx5_vdpa_dev_config(int vid) mlx5_vdpa_dev_close(vid); return -1; } - priv->configured = 1; + priv->state = MLX5_VDPA_STATE_CONFIGURED; DRV_LOG(INFO, "vDPA device %d was configured.", vid); return 0; } @@ -373,7 +376,7 @@ mlx5_vdpa_get_stats(struct rte_vdpa_device *vdev, int qid, DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (!priv->configured) { + if (priv->state == MLX5_VDPA_STATE_PROBED) { DRV_LOG(ERR, "Device %s was not configured.", vdev->device->name); return -ENODATA; @@ -401,7 +404,7 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (!priv->configured) { + if (priv->state == MLX5_VDPA_STATE_PROBED) { DRV_LOG(ERR, "Device %s was not configured.", vdev->device->name); return -ENODATA; @@ -590,7 +593,7 @@ mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev) TAILQ_REMOVE(&priv_list, priv, next); pthread_mutex_unlock(&priv_list_lock); if (found) { - if (priv->configured) + if (priv->state == MLX5_VDPA_STATE_CONFIGURED) mlx5_vdpa_dev_close(priv->vid); if (priv->var) { mlx5_glue->dv_free_var(priv->var); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 22617924eac..cc83d7cba3d 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -113,9 +113,15 @@ enum { MLX5_VDPA_EVENT_MODE_ONLY_INTERRUPT }; +enum mlx5_dev_state { + MLX5_VDPA_STATE_PROBED = 0, + MLX5_VDPA_STATE_CONFIGURED, + MLX5_VDPA_STATE_IN_PROGRESS /* Shutting down. */ +}; + struct mlx5_vdpa_priv { TAILQ_ENTRY(mlx5_vdpa_priv) next; - uint8_t configured; + enum mlx5_dev_state state; pthread_mutex_t vq_config_lock; uint64_t no_traffic_counter; pthread_t timer_tid; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index e1e05924a40..b1d584ca8b0 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -25,6 +25,11 @@ mlx5_vdpa_virtq_kick_handler(void *cb_arg) int nbytes; int retry; + if (priv->state != MLX5_VDPA_STATE_CONFIGURED && !virtq->enable) { + DRV_LOG(ERR, "device %d queue %d down, skip kick handling", + priv->vid, virtq->index); + return; + } if (rte_intr_fd_get(virtq->intr_handle) < 0) return; for (retry = 0; retry < 3; ++retry) { @@ -43,6 +48,11 @@ mlx5_vdpa_virtq_kick_handler(void *cb_arg) if (nbytes < 0) return; rte_write32(virtq->index, priv->virtq_db_addr); + if (priv->state != MLX5_VDPA_STATE_CONFIGURED && !virtq->enable) { + DRV_LOG(ERR, "device %d queue %d down, skip kick handling", + priv->vid, virtq->index); + return; + } if (virtq->notifier_state == MLX5_VDPA_NOTIFIER_STATE_DISABLED) { if (rte_vhost_host_notifier_ctrl(priv->vid, virtq->index, true)) virtq->notifier_state = MLX5_VDPA_NOTIFIER_STATE_ERR; @@ -541,7 +551,7 @@ mlx5_vdpa_virtq_enable(struct mlx5_vdpa_priv *priv, int index, int enable) DRV_LOG(INFO, "Update virtq %d status %sable -> %sable.", index, virtq->enable ? "en" : "dis", enable ? "en" : "dis"); - if (!priv->configured) { + if (priv->state == MLX5_VDPA_STATE_PROBED) { virtq->enable = !!enable; return 0; }