From patchwork Thu Feb 24 15:50:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 108305 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A0B5BA034C; Thu, 24 Feb 2022 16:51:47 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5453442733; Thu, 24 Feb 2022 16:51:34 +0100 (CET) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2055.outbound.protection.outlook.com [40.107.237.55]) by mails.dpdk.org (Postfix) with ESMTP id 5C71942712 for ; Thu, 24 Feb 2022 16:51:32 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Al3k01VgHQDtq955Ai8m3MP9F3oxmV86YsXcxe0mLlqlPsDuIbbn8M4JJ+R/Q+W3IhtG+e0zjMhD+tqFBgz38el/WKEie8zK946pzO51WpDZ2KJJtnBjDH/eid6LFhOK3DCSLXUv8Dn78yY12167froavIZLCY/Ddt5wm7z7syr9EUJVhDi3kf61rtTHSq3ngMF2trOmhlbIbigFqXUJkmnm2EM+5UIfD0+Ht0xZXzSn8ODYiyGEi23Qgi5XTyLbznGVGLuZZkXqUaODy/n21Zkqwm7LWzQFS0tjZd6fYvuE1tUChR5XnpDwU4L12t8gpaJZkgNwUYyyJgEAi/L4Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1XIJVSeYObyHggMrA4p8UzsV4f9rmy+23mKlN8dnoiM=; b=Ut3sjirUl++nwWIeNvVF47FjJvpFBCYQvY1bXfyD9/C98Kkzmwzyk+xKQQpeWfZhCbl9c+18V8/1nAIdi2+pJPgyBq3FLmoutipr2gxDiqiE0LUwjXME8NZdGqtsIB0shHYwUdOk/Tn1xXyUmAt+xpnwgeZJnEgM5tnQB7SixHIjlFWrh4ASmazAr2JkHptnqsnahErs/f6Ki+bjxsTNWUQWdG73QLK0XNFoctgG+cOy+/c/xZBJDPisy3LZkZmqQ/cNs+1u0u93Y9dOU87CIwFgeaBQGp9oyoGwEXUKWLk6mZRHbnx5v/ttdrQoAvejoED2hAtpjS9YAyb1qfEyZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1XIJVSeYObyHggMrA4p8UzsV4f9rmy+23mKlN8dnoiM=; b=XZieU1oy+teTY682xR7fM6k4ha418KKkzjGHk4rs/GGo8uGvBNLeUOtDY7yZEtzETZ5b2EFePBxvRzBY8gL1wjpf3w2YBxWnnmgIdnmTTnZjNFLQDbxsfqaJoZZD6drGebEkdqj97PZ1R3D7alUk9srvHxxCMkRVvM6JyFijQ7f2F75ogSCdUmwzAl9iIis00fCCUjLvJ17b3la7cyc5XE+Vu5+84EOrE0KMTALZ+U1xcHlY1Y7bqLGw+m4GVOG17MPMZT4+3y83PVqqEG8pEh3B5DWbBhKg11lV3aPKpoo5NRQuc2AeQsAqgzy3HXW9M45pqQDxQzRlhGGEK6eY+g== Received: from MWHPR02CA0023.namprd02.prod.outlook.com (2603:10b6:300:4b::33) by CY4PR1201MB0264.namprd12.prod.outlook.com (2603:10b6:910:1e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.22; Thu, 24 Feb 2022 15:51:25 +0000 Received: from CO1NAM11FT014.eop-nam11.prod.protection.outlook.com (2603:10b6:300:4b:cafe::53) by MWHPR02CA0023.outlook.office365.com (2603:10b6:300:4b::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.21 via Frontend Transport; Thu, 24 Feb 2022 15:51:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT014.mail.protection.outlook.com (10.13.175.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Thu, 24 Feb 2022 15:51:24 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Feb 2022 15:51:22 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Thu, 24 Feb 2022 07:51:20 -0800 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" Subject: [PATCH v2 3/7] vdpa/mlx5: no kick handling during shutdown Date: Thu, 24 Feb 2022 23:50:57 +0800 Message-ID: <20220224155101.1991626-4-xuemingl@nvidia.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220224155101.1991626-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220224155101.1991626-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b1faf5ed-38a1-4ea2-32f6-08d9f7ad8300 X-MS-TrafficTypeDiagnostic: CY4PR1201MB0264:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YnEF35EX9C9GAOkCQ9ygCVX+9ULiV+4JnFLeO1As8YQ5baH/L8fp8gPXkpicV26xcYaYtFnU1vD3hB1ZJTBzNW1M69BMzkOV7VE47kny3PjvWfln+8804R2teQHEfdCsOcoUlkKYeK/Zdo96iTFP3I8Qkkw+p0+H0tmApf1WVURNlIropD2z4MB15CIcQllKAZzuoITMzVPW+MORaHralxuOIUA6KWwEohHPIpVJ2GD5fqchMbur40pojlGDmSDKnKHDgK771aPfjtVFv21dSsVaYFJ1L1SM0bcUO4wDIisMFVo1owVBSgacFnlTha4LHReCKGjdsaTHHnuhKHFELOV0qLaog4+mFGfhrZEK55XB6veWyo4N1IUoiaalw9qJdv0I2J8CeHkSE6H+2UOI6FEZ3kt7RF/V2SFjBlSPbbfHciVJMkygy8ZwuG4V00141Fcah8DVWULGfLlROnbzWtOQqu4IQi/15PKIN2fFuwa78bRN4SmYB48mDls5rAZ4F1dLrrBnzOTxYNSGp9Gog709b6hCn1SzT3ykMPBreydBILtsLthaS/zzZBL9Twfmwjqj6WXGLIMuQLbMe+9YlXt28vrRM+yMUCUJglFp2QkIzp5nWE4IXqIKoY5u+foD1bWghQ34HG48cDXlTe9FMGQqT2qG488nO56xyxTZn5D2pWHvNigijT8E6snpxYFX5EtltdtSdM9d+uY42XZFfCXHnsQgiqK6SoydZmJGvpUrr5BbAN1ky2GxJ8bH/ZIG X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(2616005)(6666004)(36756003)(356005)(107886003)(8936002)(40460700003)(83380400001)(508600001)(426003)(336012)(2906002)(186003)(26005)(36860700001)(82310400004)(8676002)(55016003)(16526019)(4326008)(6916009)(81166007)(5660300002)(1076003)(86362001)(316002)(47076005)(54906003)(7696005)(70206006)(70586007)(6286002)(36900700001)(309714004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 15:51:24.7617 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b1faf5ed-38a1-4ea2-32f6-08d9f7ad8300 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT014.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1201MB0264 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When Qemu suspend a VM, hw notifier is un-mmapped while vCPU thread may still active and write notifier through kick socket. PMD kick handler thread tries to install hw notifier through client socket in such case will timeout and slow down device close. This patch skips hw notifier install if VQ or device in middle of shutdown. Signed-off-by: Xueming Li Reviewed-by: Maxime Coquelin --- drivers/vdpa/mlx5/mlx5_vdpa.c | 17 ++++++++++------- drivers/vdpa/mlx5/mlx5_vdpa.h | 8 +++++++- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 12 +++++++++++- 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 8dfaba791dc..a93a9e78f7f 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -252,13 +252,15 @@ mlx5_vdpa_dev_close(int vid) } mlx5_vdpa_err_event_unset(priv); mlx5_vdpa_cqe_event_unset(priv); - if (priv->configured) + if (priv->state == MLX5_VDPA_STATE_CONFIGURED) { ret |= mlx5_vdpa_lm_log(priv); + priv->state = MLX5_VDPA_STATE_IN_PROGRESS; + } mlx5_vdpa_steer_unset(priv); mlx5_vdpa_virtqs_release(priv); mlx5_vdpa_event_qp_global_release(priv); mlx5_vdpa_mem_dereg(priv); - priv->configured = 0; + priv->state = MLX5_VDPA_STATE_PROBED; priv->vid = 0; /* The mutex may stay locked after event thread cancel - initiate it. */ pthread_mutex_init(&priv->vq_config_lock, NULL); @@ -277,7 +279,8 @@ mlx5_vdpa_dev_config(int vid) DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name); return -EINVAL; } - if (priv->configured && mlx5_vdpa_dev_close(vid)) { + if (priv->state == MLX5_VDPA_STATE_CONFIGURED && + mlx5_vdpa_dev_close(vid)) { DRV_LOG(ERR, "Failed to reconfigure vid %d.", vid); return -1; } @@ -291,7 +294,7 @@ mlx5_vdpa_dev_config(int vid) mlx5_vdpa_dev_close(vid); return -1; } - priv->configured = 1; + priv->state = MLX5_VDPA_STATE_CONFIGURED; DRV_LOG(INFO, "vDPA device %d was configured.", vid); return 0; } @@ -373,7 +376,7 @@ mlx5_vdpa_get_stats(struct rte_vdpa_device *vdev, int qid, DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (!priv->configured) { + if (priv->state == MLX5_VDPA_STATE_PROBED) { DRV_LOG(ERR, "Device %s was not configured.", vdev->device->name); return -ENODATA; @@ -401,7 +404,7 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) DRV_LOG(ERR, "Invalid device: %s.", vdev->device->name); return -ENODEV; } - if (!priv->configured) { + if (priv->state == MLX5_VDPA_STATE_PROBED) { DRV_LOG(ERR, "Device %s was not configured.", vdev->device->name); return -ENODATA; @@ -590,7 +593,7 @@ mlx5_vdpa_dev_remove(struct mlx5_common_device *cdev) TAILQ_REMOVE(&priv_list, priv, next); pthread_mutex_unlock(&priv_list_lock); if (found) { - if (priv->configured) + if (priv->state == MLX5_VDPA_STATE_CONFIGURED) mlx5_vdpa_dev_close(priv->vid); if (priv->var) { mlx5_glue->dv_free_var(priv->var); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 22617924eac..cc83d7cba3d 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -113,9 +113,15 @@ enum { MLX5_VDPA_EVENT_MODE_ONLY_INTERRUPT }; +enum mlx5_dev_state { + MLX5_VDPA_STATE_PROBED = 0, + MLX5_VDPA_STATE_CONFIGURED, + MLX5_VDPA_STATE_IN_PROGRESS /* Shutting down. */ +}; + struct mlx5_vdpa_priv { TAILQ_ENTRY(mlx5_vdpa_priv) next; - uint8_t configured; + enum mlx5_dev_state state; pthread_mutex_t vq_config_lock; uint64_t no_traffic_counter; pthread_t timer_tid; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index e1e05924a40..b1d584ca8b0 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -25,6 +25,11 @@ mlx5_vdpa_virtq_kick_handler(void *cb_arg) int nbytes; int retry; + if (priv->state != MLX5_VDPA_STATE_CONFIGURED && !virtq->enable) { + DRV_LOG(ERR, "device %d queue %d down, skip kick handling", + priv->vid, virtq->index); + return; + } if (rte_intr_fd_get(virtq->intr_handle) < 0) return; for (retry = 0; retry < 3; ++retry) { @@ -43,6 +48,11 @@ mlx5_vdpa_virtq_kick_handler(void *cb_arg) if (nbytes < 0) return; rte_write32(virtq->index, priv->virtq_db_addr); + if (priv->state != MLX5_VDPA_STATE_CONFIGURED && !virtq->enable) { + DRV_LOG(ERR, "device %d queue %d down, skip kick handling", + priv->vid, virtq->index); + return; + } if (virtq->notifier_state == MLX5_VDPA_NOTIFIER_STATE_DISABLED) { if (rte_vhost_host_notifier_ctrl(priv->vid, virtq->index, true)) virtq->notifier_state = MLX5_VDPA_NOTIFIER_STATE_ERR; @@ -541,7 +551,7 @@ mlx5_vdpa_virtq_enable(struct mlx5_vdpa_priv *priv, int index, int enable) DRV_LOG(INFO, "Update virtq %d status %sable -> %sable.", index, virtq->enable ? "en" : "dis", enable ? "en" : "dis"); - if (!priv->configured) { + if (priv->state == MLX5_VDPA_STATE_PROBED) { virtq->enable = !!enable; return 0; }