From patchwork Sun May 8 14:25:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 110905 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99B45A0510; Sun, 8 May 2022 16:27:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 68D2C42834; Sun, 8 May 2022 16:26:49 +0200 (CEST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam07on2059.outbound.protection.outlook.com [40.107.212.59]) by mails.dpdk.org (Postfix) with ESMTP id ECA1442833 for ; Sun, 8 May 2022 16:26:47 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KtXEqnRseWdt7qs9ILLwH7A8G9bJtl9VZa983DZ8VZEW5LsyQnSfiiK7nonEDRsBfY5FzBskQe4ixY4NG6YU7o1kkIRZoaROKb1apoklJrh5PUf0fQPE8nUEQFiZHZhsWUinDPr07TV9my7XVZLNhLWZfBmHtIXkLWPHZH9sp4CQPCJ8gISZ/AuD7/yJ85xELib/ucKB6keiRJ9mbwl/YrD02q7XpA5xdxHk7la20Y8OgvpR5yw+tWX76L1HAhMoB15r7x24VfWImJ/+YGfOVzpMX7jDNf4w/5u2CmjgIlEVcTLfCYBvVMLEJGmJSoVBHxnijdBp3+0B3R4nQAnN5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TDM3X/u3ChT1Bj5mSzLQNzgP8gZDYpMpt15gLrdaO/U=; b=Vb7u2NgzCRDDDjZky5F0GCxGqYgP8cHf3/SXDKbAjoQPKSHDiDZnsbQgLL0sfSK3XBb0GzztGL5Q9SLgcbDGQDq0/YM5ZcmXND8VoMy6sDUHnPco5T+nXkL9YXbasdS25AFrVKnjtPoIXWzhnDz1XLg3QGaRxnasqauKI3maOSvgJPrGEugFsPGVdOPihEHHLSc++QoHpg/YcGVKphHCu1NyvMZdIYiwNqINY+IWsiUhDdBIv83EEk/BNiYlpW5Wy6vuV0QSjOmif7d8C9sOuJDmEiUJjJH4EoXOK4fETCSDJTxDdgxeSAdKNTwcr/CJfhUMy0rlctbln2oEbVMFCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TDM3X/u3ChT1Bj5mSzLQNzgP8gZDYpMpt15gLrdaO/U=; b=HhMBupzh2rrnNVAePMPVIuMz13oWCDbJRLpT0Cotn84uio3VkNSq+JMrstqplj1fF8jUrWefRDQQMEEqI+I0AlTLKw2KJ5vo1y8jaznH3OM8Tsqi85JvIyNPIJl/GI2dxu3mPk6tw77opRVPn5vP7hYq5d6neZ1nozlg9xznq+7lXoHJbFW6bcGkM2v95j0fNNx8xEeQi5qGo4HLQnwhHFwUDpkrwKtf5O7eqsIqdSweK+G+QvQ52OneTvrtNl7dAGJE0fj8QySx1AsFrqUd7PX8Lz72TczjZZJaJdvMyWwOkwgyQjDmu5xsjm78ExkwFOnvsWKw+IAJcAxOZo7rLA== Received: from DM6PR03CA0097.namprd03.prod.outlook.com (2603:10b6:5:333::30) by BY5PR12MB4164.namprd12.prod.outlook.com (2603:10b6:a03:207::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.18; Sun, 8 May 2022 14:26:46 +0000 Received: from DM6NAM11FT063.eop-nam11.prod.protection.outlook.com (2603:10b6:5:333:cafe::e) by DM6PR03CA0097.outlook.office365.com (2603:10b6:5:333::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.20 via Frontend Transport; Sun, 8 May 2022 14:26:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.236) by DM6NAM11FT063.mail.protection.outlook.com (10.13.172.219) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5227.15 via Frontend Transport; Sun, 8 May 2022 14:26:45 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Sun, 8 May 2022 14:26:45 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Sun, 8 May 2022 07:26:43 -0700 From: Xueming Li To: , Maxime Coquelin CC: Subject: [PATCH v3 6/7] vdpa/mlx5: support device cleanup callback Date: Sun, 8 May 2022 17:25:53 +0300 Message-ID: <20220508142554.560354-7-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220508142554.560354-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220508142554.560354-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bde9b762-3891-449b-16f2-08da30fec7ce X-MS-TrafficTypeDiagnostic: BY5PR12MB4164:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZIlOutgivGhqbOvWfZ33ob/srsbKBGHKwe7sv6tTz0pwITi84XbGxrXhhWZa6sn6/fZiAIfV32PTXPo7JcSWk6GW2j0QToWH7+sHRkjW+m+lQaRDO9t+1r+Muaimb7vjkdIJ+ISTGjYlD9biGCeRgm6Lt5SwQKNocTkLMb2y9vL4yIppEAdJxF07Qd0hGPL3PwMQ4hve1C6REMRSVZIOHEb6khd6e5lYaNgWwmUXgsZVFXFb64EpaCrPcBEssIB9C2Cr8Lak+pEPpN4MtKS8w1qxUEPjCXL69a6MOhtmdiapNJzT6iV0DYtrg181Ak2K6ezyFF2eVaXzF8vCKbddW7rk4AEBd0HmwW+Bh6xO2q5K8KO7m51GEaA09VBPh7v9epS6j/lSce3mk9csoOFAw8wMmkf+/aRaP85eQ9C2P5hRRU1BpS7Q5obRA7ZpX8ZvoyOaXq/j8qtU4f2uiQxqUT8/kP5iDiG1h6vD59+ZFktiEJxIy/Ckkj1xzNg7A/uK5PKitx17rq+8IIsR16Ulp3d7RI5wpVDpoW1uKQtr8NhwJsINGcMV+/xxU73syYtwfG6H8YK7Ck7XGH46MSMa5mKoCnDuOIcrsvZmBOVh9oXYi/8rKXna526UrJ2wQzyHIX5/UdKCs02BbU8fCVjmYf9kZWkxrDhp3v9TQIC8j6llHgJcl8hh7nPEmSBzPWZTPS9FYQQZf+NFpgsQA3BkCQ== X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(1076003)(83380400001)(186003)(16526019)(82310400005)(40460700003)(70586007)(70206006)(316002)(36860700001)(47076005)(336012)(8676002)(426003)(4326008)(356005)(107886003)(55016003)(2906002)(6666004)(7696005)(508600001)(5660300002)(8936002)(2616005)(36756003)(6286002)(86362001)(81166007)(110136005)(26005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2022 14:26:45.6939 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bde9b762-3891-449b-16f2-08da30fec7ce X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT063.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4164 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch supports device cleanup callback API which is called when the device is disconnected from the VM. Cached resources like VM MR and VQ memory are released. Signed-off-by: Xueming Li Reviewed-by: Maxime Coquelin --- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + 2 files changed, 24 insertions(+) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index fb5d9276621..b1d5487080d 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -270,6 +270,8 @@ mlx5_vdpa_dev_close(int vid) if (priv->lm_mr.addr) mlx5_os_wrapped_mkey_destroy(&priv->lm_mr); priv->state = MLX5_VDPA_STATE_PROBED; + if (!priv->connected) + mlx5_vdpa_dev_cache_clean(priv); priv->vid = 0; /* The mutex may stay locked after event thread cancel - initiate it. */ pthread_mutex_init(&priv->vq_config_lock, NULL); @@ -294,6 +296,7 @@ mlx5_vdpa_dev_config(int vid) return -1; } priv->vid = vid; + priv->connected = true; if (mlx5_vdpa_mtu_set(priv)) DRV_LOG(WARNING, "MTU cannot be set on device %s.", vdev->device->name); @@ -431,12 +434,32 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) return mlx5_vdpa_virtq_stats_reset(priv, qid); } +static int +mlx5_vdpa_dev_cleanup(int vid) +{ + struct rte_vdpa_device *vdev = rte_vhost_get_vdpa_device(vid); + struct mlx5_vdpa_priv *priv; + + if (vdev == NULL) + return -1; + priv = mlx5_vdpa_find_priv_resource_by_vdev(vdev); + if (priv == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name); + return -1; + } + if (priv->state == MLX5_VDPA_STATE_PROBED) + mlx5_vdpa_dev_cache_clean(priv); + priv->connected = false; + return 0; +} + static struct rte_vdpa_dev_ops mlx5_vdpa_ops = { .get_queue_num = mlx5_vdpa_get_queue_num, .get_features = mlx5_vdpa_get_vdpa_features, .get_protocol_features = mlx5_vdpa_get_protocol_features, .dev_conf = mlx5_vdpa_dev_config, .dev_close = mlx5_vdpa_dev_close, + .dev_cleanup = mlx5_vdpa_dev_cleanup, .set_vring_state = mlx5_vdpa_set_vring_state, .set_features = mlx5_vdpa_features_set, .migration_done = NULL, diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 540bf87a352..24bafe85b44 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -121,6 +121,7 @@ enum mlx5_dev_state { struct mlx5_vdpa_priv { TAILQ_ENTRY(mlx5_vdpa_priv) next; + bool connected; enum mlx5_dev_state state; pthread_mutex_t vq_config_lock; uint64_t no_traffic_counter;