From patchwork Thu Feb 24 15:51:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueming Li X-Patchwork-Id: 108307 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99A51A034C; Thu, 24 Feb 2022 16:52:02 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A4CE94271E; Thu, 24 Feb 2022 16:51:55 +0100 (CET) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2061.outbound.protection.outlook.com [40.107.237.61]) by mails.dpdk.org (Postfix) with ESMTP id 2BA8C426ED for ; Thu, 24 Feb 2022 16:51:54 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oIFmeYRn2gbYhlFqM/B35FZ8G6uzHiaY6kZgd4HxPEEmGKQy2kCrR+EUQpHj/UtRtHqiKJVKv7hbcKA2PugX00kyyuCU7FBI2sX1YRl6bTxZBdSye9f/XvDkn/c1bMcwz7eZeVJzJgkKRYD8s4kbOWa7TwBAXiFSiEus1AqQCh1LX2Pev6HCRaSA/P/7f86mo91nX4BpYIwz+n01Xx2Vs8/kpsGAd4y3qnn9YMpaI7GivtMLrWNVgXcZobnbKCnC9bA56OiYmygcbA2nCFrW/dNbXNQjYW2vooOR4s/c+ybyBIDOfVV64aroe85ge5TGv/6edq+Zd+brJPk2dNhd1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4mV/sslYekF6m2128md4WPqcDG4JNYtKWDPtINzWlIQ=; b=kjCBiNHJIoPBhDzZQJvrwhvafeZo+i1UrkYueKdlAS5gtav/foY8VoUZf+Yw9MqsFINqgnR2/owMTNoQPdISaIxFrmVbSk4PnVGkySONLEiElEMV4zSoYvXrplgvQE3qUjUzIrz5ApTPHzI15gsbq7+enA/SM0VyUDlfEmKEsAXyYDqaZqJvbIJkFar3PviSGIbJcfVrleltI+SauQEJ2P9FBqpRWsicu8jFEe23lnwdzkEV315K/SwDfz/uWmPHozzJH1+8AQzEBomXxoA3U96gPZRS8FA9Qi8TD8d0nbiaGGUBw8PsDda+JoDubgN/KR67Be5x+0cNBArl6skCnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4mV/sslYekF6m2128md4WPqcDG4JNYtKWDPtINzWlIQ=; b=jAEgWP4bI6gfMqtPFk7GhV3cbaS0J/xudx5ujjOPm6QWjvtGISWzckfy1JE/OUYh7AzvJeOf5UWYKyNVpMpdUMUB81+x/krQn+0P987uZiXFeSaAqxewfhIf+SQRyaBfDHdc95ScrXbITqf+eIFV5DMRLOvjjyptf/o2iuT1uLrAXxneHyu7VaneHFjHnDRoaFZZaiQur0z6R1Z7JTU0Wp7OR4w+taLh42PuLdf5Gt2CRaGv2Dhh+wV1N56kLgn6PGV+WrOl+fReLUC7aHKazOxjwxsf9Rbm63x2guW1sYTQqcuWK00PTfa9rEiBDt/yZMC3jTmk+VyBhqpl35olNA== Received: from MWHPR17CA0070.namprd17.prod.outlook.com (2603:10b6:300:93::32) by BN8PR12MB4595.namprd12.prod.outlook.com (2603:10b6:408:72::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.22; Thu, 24 Feb 2022 15:51:51 +0000 Received: from CO1NAM11FT056.eop-nam11.prod.protection.outlook.com (2603:10b6:300:93:cafe::9b) by MWHPR17CA0070.outlook.office365.com (2603:10b6:300:93::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.24 via Frontend Transport; Thu, 24 Feb 2022 15:51:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT056.mail.protection.outlook.com (10.13.175.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Thu, 24 Feb 2022 15:51:48 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Feb 2022 15:51:47 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Thu, 24 Feb 2022 07:51:44 -0800 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" Subject: [PATCH v2 6/7] vdpa/mlx5: support device cleanup callback Date: Thu, 24 Feb 2022 23:51:00 +0800 Message-ID: <20220224155101.1991626-7-xuemingl@nvidia.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220224155101.1991626-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220224155101.1991626-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 53b3d8f0-3a99-4f95-9b4d-08d9f7ad90f3 X-MS-TrafficTypeDiagnostic: BN8PR12MB4595:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dSVKbgHKrQLFsD+CxNqs0XS5TQE2BIVXd6OgiQYKhC1iKC++YDK5xcooGaiwotbPNZkTQif6SCibx2YU3WtVW2kDfKgn9RI6uBWF2gn7vnFrE+N9juViW1R08xJAyqBKXOm+ysFXfyu57hxkmmfmE/MYvGEhdzj1TA+SDXOxWLU8WMKm+CaMWFLBR4nm47FjEOMORVeOBwbgWqoTgCyxbl+LKS4remKtwat+ideLUCy0blkyDDHesps57THWXz0wd393ODJvBhM4R+vYxyCCx3z/pvc0Fq7MX8MhxPC22ws9WRYgF62b00opYdTg28a5LH7W9oQbTdJ/eQ1M5b+F5CLilc0212IIc9TA9VFg+w6OFd0bjyGLg1tabH8sSw9JHF+aM9GsFFGjh4jXA0LU5Hsum/ytwpvSBnbI+YkIOw32Qsfw0iMG/0lMYBlP0W/VqQO25Mn8pr9lSp82ZrGiS+X9UknJLfOaoqDtESh09XI+qssIb2i10aqKdeE4MP9JQLLkfISYvKMoqBzz7vFnqhLMr6ZNpJy+du/jAefW/yr4ud8zT76yYSf24HSbgqeaybfPkj0HuRt7rKreKEJnp5xKY1JMThJdOB65rh4+JP0Y7tiq92f0eI83mhhMamz3f360Zihh1XrWEa+elKizv9v+0d+7a4JtgjlGPrQJxddzRy3v6qcAueAcH2dsjjkZhRAdueH6cMJul98LhcpylQ== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(82310400004)(2616005)(36860700001)(26005)(426003)(186003)(508600001)(81166007)(6666004)(55016003)(7696005)(86362001)(107886003)(47076005)(6286002)(83380400001)(336012)(1076003)(2906002)(16526019)(54906003)(8676002)(6916009)(5660300002)(356005)(316002)(70586007)(8936002)(4326008)(70206006)(36756003)(40460700003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 15:51:48.1672 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 53b3d8f0-3a99-4f95-9b4d-08d9f7ad90f3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT056.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB4595 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch supports device cleanup callback API which called when device disconnected with VM. Cached resources like VM MR and VQ memory are released. Signed-off-by: Xueming Li Reviewed-by: Maxime Coquelin --- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + 2 files changed, 24 insertions(+) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 38ed45f95f7..47874c9b1ff 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -270,6 +270,8 @@ mlx5_vdpa_dev_close(int vid) if (priv->lm_mr.addr) mlx5_os_wrapped_mkey_destroy(&priv->lm_mr); priv->state = MLX5_VDPA_STATE_PROBED; + if (!priv->connected) + mlx5_vdpa_dev_cache_clean(priv); priv->vid = 0; /* The mutex may stay locked after event thread cancel - initiate it. */ pthread_mutex_init(&priv->vq_config_lock, NULL); @@ -294,6 +296,7 @@ mlx5_vdpa_dev_config(int vid) return -1; } priv->vid = vid; + priv->connected = true; if (mlx5_vdpa_mtu_set(priv)) DRV_LOG(WARNING, "MTU cannot be set on device %s.", vdev->device->name); @@ -431,12 +434,32 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) return mlx5_vdpa_virtq_stats_reset(priv, qid); } +static int +mlx5_vdpa_dev_clean(int vid) +{ + struct rte_vdpa_device *vdev = rte_vhost_get_vdpa_device(vid); + struct mlx5_vdpa_priv *priv; + + if (vdev == NULL) + return -1; + priv = mlx5_vdpa_find_priv_resource_by_vdev(vdev); + if (priv == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name); + return -1; + } + if (priv->state == MLX5_VDPA_STATE_PROBED) + mlx5_vdpa_dev_cache_clean(priv); + priv->connected = false; + return 0; +} + static struct rte_vdpa_dev_ops mlx5_vdpa_ops = { .get_queue_num = mlx5_vdpa_get_queue_num, .get_features = mlx5_vdpa_get_vdpa_features, .get_protocol_features = mlx5_vdpa_get_protocol_features, .dev_conf = mlx5_vdpa_dev_config, .dev_close = mlx5_vdpa_dev_close, + .dev_cleanup = mlx5_vdpa_dev_clean, .set_vring_state = mlx5_vdpa_set_vring_state, .set_features = mlx5_vdpa_features_set, .migration_done = NULL, diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 540bf87a352..24bafe85b44 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -121,6 +121,7 @@ enum mlx5_dev_state { struct mlx5_vdpa_priv { TAILQ_ENTRY(mlx5_vdpa_priv) next; + bool connected; enum mlx5_dev_state state; pthread_mutex_t vq_config_lock; uint64_t no_traffic_counter;