From patchwork Wed Feb 23 18:48:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 108193 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1D88AA00C2; Wed, 23 Feb 2022 19:48:59 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A18D41158; Wed, 23 Feb 2022 19:48:52 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2057.outbound.protection.outlook.com [40.107.93.57]) by mails.dpdk.org (Postfix) with ESMTP id B27734114E for ; Wed, 23 Feb 2022 19:48:49 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YlLzysAfV2R0teRj35h00ePXU1pxkqTI9cFsaFg3Mi1UJ7fROFGVCrF6oxLxk6yretB9hamcFJH52JeoS/iCxCZqFrcQhuUopltDdYr6yp1hA13ft8I6tzjATnAtCUQ2mR8EGIFnC8FMoKMokGFSkinTVibA0cZrVT3PhsbUb7rczzZcAkYiLy3RqP9HU6vzP2dai3s27V+z0+74HWzHe0C/J5ek1v2z4nWwhzyCkRIQHZ4PppiUtC7PhO/1zTxiTRrISGMUg4wiUCob+1O56pw8S6gTTbpi2gTrlVgiz/8xRnshZc46v4nDFdx2vvf1hlr/w9FQr3vPslZ2OT98GA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PrShDu9D673FvzdOk5pWzOBHE9oOYLp8YYkFtHYNKVg=; b=B+/xZJVvjwLX0baeX0aJ2l6YLFot9hr/nDRv9T2L86ORBNTh7Mz/KBPzJEcg4CGrI+yDIZYrppaLMqe6Pup8ggF+qOxkE7b+PB1cUmc47gqRM7Fz54qM+ONBO6RxDYQV9C97q+gcRQTVDKiwbcmwQ+oGShrd3GriilTPUAZR6jz5BZkmoxfbAxkvw77srVswuYFI4NK5X3MCXhD4V/4Ei3hBpfBtGnrPf7cgpdeCoGhHK0kmlWWgPUcQBDIoRROJI/xx/lSnV1zZ2fJUGMzuBWKG4zF3bWJkqkjkQcHQTn6LjUDrKXlJSZ3hqctn6AaF4jhUl4DBGUlntEr3TcscIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PrShDu9D673FvzdOk5pWzOBHE9oOYLp8YYkFtHYNKVg=; b=fX9pyHghmLWVcSEw/fRrYoiMOXk8fZ8tSrnFshV5sAKPmuKCJw0gTSclKVIJbj3xgbTgd0IiCaU5aFr9QbVxiyAY39gM/mk/1Qf4OmIjeo6p0az25Od0oUbr/z8Ew/ExLbc3aBTq0Y39YJtW3xEipStBKn8ZLa9Sp5PA6BNIt4F5HT5u/NKOxB35XxE/KUGj+PtZuAUpD4T5zfnv/PSNe+29XBAAem/vXbtA/7s+zGsGUh/SGzFdQDFp0W1B38VXeTFESzB/Q0WN3NefKDS5S5h+NAuAlmxHjONT42ZW1A0C6b6SquELnM267ut23EsBUYBge7tX4mSuhG1TkL4zbQ== Received: from DM5PR12CA0061.namprd12.prod.outlook.com (2603:10b6:3:103::23) by DM4PR12MB5913.namprd12.prod.outlook.com (2603:10b6:8:66::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.17; Wed, 23 Feb 2022 18:48:48 +0000 Received: from DM6NAM11FT014.eop-nam11.prod.protection.outlook.com (2603:10b6:3:103:cafe::54) by DM5PR12CA0061.outlook.office365.com (2603:10b6:3:103::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.23 via Frontend Transport; Wed, 23 Feb 2022 18:48:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.234) by DM6NAM11FT014.mail.protection.outlook.com (10.13.173.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Wed, 23 Feb 2022 18:48:47 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 23 Feb 2022 18:48:47 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Wed, 23 Feb 2022 10:48:46 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9 via Frontend Transport; Wed, 23 Feb 2022 10:48:45 -0800 From: Michael Baum To: CC: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Subject: [PATCH v2 2/6] common/mlx5: glue device and PD importation Date: Wed, 23 Feb 2022 20:48:31 +0200 Message-ID: <20220223184835.3061161-3-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220223184835.3061161-1-michaelba@nvidia.com> References: <20220222210416.2669519-1-michaelba@nvidia.com> <20220223184835.3061161-1-michaelba@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 596ec620-6828-4c63-a151-08d9f6fd205c X-MS-TrafficTypeDiagnostic: DM4PR12MB5913:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kCc9NU6MQnYZRMI2n1gpDI7GejxiO1PAU4bntoAS21MECRwdJ/zohw3d9EzvWIa4LWde322H6nrj0srnTIDAcM5cECksUMqqOzE3z3KCiUd7G5iWPcyp7qO78Kxs+rIATvPlVLd1ctdU3uMoWLuzsbrGrQeEgUZvxQDt9UktENjDGZYWLnxofAsg9DjBtWNL9JicJWQKO3Pylwyh6bIUPcywUn1PQ+A92GWHLgDJCSpY8rZIuqcfgV1Il9dma+d+wgQ3hwyXkxIMMAZn/qvNMR1VEVbt9VpBR6z3926dLKM6cjHIeyy+P9SuXaNoepa0WMl16u7kxQ2ABACIYnZWirpXm0XPSm0x++gQ0z51ijeBye4zBl+wVVkRhfNUIxCqsCAxuTp6bR5YnVFD75vRjiQleGfsUWtctJ1V2OO2YGIJaF758WQhMKRfnsCK5C7OeSNNTvDBCLXQL7Xt4PQ4IiOujavQRpOjgVC6JxDbIyr/ir7l4VZVLNfYktpbFETqT7vk8o+eV9N8tSq2sT7yARerniPi/0g6kPx8UK29oPmiomsLVmZ5M1RO6NfVuqw2QUmY3Dm25tujac3Dsym19MHMtvS0/mOhwxf7EP9zhVJD/u9gmW/eUwi9aLmSfgDgYRAaZrg8WGPudv6UuXEDRiuzsxAQrnumlm+g3ZpjD5tpBLthjjhRIxGIoXgpVw0cpNVG5kONyfEbOMFhgKg5HQ== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(6286002)(83380400001)(1076003)(6916009)(47076005)(2906002)(54906003)(40460700003)(356005)(186003)(26005)(8936002)(107886003)(81166007)(86362001)(6666004)(36756003)(336012)(82310400004)(316002)(36860700001)(55016003)(508600001)(2616005)(5660300002)(7696005)(4326008)(8676002)(70206006)(70586007)(426003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2022 18:48:47.8642 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 596ec620-6828-4c63-a151-08d9f6fd205c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT014.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5913 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for rdma-core API to import device. The API gets ibv_context file descriptor and returns an ibv_context pointer that is associated with the given file descriptor. Add also support for rdma-core API to import PD. The API gets ibv_context and PD handle and returns a protection domain (PD) that is associated with the given handle in the given context. Signed-off-by: Michael Baum --- drivers/common/mlx5/linux/meson.build | 2 ++ drivers/common/mlx5/linux/mlx5_glue.c | 41 +++++++++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_glue.h | 4 +++ 3 files changed, 47 insertions(+) diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index 4c7b53b9bd..ed48245c67 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -202,6 +202,8 @@ has_sym_args = [ 'mlx5dv_dr_domain_allow_duplicate_rules' ], [ 'HAVE_MLX5_IBV_REG_MR_IOVA', 'infiniband/verbs.h', 'ibv_reg_mr_iova' ], + [ 'HAVE_MLX5_IBV_IMPORT_CTX_PD_AND_MR', 'infiniband/verbs.h', + 'ibv_import_device' ], ] config = configuration_data() foreach arg:has_sym_args diff --git a/drivers/common/mlx5/linux/mlx5_glue.c b/drivers/common/mlx5/linux/mlx5_glue.c index bc6622053f..450dd6a06a 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.c +++ b/drivers/common/mlx5/linux/mlx5_glue.c @@ -34,6 +34,32 @@ mlx5_glue_dealloc_pd(struct ibv_pd *pd) return ibv_dealloc_pd(pd); } +static struct ibv_pd * +mlx5_glue_import_pd(struct ibv_context *context, uint32_t pd_handle) +{ +#ifdef HAVE_MLX5_IBV_IMPORT_CTX_PD_AND_MR + return ibv_import_pd(context, pd_handle); +#else + (void)context; + (void)pd_handle; + errno = ENOTSUP; + return NULL; +#endif +} + +static int +mlx5_glue_unimport_pd(struct ibv_pd *pd) +{ +#ifdef HAVE_MLX5_IBV_IMPORT_CTX_PD_AND_MR + ibv_unimport_pd(pd); + return 0; +#else + (void)pd; + errno = ENOTSUP; + return -errno; +#endif +} + static struct ibv_device ** mlx5_glue_get_device_list(int *num_devices) { @@ -52,6 +78,18 @@ mlx5_glue_open_device(struct ibv_device *device) return ibv_open_device(device); } +static struct ibv_context * +mlx5_glue_import_device(int cmd_fd) +{ +#ifdef HAVE_MLX5_IBV_IMPORT_CTX_PD_AND_MR + return ibv_import_device(cmd_fd); +#else + (void)cmd_fd; + errno = ENOTSUP; + return NULL; +#endif +} + static int mlx5_glue_close_device(struct ibv_context *context) { @@ -1402,9 +1440,12 @@ const struct mlx5_glue *mlx5_glue = &(const struct mlx5_glue) { .fork_init = mlx5_glue_fork_init, .alloc_pd = mlx5_glue_alloc_pd, .dealloc_pd = mlx5_glue_dealloc_pd, + .import_pd = mlx5_glue_import_pd, + .unimport_pd = mlx5_glue_unimport_pd, .get_device_list = mlx5_glue_get_device_list, .free_device_list = mlx5_glue_free_device_list, .open_device = mlx5_glue_open_device, + .import_device = mlx5_glue_import_device, .close_device = mlx5_glue_close_device, .query_device = mlx5_glue_query_device, .query_device_ex = mlx5_glue_query_device_ex, diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index 4e6d31f263..c4903a6dce 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -151,9 +151,13 @@ struct mlx5_glue { int (*fork_init)(void); struct ibv_pd *(*alloc_pd)(struct ibv_context *context); int (*dealloc_pd)(struct ibv_pd *pd); + struct ibv_pd *(*import_pd)(struct ibv_context *context, + uint32_t pd_handle); + int (*unimport_pd)(struct ibv_pd *pd); struct ibv_device **(*get_device_list)(int *num_devices); void (*free_device_list)(struct ibv_device **list); struct ibv_context *(*open_device)(struct ibv_device *device); + struct ibv_context *(*import_device)(int cmd_fd); int (*close_device)(struct ibv_context *context); int (*query_device)(struct ibv_context *context, struct ibv_device_attr *device_attr);