From patchwork Wed Jun 16 04:09:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94274 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 470D0A0C45; Wed, 16 Jun 2021 06:10:02 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 13CC5410E9; Wed, 16 Jun 2021 06:09:57 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2060.outbound.protection.outlook.com [40.107.223.60]) by mails.dpdk.org (Postfix) with ESMTP id AD2EA410DA for ; Wed, 16 Jun 2021 06:09:55 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=INKmDtkkUZlfB6wtcJhy75VMniDiKSviao/YClP6f53WqEwJbhtsXAzkJxI66SMeZBMwMSnxZhZ/HrbTzihfDJch9Gmczt4TdQvswdYC2w6BJGn904p16WvK62pzyEb9XDAIFAw29MKMh69u4XFvHmaSREIF1pWaTwNmTYKGSwDZjJtgN8+JZtdXkru2FZ2/PsF6yjs22rpckuB1RDbEn4Wuh8DOCv7/8toq9uwIfu/MJjyUYm1kvXUbER/ya+vjVm1TkQ78h0zYAiX3bOItXk+XNUVT8tZKiPGXFIyCTBNuPRjt0n+HzJ996QidLTZRm3SInW6Ive8SoA582OZo7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xPJbcr8kYTP9f5UzXb9FsY7cAwBeaSgRr8sNbdazkQ0=; b=W91LJJVrHpsgoKM+isUlAa7TPi48h+/d9Q/L8Hvf9sIt/ndi0hcqcShyxkOYXOIp+5MTmcVX3VNx9pBW6NBBWVZ4C3RWLpRGZ/kJpRZHip0/2XRsUrCO7vD3lDiXvgJL+69XXsBd5E52H4uLdO8/XqUfPQySVeS4p9rWzql457qUvVpM78TmpiujAf3c8lXSqBOsENvJnJSeRzNfUwCFhSOuh7IPsS0wyB5MGaplvRLml6Zw/IbuxX9i9uLAAAkYbTLgZXixw8rOWOYmmKkYjJSLvTzrdWFA1Kwxu4gu26/yS47wHwKVHSA+9hEIO0323Lq7D18Z7aky+stlLKhWBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=tuxdriver.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xPJbcr8kYTP9f5UzXb9FsY7cAwBeaSgRr8sNbdazkQ0=; b=gr2zv3kOqczYDvEvPPw9k49bRuwQTQ9Yf8j8/kbylT26OvLBHO+i2v6eFl9FEFI9mpUyAz+cevTXUK4qZKFHPttHbKehTn4dj3aqMx/C4dpfsbrCt7zbpO9H+vkVenkXLmH9g6D7rM1AdjX7whX81RBI4FtLHjfsYW6a0yOFVwsPuDl9pJ6XS6r3mYjnOp3ZYv/zCIKgx4w5udZeW2zt7d/xumEa/SyNreFSS76jBHFqY7N+RseFzCk5AW0cdkejNQaJ8Ds6eYwqP9QkP+oJ0HOIT+Fy+vjBPPiy+doXga+Ir3f2xIpxAVJh2KHEjkrJGLgYjetVGKuhndWRhUpcuQ== Received: from DM5PR16CA0019.namprd16.prod.outlook.com (2603:10b6:3:c0::29) by CH0PR12MB5265.namprd12.prod.outlook.com (2603:10b6:610:d0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Wed, 16 Jun 2021 04:09:53 +0000 Received: from DM6NAM11FT060.eop-nam11.prod.protection.outlook.com (2603:10b6:3:c0:cafe::93) by DM5PR16CA0019.outlook.office365.com (2603:10b6:3:c0::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; tuxdriver.com; dkim=none (message not signed) header.d=none;tuxdriver.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT060.mail.protection.outlook.com (10.13.173.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:53 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:09:51 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Ray Kinsella , Neil Horman Date: Wed, 16 Jun 2021 07:09:22 +0300 Message-ID: <20210616040935.311733-2-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c0f3ba89-b429-4b6d-d387-08d9307c981d X-MS-TrafficTypeDiagnostic: CH0PR12MB5265: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kM9N/aihYuE7c2r5yZCztdJ2VN65fUOntWNevaq8HlVnsY+7qgO7tXH2SIknJx2I6ubkeYDgg0zHW5JgHtdwgLhH33Ro2zLdWoha3DIkWRxdvMZ1G/CB31dXwn/r8CvXyyYe5NlDyj2zfvfWTOu5V8RvXi0g0uRCLh7H4RE1YaksChLkc4xESYUYL00hQmjMrL3t0wQb/xT1HP+3lAm57CmuAXtKTV+qEB5HWqZzi+BoXDqtg5LduiwwlHsRTn2ltNSB5UaU4zoaO8HfdW7+/sX6b6IPWgVQ6lnd53zm7y3E3QGIjMc+72xLEDAfgshiEeU56PAyoJbL8VedC7YfppqGOrIgiKnDyFR8+CeUcy2DI4aZ+H+RWhvPJ6kbe+FMt+ouu/KWOmlvWdRiWMmhmFgxP36q8PiIFkjajSdrXs40Mr1aitiDxsJaghpy7sCVNgF7Ac6oTIyQK93GKMo+qGYqUFLW6eSr5WAWObMYljACBH5Qdy8ES0nP7MVDK1P+/ceXb7KOnDZ8+vZv3j58A41psFPtjIUW/MigwK5FoyUtjgzPAjvFMyEO//Dl87uuy1xbcu5mtRRhhuoRDBBEwkPn6M6vJgXyn6r2Klq+Ze7LirBGJlFtsb/samCuHRt53l7/5qBQSSfRm6eDHciPXA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(346002)(39860400002)(396003)(376002)(36840700001)(46966006)(55016002)(7636003)(336012)(70586007)(82740400003)(6666004)(356005)(2906002)(86362001)(36906005)(426003)(16526019)(186003)(478600001)(5660300002)(6286002)(83380400001)(8676002)(8936002)(37006003)(1076003)(82310400003)(36860700001)(6636002)(316002)(7696005)(36756003)(47076005)(6862004)(70206006)(30864003)(54906003)(2616005)(4326008)(26005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:09:53.4306 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c0f3ba89-b429-4b6d-d387-08d9307c981d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT060.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5265 Subject: [dpdk-dev] [PATCH v1 01/14] common/mlx5: add common device driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To support auxiliary bus, introduces common device driver and callbacks, suppose to replace current mlx5 common PCI bus driver. mlx5 common PCI bus driver still used by mlx5 eth, vDPA and regex PMD, will remove once all PMD drivers adapt to new common driver. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/mlx5_common_os.c | 2 +- drivers/common/mlx5/linux/mlx5_common_os.h | 7 +- drivers/common/mlx5/linux/mlx5_common_verbs.c | 21 +- drivers/common/mlx5/mlx5_common.c | 364 +++++++++++++++++- drivers/common/mlx5/mlx5_common.h | 129 ++++++- drivers/common/mlx5/mlx5_common_pci.c | 133 ++++++- drivers/common/mlx5/mlx5_common_private.h | 41 ++ drivers/common/mlx5/mlx5_common_utils.h | 2 + drivers/common/mlx5/version.map | 4 + drivers/net/mlx5/mlx5.c | 2 +- 10 files changed, 685 insertions(+), 20 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_private.h diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index ea0b71e425..78a9723075 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -425,7 +425,7 @@ mlx5_glue_constructor(void) } struct ibv_device * -mlx5_os_get_ibv_device(struct rte_pci_addr *addr) +mlx5_os_get_ibv_device(const struct rte_pci_addr *addr) { int n; struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n); diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h index 72d6bf828b..86d0cb09b0 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.h +++ b/drivers/common/mlx5/linux/mlx5_common_os.h @@ -291,6 +291,11 @@ mlx5_os_free(void *addr) __rte_internal struct ibv_device * -mlx5_os_get_ibv_device(struct rte_pci_addr *addr); +mlx5_os_get_ibv_device(const struct rte_pci_addr *addr); + +__rte_internal +struct ibv_device * +mlx5_os_get_ibv_dev(const struct rte_device *dev); + #endif /* RTE_PMD_MLX5_COMMON_OS_H_ */ diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c index aa560f05f2..6a6ab7a7a2 100644 --- a/drivers/common/mlx5/linux/mlx5_common_verbs.c +++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c @@ -10,11 +10,31 @@ #include #include +#include +#include + +#include "mlx5_common_utils.h" +#include "mlx5_common_log.h" +#include "mlx5_common_private.h" #include "mlx5_autoconf.h" #include #include #include +struct ibv_device * +mlx5_os_get_ibv_dev(const struct rte_device *dev) +{ + struct ibv_device *ibv = NULL; + + if (mlx5_dev_is_pci(dev)) + ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr); + if (ibv == NULL) { + rte_errno = ENODEV; + DRV_LOG(ERR, "Verbs device not found: %s", dev->name); + } + return ibv; +} + /** * Register mr. Given protection domain pointer, pointer to addr and length * register the memory region. @@ -68,4 +88,3 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr) memset(pmd_mr, 0, sizeof(*pmd_mr)); } } - diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 25e9f09108..dc32a7bd90 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -8,11 +8,14 @@ #include #include +#include +#include #include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" #include "mlx5_common_pci.h" +#include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu; @@ -41,6 +44,363 @@ static inline void mlx5_cpu_id(unsigned int level, RTE_LOG_REGISTER_DEFAULT(mlx5_common_logtype, NOTICE) +/* Head of list of drivers. */ +static TAILQ_HEAD(mlx5_drivers, mlx5_class_driver) drivers_list = + TAILQ_HEAD_INITIALIZER(drivers_list); + +/* Head of devices. */ +static TAILQ_HEAD(mlx5_devices, mlx5_common_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int drv_class; +} mlx5_classes[] = { + { .name = "vdpa", .drv_class = MLX5_CLASS_VDPA }, + { .name = "eth", .drv_class = MLX5_CLASS_ETH }, + /* Keep class "net" for backward compatibility. */ + { .name = "net", .drv_class = MLX5_CLASS_ETH }, + { .name = "regex", .drv_class = MLX5_CLASS_REGEX }, + { .name = "compress", .drv_class = MLX5_CLASS_COMPRESS }, +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].drv_class; + } + return -EINVAL; +} + +static struct mlx5_class_driver * +driver_get(uint32_t class) +{ + struct mlx5_class_driver *driver; + + TAILQ_FOREACH(driver, &drivers_list, next) { + if ((uint32_t)driver->drv_class == class) + return driver; + } + return NULL; +} + +static int +devargs_class_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + int class_val; + char *scratch; + char *scratch_ref; + char *found; + char *refstr = NULL; + + *ret = 0; + scratch = strdup(class_names); + if (!scratch) { + *ret = -ENOMEM; + return *ret; + } + scratch_ref = scratch; + found = strtok_r(scratch, ":", &refstr); + if (!found) + /* Empty string. */ + goto err; + do { + /* Extract each individual class name. Multiple + * classes can be supplied as class=net:regex:foo:bar. + */ + class_val = class_name_to_value(found); + /* Check if its a valid class. */ + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + found = strtok_r(NULL, ":", &refstr); + } while (found); +err: + free(scratch_ref); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options: %s.\n", class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + if (devargs->cls != NULL && devargs->cls->name != NULL) + /* Global syntax, only one class type. */ + return class_name_to_value(devargs->cls->name); + /* Legacy devargs support multiple classes. */ + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + rte_kvargs_process(kvlist, RTE_DEVARGS_KEY_CLASS, + devargs_class_handler, &ret); + rte_kvargs_free(kvlist); + return ret; +} + +static const unsigned int mlx5_class_invalid_combinations[] = { + MLX5_CLASS_ETH | MLX5_CLASS_VDPA, + /* New class combination should be added here. */ +}; + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified unsupported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_invalid_combinations); i++) { + if ((mlx5_class_invalid_combinations[i] & user_classes) == + mlx5_class_invalid_combinations[i]) + return -EINVAL; + } + /* Not found any invalid class combination. */ + return 0; +} + +static bool +device_class_enabled(const struct mlx5_common_device *device, uint32_t class) +{ + return (device->classes_loaded & class) > 0; +} + +static bool +mlx5_bus_match(const struct mlx5_class_driver *drv, + const struct rte_device *dev) +{ + if (mlx5_dev_is_pci(dev)) + return mlx5_dev_pci_match(drv, dev); + return true; +} + +static struct mlx5_common_device * +to_mlx5_device(const struct rte_device *rte_dev) +{ + struct mlx5_common_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (rte_dev == dev->dev) + return dev; + } + return NULL; +} + +static void +dev_release(struct mlx5_common_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +drivers_remove(struct mlx5_common_device *dev, uint32_t enabled_classes) +{ + struct mlx5_class_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = driver_get(RTE_BIT64(i)); + if (driver != NULL) { + local_ret = driver->remove(dev->dev); + if (local_ret == 0) + dev->classes_loaded &= ~RTE_BIT64(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT64(i); + i++; + } + if (local_ret != 0 && ret == 0) + ret = local_ret; + return ret; +} + +static int +drivers_probe(struct mlx5_common_device *dev, uint32_t user_classes) +{ + struct mlx5_class_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drivers_list, next) { + if ((driver->drv_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, dev->dev)) + continue; + already_loaded = dev->classes_loaded & driver->drv_class; + if (already_loaded && driver->probe_again == 0) { + DRV_LOG(ERR, "Device %s is already probed", + dev->dev->name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->probe(dev->dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load driver %s", + driver->name); + goto probe_err; + } + enabled_classes |= driver->drv_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + drivers_remove(dev, enabled_classes); + return ret; +} + +int +mlx5_common_dev_probe(struct rte_device *eal_dev) +{ + struct mlx5_common_device *dev; + uint32_t classes = 0; + bool new_device = false; + int ret; + + DRV_LOG(INFO, "probe device \"%s\".", eal_dev->name); + ret = parse_class_options(eal_dev->devargs); + if (ret < 0) { + DRV_LOG(ERR, "Unsupported mlx5 class type: %s", + eal_dev->devargs->args); + return ret; + } + classes = ret; + if (classes == 0) + /* Default to net class. */ + classes = MLX5_CLASS_ETH; + dev = to_mlx5_device(eal_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_common_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->dev = eal_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } else { + /* Validate combination here. */ + ret = is_valid_class_combination(classes | + dev->classes_loaded); + if (ret != 0) { + DRV_LOG(ERR, "Unsupported mlx5 classes combination."); + return ret; + } + } + ret = drivers_probe(dev, classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +int +mlx5_common_dev_remove(struct rte_device *eal_dev) +{ + struct mlx5_common_device *dev; + int ret; + + dev = to_mlx5_device(eal_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload drivers. */ + ret = drivers_remove(dev, dev->classes_loaded); + if (ret != 0) + dev_release(dev); + return ret; +} + +int +mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova, + size_t len) +{ + struct mlx5_class_driver *driver = NULL; + struct mlx5_class_driver *temp; + struct mlx5_common_device *mdev; + int ret = -EINVAL; + + mdev = to_mlx5_device(dev); + if (!mdev) + return -ENODEV; + TAILQ_FOREACH(driver, &drivers_list, next) { + if (!device_class_enabled(mdev, driver->drv_class) || + driver->dma_map == NULL) + continue; + ret = driver->dma_map(dev, addr, iova, len); + if (ret) + goto map_err; + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drivers_list, next) { + if (temp == driver) + break; + if (device_class_enabled(mdev, temp->drv_class) && + temp->dma_map && temp->dma_unmap) + temp->dma_unmap(dev, addr, iova, len); + } + return ret; +} + +int +mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, + size_t len) +{ + struct mlx5_class_driver *driver; + struct mlx5_common_device *mdev; + int local_ret = -EINVAL; + int ret = 0; + + mdev = to_mlx5_device(dev); + if (!mdev) + return -ENODEV; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(driver, &drivers_list, mlx5_drivers, next) { + if (!device_class_enabled(mdev, driver->drv_class) || + driver->dma_unmap == NULL) + continue; + local_ret = driver->dma_unmap(dev, addr, iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + if (local_ret) + ret = local_ret; + return ret; +} + +void +mlx5_class_driver_register(struct mlx5_class_driver *driver) +{ + mlx5_common_driver_on_register_pci(driver); + TAILQ_INSERT_TAIL(&drivers_list, driver, next); +} + +static void mlx5_common_driver_init(void) +{ + mlx5_common_pci_init(); +} + static bool mlx5_common_initialized; /** @@ -55,7 +415,7 @@ mlx5_common_init(void) return; mlx5_glue_constructor(); - mlx5_common_pci_init(); + mlx5_common_driver_init(); mlx5_common_initialized = true; } @@ -214,3 +574,5 @@ mlx5_devx_alloc_uar(void *ctx, int mapping) exit: return uar; } + +RTE_PMD_EXPORT_NAME(mlx5_class_driver, __COUNTER__); diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 306f2f1ab7..4783fa6303 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -210,7 +210,7 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); enum mlx5_class { MLX5_CLASS_INVALID, - MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_ETH = RTE_BIT64(0), MLX5_CLASS_VDPA = RTE_BIT64(1), MLX5_CLASS_REGEX = RTE_BIT64(2), MLX5_CLASS_COMPRESS = RTE_BIT64(3), @@ -242,4 +242,131 @@ extern uint8_t haswell_broadwell_cpu; __rte_internal void mlx5_common_init(void); +/** + * Common Driver Interface + * ConnectX common driver supports multiple classes: net,vdpa,regex,crypto and + * compress devices. This layer enables creating such multiple class of devices + * on a single device by allowing to bind multiple class specific device + * driver to attach to common driver. + * + * ---------------- + * | mlx5 class | + * | drivers | + * ---------------- + * || + * ----------------- + * | mlx5 | + * | common driver | + * ----------------- + * | | + * ----------- ----------------- + * | mlx5 | | mlx5 | + * | pci dev | | auxiliary dev | + * ----------- ----------------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 common driver is central place that validates supported + * class combinations. + * - mlx5 common driver hide bus difference by resolving device address + * from devargs, locating target RDMA device and probing with it. + */ + +/** + * Initialization function for the driver called during device probing. + */ +typedef int (mlx5_class_driver_probe_t)(struct rte_device *dev); + +/** + * Uninitialization function for the driver called during hot-unplugging. + */ +typedef int (mlx5_class_driver_remove_t)(struct rte_device *dev); + +/** + * Driver-specific DMA mapping. After a successful call the device + * will be able to read/write from/to this segment. + * + * @param dev + * Pointer to the device. + * @param addr + * Starting virtual address of memory to be mapped. + * @param iova + * Starting IOVA address of memory to be mapped. + * @param len + * Length of memory segment being mapped. + * @return + * - 0 On success. + * - Negative value and rte_errno is set otherwise. + */ +typedef int (mlx5_class_driver_dma_map_t)(struct rte_device *dev, void *addr, + uint64_t iova, size_t len); + +/** + * Driver-specific DMA un-mapping. After a successful call the device + * will not be able to read/write from/to this segment. + * + * @param dev + * Pointer to the device. + * @param addr + * Starting virtual address of memory to be unmapped. + * @param iova + * Starting IOVA address of memory to be unmapped. + * @param len + * Length of memory segment being unmapped. + * @return + * - 0 On success. + * - Negative value and rte_errno is set otherwise. + */ +typedef int (mlx5_class_driver_dma_unmap_t)(struct rte_device *dev, void *addr, + uint64_t iova, size_t len); + +/** Device already probed can be probed again to check for new ports. */ +#define MLX5_DRV_PROBE_AGAIN 0x0004 + +/** + * A structure describing a mlx5 common class driver. + */ +struct mlx5_class_driver { + TAILQ_ENTRY(mlx5_class_driver) next; + enum mlx5_class drv_class; /**< Class of this driver. */ + const char *name; /**< Driver name. */ + mlx5_class_driver_probe_t *probe; /**< Device Probe function. */ + mlx5_class_driver_remove_t *remove; /**< Device Remove function. */ + mlx5_class_driver_dma_map_t *dma_map; /**< device dma map function. */ + mlx5_class_driver_dma_unmap_t *dma_unmap; /**< device dma unmap function. */ + const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */ + uint32_t probe_again:1; + /**< Device already probed can be probed again to check new device. */ + uint32_t intr_lsc:1; /**< Supports link state interrupt. */ + uint32_t intr_rmv:1; /**< Supports device remove interrupt. */ +}; + +/** + * Register a mlx5 device driver. + * + * @param driver + * A pointer to a mlx5_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +mlx5_class_driver_register(struct mlx5_class_driver *driver); + +/** + * Test a device is PCI bus device. + * + * @param dev + * Pointer to device. + * + * @return + * - True on device devargs is a PCI bus device. + * - False otherwise. + */ +__rte_internal +bool +mlx5_dev_is_pci(const struct rte_device *dev); + #endif /* RTE_PMD_MLX5_COMMON_H_ */ diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c index 34747c4e07..6fe28defbf 100644 --- a/drivers/common/mlx5/mlx5_common_pci.c +++ b/drivers/common/mlx5/mlx5_common_pci.c @@ -3,11 +3,19 @@ */ #include + #include +#include +#include #include #include "mlx5_common_log.h" #include "mlx5_common_pci.h" +#include "mlx5_common_private.h" + +static struct rte_pci_driver mlx5_common_pci_driver; + +/********** Legacy PCI bus driver, to be removed ********/ struct mlx5_pci_device { struct rte_pci_device *pci_dev; @@ -282,8 +290,8 @@ drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv, * 0 on success, a negative errno value otherwise and rte_errno is set. */ static int -mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) { struct mlx5_pci_device *dev; uint32_t user_classes = 0; @@ -336,7 +344,7 @@ mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, * 0 on success, the function cannot fail. */ static int -mlx5_common_pci_remove(struct rte_pci_device *pci_dev) +mlx5_pci_remove(struct rte_pci_device *pci_dev) { struct mlx5_pci_device *dev; int ret; @@ -352,8 +360,8 @@ mlx5_common_pci_remove(struct rte_pci_device *pci_dev) } static int -mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, - uint64_t iova, size_t len) +mlx5_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) { struct mlx5_pci_driver *driver = NULL; struct mlx5_pci_driver *temp; @@ -385,8 +393,8 @@ mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, } static int -mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, - uint64_t iova, size_t len) +mlx5_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) { struct mlx5_pci_driver *driver; struct mlx5_pci_device *dev; @@ -419,10 +427,10 @@ static struct rte_pci_driver mlx5_pci_driver = { .driver = { .name = MLX5_PCI_DRIVER_NAME, }, - .probe = mlx5_common_pci_probe, - .remove = mlx5_common_pci_remove, - .dma_map = mlx5_common_pci_dma_map, - .dma_unmap = mlx5_common_pci_dma_unmap, + .probe = mlx5_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_pci_dma_map, + .dma_unmap = mlx5_pci_dma_unmap, }; static int @@ -486,7 +494,7 @@ pci_ids_table_update(const struct rte_pci_id *driver_id_table) updated_table = calloc(num_ids, sizeof(*updated_table)); if (!updated_table) return -ENOMEM; - if (TAILQ_EMPTY(&drv_list)) { + if (old_table == NULL) { /* Copy the first driver's ID table. */ for (id_iter = driver_id_table; id_iter->vendor_id != 0; id_iter++, i++) @@ -502,6 +510,7 @@ pci_ids_table_update(const struct rte_pci_id *driver_id_table) /* Terminate table with empty entry. */ updated_table[i].vendor_id = 0; mlx5_pci_driver.id_table = updated_table; + mlx5_common_pci_driver.id_table = updated_table; mlx5_pci_id_table = updated_table; if (old_table) free(old_table); @@ -520,6 +529,101 @@ mlx5_pci_driver_register(struct mlx5_pci_driver *driver) TAILQ_INSERT_TAIL(&drv_list, driver, next); } +/********** New common PCI bus driver ********/ + +bool +mlx5_dev_is_pci(const struct rte_device *dev) +{ + return strcmp(dev->bus->name, "pci") == 0; +} + +bool +mlx5_dev_pci_match(const struct mlx5_class_driver *drv, + const struct rte_device *dev) +{ + const struct rte_pci_device *pci_dev; + const struct rte_pci_id *id_table; + + if (!mlx5_dev_is_pci(dev)) + return false; + pci_dev = RTE_DEV_TO_PCI_CONST(dev); + for (id_table = drv->id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != RTE_PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != RTE_PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != RTE_PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != RTE_PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + return mlx5_common_dev_probe(&pci_dev->device); +} + +static int +mlx5_common_pci_remove(struct rte_pci_device *pci_dev) +{ + return mlx5_common_dev_remove(&pci_dev->device); +} + +static int +mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_map(&pci_dev->device, addr, iova, len); +} + +static int +mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_unmap(&pci_dev->device, addr, iova, len); +} + +void +mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver) +{ + if (driver->id_table != NULL) { + if (pci_ids_table_update(driver->id_table) != 0) + return; + } + if (driver->probe_again) + mlx5_common_pci_driver.drv_flags |= RTE_PCI_DRV_PROBE_AGAIN; + if (driver->intr_lsc) + mlx5_common_pci_driver.drv_flags |= RTE_PCI_DRV_INTR_LSC; + if (driver->intr_rmv) + mlx5_common_pci_driver.drv_flags |= RTE_PCI_DRV_INTR_RMV; +} + +static struct rte_pci_driver mlx5_common_pci_driver = { + .driver = { + .name = MLX5_PCI_DRIVER_NAME, + }, + .probe = mlx5_common_pci_probe, + .remove = mlx5_common_pci_remove, + .dma_map = mlx5_common_pci_dma_map, + .dma_unmap = mlx5_common_pci_dma_unmap, +}; + void mlx5_common_pci_init(void) { const struct rte_pci_id empty_table[] = { @@ -535,7 +639,7 @@ void mlx5_common_pci_init(void) */ if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table)) return; - rte_pci_register(&mlx5_pci_driver); + rte_pci_register(&mlx5_common_pci_driver); } RTE_FINI(mlx5_common_pci_finish) @@ -544,8 +648,9 @@ RTE_FINI(mlx5_common_pci_finish) /* Constructor doesn't register with PCI bus if it failed * to build the table. */ - rte_pci_unregister(&mlx5_pci_driver); + rte_pci_unregister(&mlx5_common_pci_driver); free(mlx5_pci_id_table); } } + RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__); diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h new file mode 100644 index 0000000000..791eb3cd77 --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Mellanox Technologies, Ltd + */ + +#ifndef _MLX5_COMMON_PRIVATE_H_ +#define _MLX5_COMMON_PRIVATE_H_ + +#include + +#include "mlx5_common.h" + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +/* Common bus driver: */ + +struct mlx5_common_device { + struct rte_device *dev; + TAILQ_ENTRY(mlx5_common_device) next; + uint32_t classes_loaded; +}; + +int mlx5_common_dev_probe(struct rte_device *eal_dev); +int mlx5_common_dev_remove(struct rte_device *eal_dev); +int mlx5_common_dev_dma_map(struct rte_device *dev, void *addr, uint64_t iova, + size_t len); +int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, + size_t len); + +/* Common PCI bus driver: */ + +void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver); +bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, + const struct rte_device *dev); + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _MLX5_COMMON_PRIVATE_H_ */ diff --git a/drivers/common/mlx5/mlx5_common_utils.h b/drivers/common/mlx5/mlx5_common_utils.h index ed378ce9bd..9b067e92a8 100644 --- a/drivers/common/mlx5/mlx5_common_utils.h +++ b/drivers/common/mlx5/mlx5_common_utils.h @@ -5,6 +5,8 @@ #ifndef RTE_PMD_MLX5_COMMON_UTILS_H_ #define RTE_PMD_MLX5_COMMON_UTILS_H_ +#include + #include "mlx5_common.h" #define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index db4f13f1f7..6be882e98c 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_class_driver_register; + mlx5_common_init; mlx5_common_verbs_reg_mr; # WINDOWS_NO_EXPORT @@ -10,6 +12,7 @@ INTERNAL { mlx5_create_mr_ext; + mlx5_dev_is_pci; # WINDOWS_NO_EXPORT mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT @@ -129,6 +132,7 @@ INTERNAL { mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; + mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index d0faa45944..52573e78f9 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -2428,7 +2428,7 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { }; static struct mlx5_pci_driver mlx5_driver = { - .driver_class = MLX5_CLASS_NET, + .driver_class = MLX5_CLASS_ETH, .pci_driver = { .driver = { .name = MLX5_PCI_DRIVER_NAME, From patchwork Wed Jun 16 04:09:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94275 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 12F58A0C45; Wed, 16 Jun 2021 06:10:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 99D03410F3; Wed, 16 Jun 2021 06:09:59 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2084.outbound.protection.outlook.com [40.107.236.84]) by mails.dpdk.org (Postfix) with ESMTP id 75C67410ED for ; Wed, 16 Jun 2021 06:09:57 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F8iIVybTJuZadypXiaof73cPrdadUgUeeox1WnPB1BQC4aAb/QfZpLb6xpnzTpWiYpxxptc1X3y4GSIXhLPO6UBGQChocy6F09rSaCAbnnWKkQr5oDaNYLB/J0P4jqTrLZzANPDwrbtb5vjMW1LNdPNnrMdXAkD0tSFSYbPGCy9DWMhpCnTuhhHyWAbnCw/DevYQfzDLd70d8w8H7u+j42fBB9zCyjrH0UI2bmHQCAWjjifqYPtS9Ncl+ygRnnmypABj9bvGV9aAQQyBXXDxMQr5r69OLSvRlZJgsWSHv08xxjKSQsuuqPTOSwuZrexpRKeJ9y35pU4SizaXNzhEuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ozKzWz3zVRKjuMcYkBsRPwnk5jBo3DDPZx+a4xUJMBA=; b=adnd8/9grYZYAe51Jvh7eOIUNrQvry2eH6gh7zR9A2EuFUBj5OwoFpIu9tuqH+OWu3NzXSqw4DBKVJatSGTjIbKWh5X3RwOonhqh0mj8lPOB4Y3Y0CIq7nxcZIb6i4YyaygF/Fu/Ukjmcj2HjelV//7UiZlsMZoyF7NGl3z9bvwoG+AUZofKqXd9mbQROfe7hcdDqRopuC4Kiw6Yh0fhG5Z9V1JGbEj5kK3SmVtBxCsNtqxy6AbYLnL29siU8ci1k8iltn3076y1LqtxxiHbKZqSmT+8tlxwq6fM7mLKCvArH47LTs+IZXTTZ9bLo58PkdAIFDqaTyTM4MMpuyNsPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ozKzWz3zVRKjuMcYkBsRPwnk5jBo3DDPZx+a4xUJMBA=; b=XzvM6gOZ5DKRquFErbc9NJI4fiU0zUpPYJXP6Xkr/iNKOg120dy7LH4axZV/mCO0fA7wuNrqpl1XWYWg/j8kGKbtHxTc4/osFbHGIPTtoAHrGWapRd+gBKGLJcfUN11pmQH+k/zlc2g1gtDer4c08H1HZedLhHSkSZETw2+rHfLY1dRnvK/l9lManKnjVzLkOdh32cZsplVnBE4ZKQlwb0bupTIeyoLshICrYeMxg6TAv7hdBGLsgBvlRk0HDJ6/hgHa8zqhc2nqLBH9mVxq3CuZ4McbWJZPodHKG/bxIEFvZqdUe+VQeMnXwpOKxYYKIrl4S+MAyptoZ2PTq8zlmg== Received: from DM5PR19CA0042.namprd19.prod.outlook.com (2603:10b6:3:9a::28) by BN9PR12MB5289.namprd12.prod.outlook.com (2603:10b6:408:102::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.25; Wed, 16 Jun 2021 04:09:56 +0000 Received: from DM6NAM11FT042.eop-nam11.prod.protection.outlook.com (2603:10b6:3:9a:cafe::96) by DM5PR19CA0042.outlook.office365.com (2603:10b6:3:9a::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT042.mail.protection.outlook.com (10.13.173.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:55 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:09:53 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: Thomas Monjalon , , , Matan Azrad , Shahaf Shuler Date: Wed, 16 Jun 2021 07:09:23 +0300 Message-ID: <20210616040935.311733-3-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fcb7cb6b-e4e7-46c5-285b-08d9307c99a0 X-MS-TrafficTypeDiagnostic: BN9PR12MB5289: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:923; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g9tHvPxN5vlEmJ9xK5zzTd7edEkvl1rB5BJpZFCzAdjPBf8k47yQUT3NsLvtprBur4nk9EC1dTOiWyQNg57/EwE8C/8KuH2txtRAqug7XsCsP6AwyMwDvWzZSnfgeoHTkDWPAtdYLryjte+Wy+KOsvyIiZPabk1KPgTt5kQJuoV/0YtgFLIw5KufLKdCZDv8VSqY4+F7xy2mGujopBmkKuNzHdmf9zg2b6nDOGgBcgSlaqJf+MZfGFwozlaPeIzh2iuivMcmbybbAgioKFC2Q7ed99mEclziijaYwvWm2XySvi6ascRTCmGKTCumLTmpu9B5OdTtJp3GtZe/RXk+cr1r+5Gql+jG+D6oS1Tp4dlHC3eXlU/IDwY1MXLnaT73rwrsiFeiyGRFSaFDrFuwB7Mkzj1QWCTshShwgNi+tqJH4vssu3/IqXOt9BeK8Cc8ozEvbtLP224iUK0eg3PlsmskyJpUDR7ua670xXI3Nn8IGxlRmVOYxqkI/xA7rQdPsqKQlhDqI/tE+lbz9QTlxNGgai1zdlwR4koDhyUNNIvLxLLkplt17JAgFTnHw9d2dMrqVuRtmn11jU6ixi9EavtVVH3SrvBDQ14Ue9Hqfdtb/vln+nUVux8F/5vj51bHKTVY9lxb72SUMbjn8T0t/eiev3s+U34MK32EA4ic5JA= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(396003)(376002)(346002)(39860400002)(46966006)(36840700001)(316002)(5660300002)(8936002)(26005)(336012)(6862004)(478600001)(55016002)(70586007)(186003)(16526019)(2616005)(36906005)(70206006)(8676002)(6286002)(4326008)(2906002)(6666004)(107886003)(36756003)(86362001)(54906003)(7696005)(82740400003)(7636003)(426003)(6636002)(356005)(37006003)(1076003)(47076005)(82310400003)(36860700001)(83380400001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:09:55.9964 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fcb7cb6b-e4e7-46c5-285b-08d9307c99a0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT042.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5289 Subject: [dpdk-dev] [PATCH v1 02/14] common/mlx5: move description of PCI sysfs functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon The Linux-specific functions mlx5_get_pci_addr() and mlx5_get_ifname_sysfs() are better described in the .h file. The requirement for using mlx5_get_pci_addr() is explicit: the node /device must exist in the provided sysfs path. Signed-off-by: Thomas Monjalon --- drivers/common/mlx5/linux/mlx5_common_os.c | 22 ------------------ drivers/common/mlx5/mlx5_common.h | 26 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 22 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index 78a9723075..337e9df8cb 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -23,17 +23,6 @@ const struct mlx5_glue *mlx5_glue; #endif -/** - * Get PCI information by sysfs device path. - * - * @param dev_path - * Pointer to device sysfs folder name. - * @param[out] pci_addr - * PCI bus address output buffer. - * - * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. - */ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr) @@ -159,17 +148,6 @@ mlx5_translate_port_name(const char *port_name_in, port_info_out->name_type = MLX5_PHYS_PORT_NAME_TYPE_UNKNOWN; } -/** - * Get kernel interface name from IB device path. - * - * @param[in] ibdev_path - * Pointer to IB device path. - * @param[out] ifname - * Interface name output buffer. - * - * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. - */ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname) { diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 4783fa6303..c0c950f8f9 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -202,8 +202,34 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n, return MLX5_CQE_STATUS_SW_OWN; } +/* + * Get PCI address from sysfs of a PCI-related device. + * + * @param[in] dev_path + * The sysfs path should not point to the direct plain PCI device. + * Instead, the node "/device/" is used to access the real device. + * @param[out] pci_addr + * Parsed PCI address. + * + * @return + * - 0 on success. + * - Negative value and rte_errno is set otherwise. + */ __rte_internal int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); + +/* + * Get kernel network interface name from sysfs IB device path. + * + * @param[in] ibdev_path + * The sysfs path to IB device. + * @param[out] ifname + * Interface name output of size IF_NAMESIZE. + * + * @return + * - 0 on success. + * - Negative value and rte_errno is set otherwise. + */ __rte_internal int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); From patchwork Wed Jun 16 04:09:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94276 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 63E3BA0C45; Wed, 16 Jun 2021 06:10:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BCBD5410F7; Wed, 16 Jun 2021 06:10:00 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2081.outbound.protection.outlook.com [40.107.237.81]) by mails.dpdk.org (Postfix) with ESMTP id F0C07410F3 for ; Wed, 16 Jun 2021 06:09:58 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=odL4aswcbWcd2HFbRP4xD3iBIoalvvjl3qGXb3xTmhWbhugUUgALcYI1CEmHBSnvl+nxLwF60HeMqrFVLePwB9GRJFeDWdEe5/x88nKVjwuXttZAFtEMn7zI7dmMyRSqLEK7KrmzjDR/qS1SUrxOdlM8I4BMYtBNqClzWzPHbOwz5kM4DEV6h3pm85Bg8A7zv7y/8KQXSoDv/w9CQuizxHZ7EKDM7cr5nQ+KEXaCnxi0+7vkNSxkAOqsmqykSlBvpEBgTXmpY1o0Q8ohIC6+q251ey9nov9AspgXTa624jXdjDrnC96tSOn/0mlSIGzNjGe4ffbWDcMtzZDTKieOFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8zbOHQZXtEbLucqxUiAW1JBIwij/Sh/H6oE9ZbQgRk8=; b=k/PaX7wJp0fNw0WNGYaKJqlVfMGaTq3IweorB+MV2RH1OP91yP3bEjLzcLy6eCAQ+rQR05KC+HFdXeYIB3R7SJdIMYFGZSzGHTUThTlPT8tJV/jHo0QK/v6OWSZntq5yp3LUyJt6/wkS8J3/9aJlOEvLv1s2oNb2frN1/d1CVFxR9geRr0gMI+2GepLBjaGg5CU4oMJCV3hvOL6JHwsyWyMCRyTCodZeLQ7RBVVBdsmTaW0tHWAKnSTU4oIGm7VSeDgeK2CvOE9VDINe/4giL7fU019oDvR9D6cRdvBNp/EhjMXcPQzhkZ4HoFyobRa4rKOtNGYI4+TCAOmnXpVLuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=tuxdriver.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8zbOHQZXtEbLucqxUiAW1JBIwij/Sh/H6oE9ZbQgRk8=; b=qXALKLBZcgfiQEYwqMGxXT8f8+pyLt2baqHiFrLHdA/X6RhdLWTgeHuNpkGKpnI2uj8fiite3IQW7u3zfhWlUdl5J40WJu5Ym2Mmn/OVDl7anzFvcw6KumuF+prmzNLSzsJrKj2A5M7hiHJ/jB3sGpwLw67Uql3kOOjRO/tvbUYVt0LPUaemZbTnHoDjjQtQU0pxea21JsTOI+u4KLBMFdD6XfsPTjS+cGxEi4pZlWVy6hUmXX74L7D0RBwCS7G3w+czPhUb3cpqESBEUtLNB9+3lOzN18kQxfNspmEMOEz0P06QuhZPR3OcIIRZMB4rK0uaHc2v0BhH/RpNfE0yVw== Received: from DM6PR03CA0027.namprd03.prod.outlook.com (2603:10b6:5:40::40) by DM6PR12MB4942.namprd12.prod.outlook.com (2603:10b6:5:1be::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Wed, 16 Jun 2021 04:09:57 +0000 Received: from DM6NAM11FT003.eop-nam11.prod.protection.outlook.com (2603:10b6:5:40:cafe::37) by DM6PR03CA0027.outlook.office365.com (2603:10b6:5:40::40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.18 via Frontend Transport; Wed, 16 Jun 2021 04:09:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; tuxdriver.com; dkim=none (message not signed) header.d=none;tuxdriver.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT003.mail.protection.outlook.com (10.13.173.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:57 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:09:55 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Ray Kinsella , Neil Horman Date: Wed, 16 Jun 2021 07:09:24 +0300 Message-ID: <20210616040935.311733-4-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e3d9d1dd-e260-4374-8c1b-08d9307c9a67 X-MS-TrafficTypeDiagnostic: DM6PR12MB4942: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:147; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: noE8ItcpBQjXmv45YYSXwbtkNoT7zfvCwSXo803gHFQAZPuS/vGvWR7w1kH5MQf5EOfD3Lm3B4Zhy/xPeXabMjO+RlxjWEDJUSDN4RGK7Me7g09TJWcxg/orssKYhRFpXMJWd1UrUlW7r85BnK5I9AJ4+tlEWEbIGZWeFhLY4NZMZy79YyWtoG9fpJUKbBspORYOSIxv6IjWL7eUbSJvEPWvMMzTehf52sbfvZNpLSIKQWQJ6iXWh86Q4/IEtD5ZeiDUQtVWQIGns3ucnBAgzJOuyLWxWD4xZnD6+fl24FyjxSh2ndYxDiNRqIrLKYZjr06xhbIpi+JeZiRtN6hKIof+mlI5awEPtc8BKRTnIaFbztx2Qn6k/rs/1P8w02tLOND7ofGe8XlpMNib1x3uEft6nK1K4WmjZX3pWrR8INSMxuXk183GfqQ9AtXXxTlVDB0Lk0yyVVdnVCn2LJLhi6RVsjV8f58c1Smt9Hiz2omWLJPmF5ywhbkujksEVexQKxMfNhpVCKrbzNzMb9OQ09B8qC9JVSFSWUSOEbT2/SqncXV/jhwz8W+t2/PjljtQEEvvL51ehJBd4GVoMnYZ6FJsIv/QJ95GA1ESyGEIEAHpSsqjmXody69p9bUzScOBgaV1DxUd7IdOYcNQdq+QtY0IVCzJp2SBt2KM6YeI3OI= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(39860400002)(346002)(136003)(46966006)(36840700001)(83380400001)(5660300002)(36860700001)(82740400003)(6666004)(8936002)(6636002)(54906003)(356005)(36906005)(1076003)(7696005)(316002)(37006003)(70206006)(4326008)(70586007)(7636003)(36756003)(2906002)(47076005)(2616005)(6862004)(86362001)(8676002)(426003)(478600001)(16526019)(82310400003)(186003)(336012)(55016002)(6286002)(26005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:09:57.2651 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e3d9d1dd-e260-4374-8c1b-08d9307c9a67 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT003.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4942 Subject: [dpdk-dev] [PATCH v1 03/14] common/mlx5: support auxiliary bus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds auxiliary bus driver and delegate to registered internal mlx5 common device drivers, i.e. eth, vdpa... Current major target is to support SubFunction on auxiliary bus. As a limitation of current driver, numa node of device is detected from PCI bus of device symbol link, will remove once numa node file available on sysfs. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 172 ++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_verbs.c | 5 +- drivers/common/mlx5/meson.build | 2 +- drivers/common/mlx5/mlx5_common.c | 3 + drivers/common/mlx5/mlx5_common.h | 6 + drivers/common/mlx5/mlx5_common_private.h | 6 + drivers/common/mlx5/version.map | 2 + 8 files changed, 197 insertions(+), 2 deletions(-) create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index 5cea1b44d7..676fd05014 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -48,10 +48,13 @@ endif sources += files('mlx5_nl.c') sources += files('mlx5_common_os.c') sources += files('mlx5_common_verbs.c') +sources += files('mlx5_common_auxiliary.c') if not dlopen_ibverbs sources += files('mlx5_glue.c') endif +deps += ['bus_auxiliary'] + # To maintain the compatibility with the make build system # mlx5_autoconf.h file is still generated. # input array for meson member search: diff --git a/drivers/common/mlx5/linux/mlx5_common_auxiliary.c b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c new file mode 100644 index 0000000000..79d567087c --- /dev/null +++ b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c @@ -0,0 +1,172 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies Ltd + */ + +#include +#include +#include +#include +#include +#include +#include "eal_filesystem.h" + +#include "mlx5_common_utils.h" +#include "mlx5_common_private.h" + +#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices" +#define MLX5_AUXILIARY_PREFIX "mlx5_core.sf." + +int +mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size) +{ + DIR *dir; + struct dirent *dent; + MKSTR(path, "%s/%s%s", AUXILIARY_SYSFS_PATH, dev, node); + + dir = opendir(path); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + /* Get the first file name. */ + while ((dent = readdir(dir)) != NULL) { + if (dent->d_name[0] != '.') + break; + } + closedir(dir); + if (dent == NULL) { + rte_errno = ENOENT; + return -rte_errno; + } + if (rte_strscpy(child, dent->d_name, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev, + char *sysfs_pci, size_t size) +{ + char sysfs_real[PATH_MAX] = { 0 }; + MKSTR(sysfs_aux, "%s/%s", AUXILIARY_SYSFS_PATH, dev->name); + char *dir; + + if (realpath(sysfs_aux, sysfs_real) == NULL) { + rte_errno = errno; + return -rte_errno; + } + dir = dirname(sysfs_real); + if (dir == NULL) { + rte_errno = errno; + return -rte_errno; + } + if (rte_strscpy(sysfs_pci, dir, size) < 0) + return -rte_errno; + return 0; +} + +static int +mlx5_auxiliary_get_numa(const struct rte_auxiliary_device *dev) +{ + unsigned long numa; + char numa_path[PATH_MAX]; + + if (mlx5_auxiliary_get_pci_path(dev, numa_path, sizeof(numa_path)) != 0) + return SOCKET_ID_ANY; + if (strcat(numa_path, "/numa_node") == NULL) { + rte_errno = ENAMETOOLONG; + return SOCKET_ID_ANY; + } + if (eal_parse_sysfs_value(numa_path, &numa) != 0) { + rte_errno = EINVAL; + return SOCKET_ID_ANY; + } + return (int)numa; +} + +struct ibv_device * +mlx5_get_aux_ibv_device(const struct rte_auxiliary_device *dev) +{ + int n; + char ib_name[64] = { 0 }; + struct ibv_device **ibv_list = mlx5_glue->get_device_list(&n); + struct ibv_device *ibv_match = NULL; + + if (!ibv_list) { + rte_errno = ENOSYS; + return NULL; + } + if (mlx5_auxiliary_get_child_name(dev->name, "/infiniband", + ib_name, sizeof(ib_name)) != 0) + return NULL; + while (n-- > 0) { + if (strcmp(ibv_list[n]->name, ib_name) != 0) + continue; + ibv_match = ibv_list[n]; + break; + } + if (ibv_match == NULL) + rte_errno = ENOENT; + mlx5_glue->free_device_list(ibv_list); + return ibv_match; +} + +static bool +mlx5_common_auxiliary_match(const char *name) +{ + return strncmp(name, MLX5_AUXILIARY_PREFIX, + strlen(MLX5_AUXILIARY_PREFIX)) == 0; +} + +static int +mlx5_common_auxiliary_probe(struct rte_auxiliary_driver *drv __rte_unused, + struct rte_auxiliary_device *dev) +{ + dev->device.numa_node = mlx5_auxiliary_get_numa(dev); + return mlx5_common_dev_probe(&dev->device); +} + +static int +mlx5_common_auxiliary_remove(struct rte_auxiliary_device *auxiliary_dev) +{ + return mlx5_common_dev_remove(&auxiliary_dev->device); +} + +static int +mlx5_common_auxiliary_dma_map(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_map(&auxiliary_dev->device, addr, iova, len); +} + +static int +mlx5_common_auxiliary_dma_unmap(struct rte_auxiliary_device *auxiliary_dev, + void *addr, uint64_t iova, size_t len) +{ + return mlx5_common_dev_dma_unmap(&auxiliary_dev->device, addr, iova, + len); +} + +static struct rte_auxiliary_driver mlx5_auxiliary_driver = { + .driver = { + .name = MLX5_AUXILIARY_DRIVER_NAME, + }, + .match = mlx5_common_auxiliary_match, + .probe = mlx5_common_auxiliary_probe, + .remove = mlx5_common_auxiliary_remove, + .dma_map = mlx5_common_auxiliary_dma_map, + .dma_unmap = mlx5_common_auxiliary_dma_unmap, +}; + +void mlx5_common_auxiliary_init(void) +{ + if (mlx5_auxiliary_driver.bus == NULL) + rte_auxiliary_register(&mlx5_auxiliary_driver); +} + +RTE_FINI(mlx5_common_auxiliary_driver_finish) +{ + if (mlx5_auxiliary_driver.bus != NULL) + rte_auxiliary_unregister(&mlx5_auxiliary_driver); +} diff --git a/drivers/common/mlx5/linux/mlx5_common_verbs.c b/drivers/common/mlx5/linux/mlx5_common_verbs.c index 6a6ab7a7a2..9080bd3e87 100644 --- a/drivers/common/mlx5/linux/mlx5_common_verbs.c +++ b/drivers/common/mlx5/linux/mlx5_common_verbs.c @@ -12,6 +12,7 @@ #include #include +#include #include "mlx5_common_utils.h" #include "mlx5_common_log.h" @@ -24,10 +25,12 @@ struct ibv_device * mlx5_os_get_ibv_dev(const struct rte_device *dev) { - struct ibv_device *ibv = NULL; + struct ibv_device *ibv; if (mlx5_dev_is_pci(dev)) ibv = mlx5_os_get_ibv_device(&RTE_DEV_TO_PCI_CONST(dev)->addr); + else + ibv = mlx5_get_aux_ibv_device(RTE_DEV_TO_AUXILIARY_CONST(dev)); if (ibv == NULL) { rte_errno = ENODEV; DRV_LOG(ERR, "Verbs device not found: %s", dev->name); diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build index fa2b8b9834..6ddbde7e8f 100644 --- a/drivers/common/mlx5/meson.build +++ b/drivers/common/mlx5/meson.build @@ -7,7 +7,7 @@ if not (is_linux or (is_windows and is_ms_linker)) subdir_done() endif -deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] +deps += ['hash', 'pci', 'bus_pci', 'bus_auxiliary', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c', diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index dc32a7bd90..b4ca0a9fe1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -399,6 +399,9 @@ mlx5_class_driver_register(struct mlx5_class_driver *driver) static void mlx5_common_driver_init(void) { mlx5_common_pci_init(); +#ifdef RTE_EXEC_ENV_LINUX + mlx5_common_auxiliary_init(); +#endif } static bool mlx5_common_initialized; diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index c0c950f8f9..439f16cde1 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -22,6 +22,7 @@ /* Reported driver name. */ #define MLX5_PCI_DRIVER_NAME "mlx5_pci" +#define MLX5_AUXILIARY_DRIVER_NAME "mlx5_auxiliary" /* Bit-field manipulation. */ #define BITFIELD_DECLARE(bf, type, size) \ @@ -107,6 +108,7 @@ pmd_drv_log_basename(const char *s) int mkstr_size_##name = snprintf(NULL, 0, "" __VA_ARGS__); \ char name[mkstr_size_##name + 1]; \ \ + memset(name, 0, mkstr_size_##name + 1); \ snprintf(name, sizeof(name), "" __VA_ARGS__) enum { @@ -134,6 +136,10 @@ enum { PCI_DEVICE_ID_MELLANOX_CONNECTX7BF = 0Xa2dc, }; + +__rte_internal +int mlx5_auxiliary_get_child_name(const char *dev, const char *node, + char *child, size_t size); /* Maximum number of simultaneous unicast MAC addresses. */ #define MLX5_MAX_UC_MAC_ADDRESSES 128 /* Maximum number of simultaneous Multicast MAC addresses. */ diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h index 791eb3cd77..9f00a6c54d 100644 --- a/drivers/common/mlx5/mlx5_common_private.h +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -6,6 +6,7 @@ #define _MLX5_COMMON_PRIVATE_H_ #include +#include #include "mlx5_common.h" @@ -34,6 +35,11 @@ void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver); bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, const struct rte_device *dev); +/* Common auxiliary bus driver: */ +void mlx5_common_auxiliary_init(void); +struct ibv_device *mlx5_get_aux_ibv_device( + const struct rte_auxiliary_device *dev); + #ifdef __cplusplus } #endif /* __cplusplus */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 6be882e98c..edfd3792e0 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -3,6 +3,8 @@ INTERNAL { haswell_broadwell_cpu; + mlx5_auxiliary_get_child_name; # WINDOWS_NO_EXPORT + mlx5_class_driver_register; mlx5_common_init; From patchwork Wed Jun 16 04:09:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94277 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0AD92A0C45; Wed, 16 Jun 2021 06:10:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 047DC410FC; Wed, 16 Jun 2021 06:10:03 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2071.outbound.protection.outlook.com [40.107.94.71]) by mails.dpdk.org (Postfix) with ESMTP id 0AD12410FB for ; Wed, 16 Jun 2021 06:10:02 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=W7u8T+CTRmvLHubCGgiGWqLAwQ5Yvt6R2zNUjgfzwTxEujm7bL1a8GS95UvhETH8OVCP+VkGKI5zV0EBBOUb4SXtQGOQFi3XwWV4QZoxyoSLX1CfxQa5EeWmE87MXLlFgFjBEe2KjC4K8XZ9r8Tn4gRu4eiDjakygNDf6KoNUuswbrOpPET19+ljZob2WtAgRo0FGTa0LlWsk8QVcedxZ9ZBMn8mp0h5tyRP5Y7A+lQV/ORlWcAu/qBI1sryWp7utfFPXbtb7xKxddHjd3vn+O8pzLWgp0WcyDFxeEs3HEEp/CzmdHfLSp3vlHI081s9dM1x29wDe0X4Ae6CAJd4gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g5+gWEMH9BBbVC3em1aSIYobdQWex94Kb/cAd1rAmGE=; b=lm7aUA3k+Kc6oiX+MW2kDWCaba5xkNWDMh0OlirLjSZ9kCKGAr0HZPj6E7uSDIY8R+3TGTq73RSVHp5Ij2XmmQLIrKnGKPQEW+WibXdyN8Zy38GdIOFh4dcVNKiDWw8ySFFJUhz6dxV87idCznEsE/oPAz03R8yyO1vdR28bHsTTrye5VDyva4ISWURy4+qqfME+JnqIcrIRmvzn42S3KKoxb890XZyBaipUC534UO0uI4eKH1LhwGV40V0FXs1oO5UPpCcjI457IgBO3vHcYwhPoB88BGND06KGjEqTrLehmsXA7VvBuqOvbxcYSQMHgc+I15dskLtGCcEAho7/Ng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=ashroe.eu smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g5+gWEMH9BBbVC3em1aSIYobdQWex94Kb/cAd1rAmGE=; b=Ci/20FfrsLGvazYXqKwxl1tc6eiPP/0rt8PwmQE3f5ZRF1rX2MlmS0SJfeQTNBe7cfBHorAtXucCmUYLkCdAXVscVvq1oGWMyZ8+daAFsoCRiAmiwYhO+wX6EFBKEO535G49QN/vQUSpm5lisyzL40kYtnVudHiahL7ubK/Bqt8pIZifDRtU0VR+SIDO5xoPn217g71r3zB2N1Yk+4GFE+lK/aT2F7tHHbE6S3LVDPGyIn7D+QngOif3x7g3ceNdNccQeM76IDZoSMVCwEhD2S/R7H2s5wX5cgNnrZjkje4XS1tgpQg4lJxOWs7udXY+h+1MYZn6i1YirM/EZBLWdg== Received: from DM6PR06CA0101.namprd06.prod.outlook.com (2603:10b6:5:336::34) by MWHPR12MB1198.namprd12.prod.outlook.com (2603:10b6:300:d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Wed, 16 Jun 2021 04:10:00 +0000 Received: from DM6NAM11FT031.eop-nam11.prod.protection.outlook.com (2603:10b6:5:336:cafe::28) by DM6PR06CA0101.outlook.office365.com (2603:10b6:5:336::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.21 via Frontend Transport; Wed, 16 Jun 2021 04:10:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; ashroe.eu; dkim=none (message not signed) header.d=none;ashroe.eu; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT031.mail.protection.outlook.com (10.13.172.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:09:59 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:09:57 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: Thomas Monjalon , , , Matan Azrad , Shahaf Shuler , Ray Kinsella , Neil Horman Date: Wed, 16 Jun 2021 07:09:25 +0300 Message-ID: <20210616040935.311733-5-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c62a2125-aabc-40e8-1fc5-08d9307c9bde X-MS-TrafficTypeDiagnostic: MWHPR12MB1198: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:568; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: q/MmiNkke5Om+wO1AqlxdFntpXRfF0bPhXrkkosGg7+BnkRuehq8zh8HtjR4U5SkE6HlSDjBFkA1ZLGY6CKL9dZcYexDxhMlEFXKEYt1UFDffDmut64zxH3l+EtF1xNmEQMTENxyL9s5JNQnPHj+1EhbzD69Gx4BCTBuWqH9GmEeEpmOZanNYi4OC94ydEKoOHUaY9+3B2LVdd5MKApoJg3WGDMFL9ukzKlPOsp17D32Kd09QKJcNqYeVR1nvTRppf0KkGXbiTg5gc+5nRzkiGnTfliPUtLmn7FYa2VZtjZ1hnrHJY5YXTiEK93lNi9a/0hws4dgVhc7RVDRtlJ973Fv3y+A8H/rDM/PpvUv4hyCTdRQ1BOsSIcqZobNHsSGY6lYXueLJb6Fvzsv2fCA+79fMeTVNCRVe9KtRZUACQQJm/fmG+ed5RaF3SFP5s7W38p/9Ct5O34KmzSaDP1DJWiHA7cf3J7SjS94VjsLSNdRDIeT32Yp/VpFJ2uMeIyso4TYOc1AuFzMBCpGkgcE+HMnxxv2gMlD7fJJNrLz2o1zCA76oeD7uCWpXXr7fkP3yNdklJ3UPKHFynjBs+mSMYiQaOO3IRrv+SGR1i2LQ0A8vDy3kK5Ipg1ECEgoE5v/8XCsHpc+gzhKzUU7LCe5/w== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(136003)(376002)(396003)(346002)(46966006)(36840700001)(86362001)(7696005)(37006003)(83380400001)(6636002)(478600001)(2906002)(36756003)(55016002)(54906003)(82740400003)(26005)(316002)(186003)(16526019)(1076003)(36906005)(8676002)(6862004)(356005)(2616005)(70206006)(7636003)(70586007)(6666004)(5660300002)(47076005)(82310400003)(36860700001)(4326008)(336012)(426003)(8936002)(6286002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:09:59.7650 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c62a2125-aabc-40e8-1fc5-08d9307c9bde X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT031.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1198 Subject: [dpdk-dev] [PATCH v1 04/14] common/mlx5: get PCI device address from any bus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon A function is exported to allow retrieving the PCI address of the parent PCI device of a Sub-Function in auxiliary bus sysfs. The function mlx5_dev_to_pci_str() is accepting both PCI and auxiliary devices. In case of a PCI device, it is simply using the device name. The function mlx5_dev_to_pci_addr(), which is based on sysfs path and do not use any device object, is renamed to mlx5_get_pci_addr() for clarity purpose. Signed-off-by: Thomas Monjalon --- .../common/mlx5/linux/mlx5_common_auxiliary.c | 19 +++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 5 ++-- drivers/common/mlx5/mlx5_common.c | 23 +++++++++++++++++++ drivers/common/mlx5/mlx5_common.h | 16 ++++++++++++- drivers/common/mlx5/mlx5_common_private.h | 2 ++ drivers/common/mlx5/version.map | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 6 ++--- 7 files changed, 66 insertions(+), 8 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_common_auxiliary.c b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c index 79d567087c..aaa2407ded 100644 --- a/drivers/common/mlx5/linux/mlx5_common_auxiliary.c +++ b/drivers/common/mlx5/linux/mlx5_common_auxiliary.c @@ -4,6 +4,8 @@ #include #include +#include + #include #include #include @@ -66,6 +68,23 @@ mlx5_auxiliary_get_pci_path(const struct rte_auxiliary_device *dev, return 0; } +int +mlx5_auxiliary_get_pci_str(const struct rte_auxiliary_device *dev, + char *addr, size_t size) +{ + char sysfs_pci[PATH_MAX]; + char *base; + + if (mlx5_auxiliary_get_pci_path(dev, sysfs_pci, sizeof(sysfs_pci)) != 0) + return -ENODEV; + base = basename(sysfs_pci); + if (base == NULL) + return -errno; + if (rte_strscpy(addr, base, size) < 0) + return -rte_errno; + return 0; +} + static int mlx5_auxiliary_get_numa(const struct rte_auxiliary_device *dev) { diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index 337e9df8cb..9e0c823c97 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -24,8 +24,7 @@ const struct mlx5_glue *mlx5_glue; #endif int -mlx5_dev_to_pci_addr(const char *dev_path, - struct rte_pci_addr *pci_addr) +mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr) { FILE *file; char line[32]; @@ -417,7 +416,7 @@ mlx5_os_get_ibv_device(const struct rte_pci_addr *addr) struct rte_pci_addr paddr; DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[n]->name); - if (mlx5_dev_to_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0) + if (mlx5_get_pci_addr(ibv_list[n]->ibdev_path, &paddr) != 0) continue; if (rte_pci_addr_cmp(addr, &paddr) != 0) continue; diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index b4ca0a9fe1..4ff13cb461 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -198,6 +198,29 @@ to_mlx5_device(const struct rte_device *rte_dev) return NULL; } +int +mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size) +{ + struct rte_pci_addr pci_addr = { 0 }; + int ret; + + if (mlx5_dev_is_pci(dev)) { + /* Input might be , format PCI address to . */ + ret = rte_pci_addr_parse(dev->name, &pci_addr); + if (ret != 0) + return -ENODEV; + rte_pci_device_name(&pci_addr, addr, size); + return 0; + } +#ifdef RTE_EXEC_ENV_LINUX + return mlx5_auxiliary_get_pci_str(RTE_DEV_TO_AUXILIARY_CONST(dev), + addr, size); +#else + rte_errno = ENODEV; + return -rte_errno; +#endif +} + static void dev_release(struct mlx5_common_device *dev) { diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 439f16cde1..7d7b896517 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -208,6 +208,20 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n, return MLX5_CQE_STATUS_SW_OWN; } +/* + * Get PCI address string from EAL device. + * + * @param[out] addr + * The output address buffer string + * @param[in] size + * The output buffer size + * @return + * - 0 on success. + * - Negative value and rte_errno is set otherwise. + */ +__rte_internal +int mlx5_dev_to_pci_str(const struct rte_device *dev, char *addr, size_t size); + /* * Get PCI address from sysfs of a PCI-related device. * @@ -222,7 +236,7 @@ check_cqe(volatile struct mlx5_cqe *cqe, const uint16_t cqes_n, * - Negative value and rte_errno is set otherwise. */ __rte_internal -int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); +int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); /* * Get kernel network interface name from sysfs IB device path. diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h index 9f00a6c54d..1096fa85e7 100644 --- a/drivers/common/mlx5/mlx5_common_private.h +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -39,6 +39,8 @@ bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, void mlx5_common_auxiliary_init(void); struct ibv_device *mlx5_get_aux_ibv_device( const struct rte_auxiliary_device *dev); +int mlx5_auxiliary_get_pci_str(const struct rte_auxiliary_device *dev, + char *addr, size_t size); #ifdef __cplusplus } diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index edfd3792e0..ea4c49b7e7 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -15,7 +15,7 @@ INTERNAL { mlx5_create_mr_ext; mlx5_dev_is_pci; # WINDOWS_NO_EXPORT - mlx5_dev_to_pci_addr; # WINDOWS_NO_EXPORT + mlx5_dev_to_pci_str; # WINDOWS_NO_EXPORT mlx5_devx_alloc_uar; # WINDOWS_NO_EXPORT @@ -75,6 +75,7 @@ INTERNAL { mlx5_free; mlx5_get_ifname_sysfs; # WINDOWS_NO_EXPORT + mlx5_get_pci_addr; # WINDOWS_NO_EXPORT mlx5_glue; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 54e4a1fe60..78e101d649 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1848,7 +1848,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, /* Process slave interface names in the loop. */ snprintf(tmp_str, sizeof(tmp_str), "/sys/class/net/%s", ifname); - if (mlx5_dev_to_pci_addr(tmp_str, &pci_addr)) { + if (mlx5_get_pci_addr(tmp_str, &pci_addr)) { DRV_LOG(WARNING, "can not get PCI address" " for netdev \"%s\"", ifname); continue; @@ -2025,8 +2025,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, break; } else { /* Bonding device not found. */ - if (mlx5_dev_to_pci_addr - (ibv_list[ret]->ibdev_path, &pci_addr)) + if (mlx5_get_pci_addr(ibv_list[ret]->ibdev_path, + &pci_addr)) continue; if (owner_pci.domain != pci_addr.domain || owner_pci.bus != pci_addr.bus || From patchwork Wed Jun 16 04:09:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94278 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B8183A0C45; Wed, 16 Jun 2021 06:10:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7C85341102; Wed, 16 Jun 2021 06:10:06 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2043.outbound.protection.outlook.com [40.107.220.43]) by mails.dpdk.org (Postfix) with ESMTP id 0F3F1410FD for ; Wed, 16 Jun 2021 06:10:03 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d42NGSstb0B46siXt7WJv97z8Nwc4FhcB7pmRXXfimScDPxY+QO/lnI3aAY9wf0vF1yWKhJqsMlyQOu3n8L++ulVYczkB4/7c3kWF3wsFvJwVcsw5/+ZU9/CaR0f3I3ORoLP73tiggw3589p9DG2nn1dyjkfOgzpovq/yNZu2vMuNNtdZPXs3PSzdy4p0yaol58a+beXUD2AxyKgSs5IyO8A/ww8WyuWD2lB6cxWBew2Wpl+uqf86AV+ltX5SA7FoRhP5ndCD6BJvqcf1ol0UBMLA7xNlMaOvkXV4UbjqPluX/o9vo/rHPmGM9hE29k4gUFGLVsMA4NZUHUm4VTJcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DdJ57eZIEK065CtW3OSjejTJrbZbxxU29o4Nt5QYddk=; b=DU4OLGqSsjr45rD5e95Vpp+aJ4odG0XRQRnqkYpWY4/fJ7MuboluuyhjYp155iUDX+lpgF5U+lbDVmpOUEr1jvu/zSe5ZW3FiJx/SVy8WqEn6NYZjkSQ1GwP0212HEkp6kZ6wQBF4lF5OMx2o6DUHuEdQv9jBnFzWNcRo1j7Y+QBI08dTM3w1Wn6ycWLLVD77ovnebGpeNvcw1Y4L31OLsqFHM0giIojgxULYWj7/0rkWPGrcc0kdMJIpnw8qQXC9Cka88eJYNf1fxNYtjZi5UW/otgsxyGbxhtPPDtKO28KnW/0fJQBHZh1lCG91ASYiYAa3cuf26/uxF3ZwGpS2g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DdJ57eZIEK065CtW3OSjejTJrbZbxxU29o4Nt5QYddk=; b=PdzTIzfJ/WUWLOrU6FPCHrzLaaNlHxP7ZdANV+LgaGY2aFcDL0rG0AK3C9ZXPcsmo1Cj3hK4qEpuXNDDh7LvPJhmkTmLe8EynGSvyq6qX9vN0V+xdN74kVTeWlcMKpVKC9idb5uy0xluCLYSrxcvqN1IFnZobzJZMLHO8EAoFkUHH3UFCfLW9qDi7FOcNnZRfN2m8scl2o3sTY9oHp/swYJZKCq+hdLcBRTfhlk+Un2RJg39nozj3vetLlF+7G72/vj69QRpyyoFLsGDpDMhnDT6IFtFZNSfl+guvRx2qJbFQr7zgqUw78eNYCbpPX3+FKSESt7ER61rSCUiPMKQEg== Received: from DM6PR06CA0077.namprd06.prod.outlook.com (2603:10b6:5:336::10) by BN9PR12MB5243.namprd12.prod.outlook.com (2603:10b6:408:100::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Wed, 16 Jun 2021 04:10:01 +0000 Received: from DM6NAM11FT031.eop-nam11.prod.protection.outlook.com (2603:10b6:5:336:cafe::ab) by DM6PR06CA0077.outlook.office365.com (2603:10b6:5:336::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Wed, 16 Jun 2021 04:10:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT031.mail.protection.outlook.com (10.13.172.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:00 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:09:59 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler Date: Wed, 16 Jun 2021 07:09:26 +0300 Message-ID: <20210616040935.311733-6-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e7d54e0d-78cd-4bab-dab0-08d9307c9c90 X-MS-TrafficTypeDiagnostic: BN9PR12MB5243: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:63; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: phbMBVHBzpGXN+lJiOkYuUJE/+RKL62mFRz3+2O2wlJKarpdXTbr9oxXhGI26KQvxEeLZqaFr0Veg+j/2IT1OetRU9pIPcEghWftSpWeQ9i18t0veGUIh6Gto7SCebNsmU4FjhxCmyhURB+e1pBpj9XAdOxe5UHqCDOZ6y3Z4XUrYrn0mgPVO81rx6UKquGQNewomWgkelAbo8H+GG9OMC0adrNKNs0OOTaEWLElwxIkI6l+TnDKMqcZ5zjQLxsIS+U9MzCNfGeio0Pa4HoLFC895sY0Se1wzfPHcCMtzTzE4mcaNTBWKDZu/rMrzmUnkzpj5ZPDGfYJ11+dGYbjKrdnqxHLoxtM4eTHNZR4RRDU0gQcWSqi1JIWaKeaerNrewbxulLJXJHSf1aOY7cG5Te2BDcnP/9QY1W5ckjZV6s7XWb36cIEtWh2NC4dP9VJ3E/XswdIaJAr1xqz37kTmQD56Dgs1weFkLsEXtx45QDDFtb2OZdKCRedDe+EoyixYjvpm4NUgwIlCtZiOsGhpHbWcr2ruxHS0eibVev1YBnQc2UW+DFGGeTxj3CVynROLesEY5bln0My7Tu9w0igP+Z7QFva4Kp97YcnCD6JUETPgNddDHmBPvt4tFzJZ5R+e2bN3EUtyr1pBKUeyhBaDQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(39860400002)(376002)(346002)(136003)(46966006)(36840700001)(16526019)(186003)(5660300002)(26005)(82310400003)(6862004)(55016002)(83380400001)(2616005)(7636003)(426003)(36906005)(107886003)(316002)(356005)(6286002)(82740400003)(8936002)(54906003)(6666004)(30864003)(37006003)(336012)(1076003)(478600001)(36756003)(70586007)(86362001)(36860700001)(2906002)(7696005)(6636002)(70206006)(47076005)(4326008)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:00.8713 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e7d54e0d-78cd-4bab-dab0-08d9307c9c90 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT031.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5243 Subject: [dpdk-dev] [PATCH v1 05/14] net/mlx5: remove PCI dependency X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To support more bus types, remove PCI dependency where possible. Signed-off-by: Xueming Li --- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 2 +- drivers/net/mlx5/linux/mlx5_os.c | 4 ++-- drivers/net/mlx5/mlx5.c | 24 +++++++++++++++--------- drivers/net/mlx5/mlx5.h | 8 ++++---- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 14 +++++++------- drivers/net/mlx5/mlx5_trigger.c | 12 +++++------- drivers/net/mlx5/mlx5_txq.c | 3 ++- drivers/net/mlx5/windows/mlx5_os.c | 2 +- 9 files changed, 38 insertions(+), 33 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index ddc1371aa9..b05b9fc950 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -346,7 +346,7 @@ mlx5_find_master_dev(struct rte_eth_dev *dev) priv = dev->data->dev_private; domain_id = priv->domain_id; MLX5_ASSERT(priv->representor); - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) { struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; if (opriv && diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 78e101d649..b695929e0b 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -923,6 +923,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, sh = mlx5_alloc_shared_dev_ctx(spawn, config); if (!sh) return NULL; + sh->numa_node = dpdk_dev->numa_node; config->devx = sh->devx; #ifdef HAVE_MLX5DV_DR_ACTION_DEST_DEVX_TIR config->dest_tir = 1; @@ -1125,7 +1126,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, * Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) { const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; @@ -2548,7 +2549,6 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn, int dbmap_env; int err = 0; - sh->numa_node = spawn->pci_dev->device.numa_node; pthread_mutex_init(&sh->txpp.mutex, NULL); /* * Configure environment variable "MLX5_BF_SHUT_UP" diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 52573e78f9..6077c701e7 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1185,7 +1185,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, */ err = mlx5_mr_btree_init(&sh->share_cache.cache, MLX5_MR_BTREE_CACHE_N * 2, - spawn->pci_dev->device.numa_node); + sh->numa_node); if (err) { err = rte_errno; goto error; @@ -1620,7 +1620,7 @@ mlx5_dev_close(struct rte_eth_dev *dev) unsigned int c = 0; uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) { struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; @@ -2057,7 +2057,8 @@ void mlx5_set_min_inline(struct mlx5_dev_spawn_data *spawn, struct mlx5_dev_config *config) { - if (config->txq_inline_min != MLX5_ARG_UNSET) { + if (config->txq_inline_min != MLX5_ARG_UNSET && + spawn->pci_dev != NULL) { /* Application defines size of inlined data explicitly. */ switch (spawn->pci_dev->id.device_id) { case PCI_DEVICE_ID_MELLANOX_CONNECTX4: @@ -2124,6 +2125,11 @@ mlx5_set_min_inline(struct mlx5_dev_spawn_data *spawn, } } } + if (spawn->pci_dev == NULL) { + if (config->txq_inline_min == MLX5_ARG_UNSET) + config->txq_inline_min = MLX5_INLINE_HSIZE_NONE; + goto exit; + } /* * We get here if we are unable to deduce * inline data size with DevX. Try PCI ID @@ -2258,7 +2264,7 @@ mlx5_dev_check_sibling_config(struct mlx5_priv *priv, if (sh->refcnt == 1) return 0; /* Find the device with shared context. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) { struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; @@ -2289,25 +2295,25 @@ mlx5_dev_check_sibling_config(struct mlx5_priv *priv, * * @param[in] port_id * port_id to start looking for device. - * @param[in] pci_dev - * Pointer to the hint PCI device. When device is being probed + * @param[in] odev + * Pointer to the hint device. When device is being probed * the its siblings (master and preceding representors might * not have assigned driver yet (because the mlx5_os_pci_probe() - * is not completed yet, for this case match on hint PCI + * is not completed yet, for this case match on hint * device may be used to detect sibling device. * * @return * port_id of found device, RTE_MAX_ETHPORT if not found. */ uint16_t -mlx5_eth_find_next(uint16_t port_id, struct rte_pci_device *pci_dev) +mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) { while (port_id < RTE_MAX_ETHPORTS) { struct rte_eth_dev *dev = &rte_eth_devices[port_id]; if (dev->state != RTE_ETH_DEV_UNUSED && dev->device && - (dev->device == &pci_dev->device || + (dev->device == odev || (dev->device->driver && dev->device->driver->name && !strcmp(dev->device->driver->name, MLX5_PCI_DRIVER_NAME)))) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 32b2817bf2..42de853167 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -1404,16 +1404,16 @@ int mlx5_proc_priv_init(struct rte_eth_dev *dev); void mlx5_proc_priv_uninit(struct rte_eth_dev *dev); int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev, struct rte_eth_udp_tunnel *udp_tunnel); -uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_pci_device *pci_dev); +uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev); int mlx5_dev_close(struct rte_eth_dev *dev); bool mlx5_is_hpf(struct rte_eth_dev *dev); void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh); /* Macro to iterate over all valid ports for mlx5 driver. */ -#define MLX5_ETH_FOREACH_DEV(port_id, pci_dev) \ - for (port_id = mlx5_eth_find_next(0, pci_dev); \ +#define MLX5_ETH_FOREACH_DEV(port_id, dev) \ + for (port_id = mlx5_eth_find_next(0, dev); \ port_id < RTE_MAX_ETHPORTS; \ - port_id = mlx5_eth_find_next(port_id + 1, pci_dev)) + port_id = mlx5_eth_find_next(port_id + 1, dev)) int mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs); struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 90baee5aa4..3047b921a9 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -335,7 +335,7 @@ mlx5_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *info) if (priv->representor) { uint16_t port_id; - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, dev->device) { struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c index 0c5403e493..e6324c22c5 100644 --- a/drivers/net/mlx5/mlx5_mr.c +++ b/drivers/net/mlx5/mlx5_mr.c @@ -285,23 +285,23 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, } /** - * Finds the first ethdev that match the pci device. + * Finds the first ethdev that match the device. * The existence of multiple ethdev per pci device is only with representors. * On such case, it is enough to get only one of the ports as they all share * the same ibv context. * - * @param pdev - * Pointer to the PCI device. + * @param dev + * Pointer to the device. * * @return * Pointer to the ethdev if found, NULL otherwise. */ static struct rte_eth_dev * -pci_dev_to_eth_dev(struct rte_pci_device *pdev) +dev_to_eth_dev(struct rte_device *dev) { uint16_t port_id; - port_id = rte_eth_find_next_of(0, &pdev->device); + port_id = rte_eth_find_next_of(0, dev); if (port_id == RTE_MAX_ETHPORTS) return NULL; return &rte_eth_devices[port_id]; @@ -331,7 +331,7 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, struct mlx5_priv *priv; struct mlx5_dev_ctx_shared *sh; - dev = pci_dev_to_eth_dev(pdev); + dev = dev_to_eth_dev(&pdev->device); if (!dev) { DRV_LOG(WARNING, "unable to find matching ethdev " "to PCI device %p", (void *)pdev); @@ -381,7 +381,7 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, struct mlx5_mr *mr; struct mr_cache_entry entry; - dev = pci_dev_to_eth_dev(pdev); + dev = dev_to_eth_dev(&pdev->device); if (!dev) { DRV_LOG(WARNING, "unable to find matching ethdev " "to PCI device %p", (void *)pdev); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index ae7fcca229..4ff12eac19 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -697,7 +697,7 @@ mlx5_hairpin_bind_single_port(struct rte_eth_dev *dev, uint16_t rx_port) uint32_t explicit; uint16_t rx_queue; - if (mlx5_eth_find_next(rx_port, priv->pci_dev) != rx_port) { + if (mlx5_eth_find_next(rx_port, dev->device) != rx_port) { rte_errno = ENODEV; DRV_LOG(ERR, "Rx port %u does not belong to mlx5", rx_port); return -rte_errno; @@ -835,7 +835,7 @@ mlx5_hairpin_unbind_single_port(struct rte_eth_dev *dev, uint16_t rx_port) int ret; uint16_t cur_port = priv->dev_data->port_id; - if (mlx5_eth_find_next(rx_port, priv->pci_dev) != rx_port) { + if (mlx5_eth_find_next(rx_port, dev->device) != rx_port) { rte_errno = ENODEV; DRV_LOG(ERR, "Rx port %u does not belong to mlx5", rx_port); return -rte_errno; @@ -893,7 +893,6 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) { int ret = 0; uint16_t p, pp; - struct mlx5_priv *priv = dev->data->dev_private; /* * If the Rx port has no hairpin configuration with the current port, @@ -902,7 +901,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) * information updating. */ if (rx_port == RTE_MAX_ETHPORTS) { - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret = mlx5_hairpin_bind_single_port(dev, p); if (ret != 0) goto unbind; @@ -912,7 +911,7 @@ mlx5_hairpin_bind(struct rte_eth_dev *dev, uint16_t rx_port) return mlx5_hairpin_bind_single_port(dev, rx_port); } unbind: - MLX5_ETH_FOREACH_DEV(pp, priv->pci_dev) + MLX5_ETH_FOREACH_DEV(pp, dev->device) if (pp < p) mlx5_hairpin_unbind_single_port(dev, pp); return ret; @@ -927,10 +926,9 @@ mlx5_hairpin_unbind(struct rte_eth_dev *dev, uint16_t rx_port) { int ret = 0; uint16_t p; - struct mlx5_priv *priv = dev->data->dev_private; if (rx_port == RTE_MAX_ETHPORTS) - MLX5_ETH_FOREACH_DEV(p, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(p, dev->device) { ret = mlx5_hairpin_unbind_single_port(dev, p); if (ret != 0) return ret; diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 3e5e94444b..11770aeeef 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -816,7 +817,7 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline = #if defined(RTE_ARCH_ARM64) - (priv->pci_dev->id.device_id == + (priv->pci_dev && priv->pci_dev->id.device_id == PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? MLX5_INLINE_MAX_TXQS_BLUEFIELD : #endif diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c index 3fe3f55f49..d4de2adfc1 100644 --- a/drivers/net/mlx5/windows/mlx5_os.c +++ b/drivers/net/mlx5/windows/mlx5_os.c @@ -393,7 +393,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, * Look for sibling devices in order to reuse their switch domain * if any, otherwise allocate one. */ - MLX5_ETH_FOREACH_DEV(port_id, priv->pci_dev) { + MLX5_ETH_FOREACH_DEV(port_id, NULL) { const struct mlx5_priv *opriv = rte_eth_devices[port_id].data->dev_private; From patchwork Wed Jun 16 04:09:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94279 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1AD44A0C45; Wed, 16 Jun 2021 06:10:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 931AE4111E; Wed, 16 Jun 2021 06:10:07 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2070.outbound.protection.outlook.com [40.107.223.70]) by mails.dpdk.org (Postfix) with ESMTP id AAE4F410E8 for ; Wed, 16 Jun 2021 06:10:05 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dNxkAKazC1Sr1PhGZ+j1+nbkEq0n6KTmfOqGd2gG+/yeYh+3UcQ24BYSuN4l8psTCkBD3EpbqzZaLt9sn5gcNsqo6at+Sqo7xHjWTKcwW3nFMfo/1gPEb7m4aoToBB2xcmrcW/Z2fC5AuBVe1y/KAOAhfKuEw2w/Qg3OCMaaiKzBVgjv7Z6inn63bpw/11+6utst4Cgk9mX6vMVU/grWh14Ami035AJB24S1SaDc/ndGqau2wZ/Ld0eh2nIURMHLCw06Qncy+cUsx3Q+sOfLnMBtb7JlKTHaullrvH2odliwXbiekbr05zCk9HwAxw5MaJuJgaxsOBRRKGoBet3MwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qedLy7jH/Vr67kmod5SptQ5bSXhpz1NRriW343D2dl4=; b=fdUeo5LhxHfdsn3E71wbFQNpU3WEPRXOZlo2cdldXGA7G4HxChsxEw5VVQqWCerRW1eYhHfnCn/VmPP4PbvNATsnIk/fU7RV7RefnHyzyovBXbEaxnwS9+KE3IqWDzdXxKaGdyawSGGouvHKw9cOK6a9SXCwfnjO686FhHyae5hIz4mA3F9TApEOI8eJNJn/dxDyy1JaAprZTM72fo873G5OKOt53GVr9/lcHy4Lqs5IIiP6L6+Wnl3BotI+v6kswauoBH0ltd+nzn/zMpg2X6a9paPpqd4c5zhsddTL54UKt33hoHDg+WIcY9ht99VRNvw9VN31heIMrqgy3O+m6Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qedLy7jH/Vr67kmod5SptQ5bSXhpz1NRriW343D2dl4=; b=OUKjwUN/+aMBc2w9vMn99dHaLTV6hNJQVyDLem/cuUQx2ykhsfR1S2LxjljjccTrmQYdAQdv4YJv9EB5BNYPTvICqZ95p3hbkERkPL0HhEXZdlWhFhyEqTzYZA2gUmYDgYmV0NquwU4wlx1mn6QvaNcoYnRhHkbqzD2ZweyP13xzWiKfA0pp6CPPafxdi26B40ytm3GkZa44kF0vNJmOtgEou93o8hJ0cxttFZaUuy0jcgWa40zZJaXLvCJxEaMfWvmGzgMbhqT3tgJDFCb8wFt/t5kvpD9fdUxSrdvj1fNhvq1hkFuC+h7wmz7kvnZ7l2EUGqfX2gXMSA11DkMxKA== Received: from DM6PR03CA0012.namprd03.prod.outlook.com (2603:10b6:5:40::25) by DM5PR12MB1914.namprd12.prod.outlook.com (2603:10b6:3:109::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Wed, 16 Jun 2021 04:10:04 +0000 Received: from DM6NAM11FT003.eop-nam11.prod.protection.outlook.com (2603:10b6:5:40:cafe::15) by DM6PR03CA0012.outlook.office365.com (2603:10b6:5:40::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.18 via Frontend Transport; Wed, 16 Jun 2021 04:10:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT003.mail.protection.outlook.com (10.13.173.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:02 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:00 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Anatoly Burakov Date: Wed, 16 Jun 2021 07:09:27 +0300 Message-ID: <20210616040935.311733-7-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 21b6f074-717b-4296-8ade-08d9307c9dbf X-MS-TrafficTypeDiagnostic: DM5PR12MB1914: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:30; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3iZaj0dA5MPOesb0IU8i4Wf7MNarHZb1tr+Vbcb+UxWLevGugP4+dMCwf7d32fdh/2GjSjt4UYaurwrq54BNpnLpkOhPvI7Uz54WAcH9rPOHyk1N/HSJxRfqE4LiP1NfD/ybnaoYMMlPNONtsCghEDqxhmhWDuDQGr2cXpS5UGTFODW2OdlakgYdeHU+HaffN+82pVT8kiYHwpFl4Zy69N1mzC5nv1Wy2MdrWt1ZULyEKSXS2P7WTw2bpcZc1kXyKS0HN+4ze4q6psabIFIFyIS7hU250YlgO7WdxMUVo5UZsTtqdc4boJZc5BaWuNZieWi3/L9lR3l0QxfnjGQDe3EiTPaovjyt+fs5SV+Oj4+aPveLCJ6Rb/AvrCRfte+78cQUylEp/E+iiMOzb+/U5vvt/q9JDSwK8kjsStjJTET07cpOvSifo4+WvF7XXTWgoKbwwiLObkEiOXjsfiSbV+Ju0bAZjstGV2HQMGX7MpsIZGsFhNuZHk66Pu8VcOlmeN4Kkjj03A2wEQv1IEwFWh2Eq3uEU6fVJoNTZauOEGqDbruWD78v410qI5FAAAP8iWfqw+ywO5hgXxnKFc7VZ29mqma3Het8DqejzfqGBTMWo/ECMmEIxEwZUsGSld/SMPgrA8wjIk6A5DETaAldnyejcjoWjxwxgGeVrCVEeTw= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(136003)(346002)(376002)(39860400002)(46966006)(36840700001)(86362001)(7636003)(70586007)(356005)(55016002)(70206006)(8936002)(36756003)(83380400001)(36860700001)(82740400003)(47076005)(82310400003)(8676002)(2616005)(336012)(426003)(4326008)(6862004)(6286002)(36906005)(30864003)(7696005)(5660300002)(1076003)(54906003)(6666004)(26005)(478600001)(6636002)(186003)(16526019)(37006003)(316002)(2906002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:02.8860 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 21b6f074-717b-4296-8ade-08d9307c9dbf X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT003.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1914 Subject: [dpdk-dev] [PATCH v1 06/14] net/mlx5: migrate to bus-agnostic common driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To support SubFunction based on auxiliary bus, common driver supports new bus-agnostic driver. This patch migrates net driver to new common driver. Signed-off-by: Xueming Li --- drivers/net/mlx5/linux/mlx5_os.c | 46 ++++++++++++++++++++++---------- drivers/net/mlx5/linux/mlx5_os.h | 3 --- drivers/net/mlx5/mlx5.c | 40 +++++++++++++-------------- drivers/net/mlx5/mlx5.h | 3 +-- drivers/net/mlx5/mlx5_mr.c | 38 +++++++++++++------------- drivers/net/mlx5/mlx5_rxtx.h | 9 +++---- 6 files changed, 73 insertions(+), 66 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index b695929e0b..a941ecf641 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1969,14 +1969,6 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, struct mlx5_bond_info bond_info; int ret = -1; - if (rte_eal_process_type() == RTE_PROC_PRIMARY) - mlx5_pmd_socket_init(); - ret = mlx5_init_once(); - if (ret) { - DRV_LOG(ERR, "unable to init PMD global data: %s", - strerror(rte_errno)); - return -rte_errno; - } errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { @@ -2409,21 +2401,18 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, } /** - * DPDK callback to register a PCI device. + * Callback to register a PCI device. * * This function spawns Ethernet devices out of a given PCI device. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). * @param[in] pci_dev * PCI device information. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ -int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +static int +mlx5_os_pci_probe(struct rte_pci_device *pci_dev) { struct rte_eth_devargs eth_da = { .type = RTE_ETH_REPRESENTOR_NONE }; int ret = 0; @@ -2462,6 +2451,35 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, return ret; } +/** + * Common bus driver callback to probe a device. + * + * This function probe PCI bus device(s). + * + * @param[in] dev + * Pointer to the generic device. + * + * @return + * 0 on success, the function cannot fail. + */ +int +mlx5_os_net_probe(struct rte_device *dev) +{ + int ret; + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + mlx5_pmd_socket_init(); + ret = mlx5_init_once(); + if (ret) { + DRV_LOG(ERR, "unable to init PMD global data: %s", + strerror(rte_errno)); + return -rte_errno; + } + if (mlx5_dev_is_pci(dev)) + return mlx5_os_pci_probe(RTE_DEV_TO_PCI(dev)); + return 0; +} + static int mlx5_config_doorbell_mapping_env(const struct mlx5_dev_config *config) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 4ae7d0ef47..af7cbeb418 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -19,7 +19,4 @@ enum { #define MLX5_NAMESIZE IF_NAMESIZE -#define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ - RTE_PCI_DRV_INTR_RMV | \ - RTE_PCI_DRV_PROBE_AGAIN) #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6077c701e7..c9474a6e74 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -12,7 +12,6 @@ #include #include -#include #include #include #include @@ -28,7 +27,6 @@ #include #include #include -#include #include #include "mlx5_defs.h" @@ -2326,23 +2324,23 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) } /** - * DPDK callback to remove a PCI device. + * Callback to remove a device. * - * This function removes all Ethernet devices belong to a given PCI device. + * This function removes all Ethernet devices belong to a given device. * - * @param[in] pci_dev - * Pointer to the PCI device. + * @param[in] dev + * Pointer to the generic device. * * @return * 0 on success, the function cannot fail. */ static int -mlx5_pci_remove(struct rte_pci_device *pci_dev) +mlx5_net_remove(struct rte_device *dev) { uint16_t port_id; int ret = 0; - RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device) { + RTE_ETH_FOREACH_DEV_OF(port_id, dev) { /* * mlx5_dev_close() is not registered to secondary process, * call the close function explicitly for secondary process. @@ -2433,19 +2431,17 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_driver = { - .driver_class = MLX5_CLASS_ETH, - .pci_driver = { - .driver = { - .name = MLX5_PCI_DRIVER_NAME, - }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, - }, +static struct mlx5_class_driver mlx5_net_driver = { + .drv_class = MLX5_CLASS_ETH, + .name = "mlx5_eth", + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_net_probe, + .remove = mlx5_net_remove, + .dma_map = mlx5_net_dma_map, + .dma_unmap = mlx5_net_dma_unmap, + .probe_again = 1, + .intr_lsc = 1, + .intr_rmv = 1, }; /* Initialize driver log type. */ @@ -2463,7 +2459,7 @@ RTE_INIT(rte_mlx5_pmd_init) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - mlx5_pci_driver_register(&mlx5_driver); + mlx5_class_driver_register(&mlx5_net_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 42de853167..5ea465fa0b 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -1680,8 +1680,7 @@ int mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn, const struct mlx5_dev_config *config, struct mlx5_dev_ctx_shared *sh); int mlx5_os_get_pdn(void *pd, uint32_t *pdn); -int mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev); +int mlx5_os_net_probe(struct rte_device *dev); void mlx5_os_dev_shared_handler_install(struct mlx5_dev_ctx_shared *sh); void mlx5_os_dev_shared_handler_uninstall(struct mlx5_dev_ctx_shared *sh); void mlx5_os_set_reg_mr_cb(mlx5_reg_mr_t *reg_mr_cb, diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c index e6324c22c5..fab5470ba2 100644 --- a/drivers/net/mlx5/mlx5_mr.c +++ b/drivers/net/mlx5/mlx5_mr.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include @@ -308,10 +307,10 @@ dev_to_eth_dev(struct rte_device *dev) } /** - * DPDK callback to DMA map external memory to a PCI device. + * Callback to DMA map external memory to a device. * - * @param pdev - * Pointer to the PCI device. + * @param rte_dev + * Pointer to the generic device. * @param addr * Starting virtual address of memory to be mapped. * @param iova @@ -323,18 +322,18 @@ dev_to_eth_dev(struct rte_device *dev) * 0 on success, negative value on error. */ int -mlx5_dma_map(struct rte_pci_device *pdev, void *addr, - uint64_t iova __rte_unused, size_t len) +mlx5_net_dma_map(struct rte_device *rte_dev, void *addr, + uint64_t iova __rte_unused, size_t len) { struct rte_eth_dev *dev; struct mlx5_mr *mr; struct mlx5_priv *priv; struct mlx5_dev_ctx_shared *sh; - dev = dev_to_eth_dev(&pdev->device); + dev = dev_to_eth_dev(rte_dev); if (!dev) { DRV_LOG(WARNING, "unable to find matching ethdev " - "to PCI device %p", (void *)pdev); + "to device %s", rte_dev->name); rte_errno = ENODEV; return -1; } @@ -357,10 +356,10 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, } /** - * DPDK callback to DMA unmap external memory to a PCI device. + * Callback to DMA unmap external memory to a device. * - * @param pdev - * Pointer to the PCI device. + * @param rte_dev + * Pointer to the generic device. * @param addr * Starting virtual address of memory to be unmapped. * @param iova @@ -372,8 +371,8 @@ mlx5_dma_map(struct rte_pci_device *pdev, void *addr, * 0 on success, negative value on error. */ int -mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, - uint64_t iova __rte_unused, size_t len __rte_unused) +mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr, + uint64_t iova __rte_unused, size_t len __rte_unused) { struct rte_eth_dev *dev; struct mlx5_priv *priv; @@ -381,10 +380,10 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, struct mlx5_mr *mr; struct mr_cache_entry entry; - dev = dev_to_eth_dev(&pdev->device); + dev = dev_to_eth_dev(rte_dev); if (!dev) { - DRV_LOG(WARNING, "unable to find matching ethdev " - "to PCI device %p", (void *)pdev); + DRV_LOG(WARNING, "unable to find matching ethdev to device %s", + rte_dev->name); rte_errno = ENODEV; return -1; } @@ -394,16 +393,15 @@ mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, mr = mlx5_mr_lookup_list(&sh->share_cache, &entry, (uintptr_t)addr); if (!mr) { rte_rwlock_read_unlock(&sh->share_cache.rwlock); - DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered " - "to PCI device %p", (uintptr_t)addr, - (void *)pdev); + DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered to device %s", + (uintptr_t)addr, rte_dev->name); rte_errno = EINVAL; return -1; } LIST_REMOVE(mr, mr); - mlx5_mr_free(mr, sh->share_cache.dereg_mr_cb); DRV_LOG(DEBUG, "port %u remove MR(%p) from list", dev->data->port_id, (void *)mr); + mlx5_mr_free(mr, sh->share_cache.dereg_mr_cb); mlx5_mr_rebuild_cache(&sh->share_cache); /* * No explicit wmb is needed after updating dev_gen due to diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index e168dd46f9..ad1144e218 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -16,7 +16,6 @@ #include #include #include -#include #include #include @@ -48,10 +47,10 @@ int mlx5_queue_state_modify(struct rte_eth_dev *dev, /* mlx5_mr.c */ void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl); -int mlx5_dma_map(struct rte_pci_device *pdev, void *addr, uint64_t iova, - size_t len); -int mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, uint64_t iova, - size_t len); +int mlx5_net_dma_map(struct rte_device *rte_dev, void *addr, uint64_t iova, + size_t len); +int mlx5_net_dma_unmap(struct rte_device *rte_dev, void *addr, uint64_t iova, + size_t len); /** * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from which the From patchwork Wed Jun 16 04:09:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94280 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 50D4BA0C45; Wed, 16 Jun 2021 06:10:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1F11C41137; Wed, 16 Jun 2021 06:10:09 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2045.outbound.protection.outlook.com [40.107.92.45]) by mails.dpdk.org (Postfix) with ESMTP id E94FF4111E for ; Wed, 16 Jun 2021 06:10:06 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lRpOE5eUQzR2QakmlX5D0gmY9rko+ldeSCVlY5lvZ4LSld9JnxlQwMTDYxBGUwOFgGfHIAFplMXPIPrOg4LBOmMVEGhuPS9n+BJ3EfPj8wT9n3Yk7agt23Kq9O5zt31UakHmHIfmWnrwB5PUWFuiReUXNN9VRbL6WjN5ZI74Hhx9HL4dwuIQrWBRIFYcy5qTvllz9Huv7BgKoWv4e6TtB/zEY4AqCBwzH2lb6nLmj0hydHXMYl3P0/U8YIK5RpAipHzZgu/Tvys6F+kfOM87S8K+v+Vm5ZSjAM2ir5coUbBpBI3cc/7lBAEJD3ggkae6Tl0qESiGvjUoFpWg/YeZmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JUk1EwW6JSigkY11kduVSA5b+CluUgp0OMkTVBUjsXo=; b=LXyWxLexkU1HHAebaz3ySCv46vrdNZeKVceQbOcA8+UofHUkmWQ9ibC5WuKM1e4XM8JnL5LcDS59nfFcOVVcalbv6b/JyS6AY1TIW3RRCmlxDjo5EBhgrvItr8uThR2AGSmgKj3BSDfreVAKBbIFpCOgDpTABKLsYzjjc+2hn3EA1QHK9q2XUz8+wUNCHUXbl+N8Qi3DKWztaKh3uIO4kwoSe6CX1MlTLV9wr5Y23ZuNVhqqt3lnkFfH8BqEXfYZ2Bq5kQnTmXbK7MTZ53hqWIn/+bYIZeCE/ddtcQmTkR0L09YKvzLgkhoOzJxbg+kXMomyyKOl6uIDh2jc1cgZ4A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JUk1EwW6JSigkY11kduVSA5b+CluUgp0OMkTVBUjsXo=; b=YWsAo9SYf4ZEvjHBgcOZFDtnMtKqRomkIzYqUQYarOyvz/ABxodluZbf8E7NTAz8lDdquGDUx0hriiRgI3A39grZLc52GkZSm4WkUZJBmotjhBrzyw3W8lXK9hS0qUg7TIv+LGDSPUxtr9K7vLhzYzDbo3kPisBwklgyqKoyPrJvvN9z8LDY9MQCdJc+zMy67/l7vovyWAUv1355v2/EVg9qDlbdRcnEzDX7S5m9cBcj3WPT7MDn+3p2i+G9JIQwqqZnXl2FPeMcM8ePbHr4Iw3yyn9hxjTxG0BQqKxVyTgoVb1Z2C+GnV9mGT6FvGJfUtG4GstQSweKTZE7ozSF8Q== Received: from DM6PR03CA0017.namprd03.prod.outlook.com (2603:10b6:5:40::30) by CH2PR12MB4859.namprd12.prod.outlook.com (2603:10b6:610:62::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.21; Wed, 16 Jun 2021 04:10:05 +0000 Received: from DM6NAM11FT003.eop-nam11.prod.protection.outlook.com (2603:10b6:5:40:cafe::2f) by DM6PR03CA0017.outlook.office365.com (2603:10b6:5:40::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT003.mail.protection.outlook.com (10.13.173.162) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:05 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:02 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Anatoly Burakov Date: Wed, 16 Jun 2021 07:09:28 +0300 Message-ID: <20210616040935.311733-8-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7c88c1fb-ba53-40d9-5cdd-08d9307c9f28 X-MS-TrafficTypeDiagnostic: CH2PR12MB4859: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:199; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 64/OrraDgkRoIfyw0BELLpInQ9UdW99CTAZkk/q7PebkmFsR8IgUjsYHVuqQLFd5AnB5ntHjfdegnPVRE4m59ldD/P8fFwD7OM/3MuG0xfg+EDUKoVT2hq7SdfZe/hQMeFezXKJMgPx3fpy6ieBM6Vc5ULjHBCCkAjUY325Kv69/aPH2AslGyyuiDvZTp72yxQrBK+r8pky+DA4e2T7mxsdTfrgge09uRofwcRSalAfDIgp/QHtu3tcHVaUjg3Ab57ps2wDHW1D7Jim76+xhEWKTV29EHDHRRDGROau9EBMsGU+TSTeBeU2VFgXtSj6r3rEKLpSDZVpxBZyxm4PAiLsd6EyBNu4cAXubzJNNjfaGyN0CXz/V3klSqfV3FoE4vftf2Dx5/NthrS/3RPJ5zld+Eo9bAqrngVE19eaBtYWamwAWzwSYop0RBPG3pSVl62ccwWVdN84miSOc1xLWZcmJQtBg9ko7H7TYAZHWOwHw532JRTMr6bI54yxv5flVJHR3+cCU5hZXRG28G0DtSDf4OZgxj1EzZfkGzLS0inq7QFp+XYXs7Wn3+4aQl1yQfIpdHf7qxycHmZOptllLIabeoXSAK82+8nKmL9HsbNA5zAftAH/hyR236s0eBuYHaPiJ1FQkgKDx+XbFexTi53JeKqhLw9tOAR0YcQcuA9M= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(396003)(136003)(376002)(346002)(46966006)(36840700001)(86362001)(7696005)(5660300002)(6286002)(336012)(70586007)(70206006)(36756003)(478600001)(8936002)(6636002)(83380400001)(47076005)(6862004)(2616005)(36906005)(1076003)(30864003)(82310400003)(26005)(316002)(6666004)(55016002)(8676002)(2906002)(426003)(19627235002)(54906003)(356005)(37006003)(36860700001)(4326008)(7636003)(82740400003)(186003)(16526019); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:05.2767 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c88c1fb-ba53-40d9-5cdd-08d9307c9f28 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT003.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4859 Subject: [dpdk-dev] [PATCH v1 07/14] net/mlx5: support SubFunction X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch introduces SF support. Similar to VF, SF on auxiliary bus is a portion of hardware PF, no representor or bonding parameters for SF. Devargs to support SF: -a auxiliary:mlx5_core.sf.8,dv_flow_en=1 New global syntax to support SF: -a bus=auxiliary,name=mlx5_core.sf.8/class=eth/driver=mlx5,dv_flow_en=1 Signed-off-by: Xueming Li --- doc/guides/nics/mlx5.rst | 54 +++++---- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 12 +- drivers/net/mlx5/linux/mlx5_os.c | 142 +++++++++++++++++------- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 23 +++- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_mac.c | 2 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_trigger.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 12 +- 10 files changed, 185 insertions(+), 74 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 83299646dd..3b217d9e8e 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -110,6 +110,11 @@ Features - Flow integrity offload API. - Connection tracking. - Sub-Function representors. +- Sub-Function. + +Limitations +----------- + Limitations ----------- @@ -1438,40 +1443,51 @@ the DPDK application. echo switchdev > /sys/class/net//compat/devlink/mode -Sub-Function representor ------------------------- +Sub-Function support +-------------------- Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues(txq, rxq). A SF netdev supports E-Switch representation -offload similar to existing PF and VF representors. A SF shares PCI -level resources with other SFs and/or with its parent PCI function. +dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs +and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 1. Configure SF feature:: - mlxconfig -d set PF_BAR2_SIZE=<0/1/2/3> PF_BAR2_ENABLE=1 + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - Value of PF_BAR2_SIZE: +2. Enable switchdev mode:: - 0: 8 SFs - 1: 16 SFs - 2: 32 SFs - 3: 64 SFs + mlxdevm dev eswitch set pci/ mode switchdev -2. Reset the FW:: +3. Add SF port:: - mlxfwreset -d reset + mlxdevm port add pci/ flavour pcisf pfnum 0 sfnum -3. Enable switchdev mode:: + Get SFID from output: pci// - echo switchdev > /sys/class/net//compat/devlink/mode +4. Modify MAC address:: + + mlxdevm port function set pci// hw_addr + +5. Activate SF port:: + + mlxdevm port function set pci// state active + +6. Devargs to probe SF device:: -4. Create SF:: + auxiliary:mlx5_core.sf.,dv_flow_en=1 - mlnx-sf -d -a create +Sub-Function representor support +-------------------------------- -5. Probe SF representor:: +A SF netdev supports E-Switch representation offload similar to existing PF +and VF representors. Use to probe SF representor. - testpmd> port attach ,representor=sf0,dv_flow_en=1 + testpmd> port attach ,representor=sf,dv_flow_en=1 Performance tuning ------------------ diff --git a/drivers/net/mlx5/linux/mlx5_ethdev_os.c b/drivers/net/mlx5/linux/mlx5_ethdev_os.c index b05b9fc950..f34133e2c6 100644 --- a/drivers/net/mlx5/linux/mlx5_ethdev_os.c +++ b/drivers/net/mlx5/linux/mlx5_ethdev_os.c @@ -128,6 +128,17 @@ struct ethtool_link_settings { #define ETHTOOL_LINK_MODE_200000baseCR4_Full_BIT 2 /* 66 - 64 */ #endif +/* Get interface index from SubFunction device name. */ +int +mlx5_auxiliary_get_ifindex(const char *sf_name) +{ + char if_name[IF_NAMESIZE] = { 0 }; + + if (mlx5_auxiliary_get_child_name(sf_name, "/net", + if_name, sizeof(if_name)) != 0) + return -rte_errno; + return if_nametoindex(if_name); +} /** * Get interface name from private structure. @@ -1619,4 +1630,3 @@ mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[RTE_ETHER_ADDR_LEN]) memcpy(mac, request.ifr_hwaddr.sa_data, RTE_ETHER_ADDR_LEN); return 0; } - diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index a941ecf641..47df3b92f8 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1915,6 +1916,27 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, return pf; } +static void +mlx5_os_config_default(struct mlx5_dev_config *config) +{ + memset(config, 0, sizeof(*config)); + config->mps = MLX5_ARG_UNSET; + config->dbnc = MLX5_ARG_UNSET; + config->rx_vec_en = 1; + config->txq_inline_max = MLX5_ARG_UNSET; + config->txq_inline_min = MLX5_ARG_UNSET; + config->txq_inline_mpw = MLX5_ARG_UNSET; + config->txqs_inline = MLX5_ARG_UNSET; + config->vf_nl_en = 1; + config->mr_ext_memseg_en = 1; + config->mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN; + config->mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS; + config->dv_esw_en = 1; + config->dv_flow_en = 1; + config->decap_en = 1; + config->log_hp_size = MLX5_ARG_UNSET; +} + /** * Register a PCI device within bonding. * @@ -2327,23 +2349,8 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, uint32_t restore; /* Default configuration. */ - memset(&dev_config, 0, sizeof(struct mlx5_dev_config)); + mlx5_os_config_default(&dev_config); dev_config.vf = dev_config_vf; - dev_config.mps = MLX5_ARG_UNSET; - dev_config.dbnc = MLX5_ARG_UNSET; - dev_config.rx_vec_en = 1; - dev_config.txq_inline_max = MLX5_ARG_UNSET; - dev_config.txq_inline_min = MLX5_ARG_UNSET; - dev_config.txq_inline_mpw = MLX5_ARG_UNSET; - dev_config.txqs_inline = MLX5_ARG_UNSET; - dev_config.vf_nl_en = 1; - dev_config.mr_ext_memseg_en = 1; - dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN; - dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS; - dev_config.dv_esw_en = 1; - dev_config.dv_flow_en = 1; - dev_config.decap_en = 1; - dev_config.log_hp_size = MLX5_ARG_UNSET; list[i].eth_dev = mlx5_dev_spawn(&pci_dev->device, &list[i], &dev_config, @@ -2400,6 +2407,35 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, return ret; } +static int +mlx5_os_parse_eth_devargs(struct rte_device *dev, + struct rte_eth_devargs *eth_da) +{ + int ret = 0; + + if (dev->devargs == NULL) + return 0; + memset(eth_da, 0, sizeof(*eth_da)); + /* Parse representor information first from class argument. */ + if (dev->devargs->cls_str) + ret = rte_eth_devargs_parse(dev->devargs->cls_str, eth_da); + if (ret != 0) { + DRV_LOG(ERR, "failed to parse device arguments: %s", + dev->devargs->cls_str); + return -rte_errno; + } + if (eth_da->type == RTE_ETH_REPRESENTOR_NONE) { + /* Parse legacy device argument */ + ret = rte_eth_devargs_parse(dev->devargs->args, eth_da); + if (ret) { + DRV_LOG(ERR, "failed to parse device arguments: %s", + dev->devargs->args); + return -rte_errno; + } + } + return 0; +} + /** * Callback to register a PCI device. * @@ -2414,31 +2450,13 @@ mlx5_os_pci_probe_pf(struct rte_pci_device *pci_dev, static int mlx5_os_pci_probe(struct rte_pci_device *pci_dev) { - struct rte_eth_devargs eth_da = { .type = RTE_ETH_REPRESENTOR_NONE }; + struct rte_eth_devargs eth_da = { .nb_ports = 0 }; int ret = 0; uint16_t p; - if (pci_dev->device.devargs) { - /* Parse representor information from device argument. */ - if (pci_dev->device.devargs->cls_str) - ret = rte_eth_devargs_parse - (pci_dev->device.devargs->cls_str, ð_da); - if (ret) { - DRV_LOG(ERR, "failed to parse device arguments: %s", - pci_dev->device.devargs->cls_str); - return -rte_errno; - } - if (eth_da.type == RTE_ETH_REPRESENTOR_NONE) { - /* Support legacy device argument */ - ret = rte_eth_devargs_parse - (pci_dev->device.devargs->args, ð_da); - if (ret) { - DRV_LOG(ERR, "failed to parse device arguments: %s", - pci_dev->device.devargs->args); - return -rte_errno; - } - } - } + ret = mlx5_os_parse_eth_devargs(&pci_dev->device, ð_da); + if (ret != 0) + return ret; if (eth_da.nb_ports > 0) { /* Iterate all port if devargs pf is range: "pf[0-1]vf[...]". */ @@ -2451,10 +2469,53 @@ mlx5_os_pci_probe(struct rte_pci_device *pci_dev) return ret; } +/* Probe a single SF device on auxiliary bus, no representor support. */ +static int +mlx5_os_auxiliary_probe(struct rte_device *dev) +{ + struct rte_eth_devargs eth_da = { .nb_ports = 0 }; + struct mlx5_dev_config config; + struct mlx5_dev_spawn_data spawn = { .pf_bond = -1 }; + struct rte_auxiliary_device *adev = RTE_DEV_TO_AUXILIARY(dev); + struct rte_eth_dev *eth_dev; + int ret = 0; + + /* Parse ethdev devargs. */ + ret = mlx5_os_parse_eth_devargs(dev, ð_da); + if (ret != 0) + return ret; + /* Set default config data. */ + mlx5_os_config_default(&config); + config.sf = 1; + /* Init spawn data. */ + spawn.max_port = 1; + spawn.phys_port = 1; + spawn.phys_dev = mlx5_os_get_ibv_dev(dev); + ret = mlx5_auxiliary_get_ifindex(dev->name); + if (ret < 0) { + DRV_LOG(ERR, "failed to get ethdev ifindex: %s", dev->name); + return ret; + } + spawn.ifindex = ret; + /* Spawn device. */ + eth_dev = mlx5_dev_spawn(dev, &spawn, &config, ð_da); + if (eth_dev == NULL) + return -rte_errno; + /* Post create. */ + eth_dev->intr_handle = &adev->intr_handle; + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { + eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_LSC; + eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_RMV; + eth_dev->data->numa_node = dev->numa_node; + } + rte_eth_dev_probing_finish(eth_dev); + return 0; +} + /** * Common bus driver callback to probe a device. * - * This function probe PCI bus device(s). + * This function probe PCI bus device(s) or a single SF on auxiliary bus. * * @param[in] dev * Pointer to the generic device. @@ -2477,7 +2538,8 @@ mlx5_os_net_probe(struct rte_device *dev) } if (mlx5_dev_is_pci(dev)) return mlx5_os_pci_probe(RTE_DEV_TO_PCI(dev)); - return 0; + else + return mlx5_os_auxiliary_probe(dev); } static int diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index af7cbeb418..2991d37df2 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -19,4 +19,6 @@ enum { #define MLX5_NAMESIZE IF_NAMESIZE +int mlx5_auxiliary_get_ifindex(const char *sf_name); + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index c9474a6e74..c6d70af3a7 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -392,6 +392,24 @@ mlx5_is_hpf(struct rte_eth_dev *dev) MLX5_REPRESENTOR_REPR(-1) == repr; } +/** + * Decide whether representor ID is a SF port representor. + * + * @param dev + * Pointer to Ethernet device structure. + * + * @return + * Non-zero if HPF, otherwise 0. + */ +bool +mlx5_is_sf_repr(struct rte_eth_dev *dev) +{ + struct mlx5_priv *priv = dev->data->dev_private; + int type = MLX5_REPRESENTOR_TYPE(priv->representor_id); + + return priv->representor != 0 && type == RTE_ETH_REPRESENTOR_SF; +} + /** * Initialize the ASO aging management structure. * @@ -2314,7 +2332,10 @@ mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev) (dev->device == odev || (dev->device->driver && dev->device->driver->name && - !strcmp(dev->device->driver->name, MLX5_PCI_DRIVER_NAME)))) + ((strcmp(dev->device->driver->name, + MLX5_PCI_DRIVER_NAME) == 0) || + (strcmp(dev->device->driver->name, + MLX5_AUXILIARY_DRIVER_NAME) == 0))))) break; port_id++; } diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 5ea465fa0b..28bbc72e80 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -220,6 +220,7 @@ struct mlx5_dev_config { unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int vf:1; /* This is a VF. */ + unsigned int sf:1; /* This is a SF. */ unsigned int tunnel_en:1; /* Whether tunnel stateless offloads are supported. */ unsigned int mpls_en:1; /* MPLS over GRE/UDP is enabled. */ @@ -1407,6 +1408,7 @@ int mlx5_udp_tunnel_port_add(struct rte_eth_dev *dev, uint16_t mlx5_eth_find_next(uint16_t port_id, struct rte_device *odev); int mlx5_dev_close(struct rte_eth_dev *dev); bool mlx5_is_hpf(struct rte_eth_dev *dev); +bool mlx5_is_sf_repr(struct rte_eth_dev *dev); void mlx5_age_event_prepare(struct mlx5_dev_ctx_shared *sh); /* Macro to iterate over all valid ports for mlx5 driver. */ diff --git a/drivers/net/mlx5/mlx5_mac.c b/drivers/net/mlx5/mlx5_mac.c index 19981d26d8..a791fedc91 100644 --- a/drivers/net/mlx5/mlx5_mac.c +++ b/drivers/net/mlx5/mlx5_mac.c @@ -159,7 +159,7 @@ mlx5_mac_addr_set(struct rte_eth_dev *dev, struct rte_ether_addr *mac_addr) * Configuring the VF instead of its representor, * need to skip the special case of HPF on Bluefield. */ - if (priv->representor && !mlx5_is_hpf(dev)) { + if (priv->representor && !mlx5_is_hpf(dev) && !mlx5_is_sf_repr(dev)) { DRV_LOG(DEBUG, "VF represented by port %u setting primary MAC address", dev->data->port_id); if (priv->pf_bond >= 0) { diff --git a/drivers/net/mlx5/mlx5_rxmode.c b/drivers/net/mlx5/mlx5_rxmode.c index 25fb47c9ed..7f19b235c2 100644 --- a/drivers/net/mlx5/mlx5_rxmode.c +++ b/drivers/net/mlx5/mlx5_rxmode.c @@ -36,7 +36,7 @@ mlx5_promiscuous_enable(struct rte_eth_dev *dev) dev->data->port_id); return 0; } - if (priv->config.vf) { + if (priv->config.vf || priv->config.sf) { ret = mlx5_os_set_promisc(dev, 1); if (ret) return ret; @@ -69,7 +69,7 @@ mlx5_promiscuous_disable(struct rte_eth_dev *dev) int ret; dev->data->promiscuous = 0; - if (priv->config.vf) { + if (priv->config.vf || priv->config.sf) { ret = mlx5_os_set_promisc(dev, 0); if (ret) return ret; @@ -109,7 +109,7 @@ mlx5_allmulticast_enable(struct rte_eth_dev *dev) dev->data->port_id); return 0; } - if (priv->config.vf) { + if (priv->config.vf || priv->config.sf) { ret = mlx5_os_set_allmulti(dev, 1); if (ret) goto error; @@ -142,7 +142,7 @@ mlx5_allmulticast_disable(struct rte_eth_dev *dev) int ret; dev->data->all_multicast = 0; - if (priv->config.vf) { + if (priv->config.vf || priv->config.sf) { ret = mlx5_os_set_allmulti(dev, 0); if (ret) goto error; diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 4ff12eac19..2acfb484a5 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -1259,7 +1259,7 @@ mlx5_traffic_enable(struct rte_eth_dev *dev) } mlx5_txq_release(dev, i); } - if (priv->config.dv_esw_en && !priv->config.vf) { + if (priv->config.dv_esw_en && !priv->config.vf && !priv->config.sf) { if (mlx5_flow_create_esw_table_zero_flow(dev)) priv->fdb_def_rule = 1; else diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c index d4de2adfc1..7211c06333 100644 --- a/drivers/net/mlx5/windows/mlx5_os.c +++ b/drivers/net/mlx5/windows/mlx5_os.c @@ -921,20 +921,18 @@ mlx5_match_devx_devices_to_addr(struct devx_device_bdf *devx_bdf, /** * DPDK callback to register a PCI device. * - * This function spawns Ethernet devices out of a given PCI device. + * This function spawns Ethernet devices out of a given device. * - * @param[in] pci_drv - * PCI driver structure (mlx5_driver). - * @param[in] pci_dev - * PCI device information. + * @param[in] dev + * Pointer to the generic device. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ int -mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +mlx5_os_net_probe(struct rte_device *dev) { + struct rte_pci_device *pci_dev = RTE_DEV_TO_PCI(dev); struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs; /* * Number of found IB Devices matching with requested PCI BDF. From patchwork Wed Jun 16 04:09:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94281 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27A5BA0C45; Wed, 16 Jun 2021 06:10:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 50C6D4113D; Wed, 16 Jun 2021 06:10:11 +0200 (CEST) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam08on2062.outbound.protection.outlook.com [40.107.102.62]) by mails.dpdk.org (Postfix) with ESMTP id 6FB794113C for ; Wed, 16 Jun 2021 06:10:10 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YODQvKIPmuhDiuFBrKuwljw7GXwcvK5aQgJAI6TYUjy0+mFJvzN7CZ8VnkCR3NQzzivupnnnBy/eib1bVlz6EeD9anNKVzvAmwnKgIwj03BeqmoqUIUydG2StRf70Sj1HRRMGr1fhwP/BJsI0UOHovlXtsUAOY9P2+FchyAKGozXoEuiPdm52YWZbF4M/CWw4eKB9VwL5WW23PVOomZ7ru/qVx8LTjsgKEcTkS55KDzOsHKzj94/qPzdeAk4YdHbjRT7jMK/kX8g0XyAvu9I+8P/cWjsWCugugZbjsdtPZ+rw/CILeiK/QreMVFdatPRChrlHASQ1fBkPzLboqgYog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qjxEzJV5H8iouqlK18TY9Vpi5bnfI7ci1DlsJ+H1d0o=; b=HQZuBz+oW6icINUrrJlwpyKfDbVsFc5kM4jRaWSPk2daho100jOSLM/lPIyvh9WRPsiYbiZdOqmuD/IKwpDQxhFHCZ8rrUCcKKA+Rv0cBDxiHH9HIpEon4nLgyMLaOVmQqMRvaEEyFOegi2rPGrOapkmR9tBzNrvzqN9dlvIl9GtMvKSBjzd18PvxeO0P/AWuraPmkDvvMLfQsVV3q2zAJnyr3IpSn0ST1sTit/MAvY04vBf8L2vA65tSD465lOs+9AIKIarDbEAFn0j3GYFIdL6QOapwe+2g/Kf56PcLr7HMAi0wAIkq3GTF/MNB+JM/18L/g7T1QGBjseBLmTFHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qjxEzJV5H8iouqlK18TY9Vpi5bnfI7ci1DlsJ+H1d0o=; b=SdBNkdaHeqPx3N+OrkVA7+bNf2QDZXCrTXFUbyprE7gB8oPfqYQuASCp9gS25IjY785U5nRimjQyQnvXsoNq20K6BjYV7+3nVaumCXQvkfvQ1Fi0fIPGyxZz/7qRs3jrqt/1cVGf51tUjeiA5NAwQWhaqRhdszZ4x4pAlWbILV3vQmFtGC8EJ16ime2nsv8h5WLyq12JJFvGnIZVt39SDP1ZexOco3nc36uG5SDQqNjGyOOo2M+UYC7VmkFDkPSHmSSttvdXI+foiUempunMT8tOth4ACpW42TlWsKx25mEFZ+vCAIeB8452/70qXKNQnwU6QQ+VDAu0XnSdGPrzpQ== Received: from BN0PR04CA0102.namprd04.prod.outlook.com (2603:10b6:408:ec::17) by MN2PR12MB4144.namprd12.prod.outlook.com (2603:10b6:208:15f::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Wed, 16 Jun 2021 04:10:09 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::50) by BN0PR04CA0102.outlook.office365.com (2603:10b6:408:ec::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Wed, 16 Jun 2021 04:10:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:08 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:04 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler Date: Wed, 16 Jun 2021 07:09:29 +0300 Message-ID: <20210616040935.311733-9-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fb8921e8-f870-49b0-edc1-08d9307ca15f X-MS-TrafficTypeDiagnostic: MN2PR12MB4144: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:346; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CAsuYTXB7L3gIeOeUCqFUu5lP8Jr+Iy/UQVp7N0URHkXc3X4kAViabEg/y96Z5u+6AppHPbQZrnxG3IVNI6mPVFsWYv4gkbViMQ4YCNoC5UqJoonEW+JivHslEmVzRTuoS0G+XGxuz5+ALLEX//eHywEkcu/5D6xA7dNWRnR2CFcwtOXPI0k46Yh61/rphrXyfCtRrWhwgYf/RjP7QTrItWRqhXbIks4r5m+YWC1cnY4nzziCxu/oy59TlrPpxzRM5HBjcz9FZm+66rF3YXmbr+hVac6vksKi1vj0KtQv0tFXCfKCtsoAIt1+E25P+RwBPwy9BAeLw24yURh9S8pCcTvb2Lbi0lUkwYuH8+Fc7LNsQMReDH5v4IVOhywKLzdaqLfLNEH2i/L+WH8SCW27sSR8Mdsi2n7qfa6y1bN3o1vEsiR2T0Lc0HZB8JdABHXu1poRWlwukhlDBmvyH6My6aqFRdqiXzyexutji9t1N2EOOWmfB49o+tXKidTQSY7C3T9c0BfubHfS9UlG+Gq2UgbOfJ0j9NUx5PrOyKI3S+G1yaJxXKFa8QlnDa2Z4jEI4vDkejXyktXjP5jVfZMp9JYJG8vwIeDgFUsFZdInk6sc0icuLGwmXcaIEVz2aSQdMHIQsZmJ6S5mohTCwYwvA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(39860400002)(136003)(346002)(376002)(36840700001)(46966006)(2616005)(86362001)(16526019)(26005)(107886003)(426003)(8676002)(82740400003)(6666004)(186003)(4744005)(70586007)(356005)(55016002)(47076005)(5660300002)(36756003)(7636003)(70206006)(8936002)(6862004)(37006003)(6286002)(1076003)(2906002)(54906003)(316002)(478600001)(4326008)(82310400003)(36860700001)(7696005)(336012)(36906005)(6636002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:08.9273 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fb8921e8-f870-49b0-edc1-08d9307ca15f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4144 Subject: [dpdk-dev] [PATCH v1 08/14] net/mlx5: check max Verbs port number X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Verbs API doesn't support Device port number larger than 255 by design. Adds check and fails probing with proper error log. Signed-off-by: Xueming Li --- drivers/net/mlx5/linux/mlx5_os.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 47df3b92f8..9a3616d539 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1168,6 +1168,12 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, config->dv_flow_en = 0; } #endif + if (spawn->max_port > UINT8_MAX) { + /* Verbs can't support ports larger than 255 by design. */ + DRV_LOG(ERR, "can't support IB ports > UINT8_MAX"); + err = EINVAL; + goto error; + } config->ind_table_max_size = sh->device_attr.max_rwq_indirection_table_size; /* From patchwork Wed Jun 16 04:09:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94282 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 14205A0C45; Wed, 16 Jun 2021 06:11:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B0BAD410FB; Wed, 16 Jun 2021 06:10:14 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2082.outbound.protection.outlook.com [40.107.94.82]) by mails.dpdk.org (Postfix) with ESMTP id 7C621410E6 for ; Wed, 16 Jun 2021 06:10:13 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=G8cQdJYc+t+maJa9bpxB5hDj2Nzf+kXA+QWdY4fJVl3+2HuF//Wq7SF67pnnGyjsnjNS6UPA1VUkMhXr7oMVJxz0LsPspFWjLJfUGdhgo4kCaonz8NzzayGSp9S9eJM/wi2d2mR/kqvKBaxtSehBa9byGnt+d0kqG4EMbN4WX603IO/fcsX8EsNuSLhYlO1LlIVLwx+Jm6Wl1q9WSRBVueYfJ1BQBofzcRG99MAJjajJnAZGHCp3lVaVvcQv0Dl5+GpnS23LkYwl7/Wk6UsADhJANa/9Q2ffInMGvYTULd9BfN/w8mTfN6lV4a/q7cRJz7IFfFBSxwPJ6wdVvEVQvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=srrU3wVlzdJ2zt+Jgy1MbW7rVmmCEl+uIlCWxdbXPnY=; b=PrZyZHTE5PszadpTZM8J2MakB2lsYam8OI2uEHPviB5c2L4Duc9GDZttSXK+zdsOpEel5j2Hg8zxKc7i3M44jhAykXp+/CGkIfJbuLwWU85mHmZTe+DhGplLXVDtpl31Qrq0dPRhpt+cgF2cZRu9nI+GTW2AV5jhThaC2a7wrXFl4rwcIukr2QaugIs/N/JrqoTFvWQ6xkVUhVOkLcxYIdxCw1GKkAZ5NZCQ28ne3Z2cUMU18hXyd/7Ygqu5kUceqXdqT6kYe+EYLFaDvPOLd+EwgSfilJpCsWXD+4eV7yYhAJTt1N6aPds27C4ubpemGbyof6wtvslfKVLjuKIPDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=srrU3wVlzdJ2zt+Jgy1MbW7rVmmCEl+uIlCWxdbXPnY=; b=YFtyDHHAI+lYoF74m0A4u5ZQw1XHTzVvbHjf+6i2NhpJuzbUl0nLFTkQPiKe7iFmkHmwyWExL/otwP5YQqnOfgccMRG0SG17wsX5WVJKbb9cWSwK2YIDC5RAVv/yslAE5C4nwLzie62adUyx8MlT60wtQvrvOZRWrBUYScY6c8L9rDRQ0CWiFze89TxOb8p22/XggvZTa9u4nC8hygA6iIjtpk2Cu9D7iFMXAPRcGkGiDOWyYzQNU6knTK8dwskFaG74tXzZlBHI0hMKKAj8L2hBNaHAqrYt5x8CuEFrq6lK0nrn8Wrp6AIOK+co7DbiCQDwReUBltKi3ZNEQQf3iA== Received: from BN0PR04CA0106.namprd04.prod.outlook.com (2603:10b6:408:ec::21) by BY5PR12MB3665.namprd12.prod.outlook.com (2603:10b6:a03:1a6::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Wed, 16 Jun 2021 04:10:11 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::b) by BN0PR04CA0106.outlook.office365.com (2603:10b6:408:ec::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Wed, 16 Jun 2021 04:10:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:11 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:06 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Ori Kam Date: Wed, 16 Jun 2021 07:09:30 +0300 Message-ID: <20210616040935.311733-10-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2ae7f6b7-268d-4485-9598-08d9307ca2a6 X-MS-TrafficTypeDiagnostic: BY5PR12MB3665: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2582; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XtNmh9HtcgVBB7ZEE+OTxfUcjAugvYezX0bfl7kb6hwXzA2lkVLyxISaT5wNTzqcBQXF2WlpbGeGR40736D7B2al0XIG6czLHE/rZ0IXgVdCxkQGgDXWyEeTfBEx++VNcTzfHi2DFfJUSJA8L+f88Aq8rX6wcULeHP40wPhtQ4TyJ8T6aqakAgJHjSINwkqDhZmDGzrMz3WuSOrzN3Edi8GsfEBj5Fqdh1orCsMq1rMAV+v3XtOdAp/P4QNOUUV57WlGc9496FLsDqS7iSK9ptfm3C1EWrJbM3x5N8Mngz0G2ovS1gW6uiVi8d0d2pdVd72MzBf6wb4fJtVhyQ4eJs3zva3p2EmxK0akmlSlqibmTDfwsZ7wzC7577l0jRnkr3na2Vq+wAYX3VwkQtuf2waz3DocCUWOlxV3slg5mfzvwugRHfILXtdoLaLnJy6TfZNN664x9CNdUzIpSKeTRlHsme9izolkbjPkGDEYCnavLbPIib2Z5e1a6nDX4hBb7RslwKp01xc3vpC58+h5XVT0m70bJRXzxEJj1v49pY2b6/N9qfuc0S5Hot4D5WRzxI4PEK8wi5EyVSJAGyGEMajpyrySYJfS7re0+zxURvB8YM0whHzqSRLOTjE5pA4n9iuHR0rwAwe6nmhwzOSBCcu/v5NsBQI6nH3JCvyGfl4= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(346002)(39860400002)(376002)(396003)(46966006)(36840700001)(4326008)(36756003)(36860700001)(7696005)(2906002)(6636002)(8676002)(2616005)(36906005)(336012)(316002)(6862004)(82310400003)(54906003)(47076005)(82740400003)(7636003)(186003)(70586007)(83380400001)(70206006)(426003)(1076003)(8936002)(356005)(86362001)(37006003)(478600001)(107886003)(5660300002)(6286002)(6666004)(55016002)(26005)(16526019); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:11.0681 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2ae7f6b7-268d-4485-9598-08d9307ca2a6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB3665 Subject: [dpdk-dev] [PATCH v1 09/14] regex/mlx5: migrate to common driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To support auxiliary bus, upgrades driver to use mlx5 common driver structure. Signed-off-by: Xueming Li --- drivers/regex/mlx5/mlx5_regex.c | 49 ++++++++++++--------------------- drivers/regex/mlx5/mlx5_regex.h | 1 - 2 files changed, 18 insertions(+), 32 deletions(-) diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index dcb2ced88e..9d93eaa934 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -9,8 +9,8 @@ #include #include #include +#include -#include #include #include #include @@ -76,15 +76,13 @@ mlx5_regex_engines_status(struct ibv_context *ctx, int num_engines) } static void -mlx5_regex_get_name(char *name, struct rte_pci_device *pci_dev __rte_unused) +mlx5_regex_get_name(char *name, struct rte_device *dev) { - sprintf(name, "mlx5_regex_%02x:%02x.%02x", pci_dev->addr.bus, - pci_dev->addr.devid, pci_dev->addr.function); + sprintf(name, "mlx5_regex_%s", dev->name); } static int -mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) +mlx5_regex_dev_probe(struct rte_device *rte_dev) { struct ibv_device *ibv; struct mlx5_regex_priv *priv = NULL; @@ -94,16 +92,10 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, int ret; uint32_t val; - ibv = mlx5_os_get_ibv_device(&pci_dev->addr); - if (!ibv) { - DRV_LOG(ERR, "No matching IB device for PCI slot " - PCI_PRI_FMT ".", pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv = mlx5_os_get_ibv_dev(rte_dev); + if (ibv == NULL) return -rte_errno; - } - DRV_LOG(INFO, "PCI information matches for device \"%s\".", - ibv->name); + DRV_LOG(INFO, "Probe device \"%s\".", ibv->name); ctx = mlx5_glue->dv_open_device(ibv); if (!ctx) { DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name); @@ -146,7 +138,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->is_bf2 = 1; /* Default RXP programming mode to Shared. */ priv->prog_mode = MLX5_RXP_SHARED_PROG_MODE; - mlx5_regex_get_name(name, pci_dev); + mlx5_regex_get_name(name, rte_dev); priv->regexdev = rte_regexdev_register(name); if (priv->regexdev == NULL) { DRV_LOG(ERR, "Failed to register RegEx device."); @@ -180,7 +172,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, priv->regexdev->enqueue = mlx5_regexdev_enqueue_gga; #endif priv->regexdev->dequeue = mlx5_regexdev_dequeue; - priv->regexdev->device = (struct rte_device *)pci_dev; + priv->regexdev->device = rte_dev; priv->regexdev->data->dev_private = priv; priv->regexdev->state = RTE_REGEXDEV_READY; priv->mr_scache.reg_mr_cb = mlx5_common_verbs_reg_mr; @@ -213,13 +205,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, } static int -mlx5_regex_pci_remove(struct rte_pci_device *pci_dev) +mlx5_regex_dev_remove(struct rte_device *rte_dev) { char name[RTE_REGEXDEV_NAME_MAX_LEN]; struct rte_regexdev *dev; struct mlx5_regex_priv *priv = NULL; - mlx5_regex_get_name(name, pci_dev); + mlx5_regex_get_name(name, rte_dev); dev = rte_regexdev_get_device_by_name(name); if (!dev) return 0; @@ -254,24 +246,19 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_regex_driver = { - .driver_class = MLX5_CLASS_REGEX, - .pci_driver = { - .driver = { - .name = RTE_STR(MLX5_REGEX_DRIVER_NAME), - }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags = 0, - }, +static struct mlx5_class_driver mlx5_regex_driver = { + .drv_class = MLX5_CLASS_REGEX, + .name = RTE_STR(MLX5_REGEX_DRIVER_NAME), + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_dev_probe, + .remove = mlx5_regex_dev_remove, }; RTE_INIT(rte_mlx5_regex_init) { mlx5_common_init(); if (mlx5_glue) - mlx5_pci_driver_register(&mlx5_regex_driver); + mlx5_class_driver_register(&mlx5_regex_driver); } RTE_LOG_REGISTER_DEFAULT(mlx5_regex_logtype, NOTICE) diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h index 51a2101e53..45200bf937 100644 --- a/drivers/regex/mlx5/mlx5_regex.h +++ b/drivers/regex/mlx5/mlx5_regex.h @@ -59,7 +59,6 @@ struct mlx5_regex_db { struct mlx5_regex_priv { TAILQ_ENTRY(mlx5_regex_priv) next; struct ibv_context *ctx; /* Device context. */ - struct rte_pci_device *pci_dev; struct rte_regexdev *regexdev; /* Pointer to the RegEx dev. */ uint16_t nb_queues; /* Number of queues. */ struct mlx5_regex_qp *qps; /* Pointer to the qp array. */ From patchwork Wed Jun 16 04:09:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94283 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 82DC6A0C45; Wed, 16 Jun 2021 06:11:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D2F0541142; Wed, 16 Jun 2021 06:10:16 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2072.outbound.protection.outlook.com [40.107.223.72]) by mails.dpdk.org (Postfix) with ESMTP id 9959C410E6 for ; Wed, 16 Jun 2021 06:10:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RbeOyDrf9nTiN0svQfIvqoVbVb58H35SV8PErL39fouUhEr/WwY1O8NUHeXzWS7u2DiWgUv1/KpMd+0JgiUb7uM/N+84KpjiXdrrZfnGL2Bs7KUb9XwV+oeLXl2cuPu6M+g30gevUbscE3KMRmJdpb4m1JTmukLqj+n8C1qPDj3BXn8zsVIjJLWwj7XzYs13YKppYUVvbfRe/YfPcReizr2V5/IyKEYkMQopRlsM4kpESTUFKqtyQhkW3dLgH0vrt+qiIKIE6ynzXl/F6ry9i78E9YtKaXu1FrA/pD1BWtLNjp3N5PDYxBSZtkOlfb4TFsbJquKrXB1BtgTMffvsRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k7B0vWFrTX7M4A2lxiUFc1tA9Vhx1P1KL7OLK5f7ozI=; b=Muhy3nTwwWyqS/CfokJkGOy117SY67bG81HL3HDJ8R1PNzvrSRoann5eekLUxCdBbOGG4fy0VMyVj7PxtoDmDB73pUSXzNtwuRY9C7ZylFlcZ07150Gkovi8C4BWBXotA9pz+n3hfg6oWm0MnibAj7Z4JiEneIRlCFQhORL3VH4o5r5/3tg/IE/+aUD+fYJzKA/M6Qiq+rqYV4dmBzcXLVlZ710Aiz6/meMCS5nA1yBIj2XbxvSrk1kP5YMot2UH03yWJjWU4OvnDhpMlYNJKCbc+f14vkPt3U1wGNqbu+MW1mt/eh3QJv5D2mrzPYQD8Vp5P8DcplvvSzeL3xqnkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k7B0vWFrTX7M4A2lxiUFc1tA9Vhx1P1KL7OLK5f7ozI=; b=dzILZTxlig8fDJcf2DrGhuMeOgt2X3nijHYhB4gtGnhDXZLSIz8+kg93Q3kch3QvA2AcyWesfhtIV/Ct1L49+zpna13CtVHRqW8TzpBB36BmwdssVqCWKZnT62LIOhfhXEls+WhW6AzRJc0YFbDdsldUS2ES2eyYJ0INXzhzdHxNrz5paPjMKuUxAy/TOooSE27rJimNhztyEunB2r6lrzIbyfIsIKzPirsgE2RciLGAoGGYK/KvyXa23HsNH61x6RHnRpjz3pxGPHeVPDL+bvYMXH7kBj9HQqY8yr/Wfs2fQUPYsqjp8OEVy1cS/SVy4n2uPFL5TSDJ6Z5PFWVIxQ== Received: from BN0PR04CA0104.namprd04.prod.outlook.com (2603:10b6:408:ec::19) by DM6PR12MB2922.namprd12.prod.outlook.com (2603:10b6:5:18d::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Wed, 16 Jun 2021 04:10:13 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::d7) by BN0PR04CA0104.outlook.office365.com (2603:10b6:408:ec::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.21 via Frontend Transport; Wed, 16 Jun 2021 04:10:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:13 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:07 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: Thomas Monjalon , , , Matan Azrad Date: Wed, 16 Jun 2021 07:09:31 +0300 Message-ID: <20210616040935.311733-11-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 56190330-74f9-47ed-91e8-08d9307ca3d0 X-MS-TrafficTypeDiagnostic: DM6PR12MB2922: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1079; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3oJ5/H2op5wMAtOHHJIBuSpxCqmB2Q1gkb0AqsgCUaHTsECrv7Vb3kOHYXf2syEjdG8iM1CfmxdeUZ5ZI1l56BZwYhQubBUeNMXlHIUedU4oweDfO52MZ6xB7AKe+JFs05qrQ6Sy38Yu0GBA33kvFU6l2TizfAJYOPaDdSL1yV/EYKsLg6xJoA5eAn/kWP7GC3RBawatZJTcnDwwrshydzPD64ZA3A28HsTiFEIylkyGs/Unf041Y6HFojmTZVkuRsvmOQdjAVRbyqBrf8YN/zOCV7xPElmxf3UYTyvS6lIacrCLh7pTDQYtyaeL+T9a0EBM5klmEk9HHIxW3Evjzyy1ddLc9spTsSH8Myl/TiJ5cGtcm3erWPHcLIEEL/7E7zfQh1IqJa3FOxFUUOG6v5muTCGJWODAvqIA9/PFHiaSlVvbtk0l37p4ScLvSU35Bcvfl6Ht2rfBhgV72SMp0DMJf82/Rb9fXj08grK1cSs5/hxPzDMbPD5QVBsc5qCR5Vu5rJHfN1aB6JWMivQHzttHhzcmzw7psKq/FqDVTu/yIU1evYxuZ5lFeN9hmyhqRADYZfmh6HVvfRSg91zf5O49jV08X9w0YiDFXYKV1Vo56frl+xL0M5DC418pc6x0jRenSQdFrNSIXIbS/H+iK0Ws11WxN9l3DxoJ++l06v4= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(39860400002)(136003)(346002)(376002)(36840700001)(46966006)(26005)(2616005)(86362001)(8936002)(6666004)(82740400003)(5660300002)(107886003)(16526019)(83380400001)(426003)(6286002)(36756003)(356005)(47076005)(8676002)(6862004)(186003)(70586007)(55016002)(70206006)(37006003)(4326008)(316002)(7636003)(6636002)(82310400003)(1076003)(7696005)(36860700001)(336012)(36906005)(478600001)(54906003)(2906002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:13.0269 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 56190330-74f9-47ed-91e8-08d9307ca3d0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2922 Subject: [dpdk-dev] [PATCH v1 10/14] vdpa/mlx5: define driver name as macro X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon Uses macro for pmd driver name. Signed-off-by: Thomas Monjalon --- drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 8b5bfd8c3d..5ab7c525c2 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -24,6 +24,7 @@ #include "mlx5_vdpa_utils.h" #include "mlx5_vdpa.h" +#define MLX5_VDPA_DRIVER_NAME vdpa_mlx5 #define MLX5_VDPA_DEFAULT_FEATURES ((1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ (1ULL << VIRTIO_F_ANY_LAYOUT) | \ @@ -834,7 +835,7 @@ static struct mlx5_pci_driver mlx5_vdpa_driver = { .driver_class = MLX5_CLASS_VDPA, .pci_driver = { .driver = { - .name = "mlx5_vdpa", + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), }, .id_table = mlx5_vdpa_pci_id_map, .probe = mlx5_vdpa_pci_probe, @@ -855,6 +856,6 @@ RTE_INIT(rte_mlx5_vdpa_init) mlx5_pci_driver_register(&mlx5_vdpa_driver); } -RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); -RTE_PMD_REGISTER_PCI_TABLE(net_mlx5_vdpa, mlx5_vdpa_pci_id_map); -RTE_PMD_REGISTER_KMOD_DEP(net_mlx5_vdpa, "* ib_uverbs & mlx5_core & mlx5_ib"); +RTE_PMD_EXPORT_NAME(MLX5_VDPA_DRIVER_NAME, __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE(MLX5_VDPA_DRIVER_NAME, mlx5_vdpa_pci_id_map); +RTE_PMD_REGISTER_KMOD_DEP(MLX5_VDPA_DRIVER_NAME, "* ib_uverbs & mlx5_core & mlx5_ib"); From patchwork Wed Jun 16 04:09:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94284 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F2915A0C45; Wed, 16 Jun 2021 06:11:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F34274114E; Wed, 16 Jun 2021 06:10:18 +0200 (CEST) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08on2055.outbound.protection.outlook.com [40.107.101.55]) by mails.dpdk.org (Postfix) with ESMTP id 60DF341144 for ; Wed, 16 Jun 2021 06:10:17 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DirR2iGjiq7G3RQ7zUvcsn4dY1ka4U+zD5NQVy13DitnQHrWvVjV8NFv0l/YbceUOLnZGLY0DSqlg9XbjhHEtKfPFajjVeF+i6W1Ccu2dQrbmLEJtFPKYvpDA5Z+I8YQ1SV9w1q9zvJCkEzZxRrTf8xLxMQVFRVm8sol9jeRSZY2wHcwQhwbcznPXKXMV5/IesNjdI93YnxadfID47IBCAtboE/B90cnKz1ZFC/A4jb/7YZOt0/D8TG4oQt5WzTxlIuLKOH3f29q2r+zh0+aqvmrLqLycmiAQFRdXe6qD2NOPo2E/WV/HC2R5nOQhDk24TO5MdX/4ax1qKbK424lCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c9AC4tVhXTa2eaArjvrf7KvQEicCOI8AstYFzRVQujk=; b=a/mH3Ieh26SKrgLKbdrhCPnnYayj++ylDivCs5oX8mh8oG3qWAxU+059idxwTm+A6Ftx3O0ZVJTKssNitTpTSHlorlSCV0GR8ga7pyayycZM8815suYkho8WMLF/LpL9tovLj9g4oR09U+Gz8ry17bEGIhD8i6IkOhFvtJyo1WQhoiwAGIXO3oGqhRVqmXzU0a2+JOrBpgwL+0qBsGJshbrgfipjtZ2FHpptdkT1HKNYwKIgd9cR9vVhbJBd9pJfaUsZEgx2OSTS29Rn2CKl03FRVQYqhPZvTBgk8EKdr7PHFuiqYf9Nm7PCJQdElmtT0DZ2pOn5DBpmJF994pd2JA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c9AC4tVhXTa2eaArjvrf7KvQEicCOI8AstYFzRVQujk=; b=KizVWV5IqGBfR6AawCq61XcPHwSJ/K4SpIvzkRN8i0RCl9MUXeoR9Ua2B6Qrz89vLRbdEq6qoaJ6wxZs+SAWh+vISEqKEliI2v6lhLjayEjrmYXseZ1L+O31qxjHhtDVq49FTemNaA4QTUO16CY1uA59d35/Xz7Pf7kbwv6EGcDEgW2xoa7cg3fndNWOb2Gh5wGnXgiDWN8Xidqz7tuOGXSO5GCE/pRxbLzEOf2pcJZJUL2z1eH8bthpk13fVC9Sh4WzPZ2TcxBZfRYOCnzo8fKc4ngu8MmC0LaZGzVflSJoTS5QUBqKwIAZdGc91Gg3Zxxk9Errj3sWNz6orZVBuQ== Received: from BN0PR04CA0094.namprd04.prod.outlook.com (2603:10b6:408:ec::9) by DM6PR12MB4894.namprd12.prod.outlook.com (2603:10b6:5:209::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Wed, 16 Jun 2021 04:10:14 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::72) by BN0PR04CA0094.outlook.office365.com (2603:10b6:408:ec::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Wed, 16 Jun 2021 04:10:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:14 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:09 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: Thomas Monjalon , , , Matan Azrad Date: Wed, 16 Jun 2021 07:09:32 +0300 Message-ID: <20210616040935.311733-12-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7d2e04de-9e42-4eec-f9b7-08d9307ca4ca X-MS-TrafficTypeDiagnostic: DM6PR12MB4894: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1850; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ft+W6PmmHjRPjnxv9G3yHkzxre62kNRAT+zv2tZCv8meMWFUYc6u3BFzcyO7JIewytCi3W2LrxNxyEZS9KhST9csxexjv6QyPhbxI8kI4zddRgtJA1Y7c/xxCOC79QlAClIGnvIuDiDuJitsmIjuo2E7RbH17TVNSMeYcWBuzQXWJA/NG4xT8k+7xG7zGhC3fpJh5+L4pciiv/jx2WJQOONVF8Hx985i23OoRFz1LO+Kk53XyFylvrKghjg5vQZC6n3O0vt6cz6gKuaa9+PkkDcaLQPDasAukF8TM0gCjIFNczUnaRlLtObnyi94LiVlS25BtNC/5qvWQTX8bse5nO78rr9eSUPLpLL0e87qyqI2l1DMTnyF3AWvHiKr1HHdxJF1iT53S2FwEKNFPFeMxOT19o1K6Bq6sZ1gJkt7KPkSQOSp5d6UMAZs3LephtMXVWM/EwAc67xdZhk44T2B7JyTYoEG77umoCjL7c3DdBPjigia9o/yRyWHjRi1CC13IlHKjf7ah3cIEuxBJVN04X9brqqP1XL9Xl8RqdF6sg3+Pv7mYp1/tPX9jQCIp6uf9ElkQwNbJOiI1ITBEIMfD+ytacUMRNWNxof5Iqbehm+70xZKfKs06CAJgzMtdqK9GrGCOaU28t9ROkumxfDv7A== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(346002)(39860400002)(396003)(376002)(36840700001)(46966006)(7636003)(70586007)(55016002)(82740400003)(186003)(16526019)(478600001)(6666004)(2906002)(356005)(8936002)(36906005)(8676002)(82310400003)(336012)(36756003)(426003)(1076003)(5660300002)(7696005)(47076005)(6636002)(6862004)(316002)(36860700001)(83380400001)(70206006)(86362001)(54906003)(37006003)(26005)(4326008)(2616005)(107886003)(6286002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:14.6200 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7d2e04de-9e42-4eec-f9b7-08d9307ca4ca X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4894 Subject: [dpdk-dev] [PATCH v1 11/14] vdpa/mlx5: remove PCI specifics X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon Removes PCI specific driver, replaces with common class driver. Signed-off-by: Thomas Monjalon --- drivers/vdpa/mlx5/mlx5_vdpa.c | 119 ++++++++++------------------------ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 2 files changed, 34 insertions(+), 86 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 5ab7c525c2..9c9a552ba0 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,12 +11,11 @@ #include #include #include -#include #include +#include #include #include -#include #include #include #include @@ -552,34 +551,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) } static int -mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv) +mlx5_vdpa_roce_disable(struct rte_device *dev) { - char addr_name[64] = {0}; - - rte_pci_device_name(addr, addr_name, sizeof(addr_name)); /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ - if (mlx5_vdpa_nl_roce_disable(addr_name) == 0 || - mlx5_vdpa_sys_roce_disable(addr_name) == 0) { - /* - * Succeed to disable ROCE, wait for the IB device to appear - * again after reload. - */ - int r; - struct ibv_device *ibv_new; - - for (r = MLX5_VDPA_MAX_RETRIES; r; r--) { - ibv_new = mlx5_os_get_ibv_device(addr); - if (ibv_new) { - *ibv = ibv_new; - return 0; - } - usleep(MLX5_VDPA_USEC); - } - DRV_LOG(ERR, "Cannot much device %s after ROCE disable, " - "retries exceed %d", addr_name, MLX5_VDPA_MAX_RETRIES); - rte_errno = EAGAIN; - } - return -rte_errno; + if (mlx5_vdpa_nl_roce_disable(dev->name) != 0 && + mlx5_vdpa_sys_roce_disable(dev->name) != 0) + return -rte_errno; + return 0; } static int @@ -647,44 +625,33 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) DRV_LOG(DEBUG, "no traffic max is %u.", priv->no_traffic_max); } -/** - * DPDK callback to register a mlx5 PCI device. - * - * This function spawns vdpa device out of a given PCI device. - * - * @param[in] pci_drv - * PCI driver structure (mlx5_vpda_driver). - * @param[in] pci_dev - * PCI device information. - * - * @return - * 0 on success, 1 to skip this driver, a negative errno value otherwise - * and rte_errno is set. - */ static int -mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev __rte_unused) +mlx5_vdpa_dev_probe(struct rte_device *dev) { struct ibv_device *ibv; struct mlx5_vdpa_priv *priv = NULL; struct ibv_context *ctx = NULL; struct mlx5_hca_attr attr; + int retry; int ret; - ibv = mlx5_os_get_ibv_device(&pci_dev->addr); - if (!ibv) { - DRV_LOG(ERR, "No matching IB device for PCI slot " - PCI_PRI_FMT ".", pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + if (mlx5_vdpa_roce_disable(dev) != 0) { + DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".", + dev->name); return -rte_errno; - } else { - DRV_LOG(INFO, "PCI information matches for device \"%s\".", - ibv->name); } - if (mlx5_vdpa_roce_disable(&pci_dev->addr, &ibv) != 0) { - DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".", - ibv->name); + /* Wait for the IB device to appear again after reload. */ + for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) { + ibv = mlx5_os_get_ibv_dev(dev); + if (ibv != NULL) + break; + usleep(MLX5_VDPA_USEC); + } + if (ibv == NULL) { + DRV_LOG(ERR, "Cannot get IB device after disabling RoCE for " + "\"%s\", retries exceed %d.", + dev->name, MLX5_VDPA_MAX_RETRIES); + rte_errno = EAGAIN; return -rte_errno; } ctx = mlx5_glue->dv_open_device(ibv); @@ -722,20 +689,18 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (attr.num_lag_ports == 0) priv->num_lag_ports = 1; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->var = mlx5_glue->dv_alloc_var(ctx, 0); if (!priv->var) { DRV_LOG(ERR, "Failed to allocate VAR %u.", errno); goto error; } - priv->vdev = rte_vdpa_register_device(&pci_dev->device, - &mlx5_vdpa_ops); + priv->vdev = rte_vdpa_register_device(dev, &mlx5_vdpa_ops); if (priv->vdev == NULL) { DRV_LOG(ERR, "Failed to register vDPA device."); rte_errno = rte_errno ? rte_errno : EINVAL; goto error; } - mlx5_vdpa_config_get(pci_dev->device.devargs, priv); + mlx5_vdpa_config_get(dev->devargs, priv); SLIST_INIT(&priv->mr_list); pthread_mutex_init(&priv->vq_config_lock, NULL); pthread_mutex_lock(&priv_list_lock); @@ -754,26 +719,15 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, return -rte_errno; } -/** - * DPDK callback to remove a PCI device. - * - * This function removes all vDPA devices belong to a given PCI device. - * - * @param[in] pci_dev - * Pointer to the PCI device. - * - * @return - * 0 on success, the function cannot fail. - */ static int -mlx5_vdpa_pci_remove(struct rte_pci_device *pci_dev) +mlx5_vdpa_dev_remove(struct rte_device *dev) { struct mlx5_vdpa_priv *priv = NULL; int found = 0; pthread_mutex_lock(&priv_list_lock); TAILQ_FOREACH(priv, &priv_list, next) { - if (!rte_pci_addr_cmp(&priv->pci_dev->addr, &pci_dev->addr)) { + if (priv->vdev->device == dev) { found = 1; break; } @@ -831,17 +785,12 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_vdpa_driver = { - .driver_class = MLX5_CLASS_VDPA, - .pci_driver = { - .driver = { - .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), - }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, - }, +static struct mlx5_class_driver mlx5_vdpa_driver = { + .drv_class = MLX5_CLASS_VDPA, + .name = RTE_STR(MLX5_VDPA_DRIVER_NAME), + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_dev_probe, + .remove = mlx5_vdpa_dev_remove, }; RTE_LOG_REGISTER_DEFAULT(mlx5_vdpa_logtype, NOTICE) @@ -853,7 +802,7 @@ RTE_INIT(rte_mlx5_vdpa_init) { mlx5_common_init(); if (mlx5_glue) - mlx5_pci_driver_register(&mlx5_vdpa_driver); + mlx5_class_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(MLX5_VDPA_DRIVER_NAME, __COUNTER__); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 722c72b65e..2a04e36607 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -133,7 +133,6 @@ struct mlx5_vdpa_priv { struct rte_vdpa_device *vdev; /* vDPA device. */ int vid; /* vhost device id. */ struct ibv_context *ctx; /* Device context. */ - struct rte_pci_device *pci_dev; struct mlx5_hca_vdpa_attr caps; uint32_t pdn; /* Protection Domain number. */ struct ibv_pd *pd; From patchwork Wed Jun 16 04:09:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94286 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EB415A0C45; Wed, 16 Jun 2021 06:11:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 93A7E4115D; Wed, 16 Jun 2021 06:10:22 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2044.outbound.protection.outlook.com [40.107.94.44]) by mails.dpdk.org (Postfix) with ESMTP id E40A741151 for ; Wed, 16 Jun 2021 06:10:19 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kUpNTaEsqzKeJJbHiFvjiCwRakz8jWsW5SKwLcthVNFnOnmhCiskjid+pRAfuH5Lfbakka85Q15bA3T2Z8h86x6P0+/VMcxmKtjQ3bIlhm+JC9wMMrkpAn9vTiE96BzDeqOAvIRpEVHU36dkNQCVc7WKVwQ5yyakiK+HCdhfAbLnIm8G/Ak3lKxViy87Y8/kPol16Af2EkK6jwYDrwQB5Kfdp4H1DFW+Uhui95s0isNMvgnsyicxD4U6Z/ToCuhtGriWkzHUkjdrTrixysH6YT0zXbGwWXNY3Qa4ViFvnYASv1xraGnnzh8wpq7SuGvRrpGw7j3e19yizXQ45VaxUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iL9i+ngC/REpA8SYXQ42EDbQkof5m1JDE5Vk2wKr+Fg=; b=eRNeJHThQpWB0ldTm+e1eZLAJH1YatrtqvsoZbGh/yVuTSDfyf6G0Yr3YV9sFSGBbLuxojlHNg30X89OPsRv+KZKq1AZgD5CsP+cwxtZBy7cRj3qSR5Ghpzcjs4TqusoQMQCuz3TV9vsbIfqmu0+aYAnJFFJGyGEJvWAD6oO3y2abtFkeigK0g56b2a4yJTBUuyWp+ahoXl9I2pxuv72sD5xVmZBpQ0Z8SixVaCpyYNJtt3Uxkqko/ixavNmS5erQFaWajga+vOOSIXKFWfz7nNL4LTANqGglJse3MpZ5IP89dE54GZiR5JI7mcWWg0s97iudKMbdBRdWkNNfTPrmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iL9i+ngC/REpA8SYXQ42EDbQkof5m1JDE5Vk2wKr+Fg=; b=JSrkwCXq9XiflmCtcnge7xkwHTXvpLxk4gOLVn2+yHD3tBWJuRv/eXm5dKF8JlP1io+H/WQBYSaP1n2J6um+qB8hq4yBT5C7fMFFQm6Hh1+Ovfhd/VLtthSk9qNMgmiACiO7i6/jQOEVAW3zR2Zef243TsFy4aykIBVZ4JbcJfMarQNkQ2mbmQOmZ/YyrCvnl3KelB4W3EWnI+zdyU2NBhihxdDyB+bSQDB5QO66URWfqaxlYltoPOMiNdDfvZW0oOKtfESN63lIPwB4zXL553qEQb6PUXnqDRImlGbLxo9i6pXW0LR9d+PXknPRczbLsA3vCOx+lwDJHmXePZUBPw== Received: from BN0PR04CA0094.namprd04.prod.outlook.com (2603:10b6:408:ec::9) by BN8PR12MB3171.namprd12.prod.outlook.com (2603:10b6:408:99::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.24; Wed, 16 Jun 2021 04:10:16 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::85) by BN0PR04CA0094.outlook.office365.com (2603:10b6:408:ec::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Wed, 16 Jun 2021 04:10:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:16 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:10 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: Thomas Monjalon , , , Matan Azrad Date: Wed, 16 Jun 2021 07:09:33 +0300 Message-ID: <20210616040935.311733-13-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 37ce5800-d97b-426b-818e-08d9307ca5eb X-MS-TrafficTypeDiagnostic: BN8PR12MB3171: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:565; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: q6uHk7zM/K6njyNfUvUhsHibgfsmt9FMZUGyOdjIqhvtX1lBclmQoj1kYaIvhL09rNrYxI8iTiN6VWMQeQS72DCMYZyUZCJzZGPJWDgvux8v541/Tx0oybJ505ubqUD4BdzCeHTcWH9nRUJIG7VDF2gmZdofHlZ6L8lZNpJemOwCMyHjr3EKAo1tom7nLcXWQ5vF3ijQpC3nPTC35XfCg909eiduB+wjlatDk4gwRCWagBXNioyjoGd4Y6ZkezfgmTGTrXfh8atqNjNVbIWKMU0sqCzYxfG+Wzu6r2kebQrD+Noj0h6pw15ThMalsJrBVD+fbTQ5esziILMe0OQ/17ruXWtdFwCjCduS86nb6rd1HkGesVEqEZeGiNLPzkBfQfQ+ug8bE979pBMJv2z9zFbD81fWBTdxfCLI78VzdfyDtYW3LB47aqvygahKxJKSxFxOpPom3e2RyRiCSxBnim3F7VMUj3tJX4wXyr+V3OdIjYhyo6CeoZY+FknD4saKx3WltHCDkAZOHHtmsh95vzQ/rjpvu8dJvcZS4dW5ayXOu54MhylDdakpz+kkzmLUObgSqvYGQ9SLL/J25UtjOGlpWfIb33low02Fl2rCeoaS402dDzEExLbB9BEUfi9xrxnyDcNxSJ/A8WYGHqjijw== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(376002)(346002)(396003)(39860400002)(36840700001)(46966006)(47076005)(70586007)(2616005)(7636003)(36756003)(36906005)(356005)(8936002)(6862004)(1076003)(316002)(83380400001)(2906002)(54906003)(70206006)(86362001)(82740400003)(6636002)(336012)(36860700001)(37006003)(82310400003)(8676002)(16526019)(6666004)(55016002)(7696005)(26005)(5660300002)(6286002)(478600001)(4326008)(426003)(186003)(107886003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:16.4790 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 37ce5800-d97b-426b-818e-08d9307ca5eb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3171 Subject: [dpdk-dev] [PATCH v1 12/14] vdpa/mlx5: support SubFunction X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Thomas Monjalon Supports SubFunction on auxiliary bus. SF probe devargs: auxiliary:mlx5_core.sf.,class=vdpa Signed-off-by: Thomas Monjalon --- doc/guides/vdpadevs/mlx5.rst | 10 ++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.c | 8 ++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst index 9b2f9f12c7..e81dbd0004 100644 --- a/doc/guides/vdpadevs/mlx5.rst +++ b/doc/guides/vdpadevs/mlx5.rst @@ -162,6 +162,16 @@ Driver options - 0, HW default. +Devargs example +^^^^^^^^^^^^^^^ + +- PCI devargs: + + -a 0000:03:00.2,class=vdpa + +- Auxiliary devargs: + + -a auxiliary:mlx5_core.sf.2,class=vdpa Error handling ^^^^^^^^^^^^^^ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 9c9a552ba0..6d17d7a6f3 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -553,9 +553,13 @@ mlx5_vdpa_sys_roce_disable(const char *addr) static int mlx5_vdpa_roce_disable(struct rte_device *dev) { + char pci_addr[PCI_PRI_STR_SIZE] = { 0 }; + + if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0) + return -rte_errno; /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ - if (mlx5_vdpa_nl_roce_disable(dev->name) != 0 && - mlx5_vdpa_sys_roce_disable(dev->name) != 0) + if (mlx5_vdpa_nl_roce_disable(pci_addr) != 0 && + mlx5_vdpa_sys_roce_disable(pci_addr) != 0) return -rte_errno; return 0; } From patchwork Wed Jun 16 04:09:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94285 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D5DFA0C45; Wed, 16 Jun 2021 06:11:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6BEA54115A; Wed, 16 Jun 2021 06:10:21 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2047.outbound.protection.outlook.com [40.107.223.47]) by mails.dpdk.org (Postfix) with ESMTP id D930141153 for ; Wed, 16 Jun 2021 06:10:19 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=APB7K+6yHVWXqxYBSv/G9dfvRsvcOsnGjRbqP6dSR+jEo0poyi3kzwDcOTPcsm0p/fbcbPufNWrULKxFe9J67wGGGdpIfzfGwxYbkqk/oJnpaboTyJAaF96cDzHifJbSWsdWD5URKHMPdS3wT+XaCxZzbqRlBniTQjKe4+F2LD4S4G2dZLUCE9SshHlq29gpxyJr1FAMrhRfy6pYjKrmQqbtjLw8ECEnmo67arb3Mb0PAVCQToDBjfn7uzEh+3XI0gbWLR/pauJFhWk09cKP7qgY9h9JjwH0pGymreptWeSFDf82ZHXGxdMWXY3e4sOmQEQ6JzvPKYrj7ojDMJPFkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yIrFe2ZOlwhlHsJ1fY7U+iesWwWW5wQ8ZdM9JXNUKcg=; b=lEkPnL6T0YJn0PNr2GA4k/3mHlx+yU9H/xv/wOWU81rjsDrQeP6W3Vg/fH3MMI4vor3mqm8RRToNjqgsa8EFhy7/IPk28afPSChtLHpJ/+1hwrzvTiSHuoqbr/xVlvyc0Og71c1Lj2XWZSmsEOBGGW0X6a2I5BXYMnkzZAE8IDd9I1qpMICC4/uVXnt6NXHXEg+jLEHW73TbBLGutqlDPXBNYBsN7usqxg9qko9C6RHAMAjH+Fk5CmLHbLes/cuL7bUH0WXO1HyyNTOu3oM4J14gHs88qVtkNtPoyXF3/pu74pcrPSPlSh2fnmxqfwuWQ75EiTNIm/Bea7Rgd77QYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=marvell.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yIrFe2ZOlwhlHsJ1fY7U+iesWwWW5wQ8ZdM9JXNUKcg=; b=s/K/vZ/YTxjeaHV40i5oNkM3FxnYap6Q8nR0/0qGX/0B5qDpQO51RGJPFDOfyoia4pI2Cx1VAbbtGJtkq5vk1aLtmzlIee9VaecLpXMUvEEdSPlvSJU1wk7/TAP2tVep+1UGjOyE07bOD6FsGtBe1+A7N7lP1e0uQMQWFa4Mg760HvbR0q/BoB2JcAZ8grnZA3YVX5mXcJsxeH999AKtUcYzEwJGw7bKdWnoFw+iO4ghoecIf3WQ9WnesjaPW4b9j2WXbaqGd0mro8Gr2Zght5ZK74qcYzfteXAQYNAmYG0+wyUHE4EcftjKU8GDHzuNn30zu+gkwXIPwxwHkTjvzA== Received: from BN0PR04CA0092.namprd04.prod.outlook.com (2603:10b6:408:ec::7) by CH2PR12MB4874.namprd12.prod.outlook.com (2603:10b6:610:64::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.23; Wed, 16 Jun 2021 04:10:18 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::15) by BN0PR04CA0092.outlook.office365.com (2603:10b6:408:ec::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.21 via Frontend Transport; Wed, 16 Jun 2021 04:10:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:18 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:12 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Fiona Trahe , Ashish Gupta Date: Wed, 16 Jun 2021 07:09:34 +0300 Message-ID: <20210616040935.311733-14-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ea0776fc-9c4d-4c8b-6c93-08d9307ca6d7 X-MS-TrafficTypeDiagnostic: CH2PR12MB4874: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RsBhgirzIP69+2SeycOhcgwL2vZin9sah12W7ThkmWendjNbFkLa6A9nyzRfnC7ZREby1/CF6k3/QuDkmbTxxxnxWhBqWGpDhp02NuHjH2cgG0k0e3s/BCJBCcIYri7DSP9drc9Stdj4CckOZByJKCB4NxfFaAtfqn97nQH/JgQaKl+ulP95snvuP5Vu5xbNArcVlLcC9oX3degF6noI2dG/qgSnIlHM3M91OXUnkY+WpE34rUV3vuTecklaLlAnVIJC/zag5lSFwZsY0HJYgDtNsIyvLuvmbMSkgP5jyJZSyrKGRgQqAkKesx2uVvbv/gHdiCRh0PqLm1A1g1iChBuyx6w8anqxBDu35ZBF+eHmj2gbe3j3ih6VaHcfv5/Y3dgL2sPN3qxZVccvO5dTmy/LrOoEsKM1rnd1SApT4IpPmp/HPJfr1JL28+rX8Gk0EmoyEAmR2cQUzVev4U+aFHs3O8eJYX5CxrV/e3JdH3gL34W+zQl5L1vyEnBKVloT1KLd1CcI2PgUT61PgubmZIbp4d1PSnpmyIpRGqZvcKXHR3d13LzO+9qO18cGGmzk5nCRlcLlXx8+x6d/W9y4g6khwBAhTTt2nQXzrzJt5HLoGAYrRZKpq+TsmFlIegq3+PAsbpjEI6mMJmLrdNu8LA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(346002)(136003)(39860400002)(396003)(46966006)(36840700001)(8676002)(336012)(82740400003)(478600001)(6862004)(16526019)(83380400001)(7636003)(186003)(54906003)(426003)(1076003)(6286002)(55016002)(6666004)(6636002)(2906002)(26005)(36860700001)(36756003)(2616005)(356005)(70586007)(70206006)(316002)(47076005)(37006003)(86362001)(82310400003)(8936002)(4326008)(5660300002)(7696005)(36906005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:18.1090 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ea0776fc-9c4d-4c8b-6c93-08d9307ca6d7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4874 Subject: [dpdk-dev] [PATCH v1 13/14] compress/mlx5: migrate to common driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To support auxiliary bus, upgrades driver to use mlx5 common driver structure. Signed-off-by: Xueming Li --- drivers/compress/mlx5/mlx5_compress.c | 71 ++++++--------------------- 1 file changed, 15 insertions(+), 56 deletions(-) diff --git a/drivers/compress/mlx5/mlx5_compress.c b/drivers/compress/mlx5/mlx5_compress.c index 80c564f10b..77f426e399 100644 --- a/drivers/compress/mlx5/mlx5_compress.c +++ b/drivers/compress/mlx5/mlx5_compress.c @@ -5,7 +5,7 @@ #include #include #include -#include +#include #include #include #include @@ -13,7 +13,6 @@ #include #include -#include #include #include #include @@ -37,7 +36,6 @@ struct mlx5_compress_xform { struct mlx5_compress_priv { TAILQ_ENTRY(mlx5_compress_priv) next; struct ibv_context *ctx; /* Device context. */ - struct rte_pci_device *pci_dev; struct rte_compressdev *cdev; void *uar; uint32_t pdn; /* Protection Domain number. */ @@ -711,23 +709,8 @@ mlx5_compress_hw_global_prepare(struct mlx5_compress_priv *priv) return 0; } -/** - * DPDK callback to register a PCI device. - * - * This function spawns compress device out of a given PCI device. - * - * @param[in] pci_drv - * PCI driver structure (mlx5_compress_driver). - * @param[in] pci_dev - * PCI device information. - * - * @return - * 0 on success, 1 to skip this driver, a negative errno value otherwise - * and rte_errno is set. - */ static int -mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, - struct rte_pci_device *pci_dev) +mlx5_compress_dev_probe(struct rte_device *dev) { struct ibv_device *ibv; struct rte_compressdev *cdev; @@ -736,24 +719,17 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, struct mlx5_hca_attr att = { 0 }; struct rte_compressdev_pmd_init_params init_params = { .name = "", - .socket_id = pci_dev->device.numa_node, + .socket_id = dev->numa_node, }; - RTE_SET_USED(pci_drv); if (rte_eal_process_type() != RTE_PROC_PRIMARY) { DRV_LOG(ERR, "Non-primary process type is not supported."); rte_errno = ENOTSUP; return -rte_errno; } - ibv = mlx5_os_get_ibv_device(&pci_dev->addr); - if (ibv == NULL) { - DRV_LOG(ERR, "No matching IB device for PCI slot " - PCI_PRI_FMT ".", pci_dev->addr.domain, - pci_dev->addr.bus, pci_dev->addr.devid, - pci_dev->addr.function); + ibv = mlx5_os_get_ibv_dev(dev); + if (ibv == NULL) return -rte_errno; - } - DRV_LOG(INFO, "PCI information matches for device \"%s\".", ibv->name); ctx = mlx5_glue->dv_open_device(ibv); if (ctx == NULL) { DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name); @@ -769,7 +745,7 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, rte_errno = ENOTSUP; return -ENOTSUP; } - cdev = rte_compressdev_pmd_create(ibv->name, &pci_dev->device, + cdev = rte_compressdev_pmd_create(ibv->name, dev, sizeof(*priv), &init_params); if (cdev == NULL) { DRV_LOG(ERR, "Failed to create device \"%s\".", ibv->name); @@ -784,7 +760,6 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, cdev->feature_flags = RTE_COMPDEV_FF_HW_ACCELERATED; priv = cdev->data->dev_private; priv->ctx = ctx; - priv->pci_dev = pci_dev; priv->cdev = cdev; priv->min_block_size = att.compress_min_block_size; priv->sq_ts_format = att.sq_ts_format; @@ -810,25 +785,14 @@ mlx5_compress_pci_probe(struct rte_pci_driver *pci_drv, return 0; } -/** - * DPDK callback to remove a PCI device. - * - * This function removes all compress devices belong to a given PCI device. - * - * @param[in] pci_dev - * Pointer to the PCI device. - * - * @return - * 0 on success, the function cannot fail. - */ static int -mlx5_compress_pci_remove(struct rte_pci_device *pdev) +mlx5_compress_dev_remove(struct rte_device *dev) { struct mlx5_compress_priv *priv = NULL; pthread_mutex_lock(&priv_list_lock); TAILQ_FOREACH(priv, &mlx5_compress_priv_list, next) - if (rte_pci_addr_cmp(&priv->pci_dev->addr, &pdev->addr) != 0) + if (priv->cdev->device == dev) break; if (priv) TAILQ_REMOVE(&mlx5_compress_priv_list, priv, next); @@ -852,24 +816,19 @@ static const struct rte_pci_id mlx5_compress_pci_id_map[] = { } }; -static struct mlx5_pci_driver mlx5_compress_driver = { - .driver_class = MLX5_CLASS_COMPRESS, - .pci_driver = { - .driver = { - .name = RTE_STR(MLX5_COMPRESS_DRIVER_NAME), - }, - .id_table = mlx5_compress_pci_id_map, - .probe = mlx5_compress_pci_probe, - .remove = mlx5_compress_pci_remove, - .drv_flags = 0, - }, +static struct mlx5_class_driver mlx5_compress_driver = { + .drv_class = MLX5_CLASS_COMPRESS, + .name = RTE_STR(MLX5_COMPRESS_DRIVER_NAME), + .id_table = mlx5_compress_pci_id_map, + .probe = mlx5_compress_dev_probe, + .remove = mlx5_compress_dev_remove, }; RTE_INIT(rte_mlx5_compress_init) { mlx5_common_init(); if (mlx5_glue != NULL) - mlx5_pci_driver_register(&mlx5_compress_driver); + mlx5_class_driver_register(&mlx5_compress_driver); } RTE_LOG_REGISTER_DEFAULT(mlx5_compress_logtype, NOTICE) From patchwork Wed Jun 16 04:09:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 94287 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 740D8A0C45; Wed, 16 Jun 2021 06:11:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C9E49410E8; Wed, 16 Jun 2021 06:10:23 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2070.outbound.protection.outlook.com [40.107.243.70]) by mails.dpdk.org (Postfix) with ESMTP id AA26341159 for ; Wed, 16 Jun 2021 06:10:20 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iIxRYmusOh37vHcSB2TWdsOdrTivUso6SB5owJWd5aCGzRBfgXPGbhaxQPjxqzCCtAkCR0Ey88FRlTtIRl0NiFKmURw9uPmgdMiaxXlBvtyHcVLEoRXLm4fXHfT03T27XBof6QUT4AC4YuoLHac1T5OjWI7ymY2rMAXgMHi5DJEnqfL0P0HgfleuAcJDwGr3n+dnWcdR44dJaZyS8poDrlrcAtJTBOLplVJgY8929sANCa0t1+syU8HmRXdOXn2BmrM1b+Da2qnURttNgn8rRu6fcMUqJguYgw839OAVJQGXXX6TG+5laWm1LdKwNxQ1AdwMC7hiy11svH7KuI5c5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HIubkpiBjIx8wfIJnVZjKYKLwtl9fqJavcjVXm3nFmE=; b=kZcx+y3FG8JYgm8l4toYW5I2/QD93yCPvIga0Ut118hCYdIDOOdpKIDqY7DSTKDA2ePf0WlPPwlJvfgUPRKKkuZ72dCmFn8P+cgTw4a4Sylhc+OP8VfL3xuydJMo0I7QEfIWy9YUG1Z40w/29y39eqUDY6qfOtMJN+pP7/x5u+a33lW2c2KQfZG9Z/ysl3alPkq5qWaapSbo4bTty69BmXnQyRHFtv/cog5O8VZBEtAartooSfm2zE3jYyTOiZXkJy7Uuf7fK8NVsOF9ODRAFXd8V1bnpEfjIKk37XFBCmosEWvfQPoJIDD58czjNxkq7FP7Ga7wMrOgComeSF3KpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=tuxdriver.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HIubkpiBjIx8wfIJnVZjKYKLwtl9fqJavcjVXm3nFmE=; b=IJZUh5IoIuLM9rtka5L+dfjzaJa7jQzyvHe+tAoCAXfNAEkl+gPci0XpNOtRY2gFda4ul59kKuz0fr6EEmQbPfKDeolzw3285Yp9A51YgO7reiv1A4eNfVtwvbOH5IKjEa90FsI7QiTc07ihUM3AUjUL6i1k7Y5jGjwxk2bq1s5nWj6EIHgadq4vmCRd9WNHiVQMHFFsz4/J8WwDQ1gkegzs4MI/l/syxSE0mC0EkvhXj/04oDe571ok42vi7jSppzqoup+E6hYYStMFK9sJ93Oa3pxRn9skTWmGcBb1njrVSCD93rxpKxWDFU5s12saJ2iQmEnPfJ3ZklWlws350g== Received: from BN0PR04CA0101.namprd04.prod.outlook.com (2603:10b6:408:ec::16) by BN8PR12MB3075.namprd12.prod.outlook.com (2603:10b6:408:67::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Wed, 16 Jun 2021 04:10:19 +0000 Received: from BN8NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ec:cafe::1) by BN0PR04CA0101.outlook.office365.com (2603:10b6:408:ec::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.20 via Frontend Transport; Wed, 16 Jun 2021 04:10:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; tuxdriver.com; dkim=none (message not signed) header.d=none;tuxdriver.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT049.mail.protection.outlook.com (10.13.177.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 04:10:18 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 16 Jun 2021 04:10:14 +0000 From: Xueming Li To: Viacheslav Ovsiienko CC: , , Matan Azrad , Shahaf Shuler , Ray Kinsella , Neil Horman Date: Wed, 16 Jun 2021 07:09:35 +0300 Message-ID: <20210616040935.311733-15-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210527133759.17401-2-xuemingl@nvidia.com> References: <20210527133759.17401-2-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: db003786-c220-4307-496c-08d9307ca758 X-MS-TrafficTypeDiagnostic: BN8PR12MB3075: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:586; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lhJlOpUapESTlK7Wkh6G0nB89OJZtg86l80fd0aHAVQSBz2FRxkKlWRpAs5bHnDUmBwq+Hi5LwkKZvf2mLtC57JftmFVQZIdjFF6FCd3q0EkwPM/2UjkwKoXaNvf+g0NTbSQa0C492rYLJrLATk9Z5KYx1GQMuZscQz92S7YZRHLbBhUsU/npCf5YVvIbdDr4rzidLv/thweqw1vizkiBkiBWbtJj2BKoCIiIvluudim3OJvpwhkApQb5EOpVl94zgCxNCJUZw8wsM/jN47zw0TmBdD+t7GDAaa2sqDJd8kWNZOTv5wWcwaO9b3G2BwUAjQQwDUIp0w067b8lqFaA0Th7K0wva5JsrVPyHXH0k9fjTnIi8s95WaqQsdB8INCTOg3n6IyORHdBj/EIk1dKxyvlQBZLdkEGBIPWiBjYsmyW2ZXHVfEAU8Sz4APvHwX2ZufnFOQsCNKZv6znS3ylYTA1XaSkJUwjYCLi5fiI9PDV0DrAdFXEna9s8Wmd9lDOCvcF8DmIfCF8PdX2B+kxavcENH0w0SYVD/hH8JNb3yzKwJwr2RvVHovA2OGLchITO19/U2bnTbCFGu7DCeGaEoj40eqlcj2NE8WVXa3+m6ou3qXyATFnUXsaRWPXOtLIaLHJPI8YKmXptLVBFidMg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(376002)(39860400002)(346002)(136003)(46966006)(36840700001)(1076003)(36860700001)(2906002)(6286002)(86362001)(8676002)(8936002)(316002)(83380400001)(36756003)(82310400003)(7696005)(6666004)(6862004)(70206006)(55016002)(26005)(82740400003)(356005)(7636003)(5660300002)(16526019)(70586007)(30864003)(336012)(186003)(54906003)(426003)(2616005)(6636002)(4326008)(37006003)(36906005)(478600001)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 04:10:18.9016 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: db003786-c220-4307-496c-08d9307ca758 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3075 Subject: [dpdk-dev] [PATCH v1 14/14] common/mlx5: clean up legacy PCI bus driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Clean up legacy PCI bus driver since all mlx5 PMDs moved to new common PCI bus driver. Signed-off-by: Xueming Li --- drivers/common/mlx5/linux/mlx5_common_os.h | 1 - drivers/common/mlx5/mlx5_common.c | 1 - drivers/common/mlx5/mlx5_common.h | 1 + drivers/common/mlx5/mlx5_common_pci.c | 433 +-------------------- drivers/common/mlx5/mlx5_common_pci.h | 77 ---- drivers/common/mlx5/mlx5_common_private.h | 1 + drivers/common/mlx5/version.map | 3 - 7 files changed, 4 insertions(+), 513 deletions(-) delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h index 86d0cb09b0..2b03bf811e 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.h +++ b/drivers/common/mlx5/linux/mlx5_common_os.h @@ -289,7 +289,6 @@ mlx5_os_free(void *addr) free(addr); } -__rte_internal struct ibv_device * mlx5_os_get_ibv_device(const struct rte_pci_addr *addr); diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 4ff13cb461..6c83cf4bcd 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -14,7 +14,6 @@ #include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" uint8_t haswell_broadwell_cpu; diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 7d7b896517..85855345a9 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c index 6fe28defbf..8b38091d87 100644 --- a/drivers/common/mlx5/mlx5_common_pci.c +++ b/drivers/common/mlx5/mlx5_common_pci.c @@ -8,431 +8,17 @@ #include #include #include +#include +#include #include "mlx5_common_log.h" -#include "mlx5_common_pci.h" #include "mlx5_common_private.h" static struct rte_pci_driver mlx5_common_pci_driver; -/********** Legacy PCI bus driver, to be removed ********/ - -struct mlx5_pci_device { - struct rte_pci_device *pci_dev; - TAILQ_ENTRY(mlx5_pci_device) next; - uint32_t classes_loaded; -}; - -/* Head of list of drivers. */ -static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list = - TAILQ_HEAD_INITIALIZER(drv_list); - -/* Head of mlx5 pci devices. */ -static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = - TAILQ_HEAD_INITIALIZER(devices_list); - -static const struct { - const char *name; - unsigned int driver_class; -} mlx5_classes[] = { - { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA }, - { .name = "net", .driver_class = MLX5_CLASS_NET }, - { .name = "regex", .driver_class = MLX5_CLASS_REGEX }, - { .name = "compress", .driver_class = MLX5_CLASS_COMPRESS }, -}; - -static const unsigned int mlx5_class_combinations[] = { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, - MLX5_CLASS_REGEX, - MLX5_CLASS_COMPRESS, - MLX5_CLASS_NET | MLX5_CLASS_REGEX, - MLX5_CLASS_VDPA | MLX5_CLASS_REGEX, - MLX5_CLASS_NET | MLX5_CLASS_COMPRESS, - MLX5_CLASS_VDPA | MLX5_CLASS_COMPRESS, - MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, - MLX5_CLASS_NET | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, - MLX5_CLASS_VDPA | MLX5_CLASS_REGEX | MLX5_CLASS_COMPRESS, - /* New class combination should be added here. */ -}; - -static int -class_name_to_value(const char *class_name) -{ - unsigned int i; - - for (i = 0; i < RTE_DIM(mlx5_classes); i++) { - if (strcmp(class_name, mlx5_classes[i].name) == 0) - return mlx5_classes[i].driver_class; - } - return -EINVAL; -} - -static struct mlx5_pci_driver * -driver_get(uint32_t class) -{ - struct mlx5_pci_driver *driver; - - TAILQ_FOREACH(driver, &drv_list, next) { - if (driver->driver_class == class) - return driver; - } - return NULL; -} - -static int -bus_cmdline_options_handler(__rte_unused const char *key, - const char *class_names, void *opaque) -{ - int *ret = opaque; - char *nstr_org; - int class_val; - char *found; - char *nstr; - char *refstr = NULL; - - *ret = 0; - nstr = strdup(class_names); - if (!nstr) { - *ret = -ENOMEM; - return *ret; - } - nstr_org = nstr; - found = strtok_r(nstr, ":", &refstr); - if (!found) - goto err; - do { - /* Extract each individual class name. Multiple - * class key,value is supplied as class=net:vdpa:foo:bar. - */ - class_val = class_name_to_value(found); - /* Check if its a valid class. */ - if (class_val < 0) { - *ret = -EINVAL; - goto err; - } - *ret |= class_val; - found = strtok_r(NULL, ":", &refstr); - } while (found); -err: - free(nstr_org); - if (*ret < 0) - DRV_LOG(ERR, "Invalid mlx5 class options %s." - " Maybe typo in device class argument setting?", - class_names); - return *ret; -} - -static int -parse_class_options(const struct rte_devargs *devargs) -{ - const char *key = RTE_DEVARGS_KEY_CLASS; - struct rte_kvargs *kvlist; - int ret = 0; - - if (devargs == NULL) - return 0; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return 0; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, - &ret); - rte_kvargs_free(kvlist); - return ret; -} - -static bool -mlx5_bus_match(const struct mlx5_pci_driver *drv, - const struct rte_pci_device *pci_dev) -{ - const struct rte_pci_id *id_table; - - for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; - id_table++) { - /* Check if device's ids match the class driver's ids. */ - if (id_table->vendor_id != pci_dev->id.vendor_id && - id_table->vendor_id != RTE_PCI_ANY_ID) - continue; - if (id_table->device_id != pci_dev->id.device_id && - id_table->device_id != RTE_PCI_ANY_ID) - continue; - if (id_table->subsystem_vendor_id != - pci_dev->id.subsystem_vendor_id && - id_table->subsystem_vendor_id != RTE_PCI_ANY_ID) - continue; - if (id_table->subsystem_device_id != - pci_dev->id.subsystem_device_id && - id_table->subsystem_device_id != RTE_PCI_ANY_ID) - continue; - if (id_table->class_id != pci_dev->id.class_id && - id_table->class_id != RTE_CLASS_ANY_ID) - continue; - return true; - } - return false; -} - -static int -is_valid_class_combination(uint32_t user_classes) -{ - unsigned int i; - - /* Verify if user specified valid supported combination. */ - for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { - if (mlx5_class_combinations[i] == user_classes) - return 0; - } - /* Not found any valid class combination. */ - return -EINVAL; -} - -static struct mlx5_pci_device * -pci_to_mlx5_device(const struct rte_pci_device *pci_dev) -{ - struct mlx5_pci_device *dev; - - TAILQ_FOREACH(dev, &devices_list, next) { - if (dev->pci_dev == pci_dev) - return dev; - } - return NULL; -} - -static bool -device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) -{ - return (device->classes_loaded & class) ? true : false; -} - -static void -dev_release(struct mlx5_pci_device *dev) -{ - TAILQ_REMOVE(&devices_list, dev, next); - rte_free(dev); -} - -static int -drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) -{ - struct mlx5_pci_driver *driver; - int local_ret = -ENODEV; - unsigned int i = 0; - int ret = 0; - - enabled_classes &= dev->classes_loaded; - while (enabled_classes) { - driver = driver_get(RTE_BIT64(i)); - if (driver) { - local_ret = driver->pci_driver.remove(dev->pci_dev); - if (!local_ret) - dev->classes_loaded &= ~RTE_BIT64(i); - else if (ret == 0) - ret = local_ret; - } - enabled_classes &= ~RTE_BIT64(i); - i++; - } - if (local_ret) - ret = local_ret; - return ret; -} - -static int -drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv, - struct rte_pci_device *pci_dev, uint32_t user_classes) -{ - struct mlx5_pci_driver *driver; - uint32_t enabled_classes = 0; - bool already_loaded; - int ret; - - TAILQ_FOREACH(driver, &drv_list, next) { - if ((driver->driver_class & user_classes) == 0) - continue; - if (!mlx5_bus_match(driver, pci_dev)) - continue; - already_loaded = dev->classes_loaded & driver->driver_class; - if (already_loaded && - !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { - DRV_LOG(ERR, "Device %s is already probed", - pci_dev->device.name); - ret = -EEXIST; - goto probe_err; - } - ret = driver->pci_driver.probe(pci_drv, pci_dev); - if (ret < 0) { - DRV_LOG(ERR, "Failed to load driver %s", - driver->pci_driver.driver.name); - goto probe_err; - } - enabled_classes |= driver->driver_class; - } - dev->classes_loaded |= enabled_classes; - return 0; -probe_err: - /* Only unload drivers which are enabled which were enabled - * in this probe instance. - */ - drivers_remove(dev, enabled_classes); - return ret; -} - -/** - * DPDK callback to register to probe multiple drivers for a PCI device. - * - * @param[in] pci_drv - * PCI driver structure. - * @param[in] dev - * PCI device information. - * - * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. - */ -static int -mlx5_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, - struct rte_pci_device *pci_dev) -{ - struct mlx5_pci_device *dev; - uint32_t user_classes = 0; - bool new_device = false; - int ret; - - ret = parse_class_options(pci_dev->device.devargs); - if (ret < 0) - return ret; - user_classes = ret; - if (user_classes) { - /* Validate combination here. */ - ret = is_valid_class_combination(user_classes); - if (ret) { - DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); - return ret; - } - } else { - /* Default to net class. */ - user_classes = MLX5_CLASS_NET; - } - dev = pci_to_mlx5_device(pci_dev); - if (!dev) { - dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); - if (!dev) - return -ENOMEM; - dev->pci_dev = pci_dev; - TAILQ_INSERT_HEAD(&devices_list, dev, next); - new_device = true; - } - ret = drivers_probe(dev, pci_drv, pci_dev, user_classes); - if (ret) - goto class_err; - return 0; -class_err: - if (new_device) - dev_release(dev); - return ret; -} - -/** - * DPDK callback to remove one or more drivers for a PCI device. - * - * This function removes all drivers probed for a given PCI device. - * - * @param[in] pci_dev - * Pointer to the PCI device. - * - * @return - * 0 on success, the function cannot fail. - */ -static int -mlx5_pci_remove(struct rte_pci_device *pci_dev) -{ - struct mlx5_pci_device *dev; - int ret; - - dev = pci_to_mlx5_device(pci_dev); - if (!dev) - return -ENODEV; - /* Matching device found, cleanup and unload drivers. */ - ret = drivers_remove(dev, dev->classes_loaded); - if (!ret) - dev_release(dev); - return ret; -} - -static int -mlx5_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, - uint64_t iova, size_t len) -{ - struct mlx5_pci_driver *driver = NULL; - struct mlx5_pci_driver *temp; - struct mlx5_pci_device *dev; - int ret = -EINVAL; - - dev = pci_to_mlx5_device(pci_dev); - if (!dev) - return -ENODEV; - TAILQ_FOREACH(driver, &drv_list, next) { - if (device_class_enabled(dev, driver->driver_class) && - driver->pci_driver.dma_map) { - ret = driver->pci_driver.dma_map(pci_dev, addr, - iova, len); - if (ret) - goto map_err; - } - } - return ret; -map_err: - TAILQ_FOREACH(temp, &drv_list, next) { - if (temp == driver) - break; - if (device_class_enabled(dev, temp->driver_class) && - temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) - temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); - } - return ret; -} - -static int -mlx5_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, - uint64_t iova, size_t len) -{ - struct mlx5_pci_driver *driver; - struct mlx5_pci_device *dev; - int local_ret = -EINVAL; - int ret; - - dev = pci_to_mlx5_device(pci_dev); - if (!dev) - return -ENODEV; - ret = 0; - /* There is no unmap error recovery in current implementation. */ - TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) { - if (device_class_enabled(dev, driver->driver_class) && - driver->pci_driver.dma_unmap) { - local_ret = driver->pci_driver.dma_unmap(pci_dev, addr, - iova, len); - if (local_ret && (ret == 0)) - ret = local_ret; - } - } - if (local_ret) - ret = local_ret; - return ret; -} - /* PCI ID table is build dynamically based on registered mlx5 drivers. */ static struct rte_pci_id *mlx5_pci_id_table; -static struct rte_pci_driver mlx5_pci_driver = { - .driver = { - .name = MLX5_PCI_DRIVER_NAME, - }, - .probe = mlx5_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_pci_dma_map, - .dma_unmap = mlx5_pci_dma_unmap, -}; - static int pci_id_table_size_get(const struct rte_pci_id *id_table) { @@ -509,7 +95,6 @@ pci_ids_table_update(const struct rte_pci_id *driver_id_table) } /* Terminate table with empty entry. */ updated_table[i].vendor_id = 0; - mlx5_pci_driver.id_table = updated_table; mlx5_common_pci_driver.id_table = updated_table; mlx5_pci_id_table = updated_table; if (old_table) @@ -517,20 +102,6 @@ pci_ids_table_update(const struct rte_pci_id *driver_id_table) return 0; } -void -mlx5_pci_driver_register(struct mlx5_pci_driver *driver) -{ - int ret; - - ret = pci_ids_table_update(driver->pci_driver.id_table); - if (ret) - return; - mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags; - TAILQ_INSERT_TAIL(&drv_list, driver, next); -} - -/********** New common PCI bus driver ********/ - bool mlx5_dev_is_pci(const struct rte_device *dev) { diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h deleted file mode 100644 index de89bb98bc..0000000000 --- a/drivers/common/mlx5/mlx5_common_pci.h +++ /dev/null @@ -1,77 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright 2020 Mellanox Technologies, Ltd - */ - -#ifndef _MLX5_COMMON_PCI_H_ -#define _MLX5_COMMON_PCI_H_ - -/** - * @file - * - * RTE Mellanox PCI Driver Interface - * Mellanox ConnectX PCI device supports multiple class: net,vdpa,regex and - * compress devices. This layer enables creating such multiple class of devices - * on a single PCI device by allowing to bind multiple class specific device - * driver to attach to mlx5_pci driver. - * - * ----------- ------------ ------------- ---------------- - * | mlx5 | | mlx5 | | mlx5 | | mlx5 | - * | net pmd | | vdpa pmd | | regex pmd | | compress pmd | - * ----------- ------------ ------------- ---------------- - * \ \ / / - * \ \ / / - * \ \_--------------_/ / - * \_______________| mlx5 |_______________/ - * | pci common | - * -------------- - * | - * ----------- - * | mlx5 | - * | pci dev | - * ----------- - * - * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI - * ID table of all related mlx5 PCI devices. - * - mlx5 class driver such as net, vdpa, regex PMD defines its - * specific PCI ID table and mlx5 bus driver probes matching - * class drivers. - * - mlx5 pci bus driver is cental place that validates supported - * class combinations. - */ - -#ifdef __cplusplus -extern "C" { -#endif /* __cplusplus */ - -#include -#include - -#include - -void mlx5_common_pci_init(void); - -/** - * A structure describing a mlx5 pci driver. - */ -struct mlx5_pci_driver { - struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ - uint32_t driver_class; /**< Class of this driver, enum mlx5_class */ - TAILQ_ENTRY(mlx5_pci_driver) next; -}; - -/** - * Register a mlx5_pci device driver. - * - * @param driver - * A pointer to a mlx5_pci_driver structure describing the driver - * to be registered. - */ -__rte_internal -void -mlx5_pci_driver_register(struct mlx5_pci_driver *driver); - -#ifdef __cplusplus -} -#endif /* __cplusplus */ - -#endif /* _MLX5_COMMON_PCI_H_ */ diff --git a/drivers/common/mlx5/mlx5_common_private.h b/drivers/common/mlx5/mlx5_common_private.h index 1096fa85e7..c929840408 100644 --- a/drivers/common/mlx5/mlx5_common_private.h +++ b/drivers/common/mlx5/mlx5_common_private.h @@ -31,6 +31,7 @@ int mlx5_common_dev_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, /* Common PCI bus driver: */ +void mlx5_common_pci_init(void); void mlx5_common_driver_on_register_pci(struct mlx5_class_driver *driver); bool mlx5_dev_pci_match(const struct mlx5_class_driver *drv, const struct rte_device *dev); diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index ea4c49b7e7..39bc7bad23 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -130,13 +130,10 @@ INTERNAL { mlx5_nl_vlan_vmwa_create; # WINDOWS_NO_EXPORT mlx5_nl_vlan_vmwa_delete; # WINDOWS_NO_EXPORT - mlx5_pci_driver_register; - mlx5_os_alloc_pd; mlx5_os_dealloc_pd; mlx5_os_dereg_mr; mlx5_os_get_ibv_dev; # WINDOWS_NO_EXPORT - mlx5_os_get_ibv_device; # WINDOWS_NO_EXPORT mlx5_os_reg_mr; mlx5_os_umem_dereg; mlx5_os_umem_reg;