From patchwork Thu Jan 25 13:30:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 136155 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CCC1C439C1; Thu, 25 Jan 2024 14:32:45 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4646342E60; Thu, 25 Jan 2024 14:31:31 +0100 (CET) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by mails.dpdk.org (Postfix) with ESMTP id 74DDE42E5E for ; Thu, 25 Jan 2024 14:31:30 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I6IzF4tja3qUXal+Y8dRXdSHDCXQBVhgu8PTrEs6WCeIinoYcjPzSQxJ8zw++GIE2qHlYfzzOgKFkBHZ/jRSb/fg76EwWvcI4YwuTY5gxPKxpSE8kMDQefY+qJd9ycslCTOqeISeus67gqUs3VOoGzYooYkR8JCDFIEfbxsLbeBW16wk4YWGo/+nuOsykRwJjAMPxW7N6Ds4ZNMlASq090Tt+QLSEveeJGG6JRPMTi+zGCHVvYKQc2IUNHYKKSbAMLVYqBFB7uWvGc4wH6QE4JZVcNW+g/ZKy5lIegPKcDhNagkucD5+7+mMWQbDGVERiJgfAfYPbK/z6PfB1PEr4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OulndnG4DcNYWOGzCvk7RvTtsXvJ6GRuemVwqQROocI=; b=isLWFOb6p5sDjk7bsiAkytJuGa2yYIM8Rt07uemuk1ZGSQYeU1JSujjQJp+GVUPBIilnh+Uk3iFtjRcIiaHzM92geiahRrZRsd5jzGPK4QTifHq5r4quzKPV0WcDGL80BYZKY12cwrKB+95Sl0EGlez68iFZRL3dLYOmJp9/D+ARZYN37ufo4oKHOoTrhVgINnK+c9lAMPxhlC5ZJMc8XXECuqaYhdG2QHRujWn5tOQdiaLej6s4BJ1Mr5k3lgjjg0w7p8SQmvzM98jnePhWlsgbHLhD+IsBzEFVAUuVk6QbBpeGpEkHf5/rscw+1R+4gLqT8H8vqRe5b5bjPdTcig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OulndnG4DcNYWOGzCvk7RvTtsXvJ6GRuemVwqQROocI=; b=M44GVC1lScz/FSuEAP8837q+3uKUYz5cyUW5AhfZ7eIFc1Cb2EuGfIrsupEnCBsCkcEQ/fMfWNgeY1i+ZpUJJQgrmYlEoTUphK0Ygt1vs9Fwchly1geSE42OsFIN4+zymDadSe1F8SRvGHFvAuKKpLuXxY4dCgITNFfd6hzYIQ516vJ064Q1zeMNAdaxCPUIPdvPK35xS1nRk1e697almU0FPo3YfGJiAP/LwkhW1/FI0akQh2srNihHK4wz4Z6maCjGuPt/BqifILe1bpEOFIoeXXtJeweO0JYt4ZO9TDO/TeRV/APuWNT5/P1QDsbmqTItpM1i87n5PLd3Fn0lLA== Received: from DM5PR07CA0052.namprd07.prod.outlook.com (2603:10b6:4:ad::17) by CY5PR12MB6624.namprd12.prod.outlook.com (2603:10b6:930:40::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.26; Thu, 25 Jan 2024 13:31:27 +0000 Received: from CY4PEPF0000EDD2.namprd03.prod.outlook.com (2603:10b6:4:ad:cafe::5) by DM5PR07CA0052.outlook.office365.com (2603:10b6:4:ad::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.26 via Frontend Transport; Thu, 25 Jan 2024 13:31:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CY4PEPF0000EDD2.mail.protection.outlook.com (10.167.241.206) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.16 via Frontend Transport; Thu, 25 Jan 2024 13:31:27 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 25 Jan 2024 05:31:20 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 25 Jan 2024 05:31:19 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Thu, 25 Jan 2024 05:31:17 -0800 From: Michael Baum To: CC: Matan Azrad , Raslan Darawsheh , Dariusz Sosnowski , Viacheslav Ovsiienko , Ori Kam , Suanming Mou Subject: [PATCH v2 12/23] net/mlx5: add physical device handle Date: Thu, 25 Jan 2024 15:30:32 +0200 Message-ID: <20240125133043.575860-13-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240125133043.575860-1-michaelba@nvidia.com> References: <20231203112543.844014-1-michaelba@nvidia.com> <20240125133043.575860-1-michaelba@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD2:EE_|CY5PR12MB6624:EE_ X-MS-Office365-Filtering-Correlation-Id: 8ccc0e81-4413-42dd-35ef-08dc1da9ef07 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xZfCYXS9HagmOrf0X+VkuY0vF6koX0byTLFnVBHfsOj0iwxl/9X8C5M4SFcjHQJGo1i3k/1QnsHAiCqSJtzNRPXYvXQBBvXmQIJ2wRgQ8jHJ61KXfx5DN2iyCtdbUW77vO3jsYhPJ8jBZWSWP+5fY6Xnx19gjfeYbWpOEFf6kmtjgV3ORnRa91Q/FODHIrimhi7NdoWxUQIoaeNOpFpQht5BuXMGAlsQKYYWA+7I6b5jVML8HdgetS+YQrjHVlQNm+QkYV2KASGG1XyT4FbzYRaMBZkGMxwE2BsmiO2Fc+Fvqt3qaMWN2P1LbAZWksIiU7iSUmxmDVA+RjGyQr6MFDS7+v548lVTkeCM2rroPY0O8LT/a2Hc7a01VXp4S5u8sOyW1AXMz+/7lfEoraYSoq5eeZSE2rMZcv9BgP3ZPfSQUD2kmzqVM0nolgBI4BgiMLhe0EdAXKL2iUfEJL0iauqfvC0WdUVxEusUdLsIqIJq1mGFzf2+zhgwTbv8bgTB/zHpFonhlPpMTcPlPrfA6Uprucp1WvlM/6T+j1Pw02azT8RKZR5wBSAfDXXeU9GsUq9j6cYCnkshOVqlVb5vm7QSnApToajaOBoSiNEkkzq0bxmjZ6FeStF7XLbtrULGVFr7R3OON2WlxpC5CCRrcu055xrgTyvhZjV2AAl76vUtsapkQsUJxXF4UD5UAyctAAP77rtJK7AiNGtPzhXIPxhOh8bH8saKJbFyrEV+0MWMniciAItctZ6lY29X/Khd X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(376002)(136003)(396003)(346002)(230922051799003)(1800799012)(451199024)(64100799003)(82310400011)(186009)(46966006)(36840700001)(40470700004)(41300700001)(82740400003)(356005)(7636003)(40480700001)(54906003)(6916009)(70586007)(70206006)(316002)(86362001)(8676002)(6666004)(8936002)(55016003)(478600001)(7696005)(40460700003)(36756003)(4326008)(36860700001)(2906002)(5660300002)(47076005)(26005)(2616005)(1076003)(6286002)(426003)(336012)(107886003)(83380400001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 13:31:27.5579 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8ccc0e81-4413-42dd-35ef-08dc1da9ef07 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6624 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add structure describing physical device, and manage physical device global list. Signed-off-by: Michael Baum Acked-by: Suanming Mou --- drivers/net/mlx5/mlx5.c | 77 ++++++++++++++++++++++++++++++++++++----- drivers/net/mlx5/mlx5.h | 13 +++++++ 2 files changed, 82 insertions(+), 8 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 3a182de248..f9fc652136 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -190,9 +190,10 @@ struct mlx5_shared_data *mlx5_shared_data; /** Driver-specific log messages type. */ int mlx5_logtype; -static LIST_HEAD(, mlx5_dev_ctx_shared) mlx5_dev_ctx_list = - LIST_HEAD_INITIALIZER(); +static LIST_HEAD(mlx5_dev_ctx_list, mlx5_dev_ctx_shared) dev_ctx_list = LIST_HEAD_INITIALIZER(); +static LIST_HEAD(mlx5_phdev_list, mlx5_physical_device) phdev_list = LIST_HEAD_INITIALIZER(); static pthread_mutex_t mlx5_dev_ctx_list_mutex; + static const struct mlx5_indexed_pool_config mlx5_ipool_cfg[] = { #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || !defined(HAVE_INFINIBAND_VERBS_H) [MLX5_IPOOL_DECAP_ENCAP] = { @@ -1692,6 +1693,60 @@ mlx5_init_shared_dev_registers(struct mlx5_dev_ctx_shared *sh) mlx5_init_hws_flow_tags_registers(sh); } +static struct mlx5_physical_device * +mlx5_get_physical_device(struct mlx5_common_device *cdev) +{ + struct mlx5_physical_device *phdev; + struct mlx5_hca_attr *attr = &cdev->config.hca_attr; + + /* Search for physical device by system_image_guid. */ + LIST_FOREACH(phdev, &phdev_list, next) { + if (phdev->guid == attr->system_image_guid) { + phdev->refcnt++; + return phdev; + } + } + phdev = mlx5_malloc(MLX5_MEM_ZERO | MLX5_MEM_RTE, + sizeof(struct mlx5_physical_device), + RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); + if (!phdev) { + DRV_LOG(ERR, "Physical device allocation failure."); + rte_errno = ENOMEM; + return NULL; + } + phdev->guid = attr->system_image_guid; + phdev->refcnt = 1; + LIST_INSERT_HEAD(&phdev_list, phdev, next); + DRV_LOG(DEBUG, "Physical device is created, guid=%" PRIu64 ".", + phdev->guid); + return phdev; +} + +static void +mlx5_physical_device_destroy(struct mlx5_physical_device *phdev) +{ +#ifdef RTE_LIBRTE_MLX5_DEBUG + /* Check the object presence in the list. */ + struct mlx5_physical_device *lphdev; + + LIST_FOREACH(lphdev, &phdev_list, next) + if (lphdev == phdev) + break; + MLX5_ASSERT(lphdev); + if (lphdev != phdev) { + DRV_LOG(ERR, "Freeing non-existing physical device"); + return; + } +#endif + MLX5_ASSERT(phdev); + MLX5_ASSERT(phdev->refcnt); + if (--phdev->refcnt) + return; + /* Remove physical device from the global device list. */ + LIST_REMOVE(phdev, next); + mlx5_free(phdev); +} + /** * Allocate shared device context. If there is multiport device the * master and representors will share this context, if there is single @@ -1725,7 +1780,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); pthread_mutex_lock(&mlx5_dev_ctx_list_mutex); /* Search for IB context by device name. */ - LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) { + LIST_FOREACH(sh, &dev_ctx_list, next) { if (!strcmp(sh->ibdev_name, spawn->phys_dev_name)) { sh->refcnt++; goto exit; @@ -1765,6 +1820,9 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, sizeof(sh->ibdev_name) - 1); strncpy(sh->ibdev_path, mlx5_os_get_ctx_device_path(sh->cdev->ctx), sizeof(sh->ibdev_path) - 1); + sh->phdev = mlx5_get_physical_device(sh->cdev); + if (!sh->phdev) + goto error; /* * Setting port_id to max unallowed value means there is no interrupt * subhandler installed for the given port index i. @@ -1798,7 +1856,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, #endif } mlx5_os_dev_shared_handler_install(sh); - if (LIST_EMPTY(&mlx5_dev_ctx_list)) { + if (LIST_EMPTY(&dev_ctx_list)) { err = mlx5_flow_os_init_workspace_once(); if (err) goto error; @@ -1811,7 +1869,7 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, mlx5_flow_aging_init(sh); mlx5_flow_ipool_create(sh); /* Add context to the global device list. */ - LIST_INSERT_HEAD(&mlx5_dev_ctx_list, sh, next); + LIST_INSERT_HEAD(&dev_ctx_list, sh, next); rte_spinlock_init(&sh->geneve_tlv_opt_sl); mlx5_init_shared_dev_registers(sh); /* Init counter pool list header and lock. */ @@ -1833,6 +1891,8 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, } while (++i <= (uint32_t)sh->bond.n_port); if (sh->td) claim_zero(mlx5_devx_cmd_destroy(sh->td)); + if (sh->phdev) + mlx5_physical_device_destroy(sh->phdev); mlx5_free(sh); rte_errno = err; return NULL; @@ -1919,7 +1979,7 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) /* Check the object presence in the list. */ struct mlx5_dev_ctx_shared *lctx; - LIST_FOREACH(lctx, &mlx5_dev_ctx_list, next) + LIST_FOREACH(lctx, &dev_ctx_list, next) if (lctx == sh) break; MLX5_ASSERT(lctx); @@ -1945,7 +2005,7 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) /* Remove context from the global device list. */ LIST_REMOVE(sh, next); /* Release resources on the last device removal. */ - if (LIST_EMPTY(&mlx5_dev_ctx_list)) { + if (LIST_EMPTY(&dev_ctx_list)) { mlx5_os_net_cleanup(); mlx5_flow_os_release_workspace(); } @@ -1985,6 +2045,7 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) MLX5_ASSERT(sh->geneve_tlv_option_resource == NULL); pthread_mutex_destroy(&sh->txpp.mutex); mlx5_lwm_unset(sh); + mlx5_physical_device_destroy(sh->phdev); mlx5_free(sh); return; exit: @@ -2929,7 +2990,7 @@ mlx5_probe_again_args_validate(struct mlx5_common_device *cdev, return 0; pthread_mutex_lock(&mlx5_dev_ctx_list_mutex); /* Search for IB context by common device pointer. */ - LIST_FOREACH(sh, &mlx5_dev_ctx_list, next) + LIST_FOREACH(sh, &dev_ctx_list, next) if (sh->cdev == cdev) break; pthread_mutex_unlock(&mlx5_dev_ctx_list_mutex); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 779805bcd8..8bf7f86416 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -1419,6 +1419,18 @@ struct mlx5_dev_registers { #define HAVE_MLX5_DR_CREATE_ACTION_ASO_EXT #endif +/** + * Physical device structure. + * This device is created once per NIC to manage recourses shared by all ports + * under same physical device. + */ +struct mlx5_physical_device { + LIST_ENTRY(mlx5_physical_device) next; + struct mlx5_dev_ctx_shared *sh; /* Created on sherd context. */ + uint64_t guid; /* System image guid, the uniq ID of physical device. */ + uint32_t refcnt; +}; + /* * Shared Infiniband device context for Master/Representors * which belong to same IB device with multiple IB ports. @@ -1450,6 +1462,7 @@ struct mlx5_dev_ctx_shared { uint32_t max_port; /* Maximal IB device port index. */ struct mlx5_bond_info bond; /* Bonding information. */ struct mlx5_common_device *cdev; /* Backend mlx5 device. */ + struct mlx5_physical_device *phdev; /* Backend physical device. */ uint32_t tdn; /* Transport Domain number. */ char ibdev_name[MLX5_FS_NAME_MAX]; /* SYSFS dev name. */ char ibdev_path[MLX5_FS_PATH_MAX]; /* SYSFS dev path for secondary */