From patchwork Fri Oct 22 09:11:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xueming(Steven) Li" X-Patchwork-Id: 102631 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 821CEA0C43; Fri, 22 Oct 2021 11:12:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6D6A641186; Fri, 22 Oct 2021 11:12:12 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2083.outbound.protection.outlook.com [40.107.94.83]) by mails.dpdk.org (Postfix) with ESMTP id 2FFAB41181 for ; Fri, 22 Oct 2021 11:12:11 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g/6PCtr/AGQ2Z+4u5Qdcw96MLXP0FXhuSGjyrZDXC/X9I+CTI42zKhUxFjqtmSU786+JRybWZ/YaoL89An71vhPi6RRohYbZhlHEMA9R3Euds6JekSURjK3OOWcC7egznEeGlywNliNDIqfibZFOKnZUpu7gRduMmaYMmRYICeWMA+TAutM6pHIkuUakwJVLTIr30+mvRpcGwxNv2rcLvHWW4bHniKnei+9nEnxo9lmfJAKHgBbtS9ej9EL23yoyT3RNoQY4bnsWgadUkSE8t21v/b0BTKLk9k83cAJeuNSbWQniAbwF0F7UDsncbghrAlR237sibK7XLCBa+X5fIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PS1JRSdpPmj3i5MtiaCEnUC41334hW2C/nhSUApiFt0=; b=lIojm0+vR5JTBdacPaSFoTfwPB3cbcwjI7rQ66UrbE0B1uTsq7iauGyWYPdoP6gzRRJWr0/dVvy0idrEreA7KeRe+KlxrlGmdRCrqRz+SSc0wiLGzf4vVIko9pcVfHvqegJC1e19Fi0rBEtoh4sVLqrLS/aMiGrKI2HdxF5BDHKuCfWkkbey1ndGOAlM0pxEh3WO51lcqdGrqpo5WJEYivGIfDUUvviLxbD5jYp+/yQoYtBXpGxi2wOfxV1DFCce5ZLQ/W84HWm2yIPzXtBwgoEOFuS+mSID4IYD1GWfe97VR/qMS2soVoV9JqDHBHY7FdrsnG3v7ZJHM0hNkvrA+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=ashroe.eu smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PS1JRSdpPmj3i5MtiaCEnUC41334hW2C/nhSUApiFt0=; b=dxpl8OJISPF9wEnvUcaC7OtpvEYLHGmqDSaWzSXphqDjEOuaE4KusgHfdgW66pYc2OfCrhrVCgL6JxEGkouvOfRjWUeo95tu1ZasvbwndBSUzpsOOjYS1PfPyiwpdsiWA2C8Abseota8NTj2CigR17T3w7VbKXze+VMHEWUSI+26NFIjhUt0DgwOEHjgRmfOvQW0ptY/vLqQMbmkYdiFkHTemLGxSwTi2q1c5pMP2GwGEE7G34iG189t00HmjbIn744ibWGVolag6f4mSIC3k8s+UCpzKTep1U879Y3r9rzFvby09T4zHJ5u0zlEHDk7LvCfD5fZ1ncSlVgmUeXMCQ== Received: from MW4PR03CA0283.namprd03.prod.outlook.com (2603:10b6:303:b5::18) by MN2PR12MB3438.namprd12.prod.outlook.com (2603:10b6:208:c8::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Fri, 22 Oct 2021 09:12:08 +0000 Received: from CO1NAM11FT038.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b5:cafe::21) by MW4PR03CA0283.outlook.office365.com (2603:10b6:303:b5::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4628.15 via Frontend Transport; Fri, 22 Oct 2021 09:12:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; ashroe.eu; dkim=none (message not signed) header.d=none;ashroe.eu; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT038.mail.protection.outlook.com (10.13.174.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4628.16 via Frontend Transport; Fri, 22 Oct 2021 09:12:07 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 22 Oct 2021 09:12:03 +0000 From: Xueming Li To: CC: , Viacheslav Ovsiienko , Lior Margalit , Matan Azrad , "Ray Kinsella" Date: Fri, 22 Oct 2021 17:11:35 +0800 Message-ID: <20211022091142.51397-2-xuemingl@nvidia.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211022091142.51397-1-xuemingl@nvidia.com> References: <20210927083256.337450-1-xuemingl@nvidia.com> <20211022091142.51397-1-xuemingl@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 497be2f5-648a-4490-0d6b-08d9953c05b1 X-MS-TrafficTypeDiagnostic: MN2PR12MB3438: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: i60LFhJLqcCUofsQyugKoyeZSMKl2AGjrJ/KgfT/lNew65nQbfYxKUe7g9kcvVdsaFJeQ5229AuKR0VOKK3IEFfn/SBHtlnr70DO21fVunzpuN99sZUdjdR9ap2Yfj2/dJK1znl+Ug5auQeIO7BwHxu83TfGPCfU/icqvhANN/Hk5DVbyt0eHxNT34UhUawA4Jy9DRf8O+qmhcfhC1QOU+HUxW1UmBEZnPZSgZCpjaYWGVhGtMshoRzw1r1+ahOLDoduAgBQHzju1Ep4z7J5SIDm9M1x5+TTuiMTwTg8zdYh9lJr4q8sRX3ElMk8JR8LghwBzZKyetv2wpDp8DhzSR4WvjcIm18JZAssi4PgnDZR371dY184AMb+i09loI6p5VQ4Bz/Ipxam/T1JRE2Cgd8OGont1TH3Vyq8US9jW1loiCM2yOR/OoTFGHZSUNvJA/OfU2iiuB8O2nKalWnETch2P4deMEOxIrDBwnr1hqCRmjk1cfWnqSDPsDqQuYPFn1LEy60GqQRs/p939mrcZ3KoYco15a+DE4l9ECmsjlwDfzgV6wBcAdkm6kB1lBid38hAj7KS7aajTr6L1WrWGEYZiHOdzMONl0RHzFxMriCnCmofAdzKDqLcW8hT8qcqxZrTRsoov7v12hY1WvqDPvIbO3JDZ3LUAC9boACwxbXBGkbnqqdg7wYS3btsBXbJniKoo13+AQ6kFmhJ5luqnQ== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(8676002)(82310400003)(2906002)(2616005)(426003)(6286002)(4326008)(5660300002)(1076003)(336012)(6916009)(55016002)(316002)(356005)(7696005)(54906003)(36860700001)(7636003)(8936002)(70206006)(70586007)(16526019)(26005)(83380400001)(186003)(86362001)(508600001)(47076005)(36756003)(6666004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Oct 2021 09:12:07.4232 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 497be2f5-648a-4490-0d6b-08d9953c05b1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT038.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3438 Subject: [dpdk-dev] [PATCH v4 1/8] common/mlx5: add netlink API to get RDMA port state X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Introduce netlink API to get rdma port state. Port state is restrieved based on RDMA device name and port index. Signed-off-by: Xueming Li Acked-by: Viacheslav Ovsiienko --- drivers/common/mlx5/linux/meson.build | 2 + drivers/common/mlx5/linux/mlx5_nl.c | 136 +++++++++++++++++++------- drivers/common/mlx5/linux/mlx5_nl.h | 2 + drivers/common/mlx5/version.map | 1 + 4 files changed, 106 insertions(+), 35 deletions(-) diff --git a/drivers/common/mlx5/linux/meson.build b/drivers/common/mlx5/linux/meson.build index cbea58f557d..2dcd27b7786 100644 --- a/drivers/common/mlx5/linux/meson.build +++ b/drivers/common/mlx5/linux/meson.build @@ -175,6 +175,8 @@ has_sym_args = [ 'RDMA_NLDEV_ATTR_DEV_NAME' ], [ 'HAVE_RDMA_NLDEV_ATTR_PORT_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_PORT_INDEX' ], + [ 'HAVE_RDMA_NLDEV_ATTR_PORT_STATE', 'rdma/rdma_netlink.h', + 'RDMA_NLDEV_ATTR_PORT_STATE' ], [ 'HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX', 'rdma/rdma_netlink.h', 'RDMA_NLDEV_ATTR_NDEV_INDEX' ], [ 'HAVE_MLX5_DR_FLOW_DUMP', 'infiniband/mlx5dv.h', diff --git a/drivers/common/mlx5/linux/mlx5_nl.c b/drivers/common/mlx5/linux/mlx5_nl.c index 530d491b660..fd4c2d26253 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.c +++ b/drivers/common/mlx5/linux/mlx5_nl.c @@ -78,6 +78,9 @@ #ifndef HAVE_RDMA_NLDEV_ATTR_PORT_INDEX #define RDMA_NLDEV_ATTR_PORT_INDEX 3 #endif +#ifndef HAVE_RDMA_NLDEV_ATTR_PORT_STATE +#define RDMA_NLDEV_ATTR_PORT_STATE 12 +#endif #ifndef HAVE_RDMA_NLDEV_ATTR_NDEV_INDEX #define RDMA_NLDEV_ATTR_NDEV_INDEX 50 #endif @@ -160,14 +163,16 @@ struct mlx5_nl_mac_addr { #define MLX5_NL_CMD_GET_IB_INDEX (1 << 1) #define MLX5_NL_CMD_GET_NET_INDEX (1 << 2) #define MLX5_NL_CMD_GET_PORT_INDEX (1 << 3) +#define MLX5_NL_CMD_GET_PORT_STATE (1 << 4) /** Data structure used by mlx5_nl_cmdget_cb(). */ -struct mlx5_nl_ifindex_data { +struct mlx5_nl_port_info { const char *name; /**< IB device name (in). */ uint32_t flags; /**< found attribute flags (out). */ uint32_t ibindex; /**< IB device index (out). */ uint32_t ifindex; /**< Network interface index (out). */ uint32_t portnum; /**< IB device max port number (out). */ + uint16_t state; /**< IB device port state (out). */ }; uint32_t atomic_sn; @@ -966,8 +971,8 @@ mlx5_nl_allmulti(int nlsk_fd, unsigned int iface_idx, int enable) static int mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) { - struct mlx5_nl_ifindex_data *data = arg; - struct mlx5_nl_ifindex_data local = { + struct mlx5_nl_port_info *data = arg; + struct mlx5_nl_port_info local = { .flags = 0, }; size_t off = NLMSG_HDRLEN; @@ -1000,6 +1005,10 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) local.portnum = *(uint32_t *)payload; local.flags |= MLX5_NL_CMD_GET_PORT_INDEX; break; + case RDMA_NLDEV_ATTR_PORT_STATE: + local.state = *(uint8_t *)payload; + local.flags |= MLX5_NL_CMD_GET_PORT_STATE; + break; default: break; } @@ -1016,6 +1025,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) data->ibindex = local.ibindex; data->ifindex = local.ifindex; data->portnum = local.portnum; + data->state = local.state; } return 0; error: @@ -1024,7 +1034,7 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) } /** - * Get index of network interface associated with some IB device. + * Get port info of network interface associated with some IB device. * * This is the only somewhat safe method to avoid resorting to heuristics * when faced with port representors. Unfortunately it requires at least @@ -1032,27 +1042,20 @@ mlx5_nl_cmdget_cb(struct nlmsghdr *nh, void *arg) * * @param nl * Netlink socket of the RDMA kind (NETLINK_RDMA). - * @param[in] name - * IB device name. * @param[in] pindex * IB device port index, starting from 1 + * @param[out] data + * Pointer to port info. * @return - * A valid (nonzero) interface index on success, 0 otherwise and rte_errno - * is set. + * 0 on success, negative on error and rte_errno is set. */ -unsigned int -mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +static int +mlx5_nl_port_info(int nl, uint32_t pindex, struct mlx5_nl_port_info *data) { - struct mlx5_nl_ifindex_data data = { - .name = name, - .flags = 0, - .ibindex = 0, /* Determined during first pass. */ - .ifindex = 0, /* Determined during second pass. */ - }; union { struct nlmsghdr nh; uint8_t buf[NLMSG_HDRLEN + - NLA_HDRLEN + NLA_ALIGN(sizeof(data.ibindex)) + + NLA_HDRLEN + NLA_ALIGN(sizeof(data->ibindex)) + NLA_HDRLEN + NLA_ALIGN(sizeof(pindex))]; } req = { .nh = { @@ -1068,24 +1071,24 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) - return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + return ret; + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) - return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX)) + return ret; + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX)) goto error; - data.flags = 0; + data->flags = 0; sn = MLX5_NL_SN_GENERATE; req.nh.nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_PORT_GET); req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK; req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(req.buf) - NLMSG_HDRLEN); na = (void *)((uintptr_t)req.buf + NLMSG_HDRLEN); - na->nla_len = NLA_HDRLEN + sizeof(data.ibindex); + na->nla_len = NLA_HDRLEN + sizeof(data->ibindex); na->nla_type = RDMA_NLDEV_ATTR_DEV_INDEX; memcpy((void *)((uintptr_t)na + NLA_HDRLEN), - &data.ibindex, sizeof(data.ibindex)); + &data->ibindex, sizeof(data->ibindex)); na = (void *)((uintptr_t)na + NLA_ALIGN(na->nla_len)); na->nla_len = NLA_HDRLEN + sizeof(pindex); na->nla_type = RDMA_NLDEV_ATTR_PORT_INDEX; @@ -1093,19 +1096,82 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) &pindex, sizeof(pindex)); ret = mlx5_nl_send(nl, &req.nh, sn); if (ret < 0) - return 0; - ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, &data); + return ret; + ret = mlx5_nl_recv(nl, sn, mlx5_nl_cmdget_cb, data); if (ret < 0) - return 0; - if (!(data.flags & MLX5_NL_CMD_GET_IB_NAME) || - !(data.flags & MLX5_NL_CMD_GET_IB_INDEX) || - !(data.flags & MLX5_NL_CMD_GET_NET_INDEX) || - !data.ifindex) + return ret; + if (!(data->flags & MLX5_NL_CMD_GET_IB_NAME) || + !(data->flags & MLX5_NL_CMD_GET_IB_INDEX) || + !(data->flags & MLX5_NL_CMD_GET_NET_INDEX) || + !data->ifindex) goto error; - return data.ifindex; + return 1; error: rte_errno = ENODEV; - return 0; + return -rte_errno; +} + +/** + * Get index of network interface associated with some IB device. + * + * This is the only somewhat safe method to avoid resorting to heuristics + * when faced with port representors. Unfortunately it requires at least + * Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * A valid (nonzero) interface index on success, 0 otherwise and rte_errno + * is set. + */ +unsigned int +mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { + .ifindex = 0, + .name = name, + }; + + if (mlx5_nl_port_info(nl, pindex, &data) < 0) + return 0; + return data.ifindex; +} + +/** + * Get IB device port state. + * + * This is the only somewhat safe method to get info for port number >= 255. + * Unfortunately it requires at least Linux 4.17. + * + * @param nl + * Netlink socket of the RDMA kind (NETLINK_RDMA). + * @param[in] name + * IB device name. + * @param[in] pindex + * IB device port index, starting from 1 + * @return + * Port state (ibv_port_state) on success, negative on error + * and rte_errno is set. + */ +int +mlx5_nl_port_state(int nl, const char *name, uint32_t pindex) +{ + struct mlx5_nl_port_info data = { + .state = 0, + .name = name, + }; + + if (mlx5_nl_port_info(nl, pindex, &data) < 0) + return -rte_errno; + if ((data.flags & MLX5_NL_CMD_GET_PORT_STATE) == 0) { + rte_errno = ENOTSUP; + return -rte_errno; + } + return (int)data.state; } /** @@ -1123,7 +1189,7 @@ mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex) unsigned int mlx5_nl_portnum(int nl, const char *name) { - struct mlx5_nl_ifindex_data data = { + struct mlx5_nl_port_info data = { .flags = 0, .name = name, .ifindex = 0, diff --git a/drivers/common/mlx5/linux/mlx5_nl.h b/drivers/common/mlx5/linux/mlx5_nl.h index 202849f52ad..2063c0deeb9 100644 --- a/drivers/common/mlx5/linux/mlx5_nl.h +++ b/drivers/common/mlx5/linux/mlx5_nl.h @@ -54,6 +54,8 @@ unsigned int mlx5_nl_portnum(int nl, const char *name); __rte_internal unsigned int mlx5_nl_ifindex(int nl, const char *name, uint32_t pindex); __rte_internal +int mlx5_nl_port_state(int nl, const char *name, uint32_t pindex); +__rte_internal int mlx5_nl_vf_mac_addr_modify(int nlsk_fd, unsigned int iface_idx, struct rte_ether_addr *mac, int vf_index); __rte_internal diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index 1167fcd3236..7c95172fe87 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -120,6 +120,7 @@ INTERNAL { mlx5_nl_mac_addr_flush; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_remove; # WINDOWS_NO_EXPORT mlx5_nl_mac_addr_sync; # WINDOWS_NO_EXPORT + mlx5_nl_port_state; # WINDOWS_NO_EXPORT mlx5_nl_portnum; # WINDOWS_NO_EXPORT mlx5_nl_promisc; # WINDOWS_NO_EXPORT mlx5_nl_switch_info; # WINDOWS_NO_EXPORT