Message ID | 20210527133759.17401-1-xuemingl@nvidia.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E2F0CA0548; Thu, 27 May 2021 15:38:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 98FA840683; Thu, 27 May 2021 15:38:53 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2045.outbound.protection.outlook.com [40.107.93.45]) by mails.dpdk.org (Postfix) with ESMTP id 4E32D40150 for <dev@dpdk.org>; Thu, 27 May 2021 15:38:52 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lASakPDfquRUG0PlXJEGJK7hmZEM2pP/MtAxNBpO77/1TWRpUIKIwlPKFU+92hmlgW+VeTSaPDUrQkUc1h+GEqu7UUDwQIaDmBJVKmutgF67VA2zJ1stTVmDg3h+vXfjcELFRFA3XPXLCNynPXutUhgzK4lMlb6dejXoDrAXZ3LrOEX6yc+g3JoI5D7Bf16cPrWwy71xui/tGo8F0f6iWoS2cy5iXCwUoBFJtWO5TyOokPsYsKVwaCf5fBkqNUzD78YPqM/gXVPzcN7OeKamQtucX4Lf+oMM/GvCJvCfKmslr+gTiKRnoEtYaFOfU0aWIOEKQXpAcZ2YPJ2HXajTGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7zRo7ZyqNz4Q6C3d6A9eFxQqLvZoAtAgeJsjyKOoTJU=; b=JXAdExorJyQlYwYEyNgdZbq1sCY+eMRc8leY/ak8uRL/t7pkpgqztf6Gl6HfjK20RRKl8bTNhiBeeaf6Eo2/IpIuqnYJHJerTAKjk+fhsUhZLpvUnRNHcXBDcgiPKyCiNnz8Nsz9OVQfs5Y1pkDAjS2QkSVi8n2XDhUzQuAznzEcLIbeNEDLLyJKptLk9Xdu5fLQlx4AdkamKjdFJ6ib21bqQNc1e2CXpu2ib8FtYDcK8TaVSnOEiryEqbPbnhQTmdRCQMVp1/bdSVdLpP+sajW1pX9YaoJnW0RQMXzVTJo8jYRSwo6dwQ7zAlFBsamjzYK3TG/F6mmTZuZ+CnSTYg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7zRo7ZyqNz4Q6C3d6A9eFxQqLvZoAtAgeJsjyKOoTJU=; b=FCZrD+DSxaGqrWw3+HpirVYsTSyHfx/HJ89ObGM35Uhh86gZfhXapiPG+HOH/mLV0gKIlk2XH7YmM6t55DPD6NaVBcJHSgTVbvBJ7QvQZIQQBDlyH7fiCEQHFouNOwXQqH2GQuSgCH8ZH5VpPAZHF9TA7OVU0Cl4ht3IaturmEi7gAXcbF2rbf0+bc6jMZqL9odokNNdGbHHJdLWEPiNBV2TAARHSQR1l3xGhagLo6mV/cXLTUorj0V7sNEXM80DM3phB0K7s5EtWW/thFqjt7xHVTDZrh4r15yzUAb4j/HBQ+EkCVMNnOi4ey9X2q0VAbQlDURII6sVOMuwJ79W5A== Received: from MW4PR04CA0053.namprd04.prod.outlook.com (2603:10b6:303:6a::28) by DM4PR12MB5279.namprd12.prod.outlook.com (2603:10b6:5:39f::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.22; Thu, 27 May 2021 13:38:51 +0000 Received: from CO1NAM11FT007.eop-nam11.prod.protection.outlook.com (2603:10b6:303:6a:cafe::77) by MW4PR04CA0053.outlook.office365.com (2603:10b6:303:6a::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4173.20 via Frontend Transport; Thu, 27 May 2021 13:38:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT007.mail.protection.outlook.com (10.13.174.131) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4129.25 via Frontend Transport; Thu, 27 May 2021 13:38:50 +0000 Received: from nvidia.com (172.20.145.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 27 May 2021 13:38:49 +0000 From: Xueming Li <xuemingl@nvidia.com> To: Viacheslav Ovsiienko <viacheslavo@nvidia.com> CC: <dev@dpdk.org>, <xuemingl@nvidia.com> Date: Thu, 27 May 2021 16:37:45 +0300 Message-ID: <20210527133759.17401-1-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4acf2f25-2663-409f-1a55-08d92114c34b X-MS-TrafficTypeDiagnostic: DM4PR12MB5279: X-Microsoft-Antispam-PRVS: <DM4PR12MB5279E47DB2E80777E41AF716A1239@DM4PR12MB5279.namprd12.prod.outlook.com> X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fIY5s/Fatz26g0KS5kwDpG9VVvSLZXtAWS53R/fQUM1LHb0Kkq8QFFR/F50QKERDNm9lDCcN2AgUNJ4x03b4yQiYw9vZ/7trDNO0+vetpVoxK6F8LofBtxGkJrImWKA0EkTKI7q3cn9keSsAnr/AyxwkQ/4ObPozaSXbbcR5bA/MKgvaqVgjPkdkXTChRnQpKqixbmnkuI9bdH7TMBT44N9OdxjPQx+4DLeV8BOU0rUH9VN9GXgZLPMrS/2uuAVs6bnFLtGfkCIVmjDV6uEF/dsqZU5BQSOeFrZfpxAGPjDESCFKuxGt7w2Q9P1JxyCOc+eKJyzAS4zOI2/B5zZrHkzTvGClPApBIbS69GvBcaCh7iIF41FjQ7fcj0BlkmkmNoSl+fom24DvuHTlkV0TplGhDeAxHOz4SvNsMrYE0hI/yt62jdzpbW4RgI+Jco0sPpENPt+pdmzlNx+jb9FUizLaOHhETojfYPX+CsLYmTt8G9+NY2rD1bF8+e1HLJlxPz6Du83QCi6zLVicqGL/EPtNzse5l7v8EWgpHHHapbTTcNH7xLz3cEaxZSMyrGIf9FMsBhOqUzPw+4vaAdQQEDDVGpRCUrIkRTnNKDerR+vqUk2RtjDWbeMqkEJfJP7E6RqwehPKFsSqUqcIZ7sgQDeKWoS+D6TSB+y+otdT9qcwXCaPRuj/+E6dMjQcclWIVV3zh3nEdZw9xZSB+6EzQvuLP5RYyp9TYtIhDYHozI897BA8K2wnMpQ22k4Bf8dYc0t5rE6mYyfN0djmuCZnNA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(136003)(396003)(376002)(346002)(36840700001)(46966006)(186003)(70206006)(70586007)(2616005)(426003)(1076003)(16526019)(336012)(5660300002)(6862004)(26005)(36756003)(4326008)(6666004)(7696005)(47076005)(107886003)(55016002)(37006003)(83380400001)(86362001)(7636003)(356005)(966005)(82310400003)(478600001)(8936002)(36906005)(82740400003)(316002)(54906003)(2906002)(6636002)(6286002)(8676002)(36860700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 May 2021 13:38:50.8567 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4acf2f25-2663-409f-1a55-08d92114c34b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT007.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5279 Subject: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
mlx5: support SubFunction
|
|
Message
Xueming Li
May 27, 2021, 1:37 p.m. UTC
SubFunction [1] is a portion of the PCI device, a SF netdev has its own dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF. This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress. Depends-on: series-16904 ("bus/auxiliary: introduce auxiliary bus") Version history: RFC: initial version [1] SubFunction in kernel: https://lore.kernel.org/netdev/20201112192424.2742-1-parav@nvidia.com/ [2] Auxiliary bus: http://patchwork.dpdk.org/project/dpdk/patch/20210510134732.2174-1-xuemingl@nvidia.com/ Thomas Monjalon (5): common/mlx5: move description of PCI sysfs functions vdpa/mlx5: fix driver name vdpa/mlx5: remove PCI specifics common/mlx5: get PCI device address from any bus vdpa/mlx5: support SubFunction Xueming Li (9): common/mlx5: add common device driver net/mlx5: remove PCI dependency net/mlx5: migrate to bus-agnostic common driver regex/mlx5: migrate to common driver compress/mlx5: migrate to common driver common/mlx5: clean up legacy PCI bus driver bus/auxiliary: introduce auxiliary bus common/mlx5: support auxiliary bus net/mlx5: support SubFunction MAINTAINERS | 5 + doc/guides/nics/mlx5.rst | 339 ++++++++++- drivers/bus/auxiliary/auxiliary_common.c | 408 +++++++++++++ drivers/bus/auxiliary/auxiliary_params.c | 58 ++ drivers/bus/auxiliary/linux/auxiliary.c | 147 +++++ drivers/bus/auxiliary/meson.build | 11 + drivers/bus/auxiliary/private.h | 120 ++++ drivers/bus/auxiliary/rte_bus_auxiliary.h | 199 +++++++ drivers/bus/auxiliary/version.map | 10 + drivers/bus/meson.build | 1 + drivers/common/mlx5/linux/meson.build | 3 + .../common/mlx5/linux/mlx5_common_auxiliary.c | 188 ++++++ drivers/common/mlx5/linux/mlx5_common_os.c | 53 +- drivers/common/mlx5/linux/mlx5_common_os.h | 4 - drivers/common/mlx5/linux/mlx5_common_verbs.c | 24 +- drivers/common/mlx5/mlx5_common.c | 340 ++++++++++- drivers/common/mlx5/mlx5_common.h | 176 +++++- drivers/common/mlx5/mlx5_common_pci.c | 554 ++++-------------- drivers/common/mlx5/mlx5_common_pci.h | 77 --- drivers/common/mlx5/mlx5_common_private.h | 50 ++ drivers/common/mlx5/version.map | 14 +- drivers/compress/mlx5/mlx5_compress.c | 71 +-- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/linux/mlx5_os.c | 193 ++++-- drivers/net/mlx5/linux/mlx5_os.h | 5 +- drivers/net/mlx5/mlx5.c | 97 +-- drivers/net/mlx5/mlx5.h | 12 +- drivers/net/mlx5/mlx5_ethdev.c | 2 +- drivers/net/mlx5/mlx5_mr.c | 46 +- drivers/net/mlx5/mlx5_rxmode.c | 8 +- drivers/net/mlx5/mlx5_rxtx.h | 9 +- drivers/net/mlx5/mlx5_trigger.c | 14 +- drivers/net/mlx5/mlx5_txq.c | 2 +- drivers/net/mlx5/windows/mlx5_os.c | 20 +- drivers/regex/mlx5/mlx5_regex.c | 49 +- drivers/regex/mlx5/mlx5_regex.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 128 ++-- drivers/vdpa/mlx5/mlx5_vdpa.h | 1 - 38 files changed, 2532 insertions(+), 921 deletions(-) create mode 100644 drivers/bus/auxiliary/auxiliary_common.c create mode 100644 drivers/bus/auxiliary/auxiliary_params.c create mode 100644 drivers/bus/auxiliary/linux/auxiliary.c create mode 100644 drivers/bus/auxiliary/meson.build create mode 100644 drivers/bus/auxiliary/private.h create mode 100644 drivers/bus/auxiliary/rte_bus_auxiliary.h create mode 100644 drivers/bus/auxiliary/version.map create mode 100644 drivers/common/mlx5/linux/mlx5_common_auxiliary.c delete mode 100644 drivers/common/mlx5/mlx5_common_pci.h create mode 100644 drivers/common/mlx5/mlx5_common_private.h
Comments
On 5/27/2021 2:37 PM, Xueming Li wrote: > SubFunction [1] is a portion of the PCI device, a SF netdev has its own > dedicated queues(txq, rxq). A SF shares PCI level resources with other > SFs and/or with its parent PCI function. Auxiliary bus is the > fundamental of SF. > > This patch set introduces SubFunction support for mlx5 PMD driver > including class net, regex, vdpa and compress. > There is already an mdev patch, originated from long ago. Aren't subfunctions presented as mdev device? If so can't we use mdev for it?
10/06/2021 12:33, Ferruh Yigit: > On 5/27/2021 2:37 PM, Xueming Li wrote: > > SubFunction [1] is a portion of the PCI device, a SF netdev has its own > > dedicated queues(txq, rxq). A SF shares PCI level resources with other > > SFs and/or with its parent PCI function. Auxiliary bus is the > > fundamental of SF. > > > > This patch set introduces SubFunction support for mlx5 PMD driver > > including class net, regex, vdpa and compress. > > > > There is already an mdev patch, originated from long ago. Aren't subfunctions > presented as mdev device? If so can't we use mdev for it? No unfortunately that's different. mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.
Hi Thomas, > -----Original Message----- > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Thursday, June 10, 2021 9:23 PM > To: Yigit, Ferruh <ferruh.yigit@intel.com> > Cc: Xueming Li <xuemingl@nvidia.com>; Viacheslav Ovsiienko > <viacheslavo@nvidia.com>; dev@dpdk.org; Xia, Chenbo <chenbo.xia@intel.com> > Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction > > 10/06/2021 12:33, Ferruh Yigit: > > On 5/27/2021 2:37 PM, Xueming Li wrote: > > > SubFunction [1] is a portion of the PCI device, a SF netdev has its > own > > > dedicated queues(txq, rxq). A SF shares PCI level resources with other > > > SFs and/or with its parent PCI function. Auxiliary bus is the > > > fundamental of SF. > > > > > > This patch set introduces SubFunction support for mlx5 PMD driver > > > including class net, regex, vdpa and compress. > > > > > > > There is already an mdev patch, originated from long ago. Aren't > subfunctions > > presented as mdev device? If so can't we use mdev for it? > > No unfortunately that's different. > mlx5 SF is based on top of auxiliary bus in the kernel/sysfs. > Just out of curiosity: Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊 [1] https://patchwork.ozlabs.org/project/netdev/cover/20191107160448.20962-1-parav@mellanox.com/ [2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices
11/06/2021 07:14, Xia, Chenbo: > From: Thomas Monjalon <thomas@monjalon.net> > > 10/06/2021 12:33, Ferruh Yigit: > > > On 5/27/2021 2:37 PM, Xueming Li wrote: > > > > SubFunction [1] is a portion of the PCI device, a SF netdev has its > > own > > > > dedicated queues(txq, rxq). A SF shares PCI level resources with other > > > > SFs and/or with its parent PCI function. Auxiliary bus is the > > > > fundamental of SF. > > > > > > > > This patch set introduces SubFunction support for mlx5 PMD driver > > > > including class net, regex, vdpa and compress. > > > > > > > > > > There is already an mdev patch, originated from long ago. Aren't > > subfunctions > > > presented as mdev device? If so can't we use mdev for it? > > > > No unfortunately that's different. > > mlx5 SF is based on top of auxiliary bus in the kernel/sysfs. > > Just out of curiosity: > > Does SF use mdev before aux bus is introduced in kernel. I see some history > of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField > software v2.5 is using mdev for SF. I saw it yesterday and try to figure > out the history. Since you are here, guess you know something 😊 > > [1] https://patchwork.ozlabs.org/project/netdev/cover/20191107160448.20962-1-parav@mellanox.com/ > [2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices Kernel maintainers rejected the use of mdev for this purpose and suggested to use a real bus. You can follow the discussion here: https://lore.kernel.org/netdev/20191108205204.GB1277001@kroah.com/ Does Intel plan to use mdev for SF? For the sake of follow-up discussion, this is the official mdev doc: https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt
Hi Thomas, > -----Original Message----- > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Friday, June 11, 2021 3:54 PM > To: Yigit, Ferruh <ferruh.yigit@intel.com>; Xia, Chenbo <chenbo.xia@intel.com> > Cc: Xueming Li <xuemingl@nvidia.com>; Viacheslav Ovsiienko > <viacheslavo@nvidia.com>; dev@dpdk.org; parav@nvidia.com; jgg@nvidia.com > Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction > > 11/06/2021 07:14, Xia, Chenbo: > > From: Thomas Monjalon <thomas@monjalon.net> > > > 10/06/2021 12:33, Ferruh Yigit: > > > > On 5/27/2021 2:37 PM, Xueming Li wrote: > > > > > SubFunction [1] is a portion of the PCI device, a SF netdev has its > > > own > > > > > dedicated queues(txq, rxq). A SF shares PCI level resources with other > > > > > SFs and/or with its parent PCI function. Auxiliary bus is the > > > > > fundamental of SF. > > > > > > > > > > This patch set introduces SubFunction support for mlx5 PMD driver > > > > > including class net, regex, vdpa and compress. > > > > > > > > > > > > > There is already an mdev patch, originated from long ago. Aren't > > > subfunctions > > > > presented as mdev device? If so can't we use mdev for it? > > > > > > No unfortunately that's different. > > > mlx5 SF is based on top of auxiliary bus in the kernel/sysfs. > > > > Just out of curiosity: > > > > Does SF use mdev before aux bus is introduced in kernel. I see some history > > of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField > > software v2.5 is using mdev for SF. I saw it yesterday and try to figure > > out the history. Since you are here, guess you know something 😊 > > > > [1] https://patchwork.ozlabs.org/project/netdev/cover/20191107160448.20962- > 1-parav@mellanox.com/ > > [2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices > > Kernel maintainers rejected the use of mdev for this purpose > and suggested to use a real bus. > You can follow the discussion here: > https://lore.kernel.org/netdev/20191108205204.GB1277001@kroah.com/ OK. Thanks for the info. > > Does Intel plan to use mdev for SF? Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux) And vfio-mdev is chosen to be the software framework for it. I start to realize there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/... But ADI only focuses on the virtualization of the devices and splitting devices to logic parts and providing huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong). And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub-system (like for net, wrap it as netdev). Thanks, Chenbo > > For the sake of follow-up discussion, this is the official mdev doc: > https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt > >
Hi Chenbo, > From: Xia, Chenbo <chenbo.xia@intel.com> > Sent: Tuesday, June 15, 2021 7:41 AM > > Hi Thomas, > > > From: Thomas Monjalon <thomas@monjalon.net> > > Sent: Friday, June 11, 2021 3:54 PM [..] > > Yes. In our term it's called Assignable Device Interface (ADI) introduced in > Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel- > scalable-i/o-virtualization-linux) > > And vfio-mdev is chosen to be the software framework for it. I start to realize > there is difference between SF and ADI: SF considers multi-function devices > which may include net/regex/vdpa/... Yes. net, rdma, vdpa, regex ++. And eventually vfio_device to map to VM too. Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported. > But ADI only focuses on the > virtualization of the devices and splitting devices to logic parts and providing > huge number of interfaces to host APP. I think SF also considers this but is > mainly used for multi-function devices (like DPU in your term? > Correct me if I'm wrong). > SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a vfio_device) to map to a VM. > And I also noticed that the mdev-based interface can only be used in > userspace but aux-based interface can also be used by other kernel sub- > system (like for net, wrap it as netdev). Correct.
Hi Parav, > -----Original Message----- > From: Parav Pandit <parav@nvidia.com> > Sent: Tuesday, June 15, 2021 12:05 PM > To: Xia, Chenbo <chenbo.xia@intel.com>; NBU-Contact-Thomas Monjalon > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com> > Cc: Xueming(Steven) Li <xuemingl@nvidia.com>; Slava Ovsiienko > <viacheslavo@nvidia.com>; dev@dpdk.org; Jason Gunthorpe <jgg@nvidia.com> > Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction > > Hi Chenbo, > > > From: Xia, Chenbo <chenbo.xia@intel.com> > > Sent: Tuesday, June 15, 2021 7:41 AM > > > > Hi Thomas, > > > > > From: Thomas Monjalon <thomas@monjalon.net> > > > Sent: Friday, June 11, 2021 3:54 PM > [..] > > > > > Yes. In our term it's called Assignable Device Interface (ADI) introduced in > > Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel- > > scalable-i/o-virtualization-linux) > > > > And vfio-mdev is chosen to be the software framework for it. I start to > realize > > there is difference between SF and ADI: SF considers multi-function devices > > which may include net/regex/vdpa/... > Yes. net, rdma, vdpa, regex ++. > And eventually vfio_device to map to VM too. > > Non mdev framework is chosen so that all the use cases of kernel only, or user > only or mix modes can be supported. OK. Got it. > > > But ADI only focuses on the > > virtualization of the devices and splitting devices to logic parts and > providing > > huge number of interfaces to host APP. I think SF also considers this but is > > mainly used for multi-function devices (like DPU in your term? > > Correct me if I'm wrong). > > > SF also supports DPU mode too but it is in addition to above use cases. > SF will expose mdev (or a vfio_device) to map to a VM. So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree? Just FYI: We are introducing a new mdev bus for DPDK: http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-chenbo.xia@intel.com/ Thanks, Chenbo > > > And I also noticed that the mdev-based interface can only be used in > > userspace but aux-based interface can also be used by other kernel sub- > > system (like for net, wrap it as netdev). > Correct.
> From: Xia, Chenbo <chenbo.xia@intel.com> > Sent: Tuesday, June 15, 2021 11:03 AM > > Hi Parav, > > > -----Original Message----- > > From: Parav Pandit <parav@nvidia.com> > > Sent: Tuesday, June 15, 2021 12:05 PM > > To: Xia, Chenbo <chenbo.xia@intel.com>; NBU-Contact-Thomas Monjalon > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com> > > Cc: Xueming(Steven) Li <xuemingl@nvidia.com>; Slava Ovsiienko > > <viacheslavo@nvidia.com>; dev@dpdk.org; Jason Gunthorpe > > <jgg@nvidia.com> > > Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction > > > > Hi Chenbo, > > > > > From: Xia, Chenbo <chenbo.xia@intel.com> > > > Sent: Tuesday, June 15, 2021 7:41 AM > > > > > > Hi Thomas, > > > > > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > Sent: Friday, June 11, 2021 3:54 PM > > [..] > > > > > > > > Yes. In our term it's called Assignable Device Interface (ADI) > > > introduced in Intel Scalable IOV > > > (https://01.org/blogs/2019/assignable-interfaces-intel- > > > scalable-i/o-virtualization-linux) > > > > > > And vfio-mdev is chosen to be the software framework for it. I start > > > to > > realize > > > there is difference between SF and ADI: SF considers multi-function > > > devices which may include net/regex/vdpa/... > > Yes. net, rdma, vdpa, regex ++. > > And eventually vfio_device to map to VM too. > > > > Non mdev framework is chosen so that all the use cases of kernel only, > > or user only or mix modes can be supported. > > OK. Got it. > > > > > > But ADI only focuses on the > > > virtualization of the devices and splitting devices to logic parts > > > and > > providing > > > huge number of interfaces to host APP. I think SF also considers > > > this but is mainly used for multi-function devices (like DPU in your term? > > > Correct me if I'm wrong). > > > > > SF also supports DPU mode too but it is in addition to above use cases. > > SF will expose mdev (or a vfio_device) to map to a VM. > > So your SW actually supports vfio-mdev? I suppose the device-specific mdev > Kernel module is out-of-tree? > mlx5 driver doesn't support vfio_device for SFs. Kernel plumbing for PASID assignment to SF is WIP currently kernel community. We do not have any out-of-tree kernel module. > Just FYI: > > We are introducing a new mdev bus for DPDK: > http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1- > chenbo.xia@intel.com/ > I am yet to read about it. But I am not sure what value does it add. A user can open a vfio device using vfio subsystem and operate on it. A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to vfio_foo driver. There is kernel work in progress to use vfio core as library. So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF. For Intel, ADI will never has any netdevs or rdma dev?
Hi Parav, > -----Original Message----- > From: Parav Pandit <parav@nvidia.com> > Sent: Tuesday, June 15, 2021 1:43 PM > To: Xia, Chenbo <chenbo.xia@intel.com>; NBU-Contact-Thomas Monjalon > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com> > Cc: Xueming(Steven) Li <xuemingl@nvidia.com>; Slava Ovsiienko > <viacheslavo@nvidia.com>; dev@dpdk.org; Jason Gunthorpe <jgg@nvidia.com> > Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction > > > > > From: Xia, Chenbo <chenbo.xia@intel.com> > > Sent: Tuesday, June 15, 2021 11:03 AM > > > > Hi Parav, > > > > > -----Original Message----- > > > From: Parav Pandit <parav@nvidia.com> > > > Sent: Tuesday, June 15, 2021 12:05 PM > > > To: Xia, Chenbo <chenbo.xia@intel.com>; NBU-Contact-Thomas Monjalon > > > <thomas@monjalon.net>; Yigit, Ferruh <ferruh.yigit@intel.com> > > > Cc: Xueming(Steven) Li <xuemingl@nvidia.com>; Slava Ovsiienko > > > <viacheslavo@nvidia.com>; dev@dpdk.org; Jason Gunthorpe > > > <jgg@nvidia.com> > > > Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction > > > > > > Hi Chenbo, > > > > > > > From: Xia, Chenbo <chenbo.xia@intel.com> > > > > Sent: Tuesday, June 15, 2021 7:41 AM > > > > > > > > Hi Thomas, > > > > > > > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > > Sent: Friday, June 11, 2021 3:54 PM > > > [..] > > > > > > > > > > > Yes. In our term it's called Assignable Device Interface (ADI) > > > > introduced in Intel Scalable IOV > > > > (https://01.org/blogs/2019/assignable-interfaces-intel- > > > > scalable-i/o-virtualization-linux) > > > > > > > > And vfio-mdev is chosen to be the software framework for it. I start > > > > to > > > realize > > > > there is difference between SF and ADI: SF considers multi-function > > > > devices which may include net/regex/vdpa/... > > > Yes. net, rdma, vdpa, regex ++. > > > And eventually vfio_device to map to VM too. > > > > > > Non mdev framework is chosen so that all the use cases of kernel only, > > > or user only or mix modes can be supported. > > > > OK. Got it. > > > > > > > > > But ADI only focuses on the > > > > virtualization of the devices and splitting devices to logic parts > > > > and > > > providing > > > > huge number of interfaces to host APP. I think SF also considers > > > > this but is mainly used for multi-function devices (like DPU in your > term? > > > > Correct me if I'm wrong). > > > > > > > SF also supports DPU mode too but it is in addition to above use cases. > > > SF will expose mdev (or a vfio_device) to map to a VM. > > > > So your SW actually supports vfio-mdev? I suppose the device-specific mdev > > Kernel module is out-of-tree? > > > mlx5 driver doesn't support vfio_device for SFs. > Kernel plumbing for PASID assignment to SF is WIP currently kernel community. > We do not have any out-of-tree kernel module. > > > Just FYI: > > > > We are introducing a new mdev bus for DPDK: > > http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1- > > chenbo.xia@intel.com/ > > > I am yet to read about it. But I am not sure what value does it add. > A user can open a vfio device using vfio subsystem and operate on it. > A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci > driver or a SF by binding SF to vfio_foo driver. Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev to parent device and echo uuid to create a virtual device. VFIO APP like DPDK, as you said, should work similar with VFIO UAPI for vfio-pci devices or mdev-based devices. But currently DPDK only cares about vfio-pci devices and does not care things for other cases like mdev-based pci devices. For example, it does not scan /sys/bus/mdev and it always uses pci bdf as device address, which mdev-based pci devices do not have. Therefore I sent that patchset. > There is kernel work in progress to use vfio core as library. OK. Could you share me some link to it? Much appreciated. > So we do not anticipate to use add mdev layer and uuid to create a vfio device > for a SF. OK. For now, we are following the vfio-mdev standard, using UUID to create vfio devices. > > For Intel, ADI will never has any netdevs or rdma dev? I think technically it could have. But for some devices like our dma devices, it's just using mdev: https://www.spinics.net/lists/kvm/msg244417.html Thanks, Chenbo
> From: Xia, Chenbo <chenbo.xia@intel.com> > Sent: Tuesday, June 15, 2021 4:49 PM > > > > > > Just FYI: > > > > > > We are introducing a new mdev bus for DPDK: > > > http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1- > > > chenbo.xia@intel.com/ > > > > > I am yet to read about it. But I am not sure what value does it add. > > A user can open a vfio device using vfio subsystem and operate on it. > > A vfio device can be a create as a result of binding PCI VF/PF to > > vfio-pci driver or a SF by binding SF to vfio_foo driver. > > Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev > to parent device and echo uuid to create a virtual device. VFIO APP like > DPDK, as you said, should work similar with VFIO UAPI for vfio-pci devices or > mdev-based devices. But currently DPDK only cares about vfio-pci devices > and does not care things for other cases like mdev-based pci devices. For > example, it does not scan /sys/bus/mdev and it always uses pci bdf as device > address, which mdev-based pci devices do not have. Therefore I sent that > patchset. mdev device reside on mdev bus. So dpdk should identify the mdev object by specifying bus type = mdev, and device id = uuid. There should not be any attachment to pci as Thomas said. > > > There is kernel work in progress to use vfio core as library. > > OK. Could you share me some link to it? Much appreciated. > [1] https://lore.kernel.org/kvm/20210603160809.15845-1-mgurtovoy@nvidia.com/ > > So we do not anticipate to use add mdev layer and uuid to create a > > vfio device for a SF. > > OK. For now, we are following the vfio-mdev standard, using UUID to create > vfio devices. > If this layer is going to work on top of VFIO devices, does it really care that is it mdev? Can it identify the vfio device by vfio device and its UAPI in uniform way? such as open("/dev/vfio/98" ..); > > > > For Intel, ADI will never has any netdevs or rdma dev? > > I think technically it could have. Unlikely. As I explained in previous email that creating netdev, rdma dev using mdev bus was already rejected in my previous patches. And we step forward with auxiliary bus. > But for some devices like our dma devices, > it's just using mdev: > > https://www.spinics.net/lists/kvm/msg244417.html Possibly yes. Some devices might live on mdev bus. You should wait for kernel patches to be merged as Jason said. I still think that identifying vfio device by its /dev/vfio/<id> will go long way regardless of its bus type.
On Tue, Jun 15, 2021 at 12:47:13PM +0000, Parav Pandit wrote: > > But for some devices like our dma devices, > > it's just using mdev: > > > > https://www.spinics.net/lists/kvm/msg244417.html > Possibly yes. Some devices might live on mdev bus. > You should wait for kernel patches to be merged as Jason said. Also I would not expect dpdk to consume IDXD via vfio, but instead via the normal multi-process host IOCTL interface it has. Jason