[v4,0/4] Support VFIO sparse mmap in PCI bus

  This series introduces a VFIO standard capability, called sparse
mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
mmap whole BAR region into DPDK process, only mmap part of the
BAR region after getting sparse mmap information from kernel.
For the rest of BAR region that is not mmap-ed, DPDK process
can use pread/pwrite system calls to access. Sparse mmap is
useful when kernel does not want userspace to mmap whole BAR
region, or kernel wants to control over access to specific BAR
region. Vendors can choose to enable this feature or not for
their devices in their specific kernel modules.

In this patchset:

Patch 1-3 is mainly for introducing BAR access APIs so that
driver could use them to access specific BAR using pread/pwrite
system calls when part of the BAR is not mmap-able. Patch 4
adds the VFIO sparse mmap support finally.

v4:
1. add sparse mmap information allocation and release
2. add release note for BAR access APIs

v3:
fix variable 'pdev' and 'info' uninitialized error

v2:
1. add PCI device internal structure in bus/pci/windows/pci.c
2. fix parameter type error

Chenbo Xia (3):
  bus/pci: introduce an internal representation of PCI device
  bus/pci: avoid depending on private value in kernel source
  bus/pci: introduce helper for MMIO read and write

Miao Li (1):
  bus/pci: add VFIO sparse mmap support

 doc/guides/rel_notes/release_23_07.rst |   5 +
 drivers/bus/pci/bsd/pci.c              |  35 ++-
 drivers/bus/pci/linux/pci.c            |  78 +++++-
 drivers/bus/pci/linux/pci_init.h       |  14 +-
 drivers/bus/pci/linux/pci_uio.c        |  22 ++
 drivers/bus/pci/linux/pci_vfio.c       | 371 ++++++++++++++++++++-----
 drivers/bus/pci/pci_common.c           |  12 +-
 drivers/bus/pci/private.h              |  25 +-
 drivers/bus/pci/rte_bus_pci.h          |  48 ++++
 drivers/bus/pci/version.map            |   3 +
 drivers/bus/pci/windows/pci.c          |  14 +-
 lib/eal/include/rte_vfio.h             |   1 -
 12 files changed, 525 insertions(+), 103 deletions(-)
  

31/05/2023 07:37, Miao Li:
> This series introduces a VFIO standard capability, called sparse
> mmap to PCI bus. In linux kernel, it's defined as
> VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> mmap whole BAR region into DPDK process, only mmap part of the
> BAR region after getting sparse mmap information from kernel.
> For the rest of BAR region that is not mmap-ed, DPDK process
> can use pread/pwrite system calls to access. Sparse mmap is
> useful when kernel does not want userspace to mmap whole BAR
> region, or kernel wants to control over access to specific BAR
> region. Vendors can choose to enable this feature or not for
> their devices in their specific kernel modules.
> 
> In this patchset:
> 
> Patch 1-3 is mainly for introducing BAR access APIs so that
> driver could use them to access specific BAR using pread/pwrite
> system calls when part of the BAR is not mmap-able. Patch 4
> adds the VFIO sparse mmap support finally.
> 
> v4:
> 1. add sparse mmap information allocation and release
> 2. add release note for BAR access APIs
> 
> v3:
> fix variable 'pdev' and 'info' uninitialized error
> 
> v2:
> 1. add PCI device internal structure in bus/pci/windows/pci.c
> 2. fix parameter type error
> 
> Chenbo Xia (3):
>   bus/pci: introduce an internal representation of PCI device
>   bus/pci: avoid depending on private value in kernel source
>   bus/pci: introduce helper for MMIO read and write
> 
> Miao Li (1):
>   bus/pci: add VFIO sparse mmap support

Applied, thanks.

Is there some drivers which may reuse the new MMIO helpers?
  

Hello,

This patchseries might have introduced a bug for building DPDK on windows.
It failed on windows build when it went through our CI last week, and I am
seeing other patch series fail on the windows build now that it is merged
into main.

Tomorrow morning, I will check our windows system used for CI to verify it
is still valid for building DPDK in terms of clang version, linker etc. But
it seems fine from a quick glance right now. I will update if/when I learn
more.

Thanks,
Patrick
  

Hi Thomas,

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, June 8, 2023 12:31 AM
> To: Li, Miao <miao.li@intel.com>
> Cc: dev@dpdk.org; skori@marvell.com; david.marchand@redhat.com;
> ferruh.yigit@amd.com; Xia, Chenbo <chenbo.xia@intel.com>; Cao, Yahui
> <yahui.cao@intel.com>
> Subject: Re: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus
> 
> 31/05/2023 07:37, Miao Li:
> > This series introduces a VFIO standard capability, called sparse
> > mmap to PCI bus. In linux kernel, it's defined as
> > VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> > mmap whole BAR region into DPDK process, only mmap part of the
> > BAR region after getting sparse mmap information from kernel.
> > For the rest of BAR region that is not mmap-ed, DPDK process
> > can use pread/pwrite system calls to access. Sparse mmap is
> > useful when kernel does not want userspace to mmap whole BAR
> > region, or kernel wants to control over access to specific BAR
> > region. Vendors can choose to enable this feature or not for
> > their devices in their specific kernel modules.
> >
> > In this patchset:
> >
> > Patch 1-3 is mainly for introducing BAR access APIs so that
> > driver could use them to access specific BAR using pread/pwrite
> > system calls when part of the BAR is not mmap-able. Patch 4
> > adds the VFIO sparse mmap support finally.
> >
> > v4:
> > 1. add sparse mmap information allocation and release
> > 2. add release note for BAR access APIs
> >
> > v3:
> > fix variable 'pdev' and 'info' uninitialized error
> >
> > v2:
> > 1. add PCI device internal structure in bus/pci/windows/pci.c
> > 2. fix parameter type error
> >
> > Chenbo Xia (3):
> >   bus/pci: introduce an internal representation of PCI device
> >   bus/pci: avoid depending on private value in kernel source
> >   bus/pci: introduce helper for MMIO read and write
> >
> > Miao Li (1):
> >   bus/pci: add VFIO sparse mmap support
> 
> Applied, thanks.
> 
> Is there some drivers which may reuse the new MMIO helpers?

Yes, we will send patches to let Intel drivers use them soon. Other
drivers could start to use when they want to support sparse mmap.

Thanks,
Chenbo 

>
  

Hi Patrick,

Oops.. Seems weird as patchwork does not report anything. Please reach out to me when you get some information about why it’s failing. I could fix it ASAP.

Thanks,
Chenbo

From: Patrick Robb <probb@iol.unh.edu>
Sent: Thursday, June 8, 2023 8:29 AM
To: Thomas Monjalon <thomas@monjalon.net>
Cc: Li, Miao <miao.li@intel.com>; dev@dpdk.org; skori@marvell.com; david.marchand@redhat.com; ferruh.yigit@amd.com; Xia, Chenbo <chenbo.xia@intel.com>; Cao, Yahui <yahui.cao@intel.com>
Subject: Re: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus

Hello,

This patchseries might have introduced a bug for building DPDK on windows. It failed on windows build when it went through our CI last week, and I am seeing other patch series fail on the windows build now that it is merged into main.

Tomorrow morning, I will check our windows system used for CI to verify it is still valid for building DPDK in terms of clang version, linker etc. But it seems fine from a quick glance right now. I will update if/when I learn more.

Thanks,
Patrick

--

Patrick Robb

Technical Service Manager

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

www.iol.unh.edu<http://www.iol.unh.edu/>



[Image removed by sender.]
  

> -----Original Message-----
> From: Miao Li <miao.li@intel.com>
> Sent: Wednesday, May 31, 2023 8:38 AM
> To: dev@dpdk.org
> Cc: skori@marvell.com; NBU-Contact-Thomas Monjalon (EXTERNAL)
> <thomas@monjalon.net>; david.marchand@redhat.com;
> ferruh.yigit@amd.com; chenbo.xia@intel.com; yahui.cao@intel.com
> Subject: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus
> 
> This series introduces a VFIO standard capability, called sparse
> mmap to PCI bus. In linux kernel, it's defined as
> VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> mmap whole BAR region into DPDK process, only mmap part of the
> BAR region after getting sparse mmap information from kernel.
> For the rest of BAR region that is not mmap-ed, DPDK process
> can use pread/pwrite system calls to access. Sparse mmap is
> useful when kernel does not want userspace to mmap whole BAR
> region, or kernel wants to control over access to specific BAR
> region. Vendors can choose to enable this feature or not for
> their devices in their specific kernel modules.
> 

Hello,

I see the build failure Patrick reported as well and can confirm it's caused by 095cf6e68b28 ("bus/pci: introduce MMIO read/write").
Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245

Regards,
Ali
  

> -----Original Message-----
> From: Ali Alnubani <alialnu@nvidia.com>
> Sent: Thursday, June 8, 2023 2:43 PM
> To: Li, Miao <miao.li@intel.com>; dev@dpdk.org; NBU-Contact-Thomas
> Monjalon (EXTERNAL) <thomas@monjalon.net>
> Cc: skori@marvell.com; david.marchand@redhat.com; ferruh.yigit@amd.com;
> Xia, Chenbo <chenbo.xia@intel.com>; Cao, Yahui <yahui.cao@intel.com>;
> Patrick Robb <probb@iol.unh.edu>
> Subject: RE: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus
> 
> > -----Original Message-----
> > From: Miao Li <miao.li@intel.com>
> > Sent: Wednesday, May 31, 2023 8:38 AM
> > To: dev@dpdk.org
> > Cc: skori@marvell.com; NBU-Contact-Thomas Monjalon (EXTERNAL)
> > <thomas@monjalon.net>; david.marchand@redhat.com;
> > ferruh.yigit@amd.com; chenbo.xia@intel.com; yahui.cao@intel.com
> > Subject: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus
> >
> > This series introduces a VFIO standard capability, called sparse
> > mmap to PCI bus. In linux kernel, it's defined as
> > VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> > mmap whole BAR region into DPDK process, only mmap part of the
> > BAR region after getting sparse mmap information from kernel.
> > For the rest of BAR region that is not mmap-ed, DPDK process
> > can use pread/pwrite system calls to access. Sparse mmap is
> > useful when kernel does not want userspace to mmap whole BAR
> > region, or kernel wants to control over access to specific BAR
> > region. Vendors can choose to enable this feature or not for
> > their devices in their specific kernel modules.
> >
> 
> Hello,
> 
> I see the build failure Patrick reported as well and can confirm it's
> caused by 095cf6e68b28 ("bus/pci: introduce MMIO read/write").
> Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245

Thanks Ali. I just read the bz and understand what's missing. I will send
a patch today.

But since last time CI is not reporting the error, this time how could I make
sure the fix will perfectly work?

Regards,
Chenbo

> 
> Regards,
> Ali
  

Hello Chenbo, Patrick,


On Thu, Jun 8, 2023 at 8:50 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > > This series introduces a VFIO standard capability, called sparse
> > > mmap to PCI bus. In linux kernel, it's defined as
> > > VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> > > mmap whole BAR region into DPDK process, only mmap part of the
> > > BAR region after getting sparse mmap information from kernel.
> > > For the rest of BAR region that is not mmap-ed, DPDK process
> > > can use pread/pwrite system calls to access. Sparse mmap is
> > > useful when kernel does not want userspace to mmap whole BAR
> > > region, or kernel wants to control over access to specific BAR
> > > region. Vendors can choose to enable this feature or not for
> > > their devices in their specific kernel modules.
> > >
> >
> > Hello,
> >
> > I see the build failure Patrick reported as well and can confirm it's
> > caused by 095cf6e68b28 ("bus/pci: introduce MMIO read/write").
> > Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245
>
> Thanks Ali. I just read the bz and understand what's missing. I will send
> a patch today.
>
> But since last time CI is not reporting the error, this time how could I make
> sure the fix will perfectly work?

Chenbo,

In theory, this error should have been reported so go ahead and post your fix.

Patrick,

This missing report could be a mail delivery issue (I can see the test
ran at UNH).
I see no trace in test-report ml.
Can you look at the reason?

Thanks.
  

On Thu, Jun 8, 2023 at 3:03 AM David Marchand <david.marchand@redhat.com>
wrote:

> Hello Chenbo, Patrick,
>
>
> On Thu, Jun 8, 2023 at 8:50 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > > > This series introduces a VFIO standard capability, called sparse
> > > > mmap to PCI bus. In linux kernel, it's defined as
> > > > VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> > > > mmap whole BAR region into DPDK process, only mmap part of the
> > > > BAR region after getting sparse mmap information from kernel.
> > > > For the rest of BAR region that is not mmap-ed, DPDK process
> > > > can use pread/pwrite system calls to access. Sparse mmap is
> > > > useful when kernel does not want userspace to mmap whole BAR
> > > > region, or kernel wants to control over access to specific BAR
> > > > region. Vendors can choose to enable this feature or not for
> > > > their devices in their specific kernel modules.
> > > >
> > >
> > > Hello,
> > >
> > > I see the build failure Patrick reported as well and can confirm it's
> > > caused by 095cf6e68b28 ("bus/pci: introduce MMIO read/write").
> > > Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245
> >
> > Thanks Ali. I just read the bz and understand what's missing. I will send
> > a patch today.
> >
> > But since last time CI is not reporting the error, this time how could I
> make
> > sure the fix will perfectly work?
>
> Chenbo,
>
> In theory, this error should have been reported so go ahead and post your
> fix.
>
> Patrick,
>
> This missing report could be a mail delivery issue (I can see the test
> ran at UNH).
> I see no trace in test-report ml.
> Can you look at the reason?
>
> Thanks.
>
>
> --
> David Marchand
>
>
I see where the issue is in our reporting process for the windows results
and I will resolve it after the CI meeting this morning. Sorry everyone for
the oversight.