bus/pci: support segment value as address domain on Windows
Checks
Commit Message
Set the domain value for rte_pci_addr probing on Windows
to the value of the PCI segment returned by SPDRP_BUSNUMBER.
Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
---
drivers/bus/pci/windows/pci.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
Comments
On Tue, Aug 25, 2020 at 02:43:16PM +0300, Tal Shnaiderman wrote:
> Set the domain value for rte_pci_addr probing on Windows
> to the value of the PCI segment returned by SPDRP_BUSNUMBER.
>
> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> ---
> drivers/bus/pci/windows/pci.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
> index 489aa7902a..a40acec609 100644
> --- a/drivers/bus/pci/windows/pci.c
> +++ b/drivers/bus/pci/windows/pci.c
> @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
> return -1;
> }
>
> - addr->domain = 0;
> - addr->bus = bus_num;
> + addr->domain = bus_num >> 8;
> + addr->bus = bus_num & 0xff;
> addr->devid = dev_and_func >> 16;
> addr->function = dev_and_func & 0xffff;
> return 0;
> --
Is this needed to avoid collision of devices with the same B:D:F?
> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
> Windows
>
> On Tue, Aug 25, 2020 at 02:43:16PM +0300, Tal Shnaiderman wrote:
> > Set the domain value for rte_pci_addr probing on Windows to the value
> > of the PCI segment returned by SPDRP_BUSNUMBER.
> >
> > Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> > ---
> > drivers/bus/pci/windows/pci.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/bus/pci/windows/pci.c
> > b/drivers/bus/pci/windows/pci.c index 489aa7902a..a40acec609 100644
> > --- a/drivers/bus/pci/windows/pci.c
> > +++ b/drivers/bus/pci/windows/pci.c
> > @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
> > return -1;
> > }
> >
> > - addr->domain = 0;
> > - addr->bus = bus_num;
> > + addr->domain = bus_num >> 8;
> > + addr->bus = bus_num & 0xff;
> > addr->devid = dev_and_func >> 16;
> > addr->function = dev_and_func & 0xffff;
> > return 0;
> > --
> Is this needed to avoid collision of devices with the same B:D:F?
Right, it can happen in virtualization setups when several virtual functions can have the same BDF, e.g.:
PS > Get-NetAdapterHardwareInfo
Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed
---- ------- --- ------ -------- ---- -------- -------------
Ethernet 0 0 10 0 Unknown
Ethernet 4 58601 0 2 0 0 Unknown
Ethernet 5 52956 0 2 0 0 Unknown
DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is checked.
Unix uses the Domain value, the equivalent value for Windows is Segment.
On 9/10/2020 12:30 AM, Tal Shnaiderman wrote:
>> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
>> Windows
>>
>> On Tue, Aug 25, 2020 at 02:43:16PM +0300, Tal Shnaiderman wrote:
>>> Set the domain value for rte_pci_addr probing on Windows to the value
>>> of the PCI segment returned by SPDRP_BUSNUMBER.
>>>
>>> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
>>> ---
>>> drivers/bus/pci/windows/pci.c | 4 ++--
>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/bus/pci/windows/pci.c
>>> b/drivers/bus/pci/windows/pci.c index 489aa7902a..a40acec609 100644
>>> --- a/drivers/bus/pci/windows/pci.c
>>> +++ b/drivers/bus/pci/windows/pci.c
>>> @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
>>> return -1;
>>> }
>>>
>>> - addr->domain = 0;
>>> - addr->bus = bus_num;
>>> + addr->domain = bus_num >> 8;
>>> + addr->bus = bus_num & 0xff;
>>> addr->devid = dev_and_func >> 16;
>>> addr->function = dev_and_func & 0xffff;
>>> return 0;
>>> --
>> Is this needed to avoid collision of devices with the same B:D:F?
> Right, it can happen in virtualization setups when several virtual functions can have the same BDF, e.g.:
>
> PS > Get-NetAdapterHardwareInfo
>
> Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed
> ---- ------- --- ------ -------- ---- -------- -------------
> Ethernet 0 0 10 0 Unknown
> Ethernet 4 58601 0 2 0 0 Unknown
> Ethernet 5 52956 0 2 0 0 Unknown
>
> DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is checked.
> Unix uses the Domain value, the equivalent value for Windows is Segment.
Thanks for the explanation, Tal.
I had always been curious how Windows stores the PCIe segment (domain)
number.
On VMs hosted on Hyper-V, the VF segment numbers are always in the high
16-bit values.
Is this documented somewhere, or did you find this by experimentation?
ranjit m.
On Thu, 10 Sep 2020 07:30:39 +0000, Tal Shnaiderman wrote:
> Right, it can happen in virtualization setups when several virtual functions can have the same BDF, e.g.:
>
> PS > Get-NetAdapterHardwareInfo
>
> Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed
> ---- ------- --- ------ -------- ---- -------- -------------
> Ethernet 0 0 10 0 Unknown
> Ethernet 4 58601 0 2 0 0 Unknown
> Ethernet 5 52956 0 2 0 0 Unknown
>
> DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is checked.
> Unix uses the Domain value, the equivalent value for Windows is Segment.
Hi Tal,
I wonder how exactly this setup can be reproduced, i.e. could you
share relevant QEMU options, VMX file or some other config you're using?
Patch idea and code look clear, however, I never managed to build QEMU PCIe
hierarchy to see it working.
> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
> Windows
>
> Thanks for the explanation, Tal.
>
> I had always been curious how Windows stores the PCIe segment (domain)
> number.
>
> On VMs hosted on Hyper-V, the VF segment numbers are always in the high
> 16-bit values.
>
> Is this documented somewhere, or did you find this by experimentation?
>
Hi Ranjit,
I didn’t find documentation on it.
I found it by experimentation on a Windows VM with several VFs using the same virtual switch, Linux VFs showed the same behavior but they are being detected using the domain value which is different.
>
> ranjit m.
> Subject: Re: [PATCH] bus/pci: support segment value as address domain on
> Windows
>
> On Thu, 10 Sep 2020 07:30:39 +0000, Tal Shnaiderman wrote:
> > Right, it can happen in virtualization setups when several virtual functions
> can have the same BDF, e.g.:
> >
> > PS > Get-NetAdapterHardwareInfo
> >
> > Name Segment Bus Device Function Slot NumaNode
> PcieLinkSpeed
> > ---- ------- --- ------ -------- ---- -------- -------------
> > Ethernet 0 0 10 0 Unknown
> > Ethernet 4 58601 0 2 0 0 Unknown
> > Ethernet 5 52956 0 2 0 0 Unknown
> >
> > DPDK currently can detect either Ethernet 4 or ethernet 5 if only BDF is
> checked.
> > Unix uses the Domain value, the equivalent value for Windows is Segment.
>
> Hi Tal,
>
> I wonder how exactly this setup can be reproduced, i.e. could you share
> relevant QEMU options, VMX file or some other config you're using?
> Patch idea and code look clear, however, I never managed to build QEMU
> PCIe hierarchy to see it working.
Hi Dmitry,
I'm not sure it is relevant to all NICs but for Mellanox you can recreate it on a Hyper-V Windows VM with several SR-IOV VFs using the same virtual switch.
On 8/25/2020 4:43 AM, Tal Shnaiderman wrote:
> Set the domain value for rte_pci_addr probing on Windows
> to the value of the PCI segment returned by SPDRP_BUSNUMBER.
>
> Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> ---
> drivers/bus/pci/windows/pci.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/bus/pci/windows/pci.c b/drivers/bus/pci/windows/pci.c
> index 489aa7902a..a40acec609 100644
> --- a/drivers/bus/pci/windows/pci.c
> +++ b/drivers/bus/pci/windows/pci.c
> @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
> return -1;
> }
>
> - addr->domain = 0;
> - addr->bus = bus_num;
> + addr->domain = bus_num >> 8;
> + addr->bus = bus_num & 0xff;
> addr->devid = dev_and_func >> 16;
> addr->function = dev_and_func & 0xffff;
> return 0;
Acked-by: Ranjit Menon <ranjit.menon@intel.com>
16/09/2020 20:32, Ranjit Menon:
> On 8/25/2020 4:43 AM, Tal Shnaiderman wrote:
> > Set the domain value for rte_pci_addr probing on Windows
> > to the value of the PCI segment returned by SPDRP_BUSNUMBER.
> >
> > Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
> > ---
> > --- a/drivers/bus/pci/windows/pci.c
> > +++ b/drivers/bus/pci/windows/pci.c
> > @@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
> > - addr->domain = 0;
> > - addr->bus = bus_num;
> > + addr->domain = bus_num >> 8;
> > + addr->bus = bus_num & 0xff;
> > addr->devid = dev_and_func >> 16;
> > addr->function = dev_and_func & 0xffff;
>
>
> Acked-by: Ranjit Menon <ranjit.menon@intel.com>
I was waiting for feedback from Microsoft, but never happened, so
Applied, thanks
Microsoft, we need a better support for porting DPDK on Windows.
@@ -195,8 +195,8 @@ get_device_pci_address(HDEVINFO dev_info,
return -1;
}
- addr->domain = 0;
- addr->bus = bus_num;
+ addr->domain = bus_num >> 8;
+ addr->bus = bus_num & 0xff;
addr->devid = dev_and_func >> 16;
addr->function = dev_and_func & 0xffff;
return 0;