[v6,1/2] bus/pci: use PCI standard sysfs entry to get PIO address
Checks
Commit Message
From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
Currently virtio PMD asssumes legacy device uses PIO bar.
There are three ways to get PIO(PortIO) address for virtio legacy device.
under igb_uio, get pio address from uio/uio# sysfs attribute
under uio_pci_generic:
for X86, get PIO address from /proc/ioport
for other ARCH, get PIO address from standard PCI sysfs attribute
Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e,
pci_dev->resource[]
This patch fixes these messy things, and uses standard PCI sysfs attribute.
Signed-off-by: huawei xie <huawei.xhw@alibaba-inc.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
drivers/bus/pci/linux/pci.c | 77 -----------------------------------------
drivers/bus/pci/linux/pci_uio.c | 64 ++++++++++++++++++++++++----------
2 files changed, 46 insertions(+), 95 deletions(-)
Comments
On 1/29/21 4:18 AM, 谢华伟(此时此刻） wrote:
> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>
> Currently virtio PMD asssumes legacy device uses PIO bar.
> There are three ways to get PIO(PortIO) address for virtio legacy device.
> under igb_uio, get pio address from uio/uio# sysfs attribute
> under uio_pci_generic:
> for X86, get PIO address from /proc/ioport
> for other ARCH, get PIO address from standard PCI sysfs attribute
>
> Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e,
> pci_dev->resource[]
>
> This patch fixes these messy things, and uses standard PCI sysfs attribute.
>
> Signed-off-by: huawei xie <huawei.xhw@alibaba-inc.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> drivers/bus/pci/linux/pci.c | 77 -----------------------------------------
> drivers/bus/pci/linux/pci_uio.c | 64 ++++++++++++++++++++++++----------
> 2 files changed, 46 insertions(+), 95 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
On Fri, Jan 29, 2021 at 4:19 AM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>
> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>
> Currently virtio PMD asssumes legacy device uses PIO bar.
> There are three ways to get PIO(PortIO) address for virtio legacy device.
> under igb_uio, get pio address from uio/uio# sysfs attribute
> under uio_pci_generic:
> for X86, get PIO address from /proc/ioport
> for other ARCH, get PIO address from standard PCI sysfs attribute
>
> Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e,
> pci_dev->resource[]
>
> This patch fixes these messy things, and uses standard PCI sysfs attribute.
I can not find what is being fixed.
Could you elaborate?
On 2021/2/18 17:33, David Marchand wrote:
> On Fri, Jan 29, 2021 at 4:19 AM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>
>> Currently virtio PMD asssumes legacy device uses PIO bar.
>> There are three ways to get PIO(PortIO) address for virtio legacy device.
>> under igb_uio, get pio address from uio/uio# sysfs attribute
>> under uio_pci_generic:
>> for X86, get PIO address from /proc/ioport
>> for other ARCH, get PIO address from standard PCI sysfs attribute
>>
>> Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e,
>> pci_dev->resource[]
>>
>> This patch fixes these messy things, and uses standard PCI sysfs attribute.
> I can not find what is being fixed.
> Could you elaborate?
Maybe i wrongly use the word fix? With this patch, different drivers
use the same way to get IO address for virtio device.
Previously different driver get IO address with different method.
>
On Sun, Feb 21, 2021 at 4:58 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>
>
> On 2021/2/18 17:33, David Marchand wrote:
> > On Fri, Jan 29, 2021 at 4:19 AM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
> >> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
> >>
> >> Currently virtio PMD asssumes legacy device uses PIO bar.
> >> There are three ways to get PIO(PortIO) address for virtio legacy device.
> >> under igb_uio, get pio address from uio/uio# sysfs attribute
> >> under uio_pci_generic:
> >> for X86, get PIO address from /proc/ioport
> >> for other ARCH, get PIO address from standard PCI sysfs attribute
> >>
> >> Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e,
> >> pci_dev->resource[]
> >>
> >> This patch fixes these messy things, and uses standard PCI sysfs attribute.
> > I can not find what is being fixed.
> > Could you elaborate?
>
> Maybe i wrongly use the word fix? With this patch, different drivers
> use the same way to get IO address for virtio device.
> Previously different driver get IO address with different method.
The commitlog is confusing, mentioning pci_dev->resource[] (I guess
from the kernel sources?) and talking about a fix while it works fine
so far.
So this is a refactoring.
Did you check that virtio devices bound to uio_pci_generic still works
with legacy mode + PIO?
On 2021/2/24 20:49, David Marchand wrote:
> On Sun, Feb 21, 2021 at 4:58 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>>
>> On 2021/2/18 17:33, David Marchand wrote:
>>> On Fri, Jan 29, 2021 at 4:19 AM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>>>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>>>
>>>> Currently virtio PMD asssumes legacy device uses PIO bar.
>>>> There are three ways to get PIO(PortIO) address for virtio legacy device.
>>>> under igb_uio, get pio address from uio/uio# sysfs attribute
>>>> under uio_pci_generic:
>>>> for X86, get PIO address from /proc/ioport
>>>> for other ARCH, get PIO address from standard PCI sysfs attribute
>>>>
>>>> Actually, igb_uio sysfs attribute exports exactly the same thing as standard PCI sysfs, i.e,
>>>> pci_dev->resource[]
>>>>
>>>> This patch fixes these messy things, and uses standard PCI sysfs attribute.
>>> I can not find what is being fixed.
>>> Could you elaborate?
>> Maybe i wrongly use the word fix? With this patch, different drivers
>> use the same way to get IO address for virtio device.
>> Previously different driver get IO address with different method.
> The commitlog is confusing, mentioning pci_dev->resource[] (I guess
> from the kernel sources?) and talking about a fix while it works fine
Yes.
> so far.
> So this is a refactoring.
Ok, will change the word fix.
>
> Did you check that virtio devices bound to uio_pci_generic still works
> with legacy mode + PIO?
I had verified PIO, might under igb_uio driver.
Theoretically it works with uio_pci_generic driver. I could do the test
if needed. Give me some time. Have to create a legacy VM. Now bars in
virtio device in the cloud are basically mmio.
>
>
On Wed, Feb 24, 2021 at 4:29 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
> > Did you check that virtio devices bound to uio_pci_generic still works
> > with legacy mode + PIO?
>
> I had verified PIO, might under igb_uio driver.
Well, if you are unsure, please retest both cases, igb_uio and uio_pci_generic.
>
> Theoretically it works with uio_pci_generic driver. I could do the test
> if needed. Give me some time. Have to create a legacy VM. Now bars in
> virtio device in the cloud are basically mmio.
Sure, take the time to check.
I'll wait for your confirmation, thanks.
On 2021/2/25 1:52, David Marchand wrote:
> On Wed, Feb 24, 2021 at 4:29 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>>> Did you check that virtio devices bound to uio_pci_generic still works
>>> with legacy mode + PIO?
>> I had verified PIO, might under igb_uio driver.
> Well, if you are unsure, please retest both cases, igb_uio and uio_pci_generic.
>
I still haven't got a VM with PIO, but retested MMIO with both igb_uio
and uio_pci_generic. They both works.
@Chenbo
Could you help run with PIO?
>> Theoretically it works with uio_pci_generic driver. I could do the test
>> if needed. Give me some time. Have to create a legacy VM. Now bars in
>> virtio device in the cloud are basically mmio.
> Sure, take the time to check.
> I'll wait for your confirmation, thanks.
>
>
On 2021/2/25 1:52, David Marchand wrote:
> On Wed, Feb 24, 2021 at 4:29 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>>> Did you check that virtio devices bound to uio_pci_generic still works
>>> with legacy mode + PIO?
>> I had verified PIO, might under igb_uio driver.
> Well, if you are unsure, please retest both cases, igb_uio and uio_pci_generic.
>
>
>> Theoretically it works with uio_pci_generic driver. I could do the test
>> if needed. Give me some time. Have to create a legacy VM. Now bars in
>> virtio device in the cloud are basically mmio.
> Sure, take the time to check.
> I'll wait for your confirmation, thanks.
Hi David:
Tested PIO under igb_uio and uio_pci_generic driver. Both works.
Also tested MMIO under both drivers.
>
>
@@ -677,71 +677,6 @@ int rte_pci_write_config(const struct rte_pci_device *device,
}
}
-#if defined(RTE_ARCH_X86)
-static int
-pci_ioport_map(struct rte_pci_device *dev, int bar __rte_unused,
- struct rte_pci_ioport *p)
-{
- uint16_t start, end;
- FILE *fp;
- char *line = NULL;
- char pci_id[16];
- int found = 0;
- size_t linesz;
-
- if (rte_eal_iopl_init() != 0) {
- RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions for PCI device %s\n",
- __func__, dev->name);
- return -1;
- }
-
- snprintf(pci_id, sizeof(pci_id), PCI_PRI_FMT,
- dev->addr.domain, dev->addr.bus,
- dev->addr.devid, dev->addr.function);
-
- fp = fopen("/proc/ioports", "r");
- if (fp == NULL) {
- RTE_LOG(ERR, EAL, "%s(): can't open ioports\n", __func__);
- return -1;
- }
-
- while (getdelim(&line, &linesz, '\n', fp) > 0) {
- char *ptr = line;
- char *left;
- int n;
-
- n = strcspn(ptr, ":");
- ptr[n] = 0;
- left = &ptr[n + 1];
-
- while (*left && isspace(*left))
- left++;
-
- if (!strncmp(left, pci_id, strlen(pci_id))) {
- found = 1;
-
- while (*ptr && isspace(*ptr))
- ptr++;
-
- sscanf(ptr, "%04hx-%04hx", &start, &end);
-
- break;
- }
- }
-
- free(line);
- fclose(fp);
-
- if (!found)
- return -1;
-
- p->base = start;
- RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%x\n", start);
-
- return 0;
-}
-#endif
-
int
rte_pci_ioport_map(struct rte_pci_device *dev, int bar,
struct rte_pci_ioport *p)
@@ -756,14 +691,8 @@ int rte_pci_write_config(const struct rte_pci_device *device,
break;
#endif
case RTE_PCI_KDRV_IGB_UIO:
- ret = pci_uio_ioport_map(dev, bar, p);
- break;
case RTE_PCI_KDRV_UIO_GENERIC:
-#if defined(RTE_ARCH_X86)
- ret = pci_ioport_map(dev, bar, p);
-#else
ret = pci_uio_ioport_map(dev, bar, p);
-#endif
break;
default:
break;
@@ -830,14 +759,8 @@ int rte_pci_write_config(const struct rte_pci_device *device,
break;
#endif
case RTE_PCI_KDRV_IGB_UIO:
- ret = pci_uio_ioport_unmap(p);
- break;
case RTE_PCI_KDRV_UIO_GENERIC:
-#if defined(RTE_ARCH_X86)
- ret = 0;
-#else
ret = pci_uio_ioport_unmap(p);
-#endif
break;
default:
break;
@@ -373,10 +373,13 @@
pci_uio_ioport_map(struct rte_pci_device *dev, int bar,
struct rte_pci_ioport *p)
{
+ FILE *f = NULL;
char dirname[PATH_MAX];
char filename[PATH_MAX];
- int uio_num;
- unsigned long start;
+ char buf[BUFSIZ];
+ uint64_t phys_addr, end_addr, flags;
+ unsigned long base;
+ int i;
if (rte_eal_iopl_init() != 0) {
RTE_LOG(ERR, EAL, "%s(): insufficient ioport permissions for PCI device %s\n",
@@ -384,41 +387,66 @@
return -1;
}
- uio_num = pci_get_uio_dev(dev, dirname, sizeof(dirname), 0);
- if (uio_num < 0)
+ /* open and read addresses of the corresponding resource in sysfs */
+ snprintf(filename, sizeof(filename), "%s/" PCI_PRI_FMT "/resource",
+ rte_pci_get_sysfs_path(), dev->addr.domain, dev->addr.bus,
+ dev->addr.devid, dev->addr.function);
+ f = fopen(filename, "r");
+ if (f == NULL) {
+ RTE_LOG(ERR, EAL, "%s(): Cannot open sysfs resource: %s\n",
+ __func__, strerror(errno));
return -1;
+ }
- /* get portio start */
- snprintf(filename, sizeof(filename),
- "%s/portio/port%d/start", dirname, bar);
- if (eal_parse_sysfs_value(filename, &start) < 0) {
- RTE_LOG(ERR, EAL, "%s(): cannot parse portio start\n",
- __func__);
- return -1;
+ for (i = 0; i < bar + 1; i++) {
+ if (fgets(buf, sizeof(buf), f) == NULL) {
+ RTE_LOG(ERR, EAL, "%s(): Cannot read sysfs resource\n", __func__);
+ goto error;
+ }
}
- /* ensure we don't get anything funny here, read/write will cast to
- * uin16_t */
- if (start > UINT16_MAX)
- return -1;
+ if (pci_parse_one_sysfs_resource(buf, sizeof(buf), &phys_addr,
+ &end_addr, &flags) < 0)
+ goto error;
+
+ if (!(flags & IORESOURCE_IO)) {
+ RTE_LOG(ERR, EAL, "%s(): bar resource other than IO is not supported\n", __func__);
+ goto error;
+ }
+ base = (unsigned long)phys_addr;
+ RTE_LOG(INFO, EAL, "%s(): PIO BAR %08lx detected\n", __func__, base);
+
+ if (base > UINT16_MAX)
+ goto error;
/* FIXME only for primary process ? */
if (dev->intr_handle.type == RTE_INTR_HANDLE_UNKNOWN) {
+ int uio_num = pci_get_uio_dev(dev, dirname, sizeof(dirname), 0);
+ if (uio_num < 0) {
+ RTE_LOG(ERR, EAL, "cannot open %s: %s\n",
+ dirname, strerror(errno));
+ goto error;
+ }
snprintf(filename, sizeof(filename), "/dev/uio%u", uio_num);
dev->intr_handle.fd = open(filename, O_RDWR);
if (dev->intr_handle.fd < 0) {
RTE_LOG(ERR, EAL, "Cannot open %s: %s\n",
filename, strerror(errno));
- return -1;
+ goto error;
}
dev->intr_handle.type = RTE_INTR_HANDLE_UIO;
}
- RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%lx\n", start);
+ RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%lx\n", base);
- p->base = start;
+ p->base = base;
p->len = 0;
+ fclose(f);
return 0;
+error:
+ if (f)
+ fclose(f);
+ return -1;
}
#else
int