[v6] pci: read amd iommu virtual address width
Checks
Commit Message
Add code to read the virtual address width for AMD processors.
Updated pci_device_iommu_support_va() to use glob to find iommu
capability files.
Signed-off-by: Michael Piszczek <mpiszczek@ddn.com>
---
drivers/bus/pci/linux/pci.c | 58 ++++++++++++++++++++++---------------
1 file changed, 35 insertions(+), 23 deletions(-)
Comments
On Mon, Oct 24, 2022 at 5:35 PM Michael Piszczek <mpiszczek@ddn.com> wrote:
>
> Add code to read the virtual address width for AMD processors.
> Updated pci_device_iommu_support_va() to use glob to find iommu
> capability files.
>
> Signed-off-by: Michael Piszczek <mpiszczek@ddn.com>
Please have a look at the ci.
Unit tests are failing with this patch because the default is changed
from VA to PA.
A quick way to reproduce is to run vdev unit tests as a normal user in
no huge mode.
Before change:
$ DPDK_TEST=vdev_autotest ./build-gcc/app/test/dpdk-test --no-huge -m
2048 --log-level=*:debug
...
EAL: Bus vdev wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'DC'
EAL: Buses did not request a specific IOVA mode.
EAL: Physical addresses are unavailable, selecting IOVA as VA mode.
EAL: Selected IOVA mode 'VA'
After change:
$ DPDK_TEST=vdev_autotest ./build-gcc/app/test/dpdk-test --no-huge -m
2048 --log-level=*:debug
...
EAL: Bus vdev wants IOVA as 'DC'
EAL: Bus pci wants IOVA as 'PA'
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
On Tue, Oct 25, 2022 at 1:54 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Mon, Oct 24, 2022 at 5:35 PM Michael Piszczek <mpiszczek@ddn.com> wrote:
> >
> > Add code to read the virtual address width for AMD processors.
> > Updated pci_device_iommu_support_va() to use glob to find iommu
> > capability files.
> >
> > Signed-off-by: Michael Piszczek <mpiszczek@ddn.com>
>
> Please have a look at the ci.
>
> Unit tests are failing with this patch because the default is changed
> from VA to PA.
> A quick way to reproduce is to run vdev unit tests as a normal user in
> no huge mode.
>
> Before change:
> $ DPDK_TEST=vdev_autotest ./build-gcc/app/test/dpdk-test --no-huge -m
> 2048 --log-level=*:debug
> ...
> EAL: Bus vdev wants IOVA as 'DC'
> EAL: Bus pci wants IOVA as 'DC'
> EAL: Buses did not request a specific IOVA mode.
> EAL: Physical addresses are unavailable, selecting IOVA as VA mode.
> EAL: Selected IOVA mode 'VA'
>
> After change:
> $ DPDK_TEST=vdev_autotest ./build-gcc/app/test/dpdk-test --no-huge -m
> 2048 --log-level=*:debug
> ...
> EAL: Bus vdev wants IOVA as 'DC'
> EAL: Bus pci wants IOVA as 'PA'
> EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
> EAL: Cannot use IOVA as 'PA' since physical addresses are not available
I got no reply since a few months, marking as "Changes requested" in patchwork.
If you think this is still worth pursuing, please provide a new
revision that does not break IOVA default mode (and the CI).
Cc: Ferruh, for info
Hi David,
Sorry I do not have a solution for this. Can you close the patch?
Michael Piszczek
-----Original Message-----
From: David Marchand <david.marchand@redhat.com>
Sent: Tuesday, August 8, 2023 3:31 AM
To: Michael Piszczek <mpiszczek@ddn.com>
Cc: dev@dpdk.org; Ferruh Yigit <ferruh.yigit@amd.com>
Subject: Re: [PATCH v6] pci: read amd iommu virtual address width
On Tue, Oct 25, 2022 at 1:54 PM David Marchand <david.marchand@redhat.com> wrote:
>
> On Mon, Oct 24, 2022 at 5:35 PM Michael Piszczek <mpiszczek@ddn.com> wrote:
> >
> > Add code to read the virtual address width for AMD processors.
> > Updated pci_device_iommu_support_va() to use glob to find iommu
> > capability files.
> >
> > Signed-off-by: Michael Piszczek <mpiszczek@ddn.com>
>
> Please have a look at the ci.
>
> Unit tests are failing with this patch because the default is changed
> from VA to PA.
> A quick way to reproduce is to run vdev unit tests as a normal user in
> no huge mode.
>
> Before change:
> $ DPDK_TEST=vdev_autotest ./build-gcc/app/test/dpdk-test --no-huge -m
> 2048 --log-level=*:debug
> ...
> EAL: Bus vdev wants IOVA as 'DC'
> EAL: Bus pci wants IOVA as 'DC'
> EAL: Buses did not request a specific IOVA mode.
> EAL: Physical addresses are unavailable, selecting IOVA as VA mode.
> EAL: Selected IOVA mode 'VA'
>
> After change:
> $ DPDK_TEST=vdev_autotest ./build-gcc/app/test/dpdk-test --no-huge -m
> 2048 --log-level=*:debug
> ...
> EAL: Bus vdev wants IOVA as 'DC'
> EAL: Bus pci wants IOVA as 'PA'
> EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not
> available
> EAL: Cannot use IOVA as 'PA' since physical addresses are not
> available
I got no reply since a few months, marking as "Changes requested" in patchwork.
If you think this is still worth pursuing, please provide a new revision that does not break IOVA default mode (and the CI).
Cc: Ferruh, for info
--
David Marchand
@@ -4,6 +4,7 @@
#include <string.h>
#include <dirent.h>
+#include <glob.h>
#include <rte_log.h>
#include <rte_pci.h>
@@ -480,42 +481,53 @@ rte_pci_scan(void)
}
#if defined(RTE_ARCH_X86)
+
bool
pci_device_iommu_support_va(const struct rte_pci_device *dev)
{
-#define VTD_CAP_MGAW_SHIFT 16
-#define VTD_CAP_MGAW_MASK (0x3fULL << VTD_CAP_MGAW_SHIFT)
+#define VTD_CAP_MGAW_SHIFT 16
+#define VTD_CAP_MGAW_MASK (0x3fULL << VTD_CAP_MGAW_SHIFT)
+#define RD_AMD_CAP_VASIZE_SHIFT 15
+#define RD_AMD_CAP_VASIZE_MASK (0x7F << RD_AMD_CAP_VASIZE_SHIFT)
+ int rc;
const struct rte_pci_addr *addr = &dev->addr;
- char filename[PATH_MAX];
- FILE *fp;
- uint64_t mgaw, vtd_cap_reg = 0;
+ char pattern[PATH_MAX];
+ glob_t glob_results;
+ uint64_t mgaw = 0;
- snprintf(filename, sizeof(filename),
- "%s/" PCI_PRI_FMT "/iommu/intel-iommu/cap",
+ snprintf(pattern, sizeof(pattern),
+ "%s/" PCI_PRI_FMT "/iommu/*-iommu/cap",
rte_pci_get_sysfs_path(), addr->domain, addr->bus, addr->devid,
addr->function);
- fp = fopen(filename, "r");
- if (fp == NULL) {
- /* We don't have an Intel IOMMU, assume VA supported */
- if (errno == ENOENT)
- return true;
+ rc = glob(pattern, 0, NULL, &glob_results);
+ if (rc != 0 && glob_results.gl_pathc == 1) {
+ const char *filename = glob_results.gl_pathv[0];
+ FILE *fp = fopen(filename, "r");
- RTE_LOG(ERR, EAL, "%s(): can't open %s: %s\n",
- __func__, filename, strerror(errno));
- return false;
- }
+ if (fp != NULL) {
+ uint64_t cap_reg = 0;
- /* We have an Intel IOMMU */
- if (fscanf(fp, "%" PRIx64, &vtd_cap_reg) != 1) {
- RTE_LOG(ERR, EAL, "%s(): can't read %s\n", __func__, filename);
- fclose(fp);
- return false;
+ if (fscanf(fp, "%" PRIx64, &cap_reg) != 1) {
+ RTE_LOG(ERR, EAL, "%s(): can't read %s\n", __func__, filename);
+ }
+ else if (strstr(filename, "intel-iommu") != NULL) {
+ /* We have an Intel IOMMU */
+ mgaw = ((cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 1;
+ }
+ else if (strstr(filename, "amd-iommu") != NULL) {
+ /* We have an Amd IOMMU */
+ mgaw = ((cap_reg & RD_AMD_CAP_VASIZE_MASK) >> RD_AMD_CAP_VASIZE_SHIFT) + 1;
+ }
+
+ fclose(fp);
+ }
}
- fclose(fp);
+ globfree(&glob_results);
- mgaw = ((vtd_cap_reg & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 1;
+ if (mgaw == 0)
+ return false;
/*
* Assuming there is no limitation by now. We can not know at this point