[v1] event/dlb2: add support for disabling PASID
Checks
Commit Message
vfio-pci driver in Linux kernel 6.2 enables PASID by default.
In DLB hardware, enabling PASID puts DLB in SIOV mode. This
breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
PASID needs to be disabled for kernel 6.2.
In this commit this issue is addressed and PASID is disabled
by writing a zero to PASID control register.
Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
---
drivers/event/dlb2/pf/dlb2_main.c | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
Comments
On Thu, Jun 8, 2023 at 2:31 AM Abdullah Sevincer
<abdullah.sevincer@intel.com> wrote:
>
> vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> In DLB hardware, enabling PASID puts DLB in SIOV mode. This
> breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
> PASID needs to be disabled for kernel 6.2.
>
> In this commit this issue is addressed and PASID is disabled
> by writing a zero to PASID control register.
>
> Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
> + /* The current Linux kernel vfio driver does not expose PASID capability to
> + * users. It also enables PASID by default, which breaks DLB PF PMD. We have
> + * to use the hardcoded offset for now to disable PASID.
> + */
> + pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
> +
> + off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
+++ additional folks.
Is make sense to move this helper function to PCI common for disabling
PASID for a PCI device so that other driver can use if needed
as the implementation is not specific to DLB2.
> + if (rte_pci_read_config(pdev, &pasid_ctrl, 2, off) != 2)
> + pasid_ctrl = 0;
> +
> + if (pasid_ctrl) {
> + DLB2_INFO(dlb2_dev, "DLB2 disabling pasid...\n");
> +
> + pasid_ctrl = 0;
> + ret = rte_pci_write_config(pdev, &pasid_ctrl, 2, off);
> + if (ret != 2) {
> + DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
> + __func__, (int)off);
> + return ret;
> + }
> + }
> +
> return 0;
> }
>
> --
> 2.25.1
>
08/06/2023 07:38, Jerin Jacob:
> On Thu, Jun 8, 2023 at 2:31 AM Abdullah Sevincer
> <abdullah.sevincer@intel.com> wrote:
> >
> > vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> > In DLB hardware, enabling PASID puts DLB in SIOV mode. This
> > breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
> > PASID needs to be disabled for kernel 6.2.
> >
> > In this commit this issue is addressed and PASID is disabled
> > by writing a zero to PASID control register.
> >
> > Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
>
> > + /* The current Linux kernel vfio driver does not expose PASID capability to
> > + * users. It also enables PASID by default, which breaks DLB PF PMD. We have
> > + * to use the hardcoded offset for now to disable PASID.
> > + */
> > + pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
> > +
> > + off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
>
> +++ additional folks.
>
> Is make sense to move this helper function to PCI common for disabling
> PASID for a PCI device so that other driver can use if needed
> as the implementation is not specific to DLB2.
I guess yes it makes sense.
On Thu, Jun 8, 2023 at 7:38 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Thu, Jun 8, 2023 at 2:31 AM Abdullah Sevincer
> <abdullah.sevincer@intel.com> wrote:
> >
> > vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> > In DLB hardware, enabling PASID puts DLB in SIOV mode. This
> > breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
> > PASID needs to be disabled for kernel 6.2.
> >
> > In this commit this issue is addressed and PASID is disabled
> > by writing a zero to PASID control register.
> >
> > Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
>
> > + /* The current Linux kernel vfio driver does not expose PASID capability to
> > + * users. It also enables PASID by default, which breaks DLB PF PMD. We have
> > + * to use the hardcoded offset for now to disable PASID.
> > + */
> > + pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
> > +
> > + off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
>
> +++ additional folks.
>
> Is make sense to move this helper function to PCI common for disabling
> PASID for a PCI device so that other driver can use if needed
> as the implementation is not specific to DLB2.
Yes, having a helper sounds like a first step (and we probably have
more helpers to add seeing how drivers tend to redefine non vendor
specific pci configs, but that's another story).
Now, about PASID being enabled by default with Linux 6.2, is this
breaking of dlb PF something special? Or can we expect many (all?)
other devices to break too?
If so, maybe we should disable it in the pci common code.
On Thu, Jun 8, 2023 at 12:59 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Thu, Jun 8, 2023 at 7:38 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > On Thu, Jun 8, 2023 at 2:31 AM Abdullah Sevincer
> > <abdullah.sevincer@intel.com> wrote:
> > >
> > > vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> > > In DLB hardware, enabling PASID puts DLB in SIOV mode. This
> > > breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
> > > PASID needs to be disabled for kernel 6.2.
> > >
> > > In this commit this issue is addressed and PASID is disabled
> > > by writing a zero to PASID control register.
> > >
> > > Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
> >
> > > + /* The current Linux kernel vfio driver does not expose PASID capability to
> > > + * users. It also enables PASID by default, which breaks DLB PF PMD. We have
> > > + * to use the hardcoded offset for now to disable PASID.
> > > + */
> > > + pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
> > > +
> > > + off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
> >
> > +++ additional folks.
> >
> > Is make sense to move this helper function to PCI common for disabling
> > PASID for a PCI device so that other driver can use if needed
> > as the implementation is not specific to DLB2.
>
> Yes, having a helper sounds like a first step (and we probably have
> more helpers to add seeing how drivers tend to redefine non vendor
> specific pci configs, but that's another story).
@Abdullah Sevincer Please move the implementation to code PCI code.
>
> Now, about PASID being enabled by default with Linux 6.2, is this
> breaking of dlb PF something special? Or can we expect many (all?)
> other devices to break too?
> If so, maybe we should disable it in the pci common code.
@Abdullah Sevincer Is implicitly enabling SIOV based on PASID
configuration DLB2 behavior or general PCI behavior that may affect
other NIC's?
>
>
> --
> David Marchand
>
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Thursday, June 8, 2023 7:32 AM
> To: David Marchand <david.marchand@redhat.com>
> Cc: Sevincer, Abdullah <abdullah.sevincer@intel.com>; Gaetan Rivet <grive@u256.net>; Thomas Monjalon
> <thomas@monjalon.net>; dev@dpdk.org; jerinj@marvell.com; Chen, Mike Ximing
> <mike.ximing.chen@intel.com>
> Subject: Re: [PATCH v1] event/dlb2: add support for disabling PASID
>
> On Thu, Jun 8, 2023 at 12:59 PM David Marchand <david.marchand@redhat.com> wrote:
> >
> > On Thu, Jun 8, 2023 at 7:38 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > >
> > > On Thu, Jun 8, 2023 at 2:31 AM Abdullah Sevincer
> > > <abdullah.sevincer@intel.com> wrote:
> > > >
> > > > vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> > > > In DLB hardware, enabling PASID puts DLB in SIOV mode. This breaks
> > > > DLB PF-PMD mode. For DLB PF-PMD mode to function properly PASID
> > > > needs to be disabled for kernel 6.2.
> > > >
> > > > In this commit this issue is addressed and PASID is disabled by
> > > > writing a zero to PASID control register.
> > > >
> > > > Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
> > >
> > > > + /* The current Linux kernel vfio driver does not expose PASID capability to
> > > > + * users. It also enables PASID by default, which breaks DLB PF PMD. We have
> > > > + * to use the hardcoded offset for now to disable PASID.
> > > > + */
> > > > + pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
> > > > +
> > > > + off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
> > >
> > > +++ additional folks.
> > >
> > > Is make sense to move this helper function to PCI common for
> > > disabling PASID for a PCI device so that other driver can use if
> > > needed as the implementation is not specific to DLB2.
> >
> > Yes, having a helper sounds like a first step (and we probably have
> > more helpers to add seeing how drivers tend to redefine non vendor
> > specific pci configs, but that's another story).
>
> @Abdullah Sevincer Please move the implementation to code PCI code.
>
> >
> > Now, about PASID being enabled by default with Linux 6.2, is this
> > breaking of dlb PF something special? Or can we expect many (all?)
> > other devices to break too?
> > If so, maybe we should disable it in the pci common code.
>
> @Abdullah Sevincer Is implicitly enabling SIOV based on PASID configuration DLB2 behavior or general PCI
> behavior that may affect other NIC's?
PASID capability is required for SIOV, but not needed for PF mode.
It makes sense to disable PASID in any PF mode that uses the vfio-pci driver (which enables PASID by
Default since kernel 6.2).
>
> >
> >
> > --
> > David Marchand
> >
On Thu, Jun 8, 2023 at 9:28 AM David Marchand <david.marchand@redhat.com> wrote:
>
> On Thu, Jun 8, 2023 at 7:38 AM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > On Thu, Jun 8, 2023 at 2:31 AM Abdullah Sevincer
> > <abdullah.sevincer@intel.com> wrote:
> > >
> > > vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> > > In DLB hardware, enabling PASID puts DLB in SIOV mode. This
> > > breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
> > > PASID needs to be disabled for kernel 6.2.
> > >
> > > In this commit this issue is addressed and PASID is disabled
> > > by writing a zero to PASID control register.
> > >
> > > Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
> >
> > > + /* The current Linux kernel vfio driver does not expose PASID capability to
> > > + * users. It also enables PASID by default, which breaks DLB PF PMD. We have
> > > + * to use the hardcoded offset for now to disable PASID.
> > > + */
> > > + pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
> > > +
> > > + off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
> >
> > +++ additional folks.
> >
> > Is make sense to move this helper function to PCI common for disabling
> > PASID for a PCI device so that other driver can use if needed
> > as the implementation is not specific to DLB2.
>
> Yes, having a helper sounds like a first step (and we probably have
> more helpers to add seeing how drivers tend to redefine non vendor
> specific pci configs, but that's another story).
About this other story, I sent a cleanup series:
https://patchwork.dpdk.org/project/dpdk/list/?series=29101&state=%2A&archive=both
Comments/reviews welcome.
>
> Now, about PASID being enabled by default with Linux 6.2, is this
> breaking of dlb PF something special? Or can we expect many (all?)
> other devices to break too?
> If so, maybe we should disable it in the pci common code.
Is anybody looking into reworking this proposal and moving this code
to the pci bus driver?
Cc: pci bus maintainers.
>+Is anybody looking into reworking this proposal and moving this code to the pci bus driver?
>+Cc: pci bus maintainers.
>+--
>+David Marchand
We will work on this proposal, it is not finalized yet.
Its not DLB2 specific as commenters say, we are looking into if there is another way doing it.
If it will end up common and not DLB specific for other drivers as well we will add function to pci common.
>+We will work on this proposal, it is not finalized yet.
>+Its not DLB2 specific as commenters say, we are looking into if there is another way doing it.
>+If it will end up common and not DLB specific for other drivers as well we will add function to pci common.
As far as we know this problem is specific to DLB only, other drivers may have different design so they may not need to disable
PASID. I have moved the defines to rte_pci.h file(in case other drivers wants to disable PASID) and used function rte_pci_find_ext_capability to retrieve the offset.
On Wed, 7 Jun 2023 16:00:50 -0500
Abdullah Sevincer <abdullah.sevincer@intel.com> wrote:
> From: Abdullah Sevincer <abdullah.sevincer@intel.com>
> To: dev@dpdk.org
> Cc: jerinj@marvell.com, mike.ximing.chen@intel.com, Abdullah Sevincer <abdullah.sevincer@intel.com>
> Subject: [PATCH v1] event/dlb2: add support for disabling PASID
> Date: Wed, 7 Jun 2023 16:00:50 -0500
> X-Mailer: git-send-email 2.25.1
>
> vfio-pci driver in Linux kernel 6.2 enables PASID by default.
> In DLB hardware, enabling PASID puts DLB in SIOV mode. This
> breaks DLB PF-PMD mode. For DLB PF-PMD mode to function properly
> PASID needs to be disabled for kernel 6.2.
>
> In this commit this issue is addressed and PASID is disabled
> by writing a zero to PASID control register.
>
> Signed-off-by: Abdullah Sevincer <abdullah.sevincer@intel.com>
It is not clear if this is kernels 6.2 or later, or just that version.
Also, distribution kernel versions don't reflect upstream so if you are complaining about
RHEL etc. The code is version independent (good). It would be good to supply more
commit information (ideally upstream commit id) in this patch commit log.
Also 6.2 kernel is end-of-life (May 2023) so users should be using older or later
supported kernel version.
@@ -46,6 +46,7 @@
#define DLB2_PCI_CAP_ID_MSIX 0x11
#define DLB2_PCI_EXT_CAP_ID_PRI 0x13
#define DLB2_PCI_EXT_CAP_ID_ACS 0xD
+#define DLB2_PCI_EXT_CAP_ID_PASID 0x1B /* Process Address Space ID */
#define DLB2_PCI_PRI_CTRL_ENABLE 0x1
#define DLB2_PCI_PRI_ALLOC_REQ 0xC
@@ -64,6 +65,8 @@
#define DLB2_PCI_ACS_CR 0x8
#define DLB2_PCI_ACS_UF 0x10
#define DLB2_PCI_ACS_EC 0x20
+#define DLB2_PCI_PASID_CTRL 0x06 /* PASID control register */
+#define DLB2_PCI_PASID_CAP_OFFSET 0x148 /* PASID capability offset */
static int dlb2_pci_find_capability(struct rte_pci_device *pdev, uint32_t id)
{
@@ -257,12 +260,14 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
uint16_t rt_ctl_word;
uint32_t pri_reqs_dword;
uint16_t pri_ctrl_word;
+ uint16_t pasid_ctrl;
int pcie_cap_offset;
int pri_cap_offset;
int msix_cap_offset;
int err_cap_offset;
int acs_cap_offset;
+ int pasid_cap_offset;
int wait_count;
uint16_t devsta_busy_word;
@@ -582,6 +587,28 @@ dlb2_pf_reset(struct dlb2_dev *dlb2_dev)
}
}
+ /* The current Linux kernel vfio driver does not expose PASID capability to
+ * users. It also enables PASID by default, which breaks DLB PF PMD. We have
+ * to use the hardcoded offset for now to disable PASID.
+ */
+ pasid_cap_offset = DLB2_PCI_PASID_CAP_OFFSET;
+
+ off = pasid_cap_offset + DLB2_PCI_PASID_CTRL;
+ if (rte_pci_read_config(pdev, &pasid_ctrl, 2, off) != 2)
+ pasid_ctrl = 0;
+
+ if (pasid_ctrl) {
+ DLB2_INFO(dlb2_dev, "DLB2 disabling pasid...\n");
+
+ pasid_ctrl = 0;
+ ret = rte_pci_write_config(pdev, &pasid_ctrl, 2, off);
+ if (ret != 2) {
+ DLB2_LOG_ERR("[%s()] failed to write the pcie config space at offset %d\n",
+ __func__, (int)off);
+ return ret;
+ }
+ }
+
return 0;
}