[v3] doc: add known issue about legacy intr mode for ixgbe

  When using uio_pci_generic module or using legacy interrupt mode of
igb_uio or vfio, X550 cannot get interrupts. Because the Interrupt
Status bit is not implemented, then the irq cannot be handled correctly
and cannot report the event fd to DPDK apps.

Add this hw limitation and details into ixgbe known issue.

Signed-off-by: Xiaoyun Li <xiaoyun.li@intel.com>
---
v3:
 * Correct a misspelling.
v2:
 * Correct the link of X550 spec update.
 * Polish the known issue title.
---
 doc/guides/nics/ixgbe.rst | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
  

> -----Original Message-----
> From: Li, Xiaoyun
> Sent: Sunday, September 30, 2018 10:53 AM
> To: Xing, Beilei <beilei.xing@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>;
> dev@dpdk.org; Lu, Wenzhuo <wenzhuo.lu@intel.com>; Wu, Jingjing
> <jingjing.wu@intel.com>
> Cc: Li, Xiaoyun <xiaoyun.li@intel.com>
> Subject: [PATCH v3] doc: add known issue about legacy intr mode for ixgbe
> 
> When using uio_pci_generic module or using legacy interrupt mode of igb_uio
> or vfio, X550 cannot get interrupts. Because the Interrupt Status bit is not
> implemented, then the irq cannot be handled correctly and cannot report the
> event fd to DPDK apps.
> 
> Add this hw limitation and details into ixgbe known issue.
> 
> Signed-off-by: Xiaoyun Li <xiaoyun.li@intel.com>
> ---
> v3:
>  * Correct a misspelling.
> v2:
>  * Correct the link of X550 spec update.
>  * Polish the known issue title.
> ---
>  doc/guides/nics/ixgbe.rst | 26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
> 
> diff --git a/doc/guides/nics/ixgbe.rst b/doc/guides/nics/ixgbe.rst index
> 16d6390..af4c876 100644
> --- a/doc/guides/nics/ixgbe.rst
> +++ b/doc/guides/nics/ixgbe.rst
> @@ -200,6 +200,32 @@ There is no RTE API to add a VF's MAC address from
> the PF. On ixgbe, the  ``rte_eth_dev_mac_addr_add()`` function can be used
> to add a VF's MAC address,  as a workaround.
> 
> +X550 does not support legacy interrupt mode
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Desccription
> +^^^^^^^^^^^^
> +X550 cannot get interrupts if using ``uio_pci_generic`` module or using
> +legacy interrupt mode of ``igb_uio`` or ``vfio``. Because the errata of
> +X550 states that the Interrupt Status bit is not implemented. The
> +errata is the item #22 from `X550 spec update
> +<https://www.intel.com/content/dam/www/public/us/en/
> +documents/specification-updates/ethernet-x550-spec-update.pdf>`_
> +
> +Implication
> +^^^^^^^^^^^
> +When using ``uio_pci_generic`` module or using legacy interrupt mode of
> +``igb_uio`` or ``vfio``, the Interrupt Status bit would be checked if
> +the interrupt is coming. Since the bit is not implemented in X550, the
> +irq cannot be handled correctly and cannot report the event fd to DPDK
> +apps. Then apps cannot get interrupts and ``dmesg`` will show messages
> +like ``irq #No.: `` ``nobody cared.``
> +
> +Workaround
> +^^^^^^^^^^
> +Do not bind the ``uio_pci_generic`` module in X550 NICs.
> +Do not bind ``igb_uio`` with legacy mode in X550 NICs.
> +Before using ``vfio`` with legacy mode in X550 NICs, using ``modprobe
> +vfio `` ``nointxmask=1`` to bind ``vfio``.

I have couple questions here.
If noinitxmask=1 is set, does that mean, in vfio interrupt handler we will not check intr mask? but what if the intx is shared by another devices? 
Does that mean we will also handle interrupt from other devices which is not expected?

Not sure if we should add some statement like "if the intx is not shared with other device ...." ?

> 
>  Inline crypto processing support
>  --------------------------------
> --
> 2.7.4
  

Hi

> > +Do not bind the ``uio_pci_generic`` module in X550 NICs.
> > +Do not bind ``igb_uio`` with legacy mode in X550 NICs.
> > +Before using ``vfio`` with legacy mode in X550 NICs, using ``modprobe
> > +vfio `` ``nointxmask=1`` to bind ``vfio``.
> 
> I have couple questions here.
> If noinitxmask=1 is set, does that mean, in vfio interrupt handler we will not
> check intr mask? but what if the intx is shared by another devices?
> Does that mean we will also handle interrupt from other devices which is not
> expected?

If noinitxmask=1 is not set, vfio will check if it has broken intx msking (which is hardcoded. Only i40e is considered as broken).
If it is broken, vfio_intx_handler will disable_irq_nosync and mask this irq and then return the efd to userspace.
But if it is not broken, vfio_intx_handler will call pci_check_and_set_intx_mask. In pci_check_and_set_intx_mask, the interrupt status bit is checked.
And for X550, vfio thinks it's intx masking is not broken and will check its interrupt status bit. But this bit is not enabled in X550 in fact. Then vfio will think the interrupt isn't triggered and will not deal with it. Then "nobody cared" issue occurs.

But when with nointxmask=1, vfio will think the device has broken intx masking. Then will disable_irq_nosync and mask this irq and then return the efd to userspace.

Since it will disable_irq_nosync, the irq will be affected if it is shared by another device.
> 
> Not sure if we should add some statement like "if the intx is not shared with
> other device ...." ?
Yes. I will add it.

> 
> >
> >  Inline crypto processing support
> >  --------------------------------
> > --
> > 2.7.4