[00/14] net/mlx5: add support for multiport IB devices

  The Mellanox NICs support SR-IOV and have E-Switch feature. 
When SR-IOV is set up in switchdev mode and E-Switch is enabled
we have so called VF representors in the system. All representors
belonging to the same E-Switch are created on the basis of the
single PCI function and with current implementation each representor
has its own dedicated Infiniband device and operates within its
own Infiniband context. It is proposed to provide representors
as ports of the single Infiniband device and operate on the
shared Infiniband context.

mlx5_pci_probe() routine is planned to be extended to probe
the ports of found Infiniband devices. All active ports,
belonging to the same Infiniband device will use the signle
shared Infiniband context of that device:
  - QP are created within shared context, with assigning to
    specified port
  - Verb flows are also created with specifying port index
  - Protection Domain is shared by all representors
  - Global Memory Regions Cache is shared by all representors,
    local ones are not (not in this patchset)
  - Shared Asynchronous Event Handler distributes events
    among the device ports (not in this patchset)
  - UAR mapping to user space is also shared (not in patchset)
		  
Each representor has its own attach network interface device,
interface name and index can be retrieved via IB device and
port index, so the link control is performed via Netlink
in the same way as in existing implementation.

This patchset also superseedes http://patches.dpdk.org/patch/50486/
		  
Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>

Viacheslav Ovsiienko (14):
  net/mlx5: add representor recognition on kernels 5.x
  net/mlx5: introduce multiport IB device shared structure
  net/mlx5: modify get ifindex routine for multiport IB
  net/mlx5: add getting IB ports number for multiport IB
  net/mlx5: add multiport IB device support to probing
  net/mlx5: add IB shared context alloc/free functions
  net/mlx5: switch to the names in the shared IB context
  net/mlx5: switch to the shared Protection Domain
  net/mlx5: switch to the shared context IB attributes
  net/mlx5: switch to the shared IB device context
  net/mlx5: provide IB port for the object being created
  net/mlx5: update install/uninstall int handler routines
  net/mlx5: update event handler for multiport IB devices
  net/mlx5: add source vport match to the ingress rules

 drivers/net/mlx5/Makefile          |  10 +
 drivers/net/mlx5/meson.build       |   4 +
 drivers/net/mlx5/mlx5.c            | 549 +++++++++++++++++++++++++++----------
 drivers/net/mlx5/mlx5.h            |  38 ++-
 drivers/net/mlx5/mlx5_ethdev.c     | 241 ++++++++++------
 drivers/net/mlx5/mlx5_flow.c       |   2 +
 drivers/net/mlx5/mlx5_flow_dv.c    |  47 +++-
 drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
 drivers/net/mlx5/mlx5_mr.c         |   4 +-
 drivers/net/mlx5/mlx5_nl.c         | 101 ++++++-
 drivers/net/mlx5/mlx5_rxq.c        |  41 +--
 drivers/net/mlx5/mlx5_socket.c     |   4 +-
 drivers/net/mlx5/mlx5_stats.c      |  22 +-
 drivers/net/mlx5/mlx5_trigger.c    |   2 +-
 drivers/net/mlx5/mlx5_txq.c        |  24 +-
 15 files changed, 795 insertions(+), 301 deletions(-)
  

Thursday, March 21, 2019 10:11 AM, Viacheslav Ovsiienko:
> Subject: [PATCH 00/14] net/mlx5: add support for multiport IB devices
> 
> The Mellanox NICs support SR-IOV and have E-Switch feature.
> When SR-IOV is set up in switchdev mode and E-Switch is enabled we have
> so called VF representors in the system. All representors belonging to the
> same E-Switch are created on the basis of the single PCI function and with
> current implementation each representor has its own dedicated Infiniband
> device and operates within its own Infiniband context. It is proposed to
> provide representors as ports of the single Infiniband device and operate on
> the shared Infiniband context.
> 
> mlx5_pci_probe() routine is planned to be extended to probe the ports of
> found Infiniband devices. All active ports, belonging to the same Infiniband
> device will use the signle shared Infiniband context of that device:
>   - QP are created within shared context, with assigning to
>     specified port
>   - Verb flows are also created with specifying port index
>   - Protection Domain is shared by all representors
>   - Global Memory Regions Cache is shared by all representors,
>     local ones are not (not in this patchset)
>   - Shared Asynchronous Event Handler distributes events
>     among the device ports (not in this patchset)
>   - UAR mapping to user space is also shared (not in patchset)
> 
> Each representor has its own attach network interface device, interface
> name and index can be retrieved via IB device and port index, so the link
> control is performed via Netlink in the same way as in existing
> implementation.

Nice Work Slava,
Please see my comments. 

Also - where is the support for sharing of the memory regions? 

> 
> This patchset also superseedes http://patches.dpdk.org/patch/50486/
> 
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> 
> Viacheslav Ovsiienko (14):
>   net/mlx5: add representor recognition on kernels 5.x
>   net/mlx5: introduce multiport IB device shared structure
>   net/mlx5: modify get ifindex routine for multiport IB
>   net/mlx5: add getting IB ports number for multiport IB
>   net/mlx5: add multiport IB device support to probing
>   net/mlx5: add IB shared context alloc/free functions
>   net/mlx5: switch to the names in the shared IB context
>   net/mlx5: switch to the shared Protection Domain
>   net/mlx5: switch to the shared context IB attributes
>   net/mlx5: switch to the shared IB device context
>   net/mlx5: provide IB port for the object being created
>   net/mlx5: update install/uninstall int handler routines
>   net/mlx5: update event handler for multiport IB devices
>   net/mlx5: add source vport match to the ingress rules
> 
>  drivers/net/mlx5/Makefile          |  10 +
>  drivers/net/mlx5/meson.build       |   4 +
>  drivers/net/mlx5/mlx5.c            | 549 +++++++++++++++++++++++++++------
> ----
>  drivers/net/mlx5/mlx5.h            |  38 ++-
>  drivers/net/mlx5/mlx5_ethdev.c     | 241 ++++++++++------
>  drivers/net/mlx5/mlx5_flow.c       |   2 +
>  drivers/net/mlx5/mlx5_flow_dv.c    |  47 +++-
>  drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
>  drivers/net/mlx5/mlx5_mr.c         |   4 +-
>  drivers/net/mlx5/mlx5_nl.c         | 101 ++++++-
>  drivers/net/mlx5/mlx5_rxq.c        |  41 +--
>  drivers/net/mlx5/mlx5_socket.c     |   4 +-
>  drivers/net/mlx5/mlx5_stats.c      |  22 +-
>  drivers/net/mlx5/mlx5_trigger.c    |   2 +-
>  drivers/net/mlx5/mlx5_txq.c        |  24 +-
>  15 files changed, 795 insertions(+), 301 deletions(-)
> 
> --
> 1.8.3.1
  

Hi, Shahaf,

My special thanks for immediate review. PSB.

With best regards,
Slava

> -----Original Message-----
> From: Shahaf Shuler
> Sent: Thursday, March 21, 2019 14:14
> To: Slava Ovsiienko <viacheslavo@mellanox.com>; dev@dpdk.org
> Subject: RE: [PATCH 00/14] net/mlx5: add support for multiport IB devices
> 
> Thursday, March 21, 2019 10:11 AM, Viacheslav Ovsiienko:
> > Subject: [PATCH 00/14] net/mlx5: add support for multiport IB devices
> >
> > The Mellanox NICs support SR-IOV and have E-Switch feature.
> > When SR-IOV is set up in switchdev mode and E-Switch is enabled we
> > have so called VF representors in the system. All representors
> > belonging to the same E-Switch are created on the basis of the single
> > PCI function and with current implementation each representor has its
> > own dedicated Infiniband device and operates within its own Infiniband
> > context. It is proposed to provide representors as ports of the single
> > Infiniband device and operate on the shared Infiniband context.
> >
> > mlx5_pci_probe() routine is planned to be extended to probe the ports
> > of found Infiniband devices. All active ports, belonging to the same
> > Infiniband device will use the signle shared Infiniband context of that
> device:
> >   - QP are created within shared context, with assigning to
> >     specified port
> >   - Verb flows are also created with specifying port index
> >   - Protection Domain is shared by all representors
> >   - Global Memory Regions Cache is shared by all representors,
> >     local ones are not (not in this patchset)
> >   - Shared Asynchronous Event Handler distributes events
> >     among the device ports (not in this patchset)
> >   - UAR mapping to user space is also shared (not in patchset)
> >
> > Each representor has its own attach network interface device,
> > interface name and index can be retrieved via IB device and port
> > index, so the link control is performed via Netlink in the same way as
> > in existing implementation.
> 
> Nice Work Slava,
> Please see my comments.
> 
> Also - where is the support for sharing of the memory regions?

It is coming soon, as separate patch/patchset.
Ditto regarding UAR sharing.

> 
> >
> > This patchset also superseedes http://patches.dpdk.org/patch/50486/
> >
> > Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> >
> > Viacheslav Ovsiienko (14):
> >   net/mlx5: add representor recognition on kernels 5.x
> >   net/mlx5: introduce multiport IB device shared structure
> >   net/mlx5: modify get ifindex routine for multiport IB
> >   net/mlx5: add getting IB ports number for multiport IB
> >   net/mlx5: add multiport IB device support to probing
> >   net/mlx5: add IB shared context alloc/free functions
> >   net/mlx5: switch to the names in the shared IB context
> >   net/mlx5: switch to the shared Protection Domain
> >   net/mlx5: switch to the shared context IB attributes
> >   net/mlx5: switch to the shared IB device context
> >   net/mlx5: provide IB port for the object being created
> >   net/mlx5: update install/uninstall int handler routines
> >   net/mlx5: update event handler for multiport IB devices
> >   net/mlx5: add source vport match to the ingress rules
> >
> >  drivers/net/mlx5/Makefile          |  10 +
> >  drivers/net/mlx5/meson.build       |   4 +
> >  drivers/net/mlx5/mlx5.c            | 549 +++++++++++++++++++++++++++------
> > ----
> >  drivers/net/mlx5/mlx5.h            |  38 ++-
> >  drivers/net/mlx5/mlx5_ethdev.c     | 241 ++++++++++------
> >  drivers/net/mlx5/mlx5_flow.c       |   2 +
> >  drivers/net/mlx5/mlx5_flow_dv.c    |  47 +++-
> >  drivers/net/mlx5/mlx5_flow_verbs.c |   7 +-
> >  drivers/net/mlx5/mlx5_mr.c         |   4 +-
> >  drivers/net/mlx5/mlx5_nl.c         | 101 ++++++-
> >  drivers/net/mlx5/mlx5_rxq.c        |  41 +--
> >  drivers/net/mlx5/mlx5_socket.c     |   4 +-
> >  drivers/net/mlx5/mlx5_stats.c      |  22 +-
> >  drivers/net/mlx5/mlx5_trigger.c    |   2 +-
> >  drivers/net/mlx5/mlx5_txq.c        |  24 +-
> >  15 files changed, 795 insertions(+), 301 deletions(-)
> >
> > --
> > 1.8.3.1