mbox series

[v2,0/5] mlx5: workaround MR issues

Message ID 20211108172113.2241853-1-matan@nvidia.com (mailing list archive)
Headers
Series mlx5: workaround MR issues |

Message

Matan Azrad Nov. 8, 2021, 5:21 p.m. UTC 
  The mlx5 PMD uses the kernel mlx5 driver to map physical
memory to the HW.

Using the Verbs API ibv_reg_mr, a mkey can be created for that.
In this case, the mkey is signed on the user ID of the kernel driver.

Using the DevX API, a mkey also can be created, but it should point an
umem object (represents the specific buffer mapping).
In this case, the mkey is signed on the user ID of the process DevX 
context.

In FW DevX control commands which get mkey as a parameter, there is
a security check on the user ID and Verbs mkeys are rejected.

Unfortunately, also when using DevX mkey, there is an error in the FW
command on umem validation because the umem is not designed to be used
for any mkey parameters.

As a workaround to the kernel driver/FW issue, it is needed to use a 
wrapped MR, which is an indirect mkey(created by the DevX API) pointing to
direct mkey created by the kernel for any DevX command uses an MR.

Add an API to create and destroy this wrapped MR and use it for any
control DevX command.

v2:
- fix compilation issue on Windows.
- improve logs.

Matan Azrad (2):
  common/mlx5: add wrapped MR create API
  vdpa/mlx5: workaround dirty bitmap MR creation

Michael Baum (3):
  common/mlx5: glue MR registration with IOVA
  vdpa/mlx5: workaround guest MR registrations
  net/mlx5: workaround MR creation for flow counter

 drivers/common/mlx5/linux/meson.build        |  2 +
 drivers/common/mlx5/linux/mlx5_common_os.c   | 56 ++++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_glue.c        | 18 +++++++
 drivers/common/mlx5/linux/mlx5_glue.h        |  3 ++
 drivers/common/mlx5/mlx5_common.h            | 18 +++++++
 drivers/common/mlx5/version.map              |  3 ++
 drivers/common/mlx5/windows/mlx5_common_os.c | 40 ++++++++++++++
 drivers/net/mlx5/mlx5.c                      |  8 +--
 drivers/net/mlx5/mlx5.h                      |  5 +-
 drivers/net/mlx5/mlx5_flow.c                 | 25 +++------
 drivers/vdpa/mlx5/mlx5_vdpa.h                |  9 ++--
 drivers/vdpa/mlx5/mlx5_vdpa_lm.c             | 37 +++----------
 drivers/vdpa/mlx5/mlx5_vdpa_mem.c            | 43 +++++----------
 13 files changed, 174 insertions(+), 93 deletions(-)

Comments

Matan Azrad Nov. 9, 2021, 12:36 p.m. UTC | #1 
The mlx5 PMD uses the kernel mlx5 driver to map physical memory to the
    HW.
    
    Using the Verbs API ibv_reg_mr, a mkey can be created for that.
    In this case, the mkey is signed on the user ID of the kernel driver.
    
    Using the DevX API, a mkey also can be created, but it should point an
    umem object (represents the specific buffer mapping) created by the
    kernel. In this case, the mkey is signed on the user ID of the process
    DevX context.
    
    In FW DevX control commands which get mkey as a parameter, there is
    a security check on the user ID and Verbs mkeys are rejected.
    
    Unfortunately, also when using DevX mkey, there is an error in the FW
    command on umem validation because the umem is not designed to be used
    for any mkey parameters.
    
    As a workaround to the kernel driver/FW issue, it is needed to use a
    wrapped MR, which is an indirect mkey(created by the DevX API) pointing to
    direct mkey created by the kernel for any DevX command uses an MR.
    
    Add an API to create and destroy this wrapped MR.
Use this logic in flow counter query management and in LM.


V3:
Fix issue in 32bit compilation.

V2:
Fix missing implementation for Windows.
Improve logs.

Matan Azrad (2):
  common/mlx5: add wrapped MR create API
  vdpa/mlx5: workaround dirty bitmap MR creation

Michael Baum (3):
  common/mlx5: glue MR registration with IOVA
  vdpa/mlx5: workaround guest MR registrations
  net/mlx5: workaround MR creation for flow counter

 drivers/common/mlx5/linux/meson.build        |  2 +
 drivers/common/mlx5/linux/mlx5_common_os.c   | 56 ++++++++++++++++++++
 drivers/common/mlx5/linux/mlx5_glue.c        | 18 +++++++
 drivers/common/mlx5/linux/mlx5_glue.h        |  3 ++
 drivers/common/mlx5/mlx5_common.h            | 18 +++++++
 drivers/common/mlx5/version.map              |  3 ++
 drivers/common/mlx5/windows/mlx5_common_os.c | 40 ++++++++++++++
 drivers/net/mlx5/mlx5.c                      |  8 +--
 drivers/net/mlx5/mlx5.h                      |  5 +-
 drivers/net/mlx5/mlx5_flow.c                 | 25 +++------
 drivers/vdpa/mlx5/mlx5_vdpa.h                |  9 ++--
 drivers/vdpa/mlx5/mlx5_vdpa_lm.c             | 38 +++----------
 drivers/vdpa/mlx5/mlx5_vdpa_mem.c            | 43 +++++----------
 13 files changed, 175 insertions(+), 93 deletions(-)
Thomas Monjalon Nov. 10, 2021, 2:55 p.m. UTC | #2 
> Matan Azrad (2):
>   common/mlx5: add wrapped MR create API
>   vdpa/mlx5: workaround dirty bitmap MR creation
> 
> Michael Baum (3):
>   common/mlx5: glue MR registration with IOVA
>   vdpa/mlx5: workaround guest MR registrations
>   net/mlx5: workaround MR creation for flow counter

Applied, thanks.