From patchwork Wed Feb 23 13:48:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 108178 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1814EA034C; Wed, 23 Feb 2022 14:49:12 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B674642711; Wed, 23 Feb 2022 14:48:56 +0100 (CET) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2053.outbound.protection.outlook.com [40.107.94.53]) by mails.dpdk.org (Postfix) with ESMTP id 44E9B4270A for ; Wed, 23 Feb 2022 14:48:55 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gMsifUZ3LvjwRPgZc3oZNvfXgPIzj5EoUOC/tEKi82lq1eZSlZugQ2ityZsqCWxjFNI4LTu3UIQ4WiAb2SdAJ6rT8q+eu0jIRXZUXrZFk66O1cEVmfjU66YOBGgO1TkJKWgfb0l9uaJwE2vyokUpmRSQr7OY9rR5Ys2ctumSXvNPiB+JFJKofs0HGMJa8Kh3zzv+3JMFBkDXDG8vJRzxfskBaE5iKCWgdOk3WEkvThJgSLF9I1U+CYzsXGSad9SSTNKzjbUR1pGB7rvV+VHtel1BkHmfSX2xgECp8agmDq6b3aHqe6zvJsFbaVMc1/r44vTzNS/H5MiOL6DO4dEO6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=83mHF/fl13UTjlc27SWi64OGAahnHXqN2VLPQaVNw8Y=; b=G+NDDFX6nEcZ8lO5cWFgZcCU3+vL0VykeZW0GgkNF8sRkNiiUyve26XS3bTtvn77by4VTgUnLMz5JZ6Tlzm/hFSzems+bQLOkkPRq7ZuzG8iaVgotDYdBH0jHRPpLk1dekabJ/eezj5HkAJgGAmWIX1bOaFZtX386wm1dOi9QmqcCsox4io/BMNw7gqSASD2a7UuALrZuxuaZOEPg7tQeNzlsmP9m5eBoWT3fk6kdlAbT7HMR/lbfmxmaPNbAPZ8Eb71+QEv8WdjyIl5ARkm8g1c06yX3xKnDlbDPErb+Gfqzr+3/qO7OsBpqjM/iMrRseQoyW0NkZ36JAmfTRQ8xQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=83mHF/fl13UTjlc27SWi64OGAahnHXqN2VLPQaVNw8Y=; b=jyvauJGeyYcS/uExzbC3p/Si1mYl3nEF0lahC9z4ChOuOloLKYYcaOGFf8xRlARjKTdBW9jS71c1gZrBjN47JShgOm1slF16DjVXj+VojCtOEzrmx2c3UY/kE7xylM40nzncYIkBiUfWvjQW0mfN+u9KgwiRAtb29+/BmTuefgJUT4hIt/nrCFIybx1X239dvQGMOmMvqaTLTXdGHeCYZc+C68OFgoS6kAZIKOt/Zz0Pfw97ggHtDI0zERb1Tobru7dRSX9Yq7laVmzpLEwY/+tL2dWruVNaTpzyrjhO32Y5/sh1bUvLx8wBizaQsdJ6Slx2K+s5F0cImPpK/J6Mrg== Received: from CO2PR18CA0055.namprd18.prod.outlook.com (2603:10b6:104:2::23) by BN9PR12MB5113.namprd12.prod.outlook.com (2603:10b6:408:136::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.22; Wed, 23 Feb 2022 13:48:52 +0000 Received: from CO1NAM11FT037.eop-nam11.prod.protection.outlook.com (2603:10b6:104:2:cafe::a5) by CO2PR18CA0055.outlook.office365.com (2603:10b6:104:2::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.11 via Frontend Transport; Wed, 23 Feb 2022 13:48:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.234) by CO1NAM11FT037.mail.protection.outlook.com (10.13.174.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Wed, 23 Feb 2022 13:48:51 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 23 Feb 2022 13:48:50 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Wed, 23 Feb 2022 05:48:48 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9 via Frontend Transport; Wed, 23 Feb 2022 05:48:47 -0800 From: Michael Baum To: CC: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Subject: [PATCH v2 4/5] doc: add shared guide for mlx5 PMDs Date: Wed, 23 Feb 2022 15:48:33 +0200 Message-ID: <20220223134834.2840916-5-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220223134834.2840916-1-michaelba@nvidia.com> References: <20220222124815.2587851-1-michaelba@nvidia.com> <20220223134834.2840916-1-michaelba@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2fa781ad-8080-4947-b294-08d9f6d33994 X-MS-TrafficTypeDiagnostic: BN9PR12MB5113:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vZpmEd2EA1PuP40LiesrEaEBxLgCX0Y7R+A9lJtEoweayCIuY1m5acMBu4xVAQL/+zhG97yx7cvfhLWE+koVUEGEu0w+O4mu0NQJIz6VeIQlMv4BtCi25U0yYFfufG1MMxhiRi3hZIiYshlWwMbjkONOmBz30MmfmdGUlKB7BaejuTe/sUbhjoYbRCPkHojtC9GP6w4t4tf6JDiY7aTzIFlWWfw0Pj1+WhwZdN9uzt4Ob1NwVFm5SlJpynXeOSH3qUarUjRJ/erD7M7xI9ZtlUpgsMHnkmWyDWpUoqQrNLnv93J06od6jLJGfvc/D2slhhjuIxyRmGI5Voo2CUwNfuzt7nVy+ialKfrFcEOQC+cnnoWKh5m7/1ZWpdkAxo9HqfGp99KFsYNimmHEVe2A5J70WiunCHEHPGd7dPlFprP03VTAB2xcfbvcCG6jzlNPdLSf46myl5bcexPX/aQ4jFwMIaVyXYygSmUKETESxz78CjrTqhd4cw4gWk/zvbm6m3L4DGFTzdxYLOjS3fuNEN7EMBTkrzTAI56Z8NsuN7MWy6GAj7iRrd4qyAnK/SSdySXTiYbhGde7WoaTIp3G5CPiVh/pLc7Wv8gdhRi0TjvPP27NqaRlV+DPrwv4Wmx4KNWxrYmdWm/h+DAfWVElguJWolE+v0iffKwfc6Spwzlo8qbRN+6pH7H4f69Elnw2FEcYO2L9o6y4sq9B7TLs5bvA3bCRt7uFJDp7Pma5C+DQq3+QRaa9U1DEz/s+zyxeHFltFAZ9WleOCtK3lfs1fxE2++Gj7EMtN15KzL3rJdf9joDsnNhn2kCqX+qJBW1Sti9f7damUKrTf61iWaaT0g== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(40470700004)(36840700001)(2906002)(30864003)(82310400004)(8936002)(316002)(508600001)(86362001)(6916009)(7696005)(19627235002)(47076005)(40460700003)(5660300002)(54906003)(966005)(6666004)(36860700001)(83380400001)(426003)(356005)(336012)(8676002)(81166007)(4326008)(55016003)(70586007)(70206006)(36756003)(6286002)(186003)(26005)(1076003)(2616005)(107886003)(36900700001)(559001)(579004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2022 13:48:51.2964 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2fa781ad-8080-4947-b294-08d9f6d33994 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT037.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5113 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Adds new documentation for MLX5 common driver that contains: - Its features list (doesn't exist for now). - Its devargs description. - Device configuration information and tutorial. - Quick Start Guide for Mellanox OFED/EN. Move into this doc all shared information from other MLX5 PMD docs and add them reference to new common doc. Signed-off-by: Michael Baum Reviewed-by: Raslan Darawsheh Acked-by: Viacheslav Ovsiienko --- doc/guides/compressdevs/mlx5.rst | 34 +- doc/guides/cryptodevs/mlx5.rst | 32 +- doc/guides/nics/mlx5.rst | 522 ++------------------------- doc/guides/platform/index.rst | 1 + doc/guides/platform/mlx5.rst | 602 +++++++++++++++++++++++++++++++ doc/guides/regexdevs/mlx5.rst | 36 +- doc/guides/vdpadevs/mlx5.rst | 87 +---- 7 files changed, 662 insertions(+), 652 deletions(-) create mode 100644 doc/guides/platform/mlx5.rst diff --git a/doc/guides/compressdevs/mlx5.rst b/doc/guides/compressdevs/mlx5.rst index 7f2d6bdfff..273251e088 100644 --- a/doc/guides/compressdevs/mlx5.rst +++ b/doc/guides/compressdevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: -MLX5 compress driver +MLX5 Compress Driver ==================== -The MLX5 compress driver library +The mlx5 compress driver library (**librte_compress_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. @@ -25,30 +25,7 @@ So, using the BlueField device (starting from BlueField-2), the compress class operations can be supported in parallel to the net, vDPA and RegEx class operations. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API. -- Direct Verbs is a device-specific API. -- DevX allows to access firmware objects. - -Enabling librte_compress_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 PCI device can be probed by number of different PCI devices, -for example net / vDPA / RegEx. To select the compress PMD ``class=compress`` -should be specified as device parameter. The compress device can be probed and -used with other Mellanox classes, by adding more options in the class. -For example: ``class=net:compress`` will probe both the net PMD and the compress -PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- @@ -85,6 +62,9 @@ Limitations Driver options -------------- +Please refer to :ref:`mlx5 common options ` +for an additional list of options shared with other mlx5 drivers. + - ``log-block-size`` parameter [int] Log of the Huffman block size in the Deflate algorithm. @@ -101,4 +81,4 @@ Prerequisites ------------- - Mellanox OFED version: **5.2** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. + See :ref:`mlx5 common prerequisites ` for more details. diff --git a/doc/guides/cryptodevs/mlx5.rst b/doc/guides/cryptodevs/mlx5.rst index 9936556cc9..ef47aa65dd 100644 --- a/doc/guides/cryptodevs/mlx5.rst +++ b/doc/guides/cryptodevs/mlx5.rst @@ -28,23 +28,12 @@ when the MKEY is configured to perform crypto operations. The encryption does not require text to be aligned to the AES block size (128b). -For security reasons and to increase robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow handling virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). +See :doc:`../../platform/mlx5` guide for more design details. -The PMD uses ``libibverbs`` and ``libmlx5`` to access the device firmware -or to access the hardware components directly. -There are different levels of objects and bypassing abilities. -To get the best performances: - -- Verbs is a complete high-level generic API (Linux only). -- Direct Verbs is a device-specific API (Linux only). -- DevX allows to access firmware objects. +Configuration +------------- -Enabling ``librte_crypto_mlx5`` causes DPDK applications -to be linked against libibverbs on Linux OS. +See the :ref:`mlx5 common configuration `. In order to move the device to crypto operational mode, credential and KEK (Key Encrypting Key) should be set as the first step. @@ -109,10 +98,8 @@ The mlxreg dedicated tool should be used as follows: Driver options -------------- -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `crypto` for the mlx5 crypto driver. +Please refer to :ref:`mlx5 common options ` +for an additional list of options shared with other mlx5 drivers. - ``wcs_file`` parameter [string] - mandatory @@ -168,13 +155,12 @@ Linux Prerequisites ~~~~~~~~~~~~~~~~~~~ - Mellanox OFED version: **5.3**. - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - - Compilation can be done also with rdma-core v15+. - see :doc:`../../nics/mlx5` guide for more rdma-core details. + + See :ref:`mlx5 common prerequisites ` for more details. Windows Prerequisites ~~~~~~~~~~~~~~~~~~~~~ - Mellanox WINOF-2 version: **2.60** or higher. - see :doc:`../../nics/mlx5` guide for more Mellanox WINOF-2 details. + See :ref:`mlx5 common prerequisites ` for more details. diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 9e10a68d6f..f94ed90ef0 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -4,23 +4,16 @@ .. include:: -MLX5 poll mode driver -===================== +MLX5 Ethernet Poll Mode Driver +============================== -The MLX5 poll mode driver library (**librte_net_mlx5**) provides support +The mlx5 Ethernet poll mode driver library (**librte_net_mlx5**) provides support for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox ConnectX-5**, **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox ConnectX-6 Lx**, **Mellanox BlueField** and **Mellanox BlueField-2** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in SR-IOV context. -Information and documentation about these adapters can be found on the -`Mellanox website `__. Help is also provided by the -`Mellanox community `__. - -There is also a `section dedicated to this poll mode driver -`_. - Design ------ @@ -29,12 +22,6 @@ Besides its dependency on libibverbs (that implies libmlx5 and associated kernel support), librte_net_mlx5 relies heavily on system calls for control operations such as querying/updating the MTU and flow control parameters. -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - This capability allows the PMD to coexist with kernel network interfaces which remain functional, although they stop receiving unicast packets as long as they share the same MAC address. @@ -42,18 +29,7 @@ This means legacy linux control tools (for example: ethtool, ifconfig and more) can operate on the same network interfaces that owned by the DPDK application. -The PMD can use libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects -- Direct Rules manages flow steering at low-level hardware layer - -Enabling librte_net_mlx5 causes DPDK applications to be linked against -libibverbs. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- @@ -522,75 +498,31 @@ Extended statistics can be queried using ``rte_eth_xstats_get()``. The extended Finally per-flow statistics can by queried using ``rte_flow_query`` when attaching a count action for specific flow. The flow counter counts the number of packets received successfully by the port and match the specific flow. -Configuration -------------- - -Compilation options -~~~~~~~~~~~~~~~~~~~ - -The ibverbs libraries can be linked with this PMD in a number of ways, -configured by the ``ibverbs_link`` build option: - -- ``shared`` (default): the PMD depends on some .so files. - -- ``dlopen``: Split the dependencies glue in a separate library - loaded when needed by dlopen. - It make dependencies on libibverbs and libmlx4 optional, - and has no performance impact. - -- ``static``: Embed static flavor of the dependencies libibverbs and libmlx4 - in the PMD shared library or the executable static binary. -Environment variables -~~~~~~~~~~~~~~~~~~~~~ +Compilation +----------- -- ``MLX5_GLUE_PATH`` +See :ref:`mlx5 common compilation `. - A list of directories in which to search for the rdma-core "glue" plug-in, - separated by colons or semi-colons. -- ``MLX5_SHUT_UP_BF`` +Configuration +------------- - Configures HW Tx doorbell register as IO-mapped. +Environment Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~ - By default, the HW Tx doorbell is configured as a write-combining register. - The register would be flushed to HW usually when the write-combining buffer - becomes full, but it depends on CPU design. +See :ref:`mlx5 common configuration `. -Run-time configuration +Firmware configuration ~~~~~~~~~~~~~~~~~~~~~~ -- librte_net_mlx5 brings kernel network interfaces up during initialization - because it is affected by their state. Forcing them down prevents packets - reception. - -- **ethtool** operations on related kernel interfaces also affect the PMD. - -Run as non-root -^^^^^^^^^^^^^^^ - -In order to run as a non-root user, -some capabilities must be granted to the application:: - - setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep - -Below are the reasons of the need for each capability: - -``cap_sys_admin`` - When using physical addresses (PA mode), with Linux >= 4.0, - for access to ``/proc/self/pagemap``. - -``cap_net_admin`` - For device configuration. - -``cap_net_raw`` - For raw ethernet queue allocation through kernel driver. - -``cap_ipc_lock`` - For DMA memory pinning. +See :ref:`mlx5_firmware_config` guide. Driver options -^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~ + +Please refer to :ref:`mlx5 common options ` +for an additional list of options shared with other mlx5 drivers. - ``rxq_cqe_comp_en`` parameter [int] @@ -1059,30 +991,6 @@ Driver options Disabled by default (set to 0). -- ``mr_ext_memseg_en`` parameter [int] - - A nonzero value enables extending memseg when registering DMA memory. If - enabled, the number of entries in MR (Memory Region) lookup table on datapath - is minimized and it benefits performance. On the other hand, it worsens memory - utilization because registered memory is pinned by kernel driver. Even if a - page in the extended chunk is freed, that doesn't become reusable until the - entire memory is freed. - - Enabled by default. - -- ``mr_mempool_reg_en`` parameter [int] - - A nonzero value enables implicit registration of DMA memory of all mempools - except those having ``RTE_MEMPOOL_F_NON_IO``. This flag is set automatically - for mempools populated with non-contiguous objects or those without IOVA. - The effect is that when a packet from a mempool is transmitted, - its memory is already registered for DMA in the PMD and no registration - will happen on the data path. The tradeoff is extra work on the creation - of each mempool and increased HW resource use if some mempools - are not used with MLX5 devices. - - Enabled by default. - - ``representor`` parameter [list] This parameter can be used to instantiate DPDK Ethernet devices from @@ -1153,13 +1061,6 @@ Driver options By default, the PMD will set this value to 0. -- ``sys_mem_en`` parameter [int] - - A non-zero value enables the PMD memory management allocating memory - from system by default, without explicit rte memory flag. - - By default, the PMD will set this value to 0. - - ``decap_en`` parameter [int] Some devices do not support FCS (frame checksum) scattering for @@ -1183,253 +1084,6 @@ Driver options By default, the PMD will set this value to 1. -.. _mlx5_firmware_config: - -Firmware configuration -~~~~~~~~~~~~~~~~~~~~~~ - -Firmware features can be configured as key/value pairs. - -The command to set a value is:: - - mlxconfig -d set = - -The command to query a value is:: - - mlxconfig -d query | grep - -The device name for the command ``mlxconfig`` can be either the PCI address, -or the mst device name found with:: - - mst status - -Below are some firmware configurations listed. - -- link type:: - - LINK_TYPE_P1 - LINK_TYPE_P2 - value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense) - -- enable SR-IOV:: - - SRIOV_EN=1 - -- maximum number of SR-IOV virtual functions:: - - NUM_OF_VFS= - -- enable DevX (required by Direct Rules and other features):: - - UCTX_EN=1 - -- aggressive CQE zipping:: - - CQE_COMPRESSION=1 - -- L3 VXLAN and VXLAN-GPE destination UDP port:: - - IP_OVER_VXLAN_EN=1 - IP_OVER_VXLAN_PORT= - -- enable VXLAN-GPE tunnel flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - or - FLEX_PARSER_PROFILE_ENABLE=2 - -- enable IP-in-IP tunnel flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - -- enable MPLS flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=1 - -- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching:: - - FLEX_PARSER_PROFILE_ENABLE=2 - -- enable Geneve flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - or - FLEX_PARSER_PROFILE_ENABLE=1 - -- enable Geneve TLV option flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=0 - -- enable GTP flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=3 - -- enable eCPRI flow matching:: - - FLEX_PARSER_PROFILE_ENABLE=4 - PROG_PARSE_GRAPH=1 - -- enable dynamic flex parser for flex item:: - - FLEX_PARSER_PROFILE_ENABLE=4 - PROG_PARSE_GRAPH=1 - -- enable realtime timestamp format:: - - REAL_TIME_CLOCK_ENABLE=1 - -Linux Prerequisites -------------------- - -This driver relies on external libraries and kernel drivers for resources -allocations and initialization. The following dependencies are not part of -DPDK and must be installed separately: - -- **libibverbs** - - User space Verbs framework used by librte_net_mlx5. This library provides - a generic interface between the kernel and low-level user space drivers - such as libmlx5. - - It allows slow and privileged operations (context initialization, hardware - resources allocations) to be managed by the kernel and fast operations to - never leave user space. - -- **libmlx5** - - Low-level user space driver library for Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices, it is automatically loaded - by libibverbs. - - This library basically implements send/receive calls to the hardware - queues. - -- **Kernel modules** - - They provide the kernel-side Verbs API and low level device drivers that - manage actual hardware initialization and resources sharing with user - space processes. - - Unlike most other PMDs, these modules must remain loaded and bound to - their devices: - - - mlx5_core: hardware driver managing Mellanox - ConnectX-4/ConnectX-5/ConnectX-6/BlueField devices and related Ethernet kernel - network devices. - - mlx5_ib: InfiniBand device driver. - - ib_uverbs: user space driver for Verbs (entry point for libibverbs). - -- **Firmware update** - - Mellanox OFED/EN releases include firmware updates for - ConnectX-4/ConnectX-5/ConnectX-6/BlueField adapters. - - Because each release provides new features, these updates must be applied to - match the kernel modules and libraries they come with. - -.. note:: - - Both libraries are BSD and GPL licensed. Linux kernel modules are GPL - licensed. - -Installation -~~~~~~~~~~~~ - -Either RDMA Core library with a recent enough Linux kernel release -(recommended) or Mellanox OFED/EN, which provides compatibility with older -releases. - -RDMA Core with Linux Kernel -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- Minimal kernel version : v4.14 or the most recent 4.14-rc (see `Linux installation documentation`_) -- Minimal rdma-core version: v15+ commit 0c5f5765213a ("Merge pull request #227 from yishaih/tm") - (see `RDMA Core installation documentation`_) -- When building for i686 use: - - - rdma-core version 18.0 or above built with 32bit support. - - Kernel version 4.14.41 or above. - -- Starting with rdma-core v21, static libraries can be built:: - - cd build - CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. - ninja - -.. _`Linux installation documentation`: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/plain/Documentation/admin-guide/README.rst -.. _`RDMA Core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md - - -Mellanox OFED/EN -^^^^^^^^^^^^^^^^ - -- Mellanox OFED version: **4.5** and above / - Mellanox EN version: **4.5** and above -- firmware version: - - - ConnectX-4: **12.21.1000** and above. - - ConnectX-4 Lx: **14.21.1000** and above. - - ConnectX-5: **16.21.1000** and above. - - ConnectX-5 Ex: **16.21.1000** and above. - - ConnectX-6: **20.27.0090** and above. - - ConnectX-6 Dx: **22.27.0090** and above. - - BlueField: **18.25.1010** and above. - -While these libraries and kernel modules are available on OpenFabrics -Alliance's `website `__ and provided by package -managers on most distributions, this PMD requires Ethernet extensions that -may not be supported at the moment (this is a work in progress). - -`Mellanox OFED -`__ and -`Mellanox EN -`__ -include the necessary support and should be used in the meantime. For DPDK, -only libibverbs, libmlx5, mlnx-ofed-kernel packages and firmware updates are -required from that distribution. - -.. note:: - - Several versions of Mellanox OFED/EN are available. Installing the version - this DPDK release was developed and tested against is strongly - recommended. Please check the `linux prerequisites`_. - -Windows Prerequisites ---------------------- - -This driver relies on external libraries and kernel drivers for resources -allocations and initialization. The dependencies in the following sub-sections -are not part of DPDK, and must be installed separately. - -Compilation Prerequisites -~~~~~~~~~~~~~~~~~~~~~~~~~ - -DevX SDK installation -^^^^^^^^^^^^^^^^^^^^^ - -The DevX SDK must be installed on the machine building the Windows PMD. -Additional information can be found at -`How to Integrate Windows DevX in Your Development Environment -`__. - -Runtime Prerequisites -~~~~~~~~~~~~~~~~~~~~~ - -WinOF2 version 2.60 or higher must be installed on the machine. - -WinOF2 installation -^^^^^^^^^^^^^^^^^^^ - -The driver can be downloaded from the following site: -`WINOF2 -`__ - -DevX Enablement -^^^^^^^^^^^^^^^ - -DevX for Windows must be enabled in the Windows registry. -The keys ``DevxEnabled`` and ``DevxFsRules`` must be set. -Additional information can be found in the WinOF2 user manual. Supported NICs -------------- @@ -1475,149 +1129,21 @@ Below are detailed device names: * Mellanox\ |reg| ConnectX\ |reg|-6 Dx EN 200G MCX623105AN-VDAT (1x200G) * Mellanox\ |reg| ConnectX\ |reg|-6 Lx EN 25G MCX631102AN-ADAT (2x25G) -Quick Start Guide on OFED/EN ----------------------------- - -1. Download latest Mellanox OFED/EN. For more info check the `linux prerequisites`_. - - -2. Install the required libraries and kernel modules either by installing - only the required set, or by installing the entire Mellanox OFED/EN:: - - ./mlnxofedinstall --upstream-libs --dpdk - -3. Verify the firmware is the correct one:: - - ibv_devinfo - -4. Verify all ports links are set to Ethernet:: - - mlxconfig -d query | grep LINK_TYPE - LINK_TYPE_P1 ETH(2) - LINK_TYPE_P2 ETH(2) - - Link types may have to be configured to Ethernet:: - mlxconfig -d set LINK_TYPE_P1/2=1/2/3 +Sub-Function +------------ - * LINK_TYPE_P1=<1|2|3> , 1=Infiniband 2=Ethernet 3=VPI(auto-sense) - - For hypervisors, verify SR-IOV is enabled on the NIC:: - - mlxconfig -d query | grep SRIOV_EN - SRIOV_EN True(1) - - If needed, configure SR-IOV:: - - mlxconfig -d set SRIOV_EN=1 NUM_OF_VFS=16 - mlxfwreset -d reset - -5. Restart the driver:: - - /etc/init.d/openibd restart - - or:: - - service openibd restart - - If link type was changed, firmware must be reset as well:: - - mlxfwreset -d reset - - For hypervisors, after reset write the sysfs number of virtual functions - needed for the PF. - - To dynamically instantiate a given number of virtual functions (VFs):: - - echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs - -6. Install DPDK and you are ready to go. - See :doc:`compilation instructions <../linux_gsg/build_dpdk>`. - -Enable switchdev mode ---------------------- - -Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF. -Representor is a port in DPDK that is connected to a VF or SF in such a way -that assuming there are no offload flows, each packet that is sent from the VF or SF -will be received by the corresponding representor. While each packet that is or SF -sent to a representor will be received by the VF or SF. -This is very useful in case of SRIOV mode, where the first packet that is sent -by the VF or SF will be received by the DPDK application which will decide if this -flow should be offloaded to the E-Switch. After offloading the flow packet -that the VF or SF that are matching the flow will not be received any more by -the DPDK application. - -1. Enable SRIOV mode:: - - mlxconfig -d set SRIOV_EN=true - -2. Configure the max number of VFs:: - - mlxconfig -d set NUM_OF_VFS= - -3. Reset the FW:: - - mlxfwreset -d reset - -3. Configure the actual number of VFs:: - - echo /sys/class/net//device/sriov_numvfs - -4. Unbind the device (can be rebind after the switchdev mode):: - - echo -n " /sys/bus/pci/drivers/mlx5_core/unbind - -5. Enable switchdev mode:: - - echo switchdev > /sys/class/net//compat/devlink/mode - -Sub-Function support --------------------- - -Sub-Function is a portion of the PCI device, a SF netdev has its own -dedicated queues (txq, rxq). -A SF shares PCI level resources with other SFs and/or with its parent PCI function. - -0. Requirement:: - - OFED version >= 5.4-0.3.3.0 - -1. Configure SF feature:: - - # Run mlxconfig on both PFs on host and ECPFs on BlueField. - mlxconfig -d set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 - -2. Enable switchdev mode:: - - mlxdevm dev eswitch set pci/ mode switchdev - -3. Add SF port:: - - mlxdevm port add pci/ flavour pcisf pfnum 0 sfnum - - Get SFID from output: pci// - -4. Modify MAC address:: - - mlxdevm port function set pci// hw_addr - -5. Activate SF port:: - - mlxdevm port function set pci// state active - -6. Devargs to probe SF device:: - - auxiliary:mlx5_core.sf.,dv_flow_en=1 +See :ref:`mlx5_sub_function`. Sub-Function representor support --------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A SF netdev supports E-Switch representation offload similar to PF and VF representors. Use to probe SF representor:: - testpmd> port attach ,representor=sf,dv_flow_en=1 + testpmd> port attach ,representor=sf,dv_flow_en=1 + Performance tuning ------------------ diff --git a/doc/guides/platform/index.rst b/doc/guides/platform/index.rst index 2ff91a6018..b1ba11df73 100644 --- a/doc/guides/platform/index.rst +++ b/doc/guides/platform/index.rst @@ -14,4 +14,5 @@ The following are platform specific guides and setup information. cnxk dpaa dpaa2 + mlx5 octeontx diff --git a/doc/guides/platform/mlx5.rst b/doc/guides/platform/mlx5.rst new file mode 100644 index 0000000000..0fd5e6604d --- /dev/null +++ b/doc/guides/platform/mlx5.rst @@ -0,0 +1,602 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2022 6WIND S.A. + Copyright (c) 2022 NVIDIA Corporation & Affiliates + +.. include:: + +MLX5 Common Driver +================== + +The mlx5 common driver library (**librte_common_mlx5**) provides support for +**Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx**, **Mellanox ConnectX-5**, +**Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx**, **Mellanox ConnectX-6 Lx**, +**Mellanox BlueField** and **Mellanox BlueField-2** families of +10/25/40/50/100/200 Gb/s adapters. + +Information and documentation for these adapters can be found on the +`NVIDIA website `_. +Help is also provided by the +`Mellanox community `_. +In addition, there is a `web section dedicated to the Poll Mode Driver +`_. + + +Design +------ + +For security reasons and to enhance robustness, +this driver only handles virtual memory addresses. +The way resources allocations are handled by the kernel, +combined with hardware specifications that allow handling virtual memory addresses directly, +ensure that DPDK applications cannot access random physical memory +(or memory that does not belong to the current process). + +There are different levels of objects and bypassing abilities +which are used to get the best performance: + +- **Verbs** is a complete high-level generic API +- **Direct Verbs** is a device-specific API +- **DevX** allows accessing firmware objects +- **Direct Rules** manages flow steering at the low-level hardware layer + +On Linux, above interfaces are provided by linking with `libibverbs` and `libmlx5`. +See :ref:`mlx5_linux_prerequisites` for installation. + +On Windows, DevX is the only requirement from the above list. +See :ref:`mlx5_windows_prerequisites` for DevX SDK package installation. + + +.. _mlx5_classes: + +Classes +------- + +One mlx5 device can be probed by a number of different PMDs. +To select a specific PMD, its name should be specified as a device parameter +(e.g. ``0000:08:00.1,class=eth``). + +In order to allow probing by multiple PMDs, +several classes may be listed separated by a colon. +For example: ``class=crypto:regex`` will probe both Crypto and RegEx PMDs. + + +Supported Classes +~~~~~~~~~~~~~~~~~ + +- ``class=compress`` for :doc:`../../compressdevs/mlx5`. +- ``class=crypto`` for :doc:`../../cryptodevs/mlx5`. +- ``class=eth`` for :doc:`../../nics/mlx5`. +- ``class=regex`` for :doc:`../../regexdevs/mlx5`. +- ``class=vdpa`` for :doc:`../../vdpadevs/mlx5`. + +By default, the mlx5 device will be probed by the ``eth`` PMD. + + +Limitations +~~~~~~~~~~~ + +- ``eth`` and ``vdpa`` PMDs cannot be probed at the same time. + All other combinations are possible. + +- On Windows, only ``eth`` and ``crypto`` are supported. + + +.. _mlx5_common_compilation: + +Compilation Prerequisites +------------------------- + +.. _mlx5_linux_prerequisites: + +Linux Prerequisites +~~~~~~~~~~~~~~~~~~~ + +This driver relies on external libraries and kernel drivers for resources +allocations and initialization. +The following dependencies are not part of DPDK and must be installed separately: + +- **libibverbs** + + User space Verbs framework used by ``librte_common_mlx5``. + This library provides a generic interface between the kernel + and low-level user space drivers such as ``libmlx5``. + + It allows slow and privileged operations (context initialization, + hardware resources allocations) to be managed by the kernel + and fast operations to never leave user space. + +- **libmlx5** + + Low-level user space driver library for Mellanox devices, + it is automatically loaded by ``libibverbs``. + + This library basically implements send/receive calls to the hardware queues. + +- **Kernel modules** + + They provide the kernel-side Verbs API and low level device drivers + that manage actual hardware initialization + and resources sharing with user-space processes. + + Unlike most other PMDs, these modules must remain loaded and bound to + their devices: + + - ``mlx5_core``: hardware driver managing Mellanox devices + and related Ethernet kernel network devices. + - ``mlx5_ib``: InfiniBand device driver. + - ``ib_uverbs``: user space driver for Verbs (entry point for ``libibverbs``). + +- **Firmware update** + + Mellanox OFED/EN releases include firmware updates. + + Because each release provides new features, these updates must be applied to + match the kernel modules and libraries they come with. + +Libraries and kernel modules can be provided either by the Linux distribution, +or by installing Mellanox OFED/EN which provides compatibility with older kernels. + + +Upstream Dependencies +^^^^^^^^^^^^^^^^^^^^^ + +The mlx5 kernel modules are part of upstream Linux. +The minimal supported kernel version is 4.14. +For 32-bit, version 4.14.41 or above is required. + +The libraries `libibverbs` and `libmlx5` are part of ``rdma-core``. +It is packaged by most of Linux distributions. +The minimal supported rdma-core version is 16. +For 32-bit, version 18 or above is required. + +The rdma-core sources can be downloaded at +https://github.com/linux-rdma/rdma-core + +It is possible to build rdma-core as static libraries starting with version 21:: + + cd build + CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. + ninja + + +Mellanox OFED/EN +^^^^^^^^^^^^^^^^ + +The kernel modules and libraries are packaged with other tools +in Mellanox OFED or Mellanox EN. +The minimal supported versions are: + +- Mellanox OFED version: **4.5** and above. +- Mellanox EN version: **4.5** and above. +- Firmware version: + + - ConnectX-4: **12.21.1000** and above. + - ConnectX-4 Lx: **14.21.1000** and above. + - ConnectX-5: **16.21.1000** and above. + - ConnectX-5 Ex: **16.21.1000** and above. + - ConnectX-6: **20.27.0090** and above. + - ConnectX-6 Dx: **22.27.0090** and above. + - BlueField: **18.25.1010** and above. + - BlueField-2: **24.28.1002** and above. + +The firmware, the libraries libibverbs, libmlx5, and mlnx-ofed-kernel modules +are packaged in `Mellanox OFED +`_. +After downloading, it can be installed with this command:: + + ./mlnxofedinstall --dpdk + +`Mellanox EN +`_ +is a smaller package including what is needed for DPDK. +After downloading, it can be installed with this command:: + + ./install --dpdk + +After installing, the firmware version can be checked:: + + ibv_devinfo + +.. note:: + + Several versions of Mellanox OFED/EN are available. Installing the version + this DPDK release was developed and tested against is strongly recommended. + Please check the "Tested Platforms" section in the :doc:`../../rel_notes/index`. + + +.. _mlx5_windows_prerequisites: + +Windows Prerequisites +~~~~~~~~~~~~~~~~~~~~~ + +The mlx5 PMDs rely on external libraries and kernel drivers +for resource allocation and initialization. + + +DevX SDK Installation +^^^^^^^^^^^^^^^^^^^^^ + +The DevX SDK must be installed on the machine building the Windows PMD. +Additional information can be found at +`How to Integrate Windows DevX in Your Development Environment +`_. +The minimal supported WinOF2 version is 2.60. + + +Compilation Options +------------------- + +Compilation on Linux +~~~~~~~~~~~~~~~~~~~~ + +The ibverbs libraries can be linked with this PMD in a number of ways, +configured by the ``ibverbs_link`` build option: + +``shared`` (default) + The PMD depends on some .so files. + +``dlopen`` + Split the dependencies glue in a separate library + loaded when needed by dlopen (see ``MLX5_GLUE_PATH``). + It makes dependencies on libibverbs and libmlx5 optional, + and has no performance impact. + +``static`` + Embed static flavor of the dependencies libibverbs and libmlx5 + in the PMD shared library or the executable static binary. + + +Compilation on Windows +~~~~~~~~~~~~~~~~~~~~~~ + +The DevX SDK location must be set through two environment variables: + +``DEVX_LIB_PATH`` + path to the DevX lib file. + +``DEVX_INC_PATH`` + path to the DevX header files. + + +.. _mlx5_common_env: + +Environment Configuration +------------------------- + +Linux Environment +~~~~~~~~~~~~~~~~~ + +The kernel network interfaces are brought up during initialization. +Forcing them down prevents packets reception. + +The ethtool operations on the kernel interfaces may also affect the PMD. + +Some runtime behaviours may be configured through environment variables. + +``MLX5_GLUE_PATH`` + If built with ``ibverbs_link=dlopen``, + list of directories in which to search for the rdma-core "glue" plug-in, + separated by colons or semi-colons. + +``MLX5_SHUT_UP_BF`` + If Verbs is used (DevX disabled), + HW queue doorbell register mapping. + The value 0 means non-cached IO mapping, + while 1 is a regular memory mapping. + + With regular memory mapping, the register is flushed to HW + usually when the write-combining buffer becomes full, + but it depends on CPU design. + + +Port Link with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^ + +Ports links must be set to Ethernet:: + + mlxconfig -d query | grep LINK_TYPE + LINK_TYPE_P1 ETH(2) + LINK_TYPE_P2 ETH(2) + + mlxconfig -d set LINK_TYPE_P1/2=1/2/3 + +Link type values are: + +* ``1`` Infiniband +* ``2`` Ethernet +* ``3`` VPI (auto-sense) + +If link type was changed, firmware must be reset as well:: + + mlxfwreset -d reset + + +.. _mlx5_vf: + +SR-IOV Virtual Function with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +SR-IOV must be enabled on the NIC. +It can be checked in the following command:: + + mlxconfig -d query | grep SRIOV_EN + SRIOV_EN True(1) + +If needed, configure SR-IOV:: + + mlxconfig -d set SRIOV_EN=1 NUM_OF_VFS=16 + mlxfwreset -d reset + +After doing the change, restart the driver:: + + /etc/init.d/openibd restart + +or:: + + service openibd restart + +Then the virtual functions can be instantiated:: + + echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs + + +.. _mlx5_sub_function: + +Sub-Function with OFED/EN +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sub-Function is a portion of the PCI device, +it has its own dedicated queues. +An SF shares PCI-level resources with other SFs and/or with its parent PCI function. + +0. Requirement:: + + OFED version >= 5.4-0.3.3.0 + +1. Configure SF feature:: + + # Run mlxconfig on both PFs on host and ECPFs on BlueField. + mlxconfig -d set PER_PF_NUM_SF=1 PF_TOTAL_SF=252 PF_SF_BAR_SIZE=12 + +2. Enable switchdev mode:: + + mlxdevm dev eswitch set pci/ mode switchdev + +3. Add SF port:: + + mlxdevm port add pci/ flavour pcisf pfnum 0 sfnum + + Get SFID from output: pci// + +4. Modify MAC address:: + + mlxdevm port function set pci// hw_addr + +5. Activate SF port:: + + mlxdevm port function set pci// state active + +6. Devargs to probe SF device:: + + auxiliary:mlx5_core.sf.,class=eth:regex + + +Enable Switchdev Mode +^^^^^^^^^^^^^^^^^^^^^ + +Switchdev mode is a mode in E-Switch, that binds between representor and VF or SF. +Representor is a port in DPDK that is connected to a VF or SF in such a way +that assuming there are no offload flows, each packet that is sent from the VF or SF +will be received by the corresponding representor. +While each packet that is sent to a representor will be received by the VF or SF. + +After :ref:`configuring VF `, the device must be unbound:: + + printf "" > /sys/bus/pci/drivers/mlx5_core/unbind + +Then switchdev mode is enabled:: + + echo switchdev > /sys/class/net//compat/devlink/mode + +The device can be bound again at this point. + + +Run as Non-Root +^^^^^^^^^^^^^^^ + +In order to run as a non-root user, +some capabilities must be granted to the application:: + + setcap cap_sys_admin,cap_net_admin,cap_net_raw,cap_ipc_lock+ep + +Below are the reasons for the need of each capability: + +``cap_sys_admin`` + When using physical addresses (PA mode), with Linux >= 4.0, + for access to ``/proc/self/pagemap``. + +``cap_net_admin`` + For device configuration. + +``cap_net_raw`` + For raw ethernet queue allocation through kernel driver. + +``cap_ipc_lock`` + For DMA memory pinning. + + +Windows Environment +~~~~~~~~~~~~~~~~~~~ + +WinOF2 version 2.60 or higher must be installed on the machine. + + +WinOF2 Installation +^^^^^^^^^^^^^^^^^^^ + +The driver can be downloaded from the following site: `WINOF2 +`_. + + +DevX Enablement +^^^^^^^^^^^^^^^ + +DevX for Windows must be enabled in the Windows registry. +The keys ``DevxEnabled`` and ``DevxFsRules`` must be set. +Additional information can be found in the WinOF2 user manual. + + +.. _mlx5_firmware_config: + +Firmware Configuration +~~~~~~~~~~~~~~~~~~~~~~ + +Firmware features can be configured as key/value pairs. + +The command to set a value is:: + + mlxconfig -d set = + +The command to query a value is:: + + mlxconfig -d query + +The device name for the command ``mlxconfig`` can be either the PCI address, +or the mst device name found with:: + + mst status + +Below are some firmware configurations listed. + +- link type:: + + LINK_TYPE_P1 + LINK_TYPE_P2 + value: 1=Infiniband 2=Ethernet 3=VPI(auto-sense) + +- enable SR-IOV:: + + SRIOV_EN=1 + +- the maximum number of SR-IOV virtual functions:: + + NUM_OF_VFS= + +- enable DevX (required by Direct Rules and other features):: + + UCTX_EN=1 + +- aggressive CQE zipping:: + + CQE_COMPRESSION=1 + +- L3 VXLAN and VXLAN-GPE destination UDP port:: + + IP_OVER_VXLAN_EN=1 + IP_OVER_VXLAN_PORT= + +- enable VXLAN-GPE tunnel flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + or + FLEX_PARSER_PROFILE_ENABLE=2 + +- enable IP-in-IP tunnel flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + +- enable MPLS flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=1 + +- enable ICMP(code/type/identifier/sequence number) / ICMP6(code/type) fields matching:: + + FLEX_PARSER_PROFILE_ENABLE=2 + +- enable Geneve flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + or + FLEX_PARSER_PROFILE_ENABLE=1 + +- enable Geneve TLV option flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=0 + +- enable GTP flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=3 + +- enable eCPRI flow matching:: + + FLEX_PARSER_PROFILE_ENABLE=4 + PROG_PARSE_GRAPH=1 + +- enable dynamic flex parser for flex item:: + + FLEX_PARSER_PROFILE_ENABLE=4 + PROG_PARSE_GRAPH=1 + +- enable realtime timestamp format:: + + REAL_TIME_CLOCK_ENABLE=1 + + +.. _mlx5_common_driver_options: + +Device Arguments +---------------- + +The driver can be configured per device. +A single argument list can be used for a device managed by multiple PMDs. +The parameters must be passed through the EAL option ``-a``, +as examples below: + +- PCI device:: + + -a 0000:03:00.2,class=eth:regex,mr_mempool_reg_en=0 + +- Auxiliary SF:: + + -a auxiliary:mlx5_core.sf.2,class=compress,mr_ext_memseg_en=0 + +Each device class PMD has its own list of specific arguments, +and below are the arguments supported by the common mlx5 layer. + +- ``class`` parameter [string] + + Select the classes of the drivers that should probe the device. + See :ref:`mlx5_classes` for more explanation and details. + + The default value is ``eth``. + +- ``mr_ext_memseg_en`` parameter [int] + + A nonzero value enables extending memseg when registering DMA memory. If + enabled, the number of entries in MR (Memory Region) lookup table on datapath + is minimized and it benefits performance. On the other hand, it worsens memory + utilization because registered memory is pinned by kernel driver. Even if a + page in the extended chunk is freed, that doesn't become reusable until the + entire memory is freed. + + Enabled by default. + +- ``mr_mempool_reg_en`` parameter [int] + + A nonzero value enables implicit registration of DMA memory of all mempools + except those having ``RTE_MEMPOOL_F_NON_IO``. This flag is set automatically + for mempools populated with non-contiguous objects or those without IOVA. + The effect is that when a packet from a mempool is transmitted, + its memory is already registered for DMA in the PMD and no registration + will happen on the data path. The tradeoff is extra work on the creation + of each mempool and increased HW resource use if some mempools + are not used with MLX5 devices. + + Enabled by default. + +- ``sys_mem_en`` parameter [int] + + A non-zero value enables the PMD memory management allocating memory + from system by default, without explicit rte memory flag. + + By default, the PMD will set this value to 0. diff --git a/doc/guides/regexdevs/mlx5.rst b/doc/guides/regexdevs/mlx5.rst index fc2cacba60..83280bda40 100644 --- a/doc/guides/regexdevs/mlx5.rst +++ b/doc/guides/regexdevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: -MLX5 RegEx driver +MLX5 RegEx Driver ================= -The MLX5 RegEx (Regular Expression) driver library +The mlx5 RegEx (Regular Expression) driver library (**librte_regex_mlx5**) provides support for **Mellanox BlueField-2** families of 25/50/100/200 Gb/s adapters. @@ -17,29 +17,21 @@ This PMD is configuring the RegEx HW engine. For the PMD to work, the application must supply a precompiled rule file in rof2 format. -The PMD uses libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects - -Enabling librte_regex_mlx5 causes DPDK applications to be linked against -libibverbs. - -Mellanox mlx5 pci device can be probed by number of different pci devices, -for example net / vDPA / RegEx. To select the RegEx PMD ``class=regex`` should -be specified as device parameter. The RegEx device can be probed and used with -other Mellanox devices, by adding more options in the class. -For example: ``class=net:regex`` will probe both the net PMD and the RegEx PMD. +See :doc:`../../platform/mlx5` guide for more design details. Features -------- - Multi segments mbuf support. +Configuration +------------- + +See :ref:`mlx5 common compilation `, +:ref:`mlx5 firmware configuration `, +and :ref:`mlx5 common driver options `. + + Supported NICs -------------- @@ -52,12 +44,8 @@ Prerequisites - Enable the RegEx capabilities using system call from the BlueField-2. - Official support is not yet released. + Limitations ----------- - The firmware version must be greater than XX.31.0364 - -Run-time configuration -~~~~~~~~~~~~~~~~~~~~~~ - -- **ethtool** operations on related kernel interfaces also affect the PMD. diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst index 30f0b62eb4..acb791032a 100644 --- a/doc/guides/vdpadevs/mlx5.rst +++ b/doc/guides/vdpadevs/mlx5.rst @@ -3,10 +3,10 @@ .. include:: -MLX5 vDPA driver +MLX5 vDPA Driver ================ -The MLX5 vDPA (vhost data path acceleration) driver library +The mlx5 vDPA (vhost data path acceleration) driver library (**librte_vdpa_mlx5**) provides support for **Mellanox ConnectX-6**, **Mellanox ConnectX-6 Dx** and **Mellanox BlueField** families of 10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF) in @@ -17,33 +17,8 @@ SR-IOV context. This driver is enabled automatically when using "meson" build system which will detect dependencies. - -Design ------- - -For security reasons and robustness, this driver only deals with virtual -memory addresses. The way resources allocations are handled by the kernel, -combined with hardware specifications that allow to handle virtual memory -addresses directly, ensure that DPDK applications cannot access random -physical memory (or memory that does not belong to the current process). - -The PMD can use libibverbs and libmlx5 to access the device firmware -or directly the hardware components. -There are different levels of objects and bypassing abilities -to get the best performances: - -- Verbs is a complete high-level generic API -- Direct Verbs is a device-specific API -- DevX allows to access firmware objects -- Direct Rules manages flow steering at low-level hardware layer - -Enabling librte_vdpa_mlx5 causes DPDK applications to be linked against -libibverbs. - -A Mellanox mlx5 PCI device can be probed by either net/mlx5 driver or vdpa/mlx5 -driver but not in parallel. Hence, the user should decide the driver by the -``class`` parameter in the device argument list. -By default, the mlx5 device will be probed by the net/mlx5 driver. +See :doc:`../../platform/mlx5` guide for design details, +and which PMDs can be combined with vDPA PMD. Supported NICs -------------- @@ -58,52 +33,16 @@ Prerequisites ------------- - Mellanox OFED version: **5.0** - see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. - -Compilation option -~~~~~~~~~~~~~~~~~~ - -The meson option ``ibverbs_link`` is **shared** by default, -but can be configured to have the following values: - -- ``dlopen`` - - Build PMD with additional code to make it loadable without hard - dependencies on **libibverbs** nor **libmlx5**, which may not be installed - on the target system. - - In this mode, their presence is still required for it to run properly, - however their absence won't prevent a DPDK application from starting (with - DPDK shared build disabled) and they won't show up as missing with ``ldd(1)``. - - It works by moving these dependencies to a purpose-built rdma-core "glue" - plug-in which must be installed in a directory whose name is based - on ``RTE_EAL_PMD_PATH`` suffixed with ``-glue``. - - This option has no performance impact. - -- ``static`` - - Embed static flavor of the dependencies **libibverbs** and **libmlx5** - in the PMD shared library or the executable static binary. - -.. note:: - - Default armv8a configuration of meson build sets ``RTE_CACHE_LINE_SIZE`` - to 128 then brings performance degradation. + See :ref:`mlx5 common prerequisites ` for more details. Run-time configuration ~~~~~~~~~~~~~~~~~~~~~~ -- **ethtool** operations on related kernel interfaces also affect the PMD. - Driver options ^^^^^^^^^^^^^^ -- ``class`` parameter [string] - - Select the class of the driver that should probe the device. - `vdpa` for the mlx5 vDPA driver. +Please refer to :ref:`mlx5 common options ` +for an additional list of options shared with other mlx5 drivers. - ``event_mode`` parameter [int] @@ -163,18 +102,6 @@ Driver options - 0, HW default. -Devargs example -^^^^^^^^^^^^^^^ - -- PCI devargs:: - - -a 0000:03:00.2,class=vdpa - -- Auxiliary devargs:: - - -a auxiliary:mlx5_core.sf.2,class=vdpa - - Error handling ^^^^^^^^^^^^^^