From patchwork Tue Oct 3 10:48:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matan Azrad X-Patchwork-Id: 29522 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3BE521B19A; Tue, 3 Oct 2017 12:50:02 +0200 (CEST) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0089.outbound.protection.outlook.com [104.47.0.89]) by dpdk.org (Postfix) with ESMTP id B1D331B323 for ; Tue, 3 Oct 2017 12:50:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=RHlx28fiqcAm87vuWFAlkSFYmL3IeuIJBOo3LFNkeqc=; b=w1buD2+wLGr+gZasdT7KyxzwMvDz2db4sMeq6OjsR/DzWIQP9SVTOi8RB0lxIvqYuBsDQbWZv2tR8P4n5ftUgisqBw7a3r1zM1FyvUYz80kM4kH/YlNOFJFPx+ZSJkJ+b+y63AAsCQ+3tHxPFypRuQuZ4tUAlZd+rqzHMaAPSzA= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; Received: from mellanox.com (37.142.13.130) by AM5PR0502MB3042.eurprd05.prod.outlook.com (2603:10a6:203:a1::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Tue, 3 Oct 2017 10:49:58 +0000 From: Matan Azrad To: Adrien Mazarguil Cc: dev@dpdk.org, Moti Haimovsky Date: Tue, 3 Oct 2017 10:48:26 +0000 Message-Id: <1507027711-879-2-git-send-email-matan@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1507027711-879-1-git-send-email-matan@mellanox.com> References: <1503590050-196143-1-git-send-email-motih@mellanox.com> <1507027711-879-1-git-send-email-matan@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: HE1PR09CA0085.eurprd09.prod.outlook.com (2603:10a6:7:3d::29) To AM5PR0502MB3042.eurprd05.prod.outlook.com (2603:10a6:203:a1::18) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c6982040-1216-4223-bd88-08d50a4c7e05 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(48565401081)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:AM5PR0502MB3042; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0502MB3042; 3:RNO1zFsID9wcW1TqIZsmsqU6p9dLWlkf9CfoEX5MYZn3mmHH0Q6kJbziOtaPNtnmlp5stfc+ZWstBj0LWHta8ceEOOaDo2PNfTwGH1v5aCmvRDf/OUzeXHAFLt789CNRxwHg5UDnh+PYObAo1OqTxFQtOs4s20nSHplFV3WYRiQ9rn1CKrmzj/otqXiudFafZgz+vi/XrnITthuVVWgschoV8O7p9EGwwCu0SC1pG5Tttf2TebxVMMR6aF6J6ZH+; 25:/avQz8rmSP4wVQj0y8eEiWTI+9iIENIdpA/jXuaaCESgpR2dRycKfvXwBFlVpKVT9PnrTTMRT5fsj0Wkvf8/o3y5wk2VNXZfaRSbeqcb/YbYZ0J6REZy4I8kRlK9oj6SWeLwLzYAW6xxAw3/uaiB0/k4duz5U8GX3xOUAbDBGmM7LXAgMEy+gtEOLee3PBmZDDD6lLaQsn2kdUJELdVtwGtrSnw+JY2uJgLw/GLbVMzgHUO6k0fyF1TIHI68VKnNEmMaIfsVZjPeB9pHWTcSpx128TBwACl4uNnMd4jeNt2C+Xf5kZ5NTKOe24pJ5V5BUinModaeUzXvmBRwOrr+Xw==; 31:+kqjDg4Ni+6MwEHGlnYnKjgVgMSjRTzpREixHB+hkYdTutv57Ci5Fqs3zMv1i59SQ1uAlURwqwB50VtQ6DTvvpSoChA0mtyvkI60gOnHFd0yDKUieXPNXLLfpwXzPplueX7ZVXGXARomvJ8nYKLTS2COfwPw7xYJ7fp4YRfLj+YlwLP4Amj0mY+/Ky9loWkh+glRMueR4VrHrbkW3k2hBCbKulYUgMWMoz+fxkH+Tl0= X-MS-TrafficTypeDiagnostic: AM5PR0502MB3042: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM5PR0502MB3042; 20:ilzszw69ou/EqBZ//CEu6IA+o6ecVCstBvoQg6SYrUMMreAAHiiu0SZl3uzmuSi97aF42xTMRfurycJdpI7P5tfGapCiqw9q/0oie/hACIliOO80E5KjPlX7bsGzAW5KjvLOZ9BPG6MDS0sz3iuas/RX67FHNyii6U6okyIeVe/fE+BCTVpZdsadVrMfluFigtAEbxLq7pV/K4LfvN7wnrlOpyR4TQey7XyKVkuzna9jP0nTuRV0CWkM2ZLf2+T685VHQyQ6kpDIDzs5wAhZk2dpUzJn2Ii6E4sHQWmhMUpw7fhyWswcOXXFQEpOaANEyKamRSn7MroKVOnJOZMe6QYsoAhjX+YeSePWY6NR5lBj+6HLBMomRmrdsXZtCB0juSubfzwoOvQMw7IpeApsO/xEqdZY1bnaNFoJ57NPb/Nwqm0fIrym6U3nPLjY9u7hUXYk6MfP4czlC7XqTTv+LiIxTklcUGsLKlN0pBO0qZ2J+aS1IazPQ8n34W3dsiL5; 4:j53a+29OKE7QKgpFLpxCdVCFr5DUGHB24XyKqvZhWujNiDEjo/2Nh+8xKORU2etCaswIKrHBxdtUeAEhVqca/4NLvYRBJGQey2L7JcRyvUMhWur4BoeeLyIxifRhwvEQdK90w1FacP5UZCz3cg79j8O3mjAiQ5SbPqJx6ZDJlgEgCxdKw6Whz3htdUD8JEQJ6X3C0O39pUQRSYv8DMranoQCV/M7EUxS3Zer2gHWppXwJQsixfY4noH1qWwrqH8+cdXWlWDVQ7HKJ87Vv+EA55mqwUm6hKtX3METdb665gtlpDD7P9fniQkEug4WakskV/G+f6SKL9KddFZ87Y7zrg== X-Exchange-Antispam-Report-Test: UriScan:(60795455431006)(66839620246622); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3002001)(100000703101)(100105400095)(6055026)(2002001)(6041248)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123562025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM5PR0502MB3042; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM5PR0502MB3042; X-Forefront-PRVS: 044968D9E1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(189002)(199003)(33646002)(107886003)(5660300001)(55016002)(105586002)(47776003)(3846002)(53936002)(2906002)(66066001)(50986999)(76176999)(16586007)(86362001)(316002)(101416001)(81156014)(305945005)(8936002)(81166006)(8676002)(33026002)(50226002)(7736002)(16526017)(97736004)(48376002)(69596002)(25786009)(478600001)(50466002)(6116002)(4326008)(106356001)(5003940100001)(36756003)(6916009)(6666003)(4720700003)(21086003)(2950100002)(68736007)(189998001)(50929005); DIR:OUT; SFP:1101; SCL:1; SRVR:AM5PR0502MB3042; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0502MB3042; 23:iHqgsNBrXDXKWkEDcqc7bQlSE7LT+m1JzzaMyIl?= s/VqNkdkauqj/RLA/+vm55m0D48Ul40cWZh4TR1cu3a69eVqwir79eWHY5+Sx6zIo9W+wSdm5Ra6xTcn6lHGJXdcpZSRRpvE0QNVRWI2US6Mlt1tOiWVVTqt8JhdVczi5I9V8gLFglAGbdxq4l7ugweeldMTHRqAZkc7WKRbpLoCQzU4MO4eeJoNSicFUS/AwEiKXf5hpIuk82TPyx40ftyvMNlIZipa/LgYTZWPFewogvSWSm4u6IkMD6EuMm8265iahTF8WuTIT2vii9Z0HDd2Hj28j+RC5TA8+oYTKsuvIWO0i48eN1P9v8fGmJKxTpVH8p0TRwxTyAcMioFOSSWo2SPJ6BRYsx42kLz/uej2+CZyc+OpQ62S5mpP8PNakMYCu8hKLKQ9c29vPIFpb3H6nGPWsFmvVe9nSl60cIkzHMpr8nHBLuYyToIR/oxacC9RQGy3YZReAZZ+s3RQhFxiu2ujL07cmAwHK9QPkHc46CUYmMiPzQ5IGFabWy9QqI6ySF83qKCck1cx6fOnA9lt7jZCAeSkf8OL/xS/o3hk4nfWYCE5fGqyxWZBWh0L133cg11fu9177wfRxYbArFwlN6qqFjINJIhxPUZTui0wRdrxvRmllRzv09jHj4hM/EYQKFLXTrrp/pdYwGFzop297Mg6b6ka2AJcrVRsgetQNTWL56kPbvDK+qZVJffdIam8fICa6XCiOeLER8y+BlmHKPjFPnU48oFJ892M0w9fWyJSkGR2m++rF6tPIIafZrgZo+w9pUvvHTzo/Z76JPmzw4C3uTGXROKPW4ujxzIrz+w438Dux6rD2Yc3QgMUUefoH0UsCqhKXwjm+VHvWWXw52dl3qz2z7EcokzAvVTRVLIl1zaQXp/TsPWCvdojElte0lD6PSd8JmWY9o7nChP59c3Mzmy2hSs5Y0JP9j9lqJjUtPKKAG175KHJBs50Qha7CZyD4mKuoVS0ZZjCKCMW1RBK9Y1eR2+nkEEfwvO6PEYqEDp3dj90FTdx2KURXhHFyeFUCp/4EpIdgm6eJTaRRGt7hWY+T+CufxKz6zYeldALIlUH0NvmmNjkRmJ/0q0x+RfLxqPdtGVfPyCuelcxX2JG2t/7MEfUiqCp+5YF3Jg== X-Microsoft-Exchange-Diagnostics: 1; AM5PR0502MB3042; 6:1iIYMiE3dzD0NSUawlO1APL40GUZlQdPbaK/cryF/T0g8itLGEaBCyKbMaxfjuMc/9TuKYILHbd1fYxVS+d8dtM8RmP6f+2j8EXpln0CFOmnmZ86SuB3iWVpICrZIqMC91eDQUtLj5eNhf/0kkJEt24hehMFKlnnBymfaYJ4YcNV00lm6WQoVHS0jtOJGswuXgIaSVRGluBlW32uHB4ZwFU/QGL2xWcFBgWafaYuwQCglbeXF6skegxQKhPbSPdKoRbfYWLHNlEonzDKKfVjoZXj0+S8eWUcbFf1Afftut1hTivKbN72aDvwXwGKw2aa1pw5M+6sJPPsUg7EbrAVgw==; 5:Zr4UWLOjyJE8OVquw5xf5IcuLuGpsdP7GzY21nou8q9cnqLYx1+6/w3bAki2GdExwkMjKDoFdR8dJu6qqJLsUYmWoHmSvMKQsisMIgPESgYB4cwd2X5QQ/AjnqMFlAmxBW6oOvVIYaVpHH/uVUT7QyjFZ5s9DwokUGcgqo8PTYw=; 24:CfzkRYZbko82QRYdE4i+WKrDZA9mchheO2/Q3NSq7wZYDA5+1eFzVB/aTaYfLs/I6mPZv8mn1YfFzYM9+lMQFcaePvBxtiaatfg/daSGbLc=; 7:ANHmNyL/FrJWYlfttV/rkGlcLoj4VLIy1lWg6cx9PJ2I5EHTbMSPieW7nrfkwGxNsJl+W+GZZJc2u01hLwnP88deP4hbC1OWOWKMet5KUNBrEAhMABzyl0eEWwg7867aZUATeO8QTRgLbKITB1seO+bKKHqSBcbxt5/cxSmBHGkbfE+Z9C00J/nBNIvA3pnHq+MdMhBGqz/mwC3jzV1k72sodcAh1iicX/RWFlak3OU= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Oct 2017 10:49:58.1515 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0502MB3042 Subject: [dpdk-dev] [PATCH v2 1/6] net/mlx4: add simple Tx bypassing ibverbs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Moti Haimovsky Modify PMD to send single-buffer packets directly to the device bypassing the ibv Tx post and poll routines. Signed-off-by: Moti Haimovsky --- drivers/net/mlx4/mlx4_prm.h | 108 +++++++++++++ drivers/net/mlx4/mlx4_rxtx.c | 353 ++++++++++++++++++++++++++++++++----------- drivers/net/mlx4/mlx4_rxtx.h | 32 ++-- drivers/net/mlx4/mlx4_txq.c | 90 ++++++++--- mk/rte.app.mk | 2 +- 5 files changed, 463 insertions(+), 122 deletions(-) create mode 100644 drivers/net/mlx4/mlx4_prm.h diff --git a/drivers/net/mlx4/mlx4_prm.h b/drivers/net/mlx4/mlx4_prm.h new file mode 100644 index 0000000..6d1800a --- /dev/null +++ b/drivers/net/mlx4/mlx4_prm.h @@ -0,0 +1,108 @@ +/*- + * BSD LICENSE + * + * Copyright 2017 6WIND S.A. + * Copyright 2017 Mellanox + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of 6WIND S.A. nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RTE_PMD_MLX4_PRM_H_ +#define RTE_PMD_MLX4_PRM_H_ + +#include +#include +#include + +/* Verbs headers do not support -pedantic. */ +#ifdef PEDANTIC +#pragma GCC diagnostic ignored "-Wpedantic" +#endif +#include +#include +#ifdef PEDANTIC +#pragma GCC diagnostic error "-Wpedantic" +#endif + +/* ConnectX-3 Tx queue basic block. */ +#define MLX4_TXBB_SHIFT 6 +#define MLX4_TXBB_SIZE (1 << MLX4_TXBB_SHIFT) + +/* Typical TSO descriptor with 16 gather entries is 352 bytes. */ +#define MLX4_MAX_WQE_SIZE 512 +#define MLX4_MAX_WQE_TXBBS (MLX4_MAX_WQE_SIZE / MLX4_TXBB_SIZE) + +/* Send queue stamping/invalidating information. */ +#define MLX4_SQ_STAMP_STRIDE 64 +#define MLX4_SQ_STAMP_DWORDS (MLX4_SQ_STAMP_STRIDE / 4) +#define MLX4_SQ_STAMP_SHIFT 31 +#define MLX4_SQ_STAMP_VAL 0x7fffffff + +/* Work queue element (WQE) flags. */ +#define MLX4_BIT_WQE_OWN 0x80000000 + +#define MLX4_SIZE_TO_TXBBS(size) \ + (RTE_ALIGN((size), (MLX4_TXBB_SIZE)) >> (MLX4_TXBB_SHIFT)) + +/* Send queue information. */ +struct mlx4_sq { + char *buf; /**< SQ buffer. */ + char *eob; /**< End of SQ buffer */ + uint32_t head; /**< SQ head counter in units of TXBBS. */ + uint32_t tail; /**< SQ tail counter in units of TXBBS. */ + uint32_t txbb_cnt; /**< Num of WQEBB in the Q (should be ^2). */ + uint32_t txbb_cnt_mask; /**< txbbs_cnt mask (txbb_cnt is ^2). */ + uint32_t headroom_txbbs; /**< Num of txbbs that should be kept free. */ + uint32_t *db; /**< Pointer to the doorbell. */ + uint32_t doorbell_qpn; /**< qp number to write to the doorbell. */ +}; + +#define mlx4_get_send_wqe(sq, n) ((sq)->buf + ((n) * (MLX4_TXBB_SIZE))) + +/* Completion queue information. */ +struct mlx4_cq { + char *buf; /**< Pointer to the completion queue buffer. */ + uint32_t cqe_cnt; /**< Number of entries in the queue. */ + uint32_t cqe_64:1; /**< CQ entry size is 64 bytes. */ + uint32_t cons_index; /**< Last queue entry that was handled. */ + uint32_t *set_ci_db; /**< Pointer to the completion queue doorbell. */ +}; + +/* + * cqe = cq->buf + cons_index * cqe_size + cqe_offset + * Where cqe_size is 32 or 64 bytes and + * cqe_offset is 0 or 32 (depending on cqe_size). + */ +#define mlx4_get_cqe(cq, n) (__extension__({ \ + typeof(cq) q = (cq); \ + (q)->buf + \ + (((n) & ((q)->cqe_cnt - 1)) << \ + (5 + (q)->cqe_64)) + \ + ((q)->cqe_64 << 5); \ + })) + +#endif /* RTE_PMD_MLX4_PRM_H_ */ diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c index b5e7777..55c8e9a 100644 --- a/drivers/net/mlx4/mlx4_rxtx.c +++ b/drivers/net/mlx4/mlx4_rxtx.c @@ -40,6 +40,7 @@ #include #include #include +#include /* Verbs headers do not support -pedantic. */ #ifdef PEDANTIC @@ -52,15 +53,76 @@ #include #include +#include #include #include #include #include "mlx4.h" +#include "mlx4_prm.h" #include "mlx4_rxtx.h" #include "mlx4_utils.h" /** + * Stamp a WQE so it won't be reused by the HW. + * Routine is used when freeing WQE used by the chip or when failing + * building an WQ entry has failed leaving partial information on the queue. + * + * @param sq + * Pointer to the sq structure. + * @param index + * Index of the freed WQE. + * @param num_txbbs + * Number of blocks to stamp. + * If < 0 the routine will use the size written in the WQ entry. + * @param owner + * The value of the WQE owner bit to use in the stamp. + * + * @return + * The number of TX basic blocs (TXBB) the WQE contained. + */ +static int +mlx4_txq_stamp_freed_wqe(struct mlx4_sq *sq, uint16_t index, uint8_t owner) +{ + uint32_t stamp = + rte_cpu_to_be_32(MLX4_SQ_STAMP_VAL | + (!!owner << MLX4_SQ_STAMP_SHIFT)); + char *wqe = mlx4_get_send_wqe(sq, (index & sq->txbb_cnt_mask)); + uint32_t *ptr = (uint32_t *)wqe; + int i; + int txbbs_size; + int num_txbbs; + + /* Extract the size from the control segment of the WQE. */ + num_txbbs = MLX4_SIZE_TO_TXBBS((((struct mlx4_wqe_ctrl_seg *) + wqe)->fence_size & 0x3f) << 4); + txbbs_size = num_txbbs * MLX4_TXBB_SIZE; + /* Optimize the common case when there is no wrap-around */ + if (wqe + txbbs_size <= sq->eob) { + /* Stamp the freed descriptor */ + for (i = 0; + i < txbbs_size; + i += MLX4_SQ_STAMP_STRIDE) { + *ptr = stamp; + ptr += MLX4_SQ_STAMP_DWORDS; + } + } else { + /* Stamp the freed descriptor */ + for (i = 0; + i < txbbs_size; + i += MLX4_SQ_STAMP_STRIDE) { + *ptr = stamp; + ptr += MLX4_SQ_STAMP_DWORDS; + if ((char *)ptr >= sq->eob) { + ptr = (uint32_t *)sq->buf; + stamp ^= RTE_BE32(0x80000000); + } + } + } + return num_txbbs; +} + +/** * Manage Tx completions. * * When sending a burst, mlx4_tx_burst() posts several WRs. @@ -80,26 +142,73 @@ unsigned int elts_comp = txq->elts_comp; unsigned int elts_tail = txq->elts_tail; const unsigned int elts_n = txq->elts_n; - struct ibv_wc wcs[elts_comp]; - int wcs_n; + struct mlx4_cq *cq = &txq->mcq; + struct mlx4_sq *sq = &txq->msq; + struct mlx4_cqe *cqe; + uint32_t cons_index = cq->cons_index; + uint16_t new_index; + uint16_t nr_txbbs = 0; + int pkts = 0; if (unlikely(elts_comp == 0)) return 0; - wcs_n = ibv_poll_cq(txq->cq, elts_comp, wcs); - if (unlikely(wcs_n == 0)) + /* + * Traverse over all CQ entries reported and handle each WQ entry + * reported by them. + */ + do { + cqe = (struct mlx4_cqe *)mlx4_get_cqe(cq, cons_index); + if (unlikely(!!(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK) ^ + !!(cons_index & cq->cqe_cnt))) + break; + /* + * make sure we read the CQE after we read the + * ownership bit + */ + rte_rmb(); + if (unlikely((cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) == + MLX4_CQE_OPCODE_ERROR)) { + struct mlx4_err_cqe *cqe_err = + (struct mlx4_err_cqe *)cqe; + ERROR("%p CQE error - vendor syndrome: 0x%x" + " syndrome: 0x%x\n", + (void *)txq, cqe_err->vendor_err, cqe_err->syndrome); + } + /* Get WQE index reported in the CQE. */ + new_index = + rte_be_to_cpu_16(cqe->wqe_index) & sq->txbb_cnt_mask; + do { + /* free next descriptor */ + nr_txbbs += + mlx4_txq_stamp_freed_wqe(sq, + (sq->tail + nr_txbbs) & sq->txbb_cnt_mask, + !!((sq->tail + nr_txbbs) & sq->txbb_cnt)); + pkts++; + } while (((sq->tail + nr_txbbs) & sq->txbb_cnt_mask) != + new_index); + cons_index++; + } while (true); + if (unlikely(pkts == 0)) return 0; - if (unlikely(wcs_n < 0)) { - DEBUG("%p: ibv_poll_cq() failed (wcs_n=%d)", - (void *)txq, wcs_n); - return -1; - } - elts_comp -= wcs_n; + /* + * Update CQ. + * To prevent CQ overflow we first update CQ consumer and only then + * the ring consumer. + */ + cq->cons_index = cons_index; + *cq->set_ci_db = rte_cpu_to_be_32(cq->cons_index & 0xffffff); + rte_wmb(); + sq->tail = sq->tail + nr_txbbs; + /* + * Update the list of packets posted for transmission. + */ + elts_comp -= pkts; assert(elts_comp <= txq->elts_comp); /* - * Assume WC status is successful as nothing can be done about it - * anyway. + * Assume completion status is successful as nothing can be done about + * it anyway. */ - elts_tail += wcs_n * txq->elts_comp_cd_init; + elts_tail += pkts; if (elts_tail >= elts_n) elts_tail -= elts_n; txq->elts_tail = elts_tail; @@ -117,7 +226,7 @@ * @return * Memory pool where data is located for given mbuf. */ -static struct rte_mempool * +static inline struct rte_mempool * mlx4_txq_mb2mp(struct rte_mbuf *buf) { if (unlikely(RTE_MBUF_INDIRECT(buf))) @@ -158,7 +267,7 @@ /* Add a new entry, register MR first. */ DEBUG("%p: discovered new memory pool \"%s\" (%p)", (void *)txq, mp->name, (void *)mp); - mr = mlx4_mp2mr(txq->priv->pd, mp); + mr = mlx4_mp2mr(txq->ctrl.priv->pd, mp); if (unlikely(mr == NULL)) { DEBUG("%p: unable to configure MR, ibv_reg_mr() failed.", (void *)txq); @@ -183,6 +292,124 @@ } /** + * Posts a single work request to a send queue. + * + * @param txq + * The Tx queue to post to. + * @param wr + * The work request to handle. + * @param bad_wr + * The wr in case that posting had failed. + * + * @return + * 0 - success, negative errno value otherwise and rte_errno is set. + */ +static inline int +mlx4_post_send(struct txq *txq, + struct rte_mbuf *pkt, + uint32_t send_flags) +{ + struct mlx4_wqe_ctrl_seg *ctrl; + struct mlx4_wqe_data_seg *dseg; + struct mlx4_sq *sq = &txq->msq; + uint32_t head_idx = sq->head & sq->txbb_cnt_mask; + uint32_t lkey; + uintptr_t addr; + int wqe_real_size; + int nr_txbbs; + int rc; + + /* Calculate the needed work queue entry size for this packet. */ + wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) + + pkt->nb_segs * sizeof(struct mlx4_wqe_data_seg); + nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size); + /* Check that there is room for this WQE in the send queue and + * that the WQE size is legal. + */ + if (likely(((sq->head - sq->tail) + nr_txbbs + + sq->headroom_txbbs >= sq->txbb_cnt) || + (nr_txbbs > MLX4_MAX_WQE_TXBBS))) { + rc = ENOSPC; + goto err; + } + /* Get the control and single-data entries of the WQE */ + ctrl = (struct mlx4_wqe_ctrl_seg *)mlx4_get_send_wqe(sq, head_idx); + dseg = (struct mlx4_wqe_data_seg *)(((char *)ctrl) + + sizeof(struct mlx4_wqe_ctrl_seg)); + /* + * Fill the data segment with buffer information. + */ + addr = rte_pktmbuf_mtod(pkt, uintptr_t); + rte_prefetch0((volatile void *)addr); + dseg->addr = rte_cpu_to_be_64(addr); + /* Memory region key for this memory pool. */ + lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(pkt)); + if (unlikely(lkey == (uint32_t)-1)) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR" + " association", (void *)txq); + /* + * Restamp entry in case of failure. + * Make sure that size is written correctly. + * Note that we give ownership to the SW, not the HW. + */ + ctrl->fence_size = (wqe_real_size >> 4) & 0x3f; + mlx4_txq_stamp_freed_wqe(sq, head_idx, + (sq->head & sq->txbb_cnt) ? 0 : 1); + rc = EFAULT; + goto err; + } + dseg->lkey = rte_cpu_to_be_32(lkey); + /* + * Need a barrier here before writing the byte_count field to + * make sure that all the data is visible before the + * byte_count field is set. Otherwise, if the segment begins + * a new cacheline, the HCA prefetcher could grab the 64-byte + * chunk and get a valid (!= * 0xffffffff) byte count but + * stale data, and end up sending the wrong data. + */ + rte_io_wmb(); + if (likely(pkt->data_len)) + dseg->byte_count = rte_cpu_to_be_32(pkt->data_len); + else + /* + * Zero length segment is treated as inline segment + * with zero data. + */ + dseg->byte_count = RTE_BE32(0x80000000); + /* + * Fill the control parameters for this packet. + * For raw Ethernet, the SOLICIT flag is used to indicate that no icrc + * should be calculated + */ + ctrl->srcrb_flags = + rte_cpu_to_be_32(MLX4_WQE_CTRL_SOLICIT | + (send_flags & MLX4_WQE_CTRL_CQ_UPDATE)); + ctrl->fence_size = (wqe_real_size >> 4) & 0x3f; + /* + * The caller should prepare "imm" in advance in order to support + * VF to VF communication (when the device is a virtual-function + * device (VF)). + */ + ctrl->imm = 0; + /* + * Make sure descriptor is fully written before + * setting ownership bit (because HW can start + * executing as soon as we do). + */ + rte_wmb(); + ctrl->owner_opcode = + rte_cpu_to_be_32(MLX4_OPCODE_SEND | + ((sq->head & sq->txbb_cnt) ? + MLX4_BIT_WQE_OWN : 0)); + sq->head += nr_txbbs; + return 0; +err: + rte_errno = rc; + return -rc; +} + +/** * DPDK callback for Tx. * * @param dpdk_txq @@ -199,13 +426,11 @@ mlx4_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) { struct txq *txq = (struct txq *)dpdk_txq; - struct ibv_send_wr *wr_head = NULL; - struct ibv_send_wr **wr_next = &wr_head; - struct ibv_send_wr *wr_bad = NULL; unsigned int elts_head = txq->elts_head; const unsigned int elts_n = txq->elts_n; unsigned int elts_comp_cd = txq->elts_comp_cd; unsigned int elts_comp = 0; + unsigned int bytes_sent = 0; unsigned int i; unsigned int max; int err; @@ -229,9 +454,7 @@ (((elts_head + 1) == elts_n) ? 0 : elts_head + 1); struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; struct txq_elt *elt = &(*txq->elts)[elts_head]; - struct ibv_send_wr *wr = &elt->wr; unsigned int segs = buf->nb_segs; - unsigned int sent_size = 0; uint32_t send_flags = 0; /* Clean up old buffer. */ @@ -254,93 +477,43 @@ if (unlikely(--elts_comp_cd == 0)) { elts_comp_cd = txq->elts_comp_cd_init; ++elts_comp; - send_flags |= IBV_SEND_SIGNALED; + send_flags |= MLX4_WQE_CTRL_CQ_UPDATE; } if (likely(segs == 1)) { - struct ibv_sge *sge = &elt->sge; - uintptr_t addr; - uint32_t length; - uint32_t lkey; - - /* Retrieve buffer information. */ - addr = rte_pktmbuf_mtod(buf, uintptr_t); - length = buf->data_len; - /* Retrieve memory region key for this memory pool. */ - lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf)); - if (unlikely(lkey == (uint32_t)-1)) { - /* MR does not exist. */ - DEBUG("%p: unable to get MP <-> MR" - " association", (void *)txq); - /* Clean up Tx element. */ + /* Update element. */ + elt->buf = buf; + RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); + /* post the pkt for sending */ + err = mlx4_post_send(txq, buf, send_flags); + if (unlikely(err)) { + if (unlikely(send_flags & + MLX4_WQE_CTRL_CQ_UPDATE)) { + elts_comp_cd = 1; + --elts_comp; + } elt->buf = NULL; goto stop; } - /* Update element. */ elt->buf = buf; - if (txq->priv->vf) - rte_prefetch0((volatile void *) - (uintptr_t)addr); - RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); - sge->addr = addr; - sge->length = length; - sge->lkey = lkey; - sent_size += length; + bytes_sent += buf->pkt_len; } else { - err = -1; + err = -EINVAL; + rte_errno = -err; goto stop; } - if (sent_size <= txq->max_inline) - send_flags |= IBV_SEND_INLINE; elts_head = elts_head_next; - /* Increment sent bytes counter. */ - txq->stats.obytes += sent_size; - /* Set up WR. */ - wr->sg_list = &elt->sge; - wr->num_sge = segs; - wr->opcode = IBV_WR_SEND; - wr->send_flags = send_flags; - *wr_next = wr; - wr_next = &wr->next; } stop: /* Take a shortcut if nothing must be sent. */ if (unlikely(i == 0)) return 0; - /* Increment sent packets counter. */ + /* Increment send statistics counters. */ txq->stats.opackets += i; + txq->stats.obytes += bytes_sent; + /* Make sure that descriptors are written before doorbell record. */ + rte_wmb(); /* Ring QP doorbell. */ - *wr_next = NULL; - assert(wr_head); - err = ibv_post_send(txq->qp, wr_head, &wr_bad); - if (unlikely(err)) { - uint64_t obytes = 0; - uint64_t opackets = 0; - - /* Rewind bad WRs. */ - while (wr_bad != NULL) { - int j; - - /* Force completion request if one was lost. */ - if (wr_bad->send_flags & IBV_SEND_SIGNALED) { - elts_comp_cd = 1; - --elts_comp; - } - ++opackets; - for (j = 0; j < wr_bad->num_sge; ++j) - obytes += wr_bad->sg_list[j].length; - elts_head = (elts_head ? elts_head : elts_n) - 1; - wr_bad = wr_bad->next; - } - txq->stats.opackets -= opackets; - txq->stats.obytes -= obytes; - i -= opackets; - DEBUG("%p: ibv_post_send() failed, %" PRIu64 " packets" - " (%" PRIu64 " bytes) rejected: %s", - (void *)txq, - opackets, - obytes, - (err <= -1) ? "Internal error" : strerror(err)); - } + rte_write32(txq->msq.doorbell_qpn, txq->msq.db); txq->elts_head = elts_head; txq->elts_comp += elts_comp; txq->elts_comp_cd = elts_comp_cd; diff --git a/drivers/net/mlx4/mlx4_rxtx.h b/drivers/net/mlx4/mlx4_rxtx.h index fec998a..b515472 100644 --- a/drivers/net/mlx4/mlx4_rxtx.h +++ b/drivers/net/mlx4/mlx4_rxtx.h @@ -40,6 +40,7 @@ #ifdef PEDANTIC #pragma GCC diagnostic ignored "-Wpedantic" #endif +#include #include #ifdef PEDANTIC #pragma GCC diagnostic error "-Wpedantic" @@ -50,6 +51,7 @@ #include #include "mlx4.h" +#include "mlx4_prm.h" /** Rx queue counters. */ struct mlx4_rxq_stats { @@ -85,8 +87,6 @@ struct rxq { /** Tx element. */ struct txq_elt { - struct ibv_send_wr wr; /* Work request. */ - struct ibv_sge sge; /* Scatter/gather element. */ struct rte_mbuf *buf; /**< Buffer. */ }; @@ -100,24 +100,28 @@ struct mlx4_txq_stats { /** Tx queue descriptor. */ struct txq { - struct priv *priv; /**< Back pointer to private data. */ - struct { - const struct rte_mempool *mp; /**< Cached memory pool. */ - struct ibv_mr *mr; /**< Memory region (for mp). */ - uint32_t lkey; /**< mr->lkey copy. */ - } mp2mr[MLX4_PMD_TX_MP_CACHE]; /**< MP to MR translation table. */ - struct ibv_cq *cq; /**< Completion queue. */ - struct ibv_qp *qp; /**< Queue pair. */ - uint32_t max_inline; /**< Max inline send size. */ - unsigned int elts_n; /**< (*elts)[] length. */ - struct txq_elt (*elts)[]; /**< Tx elements. */ + struct mlx4_sq msq; /**< Info for directly manipulating the SQ. */ + struct mlx4_cq mcq; /**< Info for directly manipulating the CQ. */ unsigned int elts_head; /**< Current index in (*elts)[]. */ unsigned int elts_tail; /**< First element awaiting completion. */ unsigned int elts_comp; /**< Number of completion requests. */ unsigned int elts_comp_cd; /**< Countdown for next completion. */ unsigned int elts_comp_cd_init; /**< Initial value for countdown. */ + unsigned int elts_n; /**< (*elts)[] length. */ + struct txq_elt (*elts)[]; /**< Tx elements. */ struct mlx4_txq_stats stats; /**< Tx queue counters. */ - unsigned int socket; /**< CPU socket ID for allocations. */ + uint32_t max_inline; /**< Max inline send size. */ + struct { + const struct rte_mempool *mp; /**< Cached memory pool. */ + struct ibv_mr *mr; /**< Memory region (for mp). */ + uint32_t lkey; /**< mr->lkey copy. */ + } mp2mr[MLX4_PMD_TX_MP_CACHE]; /**< MP to MR translation table. */ + struct { + struct priv *priv; /**< Back pointer to private data. */ + unsigned int socket; /**< CPU socket ID for allocations. */ + struct ibv_cq *cq; /**< Completion queue. */ + struct ibv_qp *qp; /**< Queue pair. */ + } ctrl; }; /* mlx4_rxq.c */ diff --git a/drivers/net/mlx4/mlx4_txq.c b/drivers/net/mlx4/mlx4_txq.c index e0245b0..492779f 100644 --- a/drivers/net/mlx4/mlx4_txq.c +++ b/drivers/net/mlx4/mlx4_txq.c @@ -62,6 +62,7 @@ #include "mlx4_autoconf.h" #include "mlx4_rxtx.h" #include "mlx4_utils.h" +#include "mlx4_prm.h" /** * Allocate Tx queue elements. @@ -79,7 +80,7 @@ { unsigned int i; struct txq_elt (*elts)[elts_n] = - rte_calloc_socket("TXQ", 1, sizeof(*elts), 0, txq->socket); + rte_calloc_socket("TXQ", 1, sizeof(*elts), 0, txq->ctrl.socket); int ret = 0; if (elts == NULL) { @@ -170,10 +171,10 @@ DEBUG("cleaning up %p", (void *)txq); mlx4_txq_free_elts(txq); - if (txq->qp != NULL) - claim_zero(ibv_destroy_qp(txq->qp)); - if (txq->cq != NULL) - claim_zero(ibv_destroy_cq(txq->cq)); + if (txq->ctrl.qp != NULL) + claim_zero(ibv_destroy_qp(txq->ctrl.qp)); + if (txq->ctrl.cq != NULL) + claim_zero(ibv_destroy_cq(txq->ctrl.cq)); for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) { if (txq->mp2mr[i].mp == NULL) break; @@ -242,6 +243,42 @@ struct txq_mp2mr_mbuf_check_data { } /** + * Retrieves information needed in order to directly access the Tx queue. + * + * @param txq + * Pointer to Tx queue structure. + * @param mlxdv + * Pointer to device information for this Tx queue. + */ +static void +mlx4_txq_fill_dv_obj_info(struct txq *txq, struct mlx4dv_obj *mlxdv) +{ + struct mlx4_sq *sq = &txq->msq; + struct mlx4_cq *cq = &txq->mcq; + struct mlx4dv_qp *dqp = mlxdv->qp.out; + struct mlx4dv_cq *dcq = mlxdv->cq.out; + /* Total sq length, including headroom and spare WQEs*/ + uint32_t sq_size = (uint32_t)dqp->rq.offset - (uint32_t)dqp->sq.offset; + + sq->buf = ((char *)dqp->buf.buf) + dqp->sq.offset; + /* Total len, including headroom and spare WQEs*/ + sq->eob = sq->buf + sq_size; + sq->head = 0; + sq->tail = 0; + sq->txbb_cnt = + (dqp->sq.wqe_cnt << dqp->sq.wqe_shift) >> MLX4_TXBB_SHIFT; + sq->txbb_cnt_mask = sq->txbb_cnt - 1; + sq->db = dqp->sdb; + sq->doorbell_qpn = dqp->doorbell_qpn; + sq->headroom_txbbs = + (2048 + (1 << dqp->sq.wqe_shift)) >> MLX4_TXBB_SHIFT; + cq->buf = dcq->buf.buf; + cq->cqe_cnt = dcq->cqe_cnt; + cq->set_ci_db = dcq->set_ci_db; + cq->cqe_64 = (dcq->cqe_size & 64) ? 1 : 0; +} + +/** * Configure a Tx queue. * * @param dev @@ -263,9 +300,15 @@ struct txq_mp2mr_mbuf_check_data { unsigned int socket, const struct rte_eth_txconf *conf) { struct priv *priv = dev->data->dev_private; + struct mlx4dv_obj mlxdv; + struct mlx4dv_qp dv_qp; + struct mlx4dv_cq dv_cq; + struct txq tmpl = { - .priv = priv, - .socket = socket + .ctrl = { + .priv = priv, + .socket = socket + }, }; union { struct ibv_qp_init_attr init; @@ -284,8 +327,8 @@ struct txq_mp2mr_mbuf_check_data { goto error; } /* MRs will be registered in mp2mr[] later. */ - tmpl.cq = ibv_create_cq(priv->ctx, desc, NULL, NULL, 0); - if (tmpl.cq == NULL) { + tmpl.ctrl.cq = ibv_create_cq(priv->ctx, desc, NULL, NULL, 0); + if (tmpl.ctrl.cq == NULL) { rte_errno = ENOMEM; ERROR("%p: CQ creation failure: %s", (void *)dev, strerror(rte_errno)); @@ -297,9 +340,9 @@ struct txq_mp2mr_mbuf_check_data { priv->device_attr.max_sge); attr.init = (struct ibv_qp_init_attr){ /* CQ to be associated with the send queue. */ - .send_cq = tmpl.cq, + .send_cq = tmpl.ctrl.cq, /* CQ to be associated with the receive queue. */ - .recv_cq = tmpl.cq, + .recv_cq = tmpl.ctrl.cq, .cap = { /* Max number of outstanding WRs. */ .max_send_wr = ((priv->device_attr.max_qp_wr < desc) ? @@ -316,8 +359,8 @@ struct txq_mp2mr_mbuf_check_data { */ .sq_sig_all = 0, }; - tmpl.qp = ibv_create_qp(priv->pd, &attr.init); - if (tmpl.qp == NULL) { + tmpl.ctrl.qp = ibv_create_qp(priv->pd, &attr.init); + if (tmpl.ctrl.qp == NULL) { rte_errno = errno ? errno : EINVAL; ERROR("%p: QP creation failure: %s", (void *)dev, strerror(rte_errno)); @@ -331,7 +374,8 @@ struct txq_mp2mr_mbuf_check_data { /* Primary port number. */ .port_num = priv->port }; - ret = ibv_modify_qp(tmpl.qp, &attr.mod, IBV_QP_STATE | IBV_QP_PORT); + ret = ibv_modify_qp(tmpl.ctrl.qp, &attr.mod, + IBV_QP_STATE | IBV_QP_PORT); if (ret) { rte_errno = ret; ERROR("%p: QP state to IBV_QPS_INIT failed: %s", @@ -348,7 +392,7 @@ struct txq_mp2mr_mbuf_check_data { attr.mod = (struct ibv_qp_attr){ .qp_state = IBV_QPS_RTR }; - ret = ibv_modify_qp(tmpl.qp, &attr.mod, IBV_QP_STATE); + ret = ibv_modify_qp(tmpl.ctrl.qp, &attr.mod, IBV_QP_STATE); if (ret) { rte_errno = ret; ERROR("%p: QP state to IBV_QPS_RTR failed: %s", @@ -356,7 +400,7 @@ struct txq_mp2mr_mbuf_check_data { goto error; } attr.mod.qp_state = IBV_QPS_RTS; - ret = ibv_modify_qp(tmpl.qp, &attr.mod, IBV_QP_STATE); + ret = ibv_modify_qp(tmpl.ctrl.qp, &attr.mod, IBV_QP_STATE); if (ret) { rte_errno = ret; ERROR("%p: QP state to IBV_QPS_RTS failed: %s", @@ -370,6 +414,18 @@ struct txq_mp2mr_mbuf_check_data { DEBUG("%p: txq updated with %p", (void *)txq, (void *)&tmpl); /* Pre-register known mempools. */ rte_mempool_walk(mlx4_txq_mp2mr_iter, txq); + /* Retrieve device Q info */ + mlxdv.cq.in = txq->ctrl.cq; + mlxdv.cq.out = &dv_cq; + mlxdv.qp.in = txq->ctrl.qp; + mlxdv.qp.out = &dv_qp; + ret = mlx4dv_init_obj(&mlxdv, MLX4DV_OBJ_QP | MLX4DV_OBJ_CQ); + if (ret) { + ERROR("%p: Failed to obtain information needed for " + "accessing the device queues", (void *)dev); + goto error; + } + mlx4_txq_fill_dv_obj_info(txq, &mlxdv); return 0; error: ret = rte_errno; @@ -459,7 +515,7 @@ struct txq_mp2mr_mbuf_check_data { if (txq == NULL) return; - priv = txq->priv; + priv = txq->ctrl.priv; for (i = 0; i != priv->dev->data->nb_tx_queues; ++i) if (priv->dev->data->tx_queues[i] == txq) { DEBUG("%p: removing Tx queue %p from list", diff --git a/mk/rte.app.mk b/mk/rte.app.mk index c25fdd9..2f1286e 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -128,7 +128,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_KNI),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KNI) += -lrte_pmd_kni endif _LDLIBS-$(CONFIG_RTE_LIBRTE_LIO_PMD) += -lrte_pmd_lio -_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -libverbs +_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 -libverbs -lmlx4 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 -libverbs _LDLIBS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += -lrte_pmd_nfp _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NULL) += -lrte_pmd_null