From patchwork Thu Aug 24 15:54:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Moti Haimovsky X-Patchwork-Id: 27882 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 211557D6A; Thu, 24 Aug 2017 17:54:34 +0200 (CEST) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0055.outbound.protection.outlook.com [104.47.0.55]) by dpdk.org (Postfix) with ESMTP id BBBC77D3E for ; Thu, 24 Aug 2017 17:54:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=LOxWgPa8z33NYTUp/6A/1vlFqVSEgau+DQXARxIqLmM=; b=XDiAz5FrAqCcq3WGsD4GEkdn2c5AQigw0VzeDyZbf1cdWge9zVDAaILx7MCZenx1mYth45qN9lQXJ2OKze1Taccljye/Zv9Vmr+3OdtzSCNFDYJaN/birp5pHIfjqYYbtgttY6pNkCXbFqmXOp5OLpsSR6mBy2S06Fn+6H+JPKY= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=motih@mellanox.com; Received: from mellanox.com (37.142.13.130) by AM4PR05MB1907.eurprd05.prod.outlook.com (2603:10a6:200:15::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1362.18; Thu, 24 Aug 2017 15:54:28 +0000 From: Moti Haimovsky To: adrien.mazarguil@6wind.com Cc: dev@dpdk.org, Moti Haimovsky Date: Thu, 24 Aug 2017 18:54:07 +0300 Message-Id: <1503590050-196143-3-git-send-email-motih@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1503590050-196143-1-git-send-email-motih@mellanox.com> References: <1503590050-196143-1-git-send-email-motih@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: VI1PR0302CA0006.eurprd03.prod.outlook.com (2603:10a6:800:e9::16) To AM4PR05MB1907.eurprd05.prod.outlook.com (2603:10a6:200:15::15) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 57cec9db-03a3-4582-2148-08d4eb086784 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(48565401081)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:AM4PR05MB1907; X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB1907; 3:+aBDZafB+yDyia7bTGpWG6E9S1Y0dxN3PNR84/Ckg83HEAV8tLNkTl7LTkdc9Ij6hqO6HjybDY5nnwJIcUgkf6NlZgKmTkFDLgyMdFrMq8fz12lRpMGsgPhgP9NocauLROn5hJDncG8sSUAGJ3GlynGNEzzSiPuBw4H1osgiHTuGyIGHoc6DsnzZg2p3x2crQgCrY6zFdqVHjJTyQV7ilFLblKWYAjO1j/A6Pa65SjatfRbsA0LtyzUtx4YqJ4fb; 25:nx+qSbEpRcTqpjfdemyr7btxJH3mg7z9glRQI3tRgqHcpGWyBOigBKjiwi/zOUBWReF3WILAnfXLiCHVCwW7F8mBwjFbbuXhjgMAzgtAiMjH1HIoddH8tI12qdi2BbKGV1W0uypgtxAP9HatR4GUJIePQ+6zI6NFdOEnOqCCi7rIXfuodCVKYqWyJiWW/JCHhEP2D70gNo9iG55nlz00pozgO9nYEJ4qI9PLAK5QDdCqyhMCnzZeQ+1trLQBBpX+kApz9a0o6TT2y8GpsPzUCNL+BrU6mmpD/CtW1Vca59bS1F0cYb6350ounGPcP3/WbZ6agFX3vrLj9Dl4Lqohxw==; 31:RocfxTfF23BD6ejipbaEHwayFYEhUoVIfc/sTdiuavq28NOw7eCic7QKTNCa3CFEZENo2CEFBivhDKLFvjXerW63eytaizDI7YUmho8Cw9HdMObIJlBTvg5gC516p4BDil3O48mywLuHAbdtTKFaLETZDV61hDaV5cik/iL30Hg4nneeUYVubvB1pR634WJDtjC3Ei3UFswuG6H3FIT44GoGPApO8XCwJBKzccprOak= X-MS-TrafficTypeDiagnostic: AM4PR05MB1907: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB1907; 20:l/aF3daR3YiIwKvSQ4eMogd4m0WGlTiFrXSTUtzOIzzTeF6PT7XeNL1PyzkhdVffmtNxgvjAio4NrCYmUEVVy/L0Urlh8/5tT6+TfQkCHdYss2JAGyVyhI/qiDnhMUlQndk9hBm/+S8vU6qM9nEVRbsTjU3F5cQICjNdnaStMYO3OEoSZE+6zzlJRjMm3PaV7aneixsmgVUtbSjd0n03YtDLueoorYsvYkPkOG+5nFvZoT5K6ho2IRJhgeinObRyKnKOGagRHh9moFisBVEti917ddwoww+3Rqn5I2h+f5YNMVtp7eWZk1Zs2zYH8/RqxdLkd4l1H7R+wp2z0tgPRgjp20U4FkXfYYD/mIn/3qte/1SaLVIRtOr81czLGVGWR1aMwZzzo8krIEXFF52RMz8D2w3PcwC8aEApZKK0lb3D41T6vdlwVuyNlDg1x2nZK0xuxugo4aM+J6GyToN+KVXEpCPVw4TZATh8bfZ0CT4d8WK9XLaS4IgtFdIynFo8; 4:JVIWrZioy2YU6vfWjPXmQ8eCsdojO3eY1fE6TYXcJjCwSqMD1ZdDvnnHqrBTCUZW6OGq9VFczs5xXqs8qfMNlgyCe4sK8pikNCGs1sbMm6Y+fW+HWZPgnUE6/JmcpJA44as5aPTmjDDtLmXNEgvcW6WPKhTZxMA2cFnuAv1gWFy1C9EYMH8KDq5bAB0Zx6oAqxD0VgtzSAC25LDoR7g17xXJCCANqKekWEvjNRtqr/EF63uD5KydNlk67SPl2OCeTI12I4xx0M1sJD8ff2Y1hJdCJf932C7ozxKQ2fRBmBw= X-Exchange-Antispam-Report-Test: UriScan:(60795455431006); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(3002001)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(20161123562025)(20161123555025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:AM4PR05MB1907; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:AM4PR05MB1907; X-Forefront-PRVS: 04097B7F7F X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(7370300001)(6009001)(39860400002)(199003)(189002)(22813001)(66066001)(47776003)(5660300001)(50226002)(2361001)(50986999)(81166006)(8676002)(69596002)(33026002)(478600001)(105586002)(7736002)(42186005)(76176999)(106356001)(305945005)(33646002)(4326008)(81156014)(2906002)(2351001)(101416001)(21086003)(48376002)(25786009)(189998001)(2950100002)(4720700003)(86362001)(6916009)(575784001)(55016002)(6666003)(53936002)(50466002)(107886003)(110136004)(7350300001)(6116002)(36756003)(5003940100001)(97736004)(3846002)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR05MB1907; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM4PR05MB1907; 23:42QA4LHxL80v1QeG+PPK0Nz4XjD6WPF48Oumjencj?= u2DKlk0iPeUx//OEnyzFt3++GdDcxJJvxUIeFEyOaEs94DBzxOdYizy/BzjLgT+XVRLEWT8U/HJCuQNgRdtfI6N6QlMDXv3Iaov2aK+5/j0lWpMtxYnhVFtl9n8Y/rmcsXLKAjzxzVhDOwYScBZhofdk0untvbAPxWoUPmxFrM1U+XhgfkZ/fSwZcrMXnZotU4wb2tURxdRIGspTdUbdCjIyYx2o6lYti13zQ3j8a2RxvXsJVucjWOrK8y8gj9JmjNlS2AXe/hVJcQcyBi/o45zkJ1kL7BGvj8Mxhnlf38xfqTeB+tRLHY9DaLx03eHbAXcTw1l2LMu+U9MKTJ3ME9Jbs93QCGNfUVc1aeYgiXG2iqtRoK1Hb5a7FtYwhFbbvtmDxHJSMkTVH16ExmsF4URJQPCfLTCB70gF1hEOtKe2PD6Rm5NhKJ0RXtgTby0FvvBI+9fAFJvINe1LjB48+XPCCJVQ3YZzI6bwDOvInUCSU2uY5IyPb79Z/4hnkJBHT48/kXQLFaKdfBongQimOvkpmFqzj/5zTA6bLQzG0dG8FxkuI4sdhoZSi8kRvGg7seHXCOFr8Tha8UufZNheAV7r2f06/vNn5pNhmLRY4N1h8TnQNKn6uK4Hb8+2WNv1fgk2Rr0CQ3xoswHYLHsO4l6Pj5QhIy+IC74lVGK/uQMgS4/P5cTc710QAHDHjEe7r4zvI704p2tPUmeD5GWOIWotVlLQVIywsYtKMAUGQdZUIh0Wey6UIkfZnzXX3i2qpHPQwKtSgncMaPKXA0vL7RIEqOSLIYQ7eG5gb6j3RM6Nz4KQ2IkMXcAMGUJ8Lzuohdp6UTUBgjIcdZ10swhhIGYQSX40lZPZYTUIcRzJZShnOL6ncdSdktAlnKsvc9uIsilP9eNbePXxXSzXXHOVrvwKH+iWQfIoae2c688PNk5hxvabvmCfDz0A4jDS1YzGmG+3UNL/Og5/MgtZN5CA24FZBHvPg/0rcPZcld1hgw28Jq72XsM/i4b1vAi5EIKcBovl1E2dMs6z5dCuob/glkaCizkav1cEaWKc0Xl8NBqTb5PrtEUiugzoU3rtAx9ZmbHrJUACz6lhqo+YHH5DOwotFG9rGe1SeMVKIF7xiAvj2u2c5Z9b/YyRCDQw37CU/xhEnRSga+xx3GlwDT2qCH45QLFKcthO5TYr8ayVDN84k+RFp39uvJwZ7S1YcTXDCIlRPZkGfxUyREQg0QcY0RY X-Microsoft-Exchange-Diagnostics: 1; AM4PR05MB1907; 6:o0RKC32eYRuZYL4j5/tA9fX0zAUyCokkEhAa7Q8j4N20RrDdOdT8H6VK5k3NcrtcrdV7Cc5sLldYQ5gDBh0Pe932Dwu6uZHu6gkNdsh4eR6rCxwidn7hmwaQdcV1UBv7CAmadj3WAb87ZKVZhzca85FZjtO9tXgAb+A87gaVgA+/y8i3w2ChyfsbjT9XyFHYLc6eo6W58m5CDhtkxAeERfC5L88q/0DlIrF35CNBBqMoSyeQncWr9XeyqJpYiZiyFgxEtvUXNLG4EqAgu9evu7nl1jxmSeXzRtLyozZlsG+F0mRlsWJcVn6jBk3zLWcMuu52GAWzj+xnQ25gp/5KYw==; 5:YethAwYrpVLgtUOdzgTIx84CAsA/y3kq7rhM6ZnQ+FtYFZcJEvEqyBCpGthQtAUJ6DunfZZkxad4sT9hRk3vC8I8HHIC1/s7Vr4IUwEQtpbk8U/MKIdN4NGwN9a5HlzaXM+pWJJs3MhTJigztmylgA==; 24:jRSyEUzIOGwdw1cvNzKUTv+6WUv1xR9EaXeHmx5uDcsq2PIJm2dRhHlqBfLMC2+j1cizK3Vqm9oJTutWjsWDi4VOFDK7gU/VT672HRkgzOs=; 7:ToNZ7Pq1U1iDDkSwsC8VVrqiNokidNDW3v6TSwZJ5W7oErlmiUnokoa4bAJ7Qh0lcj2Ar24+7BAspGbC8P32E89WAyi/kCL6d214Y1fFUPGNEbLi9OvRYbmpqu8ZtIwGr/JBwHFqo/hP7IBo93XoI7kLV0PMdby5QzlXSBUX3ePFZ04RDg0Ldy2kiXEHP1IshrrdCfcUcZYRmfxsmtGUpwpzT+w+UWTiZ0FyjnPvMAM= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Aug 2017 15:54:28.8029 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB1907 Subject: [dpdk-dev] [PATCH 2/5] net/mlx4: support multi-segments Tx X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" PRM now supports transmitting packets spanning over arbitrary amount of buffers. Signed-off-by: Moti Haimovsky --- drivers/net/mlx4/mlx4_prm.h | 16 +--- drivers/net/mlx4/mlx4_rxtx.c | 213 +++++++++++++++++++++++++++++++------------ drivers/net/mlx4/mlx4_rxtx.h | 3 +- drivers/net/mlx4/mlx4_txq.c | 12 ++- 4 files changed, 170 insertions(+), 74 deletions(-) diff --git a/drivers/net/mlx4/mlx4_prm.h b/drivers/net/mlx4/mlx4_prm.h index c5ce33b..8b0248a 100644 --- a/drivers/net/mlx4/mlx4_prm.h +++ b/drivers/net/mlx4/mlx4_prm.h @@ -61,7 +61,7 @@ #define MLX4_OPCODE_SEND 0x0a #define MLX4_EN_BIT_WQE_OWN 0x80000000 -#define SIZE_TO_TXBBS(size) (RTE_ALIGN((size), (TXBB_SIZE)) / (TXBB_SIZE)) +#define SIZE_TO_TXBBS(size) (RTE_ALIGN((size), (TXBB_SIZE)) / (TXBB_SIZE)) /** * Update the HW with the new CQ consumer value. @@ -148,6 +148,7 @@ /** * Fills the ctrl segment of a WQE with info needed for transmitting the packet. + * Owner field is filled later. * * @param seg * Pointer to the control structure in the WQE. @@ -161,8 +162,8 @@ * Immediate data/Invalidation key.. */ static inline void -mlx4_set_ctrl_seg(struct mlx4_wqe_ctrl_seg *seg, uint32_t owner, - uint8_t fence_size, uint32_t srcrb_flags, uint32_t imm) +mlx4_set_ctrl_seg(struct mlx4_wqe_ctrl_seg *seg, uint8_t fence_size, + uint32_t srcrb_flags, uint32_t imm) { seg->fence_size = fence_size; seg->srcrb_flags = rte_cpu_to_be_32(srcrb_flags); @@ -173,13 +174,6 @@ * For the IBV_WR_SEND_WITH_INV, it should be htobe32(imm). */ seg->imm = imm; - /* - * Make sure descriptor is fully written before - * setting ownership bit (because HW can start - * executing as soon as we do). - */ - rte_wmb(); - seg->owner_opcode = rte_cpu_to_be_32(owner); } /** @@ -241,7 +235,7 @@ * The number of data-segments the WQE contains. * * @return - * WQE size in bytes. + * The calculated WQE size in bytes. */ static inline int mlx4_wqe_calc_real_size(unsigned int count) diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c index 0720e34..e41ea9e 100644 --- a/drivers/net/mlx4/mlx4_rxtx.c +++ b/drivers/net/mlx4/mlx4_rxtx.c @@ -309,6 +309,101 @@ } /** + * Copy a WQE written in the bounce buffer back to the SQ. + * Routine is used when a WQE wraps-around the SQ and therefore needs a + * special attention. note that the WQE is written backward to the SQ. + * + * @param txq + * Pointer to mlx4 Tx queue structure. + * @param index + * First SQ TXBB index for this WQE. + * @param desc_size + * TXBB-aligned sixe of the WQE. + * + * @return + * A pointer to the control segment of this WQE in the SQ. + */ +static struct mlx4_wqe_ctrl_seg +*mlx4_bounce_to_desc(struct txq *txq, + uint32_t index, + unsigned int desc_size) +{ + struct mlx4_sq *sq = &txq->msq; + uint32_t copy = (sq->txbb_cnt - index) * TXBB_SIZE; + int i; + + for (i = desc_size - copy - 4; i >= 0; i -= 4) { + if ((i & (TXBB_SIZE - 1)) == 0) + rte_wmb(); + *((uint32_t *)(sq->buf + i)) = + *((uint32_t *)(txq->bounce_buf + copy + i)); + } + for (i = copy - 4; i >= 4; i -= 4) { + if ((i & (TXBB_SIZE - 1)) == 0) + rte_wmb(); + *((uint32_t *)(sq->buf + index * TXBB_SIZE + i)) = + *((uint32_t *)(txq->bounce_buf + i)); + } + /* Return real descriptor location */ + return (struct mlx4_wqe_ctrl_seg *)(sq->buf + index * TXBB_SIZE); +} + +/** + * Handle address translation of scattered buffers for mlx4_tx_burst(). + * + * @param txq + * TX queue structure. + * @param segs + * Number of segments in buf. + * @param elt + * TX queue element to fill. + * @param[in] buf + * Buffer to process. + * @param elts_head + * Index of the linear buffer to use if necessary (normally txq->elts_head). + * @param[out] sges + * Array filled with SGEs on success. + * + * @return + * A structure containing the processed packet size in bytes and the + * number of SGEs. Both fields are set to (unsigned int)-1 in case of + * failure. + */ +static inline int +mlx4_tx_sg_virt_to_lkey(struct txq *txq, struct rte_mbuf *buf, + struct ibv_sge *sges, unsigned int segs) +{ + unsigned int j; + + /* Register segments as SGEs. */ + for (j = 0; (j != segs); ++j) { + struct ibv_sge *sge = &sges[j]; + uint32_t lkey; + + /* Retrieve Memory Region key for this memory pool. */ + lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf)); + if (unlikely(lkey == (uint32_t)-1)) { + /* MR does not exist. */ + DEBUG("%p: unable to get MP <-> MR association", + (void *)txq); + goto stop; + } + /* Update SGE. */ + sge->addr = rte_pktmbuf_mtod(buf, uintptr_t); + if (txq->priv->vf) + rte_prefetch0((volatile void *) + (uintptr_t)sge->addr); + sge->length = buf->data_len; + sge->lkey = lkey; + buf = buf->next; + } + return 0; +stop: + return -1; +} + + +/** * Posts a single work requests to a send queue. * * @param txq @@ -323,36 +418,53 @@ */ static int mlx4_post_send(struct txq *txq, + struct rte_mbuf *pkt, struct ibv_send_wr *wr, struct ibv_send_wr **bad_wr) { struct mlx4_wqe_ctrl_seg *ctrl; struct mlx4_wqe_data_seg *dseg; struct mlx4_sq *sq = &txq->msq; + struct ibv_sge sge[wr->num_sge]; uint32_t srcrb_flags; uint8_t fence_size; uint32_t head_idx = sq->head & sq->txbb_cnt_mask; uint32_t owner_opcode; - int wqe_real_size, nr_txbbs; + int wqe_real_size, wqe_size, nr_txbbs, i; + bool bounce = FALSE; - /* for now we support pkts with one buf only */ - if (wr->num_sge != 1) + if (unlikely(mlx4_tx_sg_virt_to_lkey(txq, pkt, sge, wr->num_sge))) goto err; + wr->sg_list = sge; /* Calc the needed wqe size for this packet */ wqe_real_size = mlx4_wqe_calc_real_size(wr->num_sge); if (unlikely(!wqe_real_size)) goto err; + wqe_size = RTE_ALIGN(wqe_real_size, TXBB_SIZE); nr_txbbs = SIZE_TO_TXBBS(wqe_real_size); /* Are we too big to handle ? */ if (unlikely(mlx4_wq_overflow(sq, nr_txbbs))) goto err; - /* Get ctrl and single-data wqe entries */ - ctrl = mlx4_get_send_wqe(sq, head_idx); + /* Get ctrl entry */ + if (likely(head_idx + nr_txbbs <= sq->txbb_cnt)) { + ctrl = mlx4_get_send_wqe(sq, head_idx); + } else { + /* handle the case of wqe wraps around the SQ by working with + * a side-buf and when done copying it back to the SQ + */ + ctrl = (struct mlx4_wqe_ctrl_seg *)txq->bounce_buf; + bounce = TRUE; + } + /* Get data-seg entry */ dseg = (struct mlx4_wqe_data_seg *)(((char *)ctrl) + sizeof(struct mlx4_wqe_ctrl_seg)); - mlx4_set_data_seg(dseg, wr->sg_list); - /* For raw eth, the SOLICIT flag is used - * to indicate that no icrc should be calculated + /* Fill-in date from last to first */ + for (i = wr->num_sge - 1; i >= 0; --i) + mlx4_set_data_seg(dseg + i, wr->sg_list + i); + /* Handle control info + * + * For raw eth, the SOLICIT flag is used to indicate that + * no icrc should be calculated */ srcrb_flags = MLX4_WQE_CTRL_SOLICIT | ((wr->send_flags & IBV_SEND_SIGNALED) ? @@ -361,7 +473,19 @@ MLX4_WQE_CTRL_FENCE : 0) | ((wqe_real_size / 16) & 0x3f); owner_opcode = MLX4_OPCODE_SEND | ((sq->head & sq->txbb_cnt) ? MLX4_EN_BIT_WQE_OWN : 0); - mlx4_set_ctrl_seg(ctrl, owner_opcode, fence_size, srcrb_flags, 0); + /* fill in ctrl info but ownership */ + mlx4_set_ctrl_seg(ctrl, fence_size, srcrb_flags, 0); + /* If we used a bounce buffer then copy wqe back into sq */ + if (unlikely(bounce)) + ctrl = mlx4_bounce_to_desc(txq, head_idx, wqe_size); + /* + * Make sure descriptor is fully written before + * setting ownership bit (because HW can start + * executing as soon as we do). + */ + rte_wmb(); + ctrl->owner_opcode = rte_cpu_to_be_32(owner_opcode); + sq->head += nr_txbbs; rte_wmb(); return 0; @@ -439,62 +563,31 @@ /* Request Tx completion. */ if (unlikely(--elts_comp_cd == 0)) { elts_comp_cd = txq->elts_comp_cd_init; - ++elts_comp; send_flags |= IBV_SEND_SIGNALED; } - if (likely(segs == 1)) { - struct ibv_sge *sge = &elt->sge; - uintptr_t addr; - uint32_t length; - uint32_t lkey; - - /* Retrieve buffer information. */ - addr = rte_pktmbuf_mtod(buf, uintptr_t); - length = buf->data_len; - /* Retrieve memory region key for this memory pool. */ - lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf)); - if (unlikely(lkey == (uint32_t)-1)) { - /* MR does not exist. */ - DEBUG("%p: unable to get MP <-> MR" - " association", (void *)txq); - /* Clean up Tx element. */ - elt->buf = NULL; - goto stop; - } - if (buf->pkt_len <= txq->max_inline) - send_flags |= IBV_SEND_INLINE; - /* Update element. */ - elt->buf = buf; - if (txq->priv->vf) - rte_prefetch0((volatile void *) - (uintptr_t)addr); + if (buf->pkt_len <= txq->max_inline) + send_flags |= IBV_SEND_INLINE; + /* Update element. */ + elt->buf = buf; + if (txq->priv->vf) RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf); - sge->addr = addr; - sge->length = length; - sge->lkey = lkey; - sent_size += length; - /* Set up WR. */ - wr->sg_list = sge; - wr->num_sge = segs; - wr->opcode = IBV_WR_SEND; - wr->send_flags = send_flags; - wr->next = NULL; - /* post the pkt for sending */ - err = mlx4_post_send(txq, wr, &wr_bad); - if (unlikely(err)) { - if (unlikely(wr_bad->send_flags & - IBV_SEND_SIGNALED)) { - elts_comp_cd = 1; - --elts_comp; - } - elt->buf = NULL; - goto stop; - } - sent_size += length; - } else { - err = -1; + /* Set up WR. */ + wr->sg_list = NULL; /* handled in post_send */ + wr->num_sge = segs; + wr->opcode = IBV_WR_SEND; + wr->send_flags = send_flags; + wr->next = NULL; + /* post the pkt for sending */ + err = mlx4_post_send(txq, buf, wr, &wr_bad); + if (unlikely(err)) { + if (unlikely(wr_bad->send_flags & + IBV_SEND_SIGNALED)) + elts_comp_cd = 1; + elt->buf = NULL; goto stop; } + ++elts_comp; + sent_size += buf->pkt_len; elts_head = elts_head_next; /* Increment sent bytes counter. */ txq->stats.obytes += sent_size; diff --git a/drivers/net/mlx4/mlx4_rxtx.h b/drivers/net/mlx4/mlx4_rxtx.h index e442730..7cae7e2 100644 --- a/drivers/net/mlx4/mlx4_rxtx.h +++ b/drivers/net/mlx4/mlx4_rxtx.h @@ -139,13 +139,14 @@ struct txq { struct txq_elt (*elts)[]; /**< Tx elements. */ unsigned int elts_head; /**< Current index in (*elts)[]. */ unsigned int elts_tail; /**< First element awaiting completion. */ - unsigned int elts_comp; /**< Number of completion requests. */ + unsigned int elts_comp; /**< Number of pkts waiting for completion. */ unsigned int elts_comp_cd; /**< Countdown for next completion. */ unsigned int elts_comp_cd_init; /**< Initial value for countdown. */ struct mlx4_txq_stats stats; /**< Tx queue counters. */ unsigned int socket; /**< CPU socket ID for allocations. */ struct mlx4_sq msq; /**< Info for directly manipulating the SQ. */ struct mlx4_cq mcq; /**< Info for directly manipulating the CQ. */ + char *bounce_buf; /**< Side memory to be used when wqe wraps around */ }; /* mlx4_rxq.c */ diff --git a/drivers/net/mlx4/mlx4_txq.c b/drivers/net/mlx4/mlx4_txq.c index 1273738..6f6ea9c 100644 --- a/drivers/net/mlx4/mlx4_txq.c +++ b/drivers/net/mlx4/mlx4_txq.c @@ -83,8 +83,14 @@ rte_calloc_socket("TXQ", 1, sizeof(*elts), 0, txq->socket); int ret = 0; - if (elts == NULL) { - ERROR("%p: can't allocate packets array", (void *)txq); + /* Allocate Bounce-buf memory */ + txq->bounce_buf = (char *)rte_zmalloc_socket("TXQ", + MAX_WQE_SIZE, + RTE_CACHE_LINE_MIN_SIZE, + txq->socket); + + if ((elts == NULL) || (txq->bounce_buf == NULL)) { + ERROR("%p: can't allocate TXQ memory", (void *)txq); ret = ENOMEM; goto error; } @@ -110,6 +116,8 @@ assert(ret == 0); return 0; error: + if (txq->bounce_buf != NULL) + rte_free(txq->bounce_buf); if (elts != NULL) rte_free(elts); DEBUG("%p: failed, freed everything", (void *)txq);