From patchwork Thu Nov 2 16:42:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matan Azrad X-Patchwork-Id: 31127 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 116D71B682; Thu, 2 Nov 2017 17:43:26 +0100 (CET) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30062.outbound.protection.outlook.com [40.107.3.62]) by dpdk.org (Postfix) with ESMTP id 7EDAB1B66A for ; Thu, 2 Nov 2017 17:43:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=II8jsePJJzkOxA2w5SDm5bdD74ITfCZRKwiZpK+TK9g=; b=Jta79KdXR0FesejRyohvkApXQ2FiJmVKF8X50zxdQOOiMf6L6Sa91NteUWma7n8TpAP2V3LRywHY5JI/sqT1WJ+x8thfr4sTPDv+Ry+L9aYRUmWQqzUVjOLxxQBS2g9AQDJBuP0RPEjBxrPXdwdQncGzDEQsW29+Z3EtPlD17sU= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; Received: from mellanox.com (37.142.13.130) by HE1PR0502MB3659.eurprd05.prod.outlook.com (2603:10a6:7:85::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.197.13; Thu, 2 Nov 2017 16:43:21 +0000 From: Matan Azrad To: Adrien Mazarguil Cc: dev@dpdk.org, Ophir Munk Date: Thu, 2 Nov 2017 16:42:50 +0000 Message-Id: <1509640971-8637-8-git-send-email-matan@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1509640971-8637-1-git-send-email-matan@mellanox.com> References: <1509358049-18854-1-git-send-email-matan@mellanox.com> <1509640971-8637-1-git-send-email-matan@mellanox.com> MIME-Version: 1.0 X-Originating-IP: [37.142.13.130] X-ClientProxiedBy: VI1PR0801CA0068.eurprd08.prod.outlook.com (2603:10a6:800:7d::12) To HE1PR0502MB3659.eurprd05.prod.outlook.com (2603:10a6:7:85::17) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f7b59cf5-a84d-464e-f945-08d52210d480 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(2017052603199); SRVR:HE1PR0502MB3659; X-Microsoft-Exchange-Diagnostics: 1; HE1PR0502MB3659; 3:KSaHCFSsYy/SN/cNVFJQdioBMfJB2Gc0STa2FBPdAVUY1ELjdx7reP+w7WT9FAVadJM1suzrCqz9tsKajJ8uH2mA/Q6063ognvlmICPMLv1UK/aWCdKZPn80Xz8G8Cjb7O9puf5YFiv6jnF0Z7g67njbCuiPhpma1GlA3bZbP/UBkCybDFmRsjA67Baxqz8jMKvKHfK0N4SQEup2iuBeILKJRh0j7PlVz7xgpgG++Aa1MRqCR0p4mHpFjzFjUdha; 25:+2QOgKCgW/fhRub1xWtxvIVHqTKcP1ov47Mv44sUfhLoHevm28x6i99xqsAXqNxTXeCXp70TxRjsG5lOJLBMXrFnKQO8FL7ENtNjikrEY5ZbtbxWtXH77ZAbchlLyIEmVFehBGscMQkiMsa26jEp+tky6osTzkneU7tgRB8T2KZjbsdJJDpehYxYuXR3E1z2w6wBI3+vVngR4bu0WJZezwX0dZ6IefwO5/90IVe2l6scBmkRu6FsV63OOQvEpHgXzifmNeCdXOVUJ0vcRzyCgu76R7XadIXT1zuotfcuDYudAyKLcKb74hrSechnk+I8waFD8sPL644DHXbTUWRG8Q==; 31:WlEG1m73+dmA8fDtDPYixXT/rl2i+12C9wC4WvmgzML8aXR0c/ceEPFeLQCRoUk+nflstsFjjuvqKAVIcpFSbRWphJWkoyb4Ap+uzjmRJGpAHaLY1qSS7RjJhtgrc5cMqctqDdJdbv53mO9qmNkcGdFPActF3PIRu6jh0uwLYeaV5/V+em3Gw4+g7+vtyitUIUza8AwOJ0snLLlCLwDT9Ej70lz7ha/OY8+gh2PzsSA= X-MS-TrafficTypeDiagnostic: HE1PR0502MB3659: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; HE1PR0502MB3659; 20:8MaQXDKqUMlmTdPf9h12b6qrSIYEJv57EDNUSmcToy0TCkwaXt4XZYqRV3dxBVzNdqJlkcZYQtuRcuQDxnSUOrcaxRaqMTkKfRCjlcBoiD+eFjg0i4pVMuEpTpQktkJ6n6sFvgKhtxoEhSXCEwRKGh3B+azgorSWviwOuWipSfylVMb2cImpBqwabHcgE7Ka3vt9FARVB8lOtIsX1AqAReRZLMeAaH2ZhQEhA9TSiL7+IZPYybmr3wKFRdXGGwqfVW/nA8ov90cZgrLqhXUeMCOGu0WooJ7GrTXL5CfGgJ/84+5C3PKj7/jzx1bUl1sXEXNsz0XOaHKuO4YJlJdHILMkfbcHnIkCmMfczLeUAqUBCCIsnpl4M33jfqtK1M+9SNbCLzXG6140wQxkKrqFMTbrcGp1pfz2LDRJqTsRJvETDzT1ZpfKpS56EFqvwGZm4G0ebWSm1sS03XGad9oFkQwkt6nNcwkQk364NykZYW/VEIoivuC1QN3sktCC+wJR; 4:eTPrcZzfyXBAVNCWwM1ehMIsB5LhYE6J1zeBdDSsGV89MF3ZjOnniMo++MIbGKFrzH9ryrvkpNKBYYPx0z35DlJw2YOHkOwGBxs9Fp14c0FuXx3JQ7gzdsMoPP9/g1Yv/GER95kIZoqVME95vJUa8jO8hjc8rLH76ta0EJpWyZuNjMrtOslMzLWoxVolemzGcc9x090fBt6AIcIyHtMAXeYAu7MmETIQJrys6VwW6hg095Gl23ebpA5jvQlKP5NTagRaldzk8m3z6+MhkV5HJA== X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(10201501046)(3231020)(93006095)(93001095)(3002001)(100000703101)(100105400095)(6055026)(6041248)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123555025)(20161123564025)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR0502MB3659; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR0502MB3659; X-Forefront-PRVS: 047999FF16 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(39860400002)(189002)(199003)(16586007)(50226002)(316002)(106356001)(105586002)(101416001)(21086003)(25786009)(8936002)(478600001)(4720700003)(8676002)(81156014)(189998001)(36756003)(33026002)(81166006)(97736004)(4326008)(5003940100001)(2906002)(55016002)(68736007)(33646002)(5660300001)(305945005)(107886003)(7736002)(50466002)(53936002)(48376002)(66066001)(16526018)(2950100002)(6916009)(6666003)(50986999)(47776003)(76176999)(3846002)(86362001)(69596002)(575784001)(6116002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0502MB3659; H:mellanox.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; HE1PR0502MB3659; 23:vT+FPq9UeKzGBILIZD7wegfXKA0ij7BYXo2y7jj?= KbtRX3FPVy9cwPMittltJ2gcMXINyqxSN1iSMzXm8t8rKFOcKm4uzUzVeU3CKTPj9+QPRp4q+CqCte4fH8InHi9D3UAu4SiOC0KIVTx3zXBDIBWCP20fr/0NkFgmZWq676+9FDmZg8kP+qEpgR7XVgASycsX5fS7/amRes+YCTjZR/+kBvz3STFWBbV5j+foJFBdeM8SkEgt5e168rAgwy43IeKokBLVkmojceQWDspYpcNpEea4bS5HEXpJwcfcWK3/TZ6AYAUHYe5sXfRw/vSz/4S8/WlG8l/F2TlLm+tD+jQV3IDvyGHZDcaSBMjWufzOnxqltwRLnFTdLK54P8E1i0AxX9E5C8uns4rVIsQkzei+SN5z2U7814lEPyFSyyKOyBBMVytmu/CW3cUlxdEu8W5Ja7yXip+6P2oz4goPqabEoQr5YZ2u5e8vdw8aga9OFJzkSl3giua9ELTIM5mNhyGTAf7yKgtDuLUFRZ8u8UfnT/n/hreDpJkcRTP0VEkqzsZJOBD1O0qBnVXyBPsfSxcVd9fxcQWwpJXuQxKFJfZsCC25ASOsbxgEv0cjMYedbkSpmLU0E/PFVtlOzCaJlx7cjtyG6TdSJIpFIvdU2n484ou+9L0gYlISkGxDY97kv7KDIcq3JLozS9afMAHZH7Cp565AR8e/f7bOq9kMo5Jsq/hiZlcT42RXWsizK5kpFIWGSZOxFHg2iEPE8/7Oo/EFBnXclMeUg7tKHEyCpdAvcaSg6UHtvcv8CzsWVAyPul8/1um3Xpl8YcnKAizPfblwZzw2xDi7JdukR3GvaGRehtOwSKZpqfzTIzhoUvyzV5C7pYeh660tGOMIfaSztOngiAft1uk9PO5KF0FMJoKHIniremAW2HmNgYQsFo8W/R5RKTuM7iNNrbix3PtaTsy2Ct6GoeqKQMWHX/X6ic5l4SwMBzVQM//SIOOr4lrgQnTFXAA6JQRfr68lv2NYWr4WJ/f2ZX5CXHWdiKOC1F4NmvcRSvhIHc4L0lJP1Yusxs5DH18v2+XkKKnlw1QBodGgn0ghn4NuHxhzz72K+Kajjt6FYTzBQ9BhusmDv9aH5mdG1mxI6X26YfpV5ksCNKbpRrBh61jZJ6ixtO5ma8+aq5zB7ENwd/tRZfdYuye8= X-Microsoft-Exchange-Diagnostics: 1; HE1PR0502MB3659; 6:I76TAQLfW/dbhGLgKHPXWs2u1WXWFp5xj8SW9gv28iE+VoI4p8gEVjQ7lqf7weuusk8HI4SiegPv9haISAeEefkKIqHDO+d2KMpHbin+ZMi5evMgUrP6ibLsOc14rU3EZnnqPvVx/DVHfkB+hBfZE/8TGRhwp1bCGrIvGtplBHfCvBdKjNN86oT+i6V3ktBtUZvZOYWlRvC3oWsm3v0CRbAwIH2RbJFN/7bNCGBS1hyE/7Z8SeFqUvUX6fvZUdu1bSaqJ+x7wCvWQgIuK9eRn+zGCRLtQRX42lhAP1DfF61VKOby+14JlnM/cB2ZJ7jJXS2sWISdsfstKFNiAY6j4ju96SxXnIG2t630zSjFLZs=; 5:Iu++2u/YnUumxHit4ZLnAgN6vdFKnu4FKjIDnZMnOfZsrmZvauHO5ZkDTd3WkbHzUpCPCUcXepKa8YPBbss1uo25l+TsnvkcB0/SPNmLjCvVfUm1TtY44uk8g58iLVdCkl/A5XFQ50GLAPHEqpWhgD1k01l1JgH4QrnGFo4MAeI=; 24:BEe+76oxt8UWkTKyLRuWBYO46IDVB4nRiZHy8zaq5nvpZZK53JM5q6LwPqKNnxjbSsR2qX1VHrj60cjtouqEt4CuwodBO5Q+AD+2kRSTlFQ=; 7:TWJnx4RDJknPWrUlTu8MbcDHIykuNHNEQebZMVcgyGNPBq/GYf69iSubxtQ95gWGOuSVdxP5vWgJubfhcMWXCNFQhey2z4QHx47Lc7gW61XFbdaad6D7MVsfKg/iJ7qodgThB3N0K67LXy/u5HAcXOPsmmcnkUBOi9unVmfdE6Et3fFw7J5tSx6UwAefhteRc0nFU2mTW2a0f/KfpSc7QXiBo9b9bn76HItRzYcP2sgQnSkJb1rSCHkxywXSyIXp SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2017 16:43:21.5023 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f7b59cf5-a84d-464e-f945-08d52210d480 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0502MB3659 Subject: [dpdk-dev] [PATCH v5 7/8] net/mlx4: fix HW memory optimizations careless X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Volatilize all Rx/Tx HW negotiation memories to be sure no compiler optimization prevents either load or store commands. Fixes: c3c977bbecbd ("net/mlx4: add Tx bypassing Verbs") Fixes: 9f57340a8087 ("net/mlx4: restore Rx offloads") Fixes: 6681b845034c ("net/mlx4: add Rx bypassing Verbs") Fixes: 62e96ffb93ad ("net/mlx4: fix no Rx interrupts") Signed-off-by: Matan Azrad Acked-by: Adrien Mazarguil --- drivers/net/mlx4/mlx4_prm.h | 20 ++++++------- drivers/net/mlx4/mlx4_rxtx.c | 67 ++++++++++++++++++++++++-------------------- 2 files changed, 46 insertions(+), 41 deletions(-) diff --git a/drivers/net/mlx4/mlx4_prm.h b/drivers/net/mlx4/mlx4_prm.h index b0fd982..339831a 100644 --- a/drivers/net/mlx4/mlx4_prm.h +++ b/drivers/net/mlx4/mlx4_prm.h @@ -80,14 +80,14 @@ enum { /* Send queue information. */ struct mlx4_sq { - uint8_t *buf; /**< SQ buffer. */ - uint8_t *eob; /**< End of SQ buffer */ + volatile uint8_t *buf; /**< SQ buffer. */ + volatile uint8_t *eob; /**< End of SQ buffer */ uint32_t head; /**< SQ head counter in units of TXBBS. */ uint32_t tail; /**< SQ tail counter in units of TXBBS. */ uint32_t txbb_cnt; /**< Num of WQEBB in the Q (should be ^2). */ uint32_t txbb_cnt_mask; /**< txbbs_cnt mask (txbb_cnt is ^2). */ uint32_t headroom_txbbs; /**< Num of txbbs that should be kept free. */ - uint32_t *db; /**< Pointer to the doorbell. */ + volatile uint32_t *db; /**< Pointer to the doorbell. */ uint32_t doorbell_qpn; /**< qp number to write to the doorbell. */ }; @@ -100,11 +100,11 @@ struct mlx4_sq { /* Completion queue information. */ struct mlx4_cq { - void *cq_uar; /**< CQ user access region. */ - void *cq_db_reg; /**< CQ doorbell register. */ - uint32_t *set_ci_db; /**< Pointer to the completion queue doorbell. */ - uint32_t *arm_db; /**< Pointer to doorbell for arming Rx events. */ - uint8_t *buf; /**< Pointer to the completion queue buffer. */ + volatile void *cq_uar; /**< CQ user access region. */ + volatile void *cq_db_reg; /**< CQ doorbell register. */ + volatile uint32_t *set_ci_db; /**< Pointer to the CQ doorbell. */ + volatile uint32_t *arm_db; /**< Arming Rx events doorbell. */ + volatile uint8_t *buf; /**< Pointer to the completion queue buffer. */ uint32_t cqe_cnt; /**< Number of entries in the queue. */ uint32_t cqe_64:1; /**< CQ entry size is 64 bytes. */ uint32_t cons_index; /**< Last queue entry that was handled. */ @@ -128,10 +128,10 @@ struct mlx4_cq { * @return * Pointer to CQE entry. */ -static inline struct mlx4_cqe * +static inline volatile struct mlx4_cqe * mlx4_get_cqe(struct mlx4_cq *cq, uint32_t index) { - return (struct mlx4_cqe *)(cq->buf + + return (volatile struct mlx4_cqe *)(cq->buf + ((index & (cq->cqe_cnt - 1)) << (5 + cq->cqe_64)) + (cq->cqe_64 << 5)); diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c index d13c8d2..20060ce 100644 --- a/drivers/net/mlx4/mlx4_rxtx.c +++ b/drivers/net/mlx4/mlx4_rxtx.c @@ -69,7 +69,7 @@ * DWORD (32 byte) of a TXBB. */ struct pv { - struct mlx4_wqe_data_seg *dseg; + volatile struct mlx4_wqe_data_seg *dseg; uint32_t val; }; @@ -97,14 +97,15 @@ struct pv { { uint32_t stamp = rte_cpu_to_be_32(MLX4_SQ_STAMP_VAL | (!!owner << MLX4_SQ_STAMP_SHIFT)); - uint8_t *wqe = mlx4_get_send_wqe(sq, (index & sq->txbb_cnt_mask)); - uint32_t *ptr = (uint32_t *)wqe; + volatile uint8_t *wqe = mlx4_get_send_wqe(sq, + (index & sq->txbb_cnt_mask)); + volatile uint32_t *ptr = (volatile uint32_t *)wqe; int i; int txbbs_size; int num_txbbs; /* Extract the size from the control segment of the WQE. */ - num_txbbs = MLX4_SIZE_TO_TXBBS((((struct mlx4_wqe_ctrl_seg *) + num_txbbs = MLX4_SIZE_TO_TXBBS((((volatile struct mlx4_wqe_ctrl_seg *) wqe)->fence_size & 0x3f) << 4); txbbs_size = num_txbbs * MLX4_TXBB_SIZE; /* Optimize the common case when there is no wrap-around. */ @@ -119,8 +120,8 @@ struct pv { for (i = 0; i < txbbs_size; i += MLX4_SQ_STAMP_STRIDE) { *ptr = stamp; ptr += MLX4_SQ_STAMP_DWORDS; - if ((uint8_t *)ptr >= sq->eob) { - ptr = (uint32_t *)sq->buf; + if ((volatile uint8_t *)ptr >= sq->eob) { + ptr = (volatile uint32_t *)sq->buf; stamp ^= RTE_BE32(0x80000000); } } @@ -149,7 +150,7 @@ struct pv { unsigned int elts_comp = txq->elts_comp; unsigned int elts_tail = txq->elts_tail; struct mlx4_cq *cq = &txq->mcq; - struct mlx4_cqe *cqe; + volatile struct mlx4_cqe *cqe; uint32_t cons_index = cq->cons_index; uint16_t new_index; uint16_t nr_txbbs = 0; @@ -160,7 +161,7 @@ struct pv { * reported by them. */ do { - cqe = (struct mlx4_cqe *)mlx4_get_cqe(cq, cons_index); + cqe = (volatile struct mlx4_cqe *)mlx4_get_cqe(cq, cons_index); if (unlikely(!!(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK) ^ !!(cons_index & cq->cqe_cnt))) break; @@ -171,8 +172,8 @@ struct pv { #ifndef NDEBUG if (unlikely((cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) == MLX4_CQE_OPCODE_ERROR)) { - struct mlx4_err_cqe *cqe_err = - (struct mlx4_err_cqe *)cqe; + volatile struct mlx4_err_cqe *cqe_err = + (volatile struct mlx4_err_cqe *)cqe; ERROR("%p CQE error - vendor syndrome: 0x%x" " syndrome: 0x%x\n", (void *)txq, cqe_err->vendor_err, @@ -239,15 +240,15 @@ struct pv { static int mlx4_tx_burst_segs(struct rte_mbuf *buf, struct txq *txq, - struct mlx4_wqe_ctrl_seg **pctrl) + volatile struct mlx4_wqe_ctrl_seg **pctrl) { int wqe_real_size; int nr_txbbs; struct pv *pv = (struct pv *)txq->bounce_buf; struct mlx4_sq *sq = &txq->msq; uint32_t head_idx = sq->head & sq->txbb_cnt_mask; - struct mlx4_wqe_ctrl_seg *ctrl; - struct mlx4_wqe_data_seg *dseg; + volatile struct mlx4_wqe_ctrl_seg *ctrl; + volatile struct mlx4_wqe_data_seg *dseg; struct rte_mbuf *sbuf; uint32_t lkey; uintptr_t addr; @@ -255,8 +256,8 @@ struct pv { int pv_counter = 0; /* Calculate the needed work queue entry size for this packet. */ - wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) + - buf->nb_segs * sizeof(struct mlx4_wqe_data_seg); + wqe_real_size = sizeof(volatile struct mlx4_wqe_ctrl_seg) + + buf->nb_segs * sizeof(volatile struct mlx4_wqe_data_seg); nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size); /* * Check that there is room for this WQE in the send queue and that @@ -268,17 +269,18 @@ struct pv { return -1; } /* Get the control and data entries of the WQE. */ - ctrl = (struct mlx4_wqe_ctrl_seg *)mlx4_get_send_wqe(sq, head_idx); - dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + - sizeof(struct mlx4_wqe_ctrl_seg)); + ctrl = (volatile struct mlx4_wqe_ctrl_seg *) + mlx4_get_send_wqe(sq, head_idx); + dseg = (volatile struct mlx4_wqe_data_seg *) + ((uintptr_t)ctrl + sizeof(struct mlx4_wqe_ctrl_seg)); *pctrl = ctrl; /* Fill the data segments with buffer information. */ for (sbuf = buf; sbuf != NULL; sbuf = sbuf->next, dseg++) { addr = rte_pktmbuf_mtod(sbuf, uintptr_t); rte_prefetch0((volatile void *)addr); /* Handle WQE wraparound. */ - if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob) - dseg = (struct mlx4_wqe_data_seg *)sq->buf; + if (dseg >= (volatile struct mlx4_wqe_data_seg *)sq->eob) + dseg = (volatile struct mlx4_wqe_data_seg *)sq->buf; dseg->addr = rte_cpu_to_be_64(addr); /* Memory region key (big endian) for this memory pool. */ lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(sbuf)); @@ -395,8 +397,8 @@ struct pv { struct txq_elt *elt_next = &(*txq->elts)[elts_head_next]; struct txq_elt *elt = &(*txq->elts)[elts_head]; uint32_t owner_opcode = MLX4_OPCODE_SEND; - struct mlx4_wqe_ctrl_seg *ctrl; - struct mlx4_wqe_data_seg *dseg; + volatile struct mlx4_wqe_ctrl_seg *ctrl; + volatile struct mlx4_wqe_data_seg *dseg; union { uint32_t flags; uint16_t flags16[2]; @@ -433,15 +435,18 @@ struct pv { break; } /* Get the control and data entries of the WQE. */ - ctrl = (struct mlx4_wqe_ctrl_seg *) + ctrl = (volatile struct mlx4_wqe_ctrl_seg *) mlx4_get_send_wqe(sq, head_idx); - dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl + + dseg = (volatile struct mlx4_wqe_data_seg *) + ((uintptr_t)ctrl + sizeof(struct mlx4_wqe_ctrl_seg)); addr = rte_pktmbuf_mtod(buf, uintptr_t); rte_prefetch0((volatile void *)addr); /* Handle WQE wraparound. */ - if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob) - dseg = (struct mlx4_wqe_data_seg *)sq->buf; + if (dseg >= + (volatile struct mlx4_wqe_data_seg *)sq->eob) + dseg = (volatile struct mlx4_wqe_data_seg *) + sq->buf; dseg->addr = rte_cpu_to_be_64(addr); /* Memory region key (big endian). */ lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf)); @@ -633,7 +638,7 @@ struct pv { * CQE checksum information. */ static inline uint32_t -mlx4_cqe_flags(struct mlx4_cqe *cqe, int csum, int csum_l2tun) +mlx4_cqe_flags(volatile struct mlx4_cqe *cqe, int csum, int csum_l2tun) { uint32_t flags = 0; @@ -666,13 +671,13 @@ struct pv { * Number of bytes of the CQE, 0 in case there is no completion. */ static unsigned int -mlx4_cq_poll_one(struct rxq *rxq, struct mlx4_cqe **out) +mlx4_cq_poll_one(struct rxq *rxq, volatile struct mlx4_cqe **out) { int ret = 0; - struct mlx4_cqe *cqe = NULL; + volatile struct mlx4_cqe *cqe = NULL; struct mlx4_cq *cq = &rxq->mcq; - cqe = (struct mlx4_cqe *)mlx4_get_cqe(cq, cq->cons_index); + cqe = (volatile struct mlx4_cqe *)mlx4_get_cqe(cq, cq->cons_index); if (!!(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK) ^ !!(cq->cons_index & cq->cqe_cnt)) goto out; @@ -717,7 +722,7 @@ struct pv { int len = 0; while (pkts_n) { - struct mlx4_cqe *cqe; + volatile struct mlx4_cqe *cqe; uint32_t idx = rq_ci & wr_cnt; struct rte_mbuf *rep = (*rxq->elts)[idx]; volatile struct mlx4_wqe_data_seg *scat = &(*rxq->wqes)[idx];