From patchwork Mon Jan 20 17:23:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 64980 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E256BA0526; Mon, 20 Jan 2020 18:23:40 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 196B41C0DC; Mon, 20 Jan 2020 18:23:35 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 3EEAD1C11C for ; Mon, 20 Jan 2020 18:23:33 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from viacheslavo@mellanox.com) with ESMTPS (AES256-SHA encrypted); 20 Jan 2020 19:23:29 +0200 Received: from pegasus11.mtr.labs.mlnx (pegasus11.mtr.labs.mlnx [10.210.16.104]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 00KHNTnH011351; Mon, 20 Jan 2020 19:23:29 +0200 Received: from pegasus11.mtr.labs.mlnx (localhost [127.0.0.1]) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7) with ESMTP id 00KHNTr3002505; Mon, 20 Jan 2020 17:23:29 GMT Received: (from viacheslavo@localhost) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7/Submit) id 00KHNTZL002504; Mon, 20 Jan 2020 17:23:29 GMT X-Authentication-Warning: pegasus11.mtr.labs.mlnx: viacheslavo set sender to viacheslavo@mellanox.com using -f From: Viacheslav Ovsiienko To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, orika@mellanox.com, shahafs@mellanox.com, olivier.matz@6wind.com, stephen@networkplumber.org, thomas@mellanox.net Date: Mon, 20 Jan 2020 17:23:19 +0000 Message-Id: <1579541003-2399-2-git-send-email-viacheslavo@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> References: <20191118094938.192850-1-shahafs@mellanox.com> <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> Subject: [dpdk-dev] [PATCH v5 1/5] mbuf: introduce routine to get private mbuf pool flags X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The routine rte_pktmbuf_priv_flags is introduced to fetch the flags from the mbuf memory pool private structure in unified fashion. Signed-off-by: Viacheslav Ovsiienko Acked-by: Olivier Matz --- lib/librte_mbuf/rte_mbuf.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h index 2d4bda2..9b0691d 100644 --- a/lib/librte_mbuf/rte_mbuf.h +++ b/lib/librte_mbuf/rte_mbuf.h @@ -306,6 +306,23 @@ struct rte_pktmbuf_pool_private { uint32_t flags; /**< reserved for future use. */ }; +/** + * Return the flags from private data in an mempool structure. + * + * @param mp + * A pointer to the mempool structure. + * @return + * The flags from the private data structure. + */ +static inline uint32_t +rte_pktmbuf_priv_flags(struct rte_mempool *mp) +{ + struct rte_pktmbuf_pool_private *mbp_priv; + + mbp_priv = (struct rte_pktmbuf_pool_private *)rte_mempool_get_priv(mp); + return mbp_priv->flags; +} + #ifdef RTE_LIBRTE_MBUF_DEBUG /** check mbuf type in debug mode */ From patchwork Mon Jan 20 17:23:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 64981 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 23644A0526; Mon, 20 Jan 2020 18:23:47 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C931D1C129; Mon, 20 Jan 2020 18:23:36 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 582681C11D for ; Mon, 20 Jan 2020 18:23:33 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from viacheslavo@mellanox.com) with ESMTPS (AES256-SHA encrypted); 20 Jan 2020 19:23:31 +0200 Received: from pegasus11.mtr.labs.mlnx (pegasus11.mtr.labs.mlnx [10.210.16.104]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 00KHNVl6011366; Mon, 20 Jan 2020 19:23:31 +0200 Received: from pegasus11.mtr.labs.mlnx (localhost [127.0.0.1]) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7) with ESMTP id 00KHNUmH002508; Mon, 20 Jan 2020 17:23:30 GMT Received: (from viacheslavo@localhost) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7/Submit) id 00KHNUSd002507; Mon, 20 Jan 2020 17:23:30 GMT X-Authentication-Warning: pegasus11.mtr.labs.mlnx: viacheslavo set sender to viacheslavo@mellanox.com using -f From: Viacheslav Ovsiienko To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, orika@mellanox.com, shahafs@mellanox.com, olivier.matz@6wind.com, stephen@networkplumber.org, thomas@mellanox.net Date: Mon, 20 Jan 2020 17:23:20 +0000 Message-Id: <1579541003-2399-3-git-send-email-viacheslavo@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> References: <20191118094938.192850-1-shahafs@mellanox.com> <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> Subject: [dpdk-dev] [PATCH v5 2/5] mbuf: detach mbuf with pinned external buffer X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Update detach routine to check the mbuf pool type. Introduce the special internal version of detach routine to handle the special case of pinned external bufferon mbuf freeing. Signed-off-by: Shahaf Shuler Signed-off-by: Viacheslav Ovsiienko Acked-by: Olivier Matz --- lib/librte_mbuf/rte_mbuf.h | 102 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 93 insertions(+), 9 deletions(-) diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h index 9b0691d..7a41aad 100644 --- a/lib/librte_mbuf/rte_mbuf.h +++ b/lib/librte_mbuf/rte_mbuf.h @@ -323,6 +323,24 @@ struct rte_pktmbuf_pool_private { return mbp_priv->flags; } +/** + * When set, pktmbuf mempool will hold only mbufs with pinned external + * buffer. The external buffer will be attached to the mbuf at the + * memory pool creation and will never be detached by the mbuf free calls. + * mbuf should not contain any room for data after the mbuf structure. + */ +#define RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF (1 << 0) + +/** + * Returns non zero if given mbuf has a pinned external buffer, or zero + * otherwise. The pinned external buffer is allocated at pool creation + * time and should not be freed on mbuf freeing. + * + * External buffer is a user-provided anonymous buffer. + */ +#define RTE_MBUF_HAS_PINNED_EXTBUF(mb) \ + (rte_pktmbuf_priv_flags(mb->pool) & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) + #ifdef RTE_LIBRTE_MBUF_DEBUG /** check mbuf type in debug mode */ @@ -588,7 +606,8 @@ static inline struct rte_mbuf *rte_mbuf_raw_alloc(struct rte_mempool *mp) static __rte_always_inline void rte_mbuf_raw_free(struct rte_mbuf *m) { - RTE_ASSERT(RTE_MBUF_DIRECT(m)); + RTE_ASSERT(!RTE_MBUF_CLONED(m) && + (!RTE_MBUF_HAS_EXTBUF(m) || RTE_MBUF_HAS_PINNED_EXTBUF(m))); RTE_ASSERT(rte_mbuf_refcnt_read(m) == 1); RTE_ASSERT(m->next == NULL); RTE_ASSERT(m->nb_segs == 1); @@ -794,7 +813,7 @@ static inline void rte_pktmbuf_reset(struct rte_mbuf *m) m->nb_segs = 1; m->port = MBUF_INVALID_PORT; - m->ol_flags = 0; + m->ol_flags &= EXT_ATTACHED_MBUF; m->packet_type = 0; rte_pktmbuf_reset_headroom(m); @@ -1153,6 +1172,11 @@ static inline void rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *m) * * All other fields of the given packet mbuf will be left intact. * + * If the packet mbuf was allocated from the pool with pinned + * external buffers the rte_pktmbuf_detach does nothing with the + * mbuf of this kind, because the pinned buffers are not supposed + * to be detached. + * * @param m * The indirect attached packet mbuf. */ @@ -1162,11 +1186,26 @@ static inline void rte_pktmbuf_detach(struct rte_mbuf *m) uint32_t mbuf_size, buf_len; uint16_t priv_size; - if (RTE_MBUF_HAS_EXTBUF(m)) + if (RTE_MBUF_HAS_EXTBUF(m)) { + /* + * The mbuf has the external attached buffer, + * we should check the type of the memory pool where + * the mbuf was allocated from to detect the pinned + * external buffer. + */ + uint32_t flags = rte_pktmbuf_priv_flags(mp); + + if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { + /* + * The pinned external buffer should not be + * detached from its backing mbuf, just exit. + */ + return; + } __rte_pktmbuf_free_extbuf(m); - else + } else { __rte_pktmbuf_free_direct(m); - + } priv_size = rte_pktmbuf_priv_size(mp); mbuf_size = (uint32_t)(sizeof(struct rte_mbuf) + priv_size); buf_len = rte_pktmbuf_data_room_size(mp); @@ -1181,6 +1220,41 @@ static inline void rte_pktmbuf_detach(struct rte_mbuf *m) } /** + * @internal Handle the packet mbufs with attached pinned external buffer + * on the mbuf freeing: + * + * - return zero if reference counter in shinfo is one. It means there is + * no more reference to this pinned buffer and mbuf can be returned to + * the pool + * + * - otherwise (if reference counter is not one), decrement reference + * counter and return non-zero value to prevent freeing the backing mbuf. + * + * Returns non zero if mbuf should not be freed. + */ +static inline int __rte_pktmbuf_pinned_extbuf_decref(struct rte_mbuf *m) +{ + struct rte_mbuf_ext_shared_info *shinfo; + + /* Clear flags, mbuf is being freed. */ + m->ol_flags = EXT_ATTACHED_MBUF; + shinfo = m->shinfo; + /* Optimize for performance - do not dec/reinit */ + if (likely(rte_mbuf_ext_refcnt_read(shinfo) == 1)) + return 0; + /* + * Direct usage of add primitive to avoid + * duplication of comparing with one. + */ + if (likely(rte_atomic16_add_return + (&shinfo->refcnt_atomic, -1))) + return 1; + /* Reinitialize counter before mbuf freeing. */ + rte_mbuf_ext_refcnt_set(shinfo, 1); + return 0; +} + +/** * Decrease reference counter and unlink a mbuf segment * * This function does the same than a free, except that it does not @@ -1201,8 +1275,13 @@ static inline void rte_pktmbuf_detach(struct rte_mbuf *m) if (likely(rte_mbuf_refcnt_read(m) == 1)) { - if (!RTE_MBUF_DIRECT(m)) - rte_pktmbuf_detach(m); + if (!RTE_MBUF_DIRECT(m)) { + if (!RTE_MBUF_HAS_EXTBUF(m) || + !RTE_MBUF_HAS_PINNED_EXTBUF(m)) + rte_pktmbuf_detach(m); + else if (__rte_pktmbuf_pinned_extbuf_decref(m)) + return NULL; + } if (m->next != NULL) { m->next = NULL; @@ -1213,8 +1292,13 @@ static inline void rte_pktmbuf_detach(struct rte_mbuf *m) } else if (__rte_mbuf_refcnt_update(m, -1) == 0) { - if (!RTE_MBUF_DIRECT(m)) - rte_pktmbuf_detach(m); + if (!RTE_MBUF_DIRECT(m)) { + if (!RTE_MBUF_HAS_EXTBUF(m) || + !RTE_MBUF_HAS_PINNED_EXTBUF(m)) + rte_pktmbuf_detach(m); + else if (__rte_pktmbuf_pinned_extbuf_decref(m)) + return NULL; + } if (m->next != NULL) { m->next = NULL; From patchwork Mon Jan 20 17:23:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 64982 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B6691A0526; Mon, 20 Jan 2020 18:23:55 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9706C1C138; Mon, 20 Jan 2020 18:23:38 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 5D92B1C11F for ; Mon, 20 Jan 2020 18:23:33 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from viacheslavo@mellanox.com) with ESMTPS (AES256-SHA encrypted); 20 Jan 2020 19:23:32 +0200 Received: from pegasus11.mtr.labs.mlnx (pegasus11.mtr.labs.mlnx [10.210.16.104]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 00KHNW0A011376; Mon, 20 Jan 2020 19:23:32 +0200 Received: from pegasus11.mtr.labs.mlnx (localhost [127.0.0.1]) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7) with ESMTP id 00KHNWDe002510; Mon, 20 Jan 2020 17:23:32 GMT Received: (from viacheslavo@localhost) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7/Submit) id 00KHNWDq002509; Mon, 20 Jan 2020 17:23:32 GMT X-Authentication-Warning: pegasus11.mtr.labs.mlnx: viacheslavo set sender to viacheslavo@mellanox.com using -f From: Viacheslav Ovsiienko To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, orika@mellanox.com, shahafs@mellanox.com, olivier.matz@6wind.com, stephen@networkplumber.org, thomas@mellanox.net Date: Mon, 20 Jan 2020 17:23:21 +0000 Message-Id: <1579541003-2399-4-git-send-email-viacheslavo@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> References: <20191118094938.192850-1-shahafs@mellanox.com> <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> Subject: [dpdk-dev] [PATCH v5 3/5] mbuf: create packet pool with external memory buffers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The dedicated routine rte_pktmbuf_pool_create_extbuf() is provided to create mbuf pool with data buffers located in the pinned external memory. The application provides the external memory description and routine initializes each mbuf with appropriate virtual and physical buffer address. It is entirely application responsibility to register external memory with rte_extmem_register() API, map this memory, etc. The new introduced flag RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF is set in private pool structure, specifying the new special pool type. The allocated mbufs from pool of this kind will have the EXT_ATTACHED_MBUF flag set and initialiazed shared info structure, allowing cloning with regular mbufs (without attached external buffers of any kind). Signed-off-by: Viacheslav Ovsiienko Acked-by: Olivier Matz --- lib/librte_mbuf/rte_mbuf.c | 198 ++++++++++++++++++++++++++++++++++- lib/librte_mbuf/rte_mbuf.h | 64 +++++++++++ lib/librte_mbuf/rte_mbuf_version.map | 1 + 3 files changed, 260 insertions(+), 3 deletions(-) diff --git a/lib/librte_mbuf/rte_mbuf.c b/lib/librte_mbuf/rte_mbuf.c index 8fa7f49..a709246 100644 --- a/lib/librte_mbuf/rte_mbuf.c +++ b/lib/librte_mbuf/rte_mbuf.c @@ -59,9 +59,12 @@ } RTE_ASSERT(mp->elt_size >= sizeof(struct rte_mbuf) + - user_mbp_priv->mbuf_data_room_size + + ((user_mbp_priv->flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) ? + sizeof(struct rte_mbuf_ext_shared_info) : + user_mbp_priv->mbuf_data_room_size) + user_mbp_priv->mbuf_priv_size); - RTE_ASSERT(user_mbp_priv->flags == 0); + RTE_ASSERT((user_mbp_priv->flags & + ~RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) == 0); mbp_priv = rte_mempool_get_priv(mp); memcpy(mbp_priv, user_mbp_priv, sizeof(*mbp_priv)); @@ -107,6 +110,108 @@ m->next = NULL; } +/* + * @internal The callback routine called when reference counter in shinfo + * for mbufs with pinned external buffer reaches zero. It means there is + * no more reference to buffer backing mbuf and this one should be freed. + * This routine is called for the regular (not with pinned external or + * indirect buffer) mbufs on detaching from the mbuf with pinned external + * buffer. + */ +static void rte_pktmbuf_free_pinned_extmem(void *addr, void *opaque) +{ + struct rte_mbuf *m = opaque; + + RTE_SET_USED(addr); + RTE_ASSERT(RTE_MBUF_HAS_EXTBUF(m)); + RTE_ASSERT(RTE_MBUF_HAS_PINNED_EXTBUF(m)); + RTE_ASSERT(m->shinfo->fcb_opaque == m); + + rte_mbuf_ext_refcnt_set(m->shinfo, 1); + m->ol_flags = EXT_ATTACHED_MBUF; + if (m->next != NULL) { + m->next = NULL; + m->nb_segs = 1; + } + rte_mbuf_raw_free(m); +} + +/** + * @internal Packet mbuf constructor for pools with pinned external memory. + * + * This function initializes some fields in the mbuf structure that are + * not modified by the user once created (origin pool, buffer start + * address, and so on). This function is given as a callback function to + * rte_mempool_obj_iter() called from rte_mempool_create_extmem(). + * + * @param mp + * The mempool from which mbufs originate. + * @param opaque_arg + * A pointer to the rte_pktmbuf_extmem_init_ctx - initialization + * context structure + * @param m + * The mbuf to initialize. + * @param i + * The index of the mbuf in the pool table. + */ +static void +__rte_pktmbuf_init_extmem(struct rte_mempool *mp, + void *opaque_arg, + void *_m, + __attribute__((unused)) unsigned int i) +{ + struct rte_mbuf *m = _m; + struct rte_pktmbuf_extmem_init_ctx *ctx = opaque_arg; + const struct rte_pktmbuf_extmem *ext_mem; + uint32_t mbuf_size, buf_len, priv_size; + struct rte_mbuf_ext_shared_info *shinfo; + + priv_size = rte_pktmbuf_priv_size(mp); + mbuf_size = sizeof(struct rte_mbuf) + priv_size; + buf_len = rte_pktmbuf_data_room_size(mp); + + RTE_ASSERT(RTE_ALIGN(priv_size, RTE_MBUF_PRIV_ALIGN) == priv_size); + RTE_ASSERT(mp->elt_size >= mbuf_size); + RTE_ASSERT(buf_len <= UINT16_MAX); + + memset(m, 0, mbuf_size); + m->priv_size = priv_size; + m->buf_len = (uint16_t)buf_len; + + /* set the data buffer pointers to external memory */ + ext_mem = ctx->ext_mem + ctx->ext; + + RTE_ASSERT(ctx->ext < ctx->ext_num); + RTE_ASSERT(ctx->off < ext_mem->buf_len); + + m->buf_addr = RTE_PTR_ADD(ext_mem->buf_ptr, ctx->off); + m->buf_iova = ext_mem->buf_iova == RTE_BAD_IOVA ? + RTE_BAD_IOVA : (ext_mem->buf_iova + ctx->off); + + ctx->off += ext_mem->elt_size; + if (ctx->off >= ext_mem->buf_len) { + ctx->off = 0; + ++ctx->ext; + } + /* keep some headroom between start of buffer and data */ + m->data_off = RTE_MIN(RTE_PKTMBUF_HEADROOM, (uint16_t)m->buf_len); + + /* init some constant fields */ + m->pool = mp; + m->nb_segs = 1; + m->port = MBUF_INVALID_PORT; + m->ol_flags = EXT_ATTACHED_MBUF; + rte_mbuf_refcnt_set(m, 1); + m->next = NULL; + + /* init external buffer shared info items */ + shinfo = RTE_PTR_ADD(m, mbuf_size); + m->shinfo = shinfo; + shinfo->free_cb = rte_pktmbuf_free_pinned_extmem; + shinfo->fcb_opaque = m; + rte_mbuf_ext_refcnt_set(shinfo, 1); +} + /* Helper to create a mbuf pool with given mempool ops name*/ struct rte_mempool * rte_pktmbuf_pool_create_by_ops(const char *name, unsigned int n, @@ -169,6 +274,93 @@ struct rte_mempool * data_room_size, socket_id, NULL); } +/* Helper to create a mbuf pool with pinned external data buffers. */ +struct rte_mempool * +rte_pktmbuf_pool_create_extbuf(const char *name, unsigned int n, + unsigned int cache_size, uint16_t priv_size, + uint16_t data_room_size, int socket_id, + const struct rte_pktmbuf_extmem *ext_mem, + unsigned int ext_num) +{ + struct rte_mempool *mp; + struct rte_pktmbuf_pool_private mbp_priv; + struct rte_pktmbuf_extmem_init_ctx init_ctx; + const char *mp_ops_name; + unsigned int elt_size; + unsigned int i, n_elts = 0; + int ret; + + if (RTE_ALIGN(priv_size, RTE_MBUF_PRIV_ALIGN) != priv_size) { + RTE_LOG(ERR, MBUF, "mbuf priv_size=%u is not aligned\n", + priv_size); + rte_errno = EINVAL; + return NULL; + } + /* Check the external memory descriptors. */ + for (i = 0; i < ext_num; i++) { + const struct rte_pktmbuf_extmem *extm = ext_mem + i; + + if (!extm->elt_size || !extm->buf_len || !extm->buf_ptr) { + RTE_LOG(ERR, MBUF, "invalid extmem descriptor\n"); + rte_errno = EINVAL; + return NULL; + } + if (data_room_size > extm->elt_size) { + RTE_LOG(ERR, MBUF, "ext elt_size=%u is too small\n", + priv_size); + rte_errno = EINVAL; + return NULL; + } + n_elts += extm->buf_len / extm->elt_size; + } + /* Check whether enough external memory provided. */ + if (n_elts < n) { + RTE_LOG(ERR, MBUF, "not enough extmem\n"); + rte_errno = ENOMEM; + return NULL; + } + elt_size = sizeof(struct rte_mbuf) + + (unsigned int)priv_size + + sizeof(struct rte_mbuf_ext_shared_info); + + memset(&mbp_priv, 0, sizeof(mbp_priv)); + mbp_priv.mbuf_data_room_size = data_room_size; + mbp_priv.mbuf_priv_size = priv_size; + mbp_priv.flags = RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF; + + mp = rte_mempool_create_empty(name, n, elt_size, cache_size, + sizeof(struct rte_pktmbuf_pool_private), socket_id, 0); + if (mp == NULL) + return NULL; + + mp_ops_name = rte_mbuf_best_mempool_ops(); + ret = rte_mempool_set_ops_byname(mp, mp_ops_name, NULL); + if (ret != 0) { + RTE_LOG(ERR, MBUF, "error setting mempool handler\n"); + rte_mempool_free(mp); + rte_errno = -ret; + return NULL; + } + rte_pktmbuf_pool_init(mp, &mbp_priv); + + ret = rte_mempool_populate_default(mp); + if (ret < 0) { + rte_mempool_free(mp); + rte_errno = -ret; + return NULL; + } + + init_ctx = (struct rte_pktmbuf_extmem_init_ctx){ + .ext_mem = ext_mem, + .ext_num = ext_num, + .ext = 0, + .off = 0, + }; + rte_mempool_obj_iter(mp, __rte_pktmbuf_init_extmem, &init_ctx); + + return mp; +} + /* do some sanity checks on a mbuf: panic if it fails */ void rte_mbuf_sanity_check(const struct rte_mbuf *m, int is_header) @@ -247,7 +439,7 @@ int rte_mbuf_check(const struct rte_mbuf *m, int is_header, return 0; } -/** +/* * @internal helper function for freeing a bulk of packet mbuf segments * via an array holding the packet mbuf segments from the same mempool * pending to be freed. diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h index 7a41aad..eaeda04 100644 --- a/lib/librte_mbuf/rte_mbuf.h +++ b/lib/librte_mbuf/rte_mbuf.h @@ -637,6 +637,13 @@ static inline struct rte_mbuf *rte_mbuf_raw_alloc(struct rte_mempool *mp) void rte_pktmbuf_init(struct rte_mempool *mp, void *opaque_arg, void *m, unsigned i); +/** The context to initialize the mbufs with pinned external buffers. */ +struct rte_pktmbuf_extmem_init_ctx { + const struct rte_pktmbuf_extmem *ext_mem; /* descriptor array. */ + unsigned int ext_num; /* number of descriptors in array. */ + unsigned int ext; /* loop descriptor index. */ + size_t off; /* loop buffer offset. */ +}; /** * A packet mbuf pool constructor. @@ -738,6 +745,63 @@ struct rte_mempool * unsigned int cache_size, uint16_t priv_size, uint16_t data_room_size, int socket_id, const char *ops_name); +/** A structure that describes the pinned external buffer segment. */ +struct rte_pktmbuf_extmem { + void *buf_ptr; /**< The virtual address of data buffer. */ + rte_iova_t buf_iova; /**< The IO address of the data buffer. */ + size_t buf_len; /**< External buffer length in bytes. */ + uint16_t elt_size; /**< mbuf element size in bytes. */ +}; + +/** + * Create a mbuf pool with external pinned data buffers. + * + * This function creates and initializes a packet mbuf pool that contains + * only mbufs with external buffer. It is a wrapper to rte_mempool functions. + * + * @param name + * The name of the mbuf pool. + * @param n + * The number of elements in the mbuf pool. The optimum size (in terms + * of memory usage) for a mempool is when n is a power of two minus one: + * n = (2^q - 1). + * @param cache_size + * Size of the per-core object cache. See rte_mempool_create() for + * details. + * @param priv_size + * Size of application private are between the rte_mbuf structure + * and the data buffer. This value must be aligned to RTE_MBUF_PRIV_ALIGN. + * @param data_room_size + * Size of data buffer in each mbuf, including RTE_PKTMBUF_HEADROOM. + * @param socket_id + * The socket identifier where the memory should be allocated. The + * value can be *SOCKET_ID_ANY* if there is no NUMA constraint for the + * reserved zone. + * @param ext_mem + * Pointer to the array of structures describing the external memory + * for data buffers. It is caller responsibility to register this memory + * with rte_extmem_register() (if needed), map this memory to appropriate + * physical device, etc. + * @param ext_num + * Number of elements in the ext_mem array. + * @return + * The pointer to the new allocated mempool, on success. NULL on error + * with rte_errno set appropriately. Possible rte_errno values include: + * - E_RTE_NO_CONFIG - function could not get pointer to rte_config structure + * - E_RTE_SECONDARY - function was called from a secondary process instance + * - EINVAL - cache size provided is too large, or priv_size is not aligned. + * - ENOSPC - the maximum number of memzones has already been allocated + * - EEXIST - a memzone with the same name already exists + * - ENOMEM - no appropriate memory area found in which to create memzone + */ +__rte_experimental +struct rte_mempool * +rte_pktmbuf_pool_create_extbuf(const char *name, unsigned int n, + unsigned int cache_size, uint16_t priv_size, + uint16_t data_room_size, int socket_id, + const struct rte_pktmbuf_extmem *ext_mem, + unsigned int ext_num); + /** * Get the data room size of mbufs stored in a pktmbuf_pool * diff --git a/lib/librte_mbuf/rte_mbuf_version.map b/lib/librte_mbuf/rte_mbuf_version.map index 3bbb476..ab161bc 100644 --- a/lib/librte_mbuf/rte_mbuf_version.map +++ b/lib/librte_mbuf/rte_mbuf_version.map @@ -44,5 +44,6 @@ EXPERIMENTAL { rte_mbuf_dyn_dump; rte_pktmbuf_copy; rte_pktmbuf_free_bulk; + rte_pktmbuf_pool_create_extbuf; }; From patchwork Mon Jan 20 17:23:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 64983 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7555FA0526; Mon, 20 Jan 2020 18:24:04 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7E9321C192; Mon, 20 Jan 2020 18:23:40 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 5405A1C134 for ; Mon, 20 Jan 2020 18:23:38 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from viacheslavo@mellanox.com) with ESMTPS (AES256-SHA encrypted); 20 Jan 2020 19:23:33 +0200 Received: from pegasus11.mtr.labs.mlnx (pegasus11.mtr.labs.mlnx [10.210.16.104]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 00KHNXam011381; Mon, 20 Jan 2020 19:23:33 +0200 Received: from pegasus11.mtr.labs.mlnx (localhost [127.0.0.1]) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7) with ESMTP id 00KHNXwG002512; Mon, 20 Jan 2020 17:23:33 GMT Received: (from viacheslavo@localhost) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7/Submit) id 00KHNXg7002511; Mon, 20 Jan 2020 17:23:33 GMT X-Authentication-Warning: pegasus11.mtr.labs.mlnx: viacheslavo set sender to viacheslavo@mellanox.com using -f From: Viacheslav Ovsiienko To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, orika@mellanox.com, shahafs@mellanox.com, olivier.matz@6wind.com, stephen@networkplumber.org, thomas@mellanox.net Date: Mon, 20 Jan 2020 17:23:22 +0000 Message-Id: <1579541003-2399-5-git-send-email-viacheslavo@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> References: <20191118094938.192850-1-shahafs@mellanox.com> <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> Subject: [dpdk-dev] [PATCH v5 4/5] app/testpmd: add mempool with external data buffers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The new mbuf pool type is added to testpmd. To engage the mbuf pool with externally attached data buffers the parameter "--mp-alloc=xbuf" should be specified in testpmd command line. The objective of this patch is just to test whether mbuf pool with externally attached data buffers works OK. The memory for data buffers is allocated from DPDK memory, so this is not "true" external memory from some physical device (this is supposed the most common use case for such kind of mbuf pool). The user should be aware that not all drivers support the mbuf with EXT_ATTACHED_BUF flags set in newly allocated mbuf (many PMDs just overwrite ol_flags field and flag value is getting lost). Signed-off-by: Viacheslav Ovsiienko Acked-by: Olivier Matz --- app/test-pmd/config.c | 2 ++ app/test-pmd/flowgen.c | 3 +- app/test-pmd/parameters.c | 2 ++ app/test-pmd/testpmd.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++ app/test-pmd/testpmd.h | 4 ++- app/test-pmd/txonly.c | 3 +- 6 files changed, 92 insertions(+), 3 deletions(-) diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c index 52f1d9d..9669cbd 100644 --- a/app/test-pmd/config.c +++ b/app/test-pmd/config.c @@ -2422,6 +2422,8 @@ struct igb_ring_desc_16_bytes { return "xmem"; case MP_ALLOC_XMEM_HUGE: return "xmemhuge"; + case MP_ALLOC_XBUF: + return "xbuf"; default: return "invalid"; } diff --git a/app/test-pmd/flowgen.c b/app/test-pmd/flowgen.c index 03b72aa..ae50cdc 100644 --- a/app/test-pmd/flowgen.c +++ b/app/test-pmd/flowgen.c @@ -199,7 +199,8 @@ sizeof(*ip_hdr)); pkt->nb_segs = 1; pkt->pkt_len = pkt_size; - pkt->ol_flags = ol_flags; + pkt->ol_flags &= EXT_ATTACHED_MBUF; + pkt->ol_flags |= ol_flags; pkt->vlan_tci = vlan_tci; pkt->vlan_tci_outer = vlan_tci_outer; pkt->l2_len = sizeof(struct rte_ether_hdr); diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index 2e7a504..6340104 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -841,6 +841,8 @@ mp_alloc_type = MP_ALLOC_XMEM; else if (!strcmp(optarg, "xmemhuge")) mp_alloc_type = MP_ALLOC_XMEM_HUGE; + else if (!strcmp(optarg, "xbuf")) + mp_alloc_type = MP_ALLOC_XBUF; else rte_exit(EXIT_FAILURE, "mp-alloc %s invalid - must be: " diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 38dbb12..f9f4cd1 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -78,6 +78,7 @@ #endif #define EXTMEM_HEAP_NAME "extmem" +#define EXTBUF_ZONE_SIZE RTE_PGSIZE_2M uint16_t verbose_level = 0; /**< Silent by default. */ int testpmd_logtype; /**< Log type for testpmd logs */ @@ -868,6 +869,66 @@ struct extmem_param { } } +static unsigned int +setup_extbuf(uint32_t nb_mbufs, uint16_t mbuf_sz, unsigned int socket_id, + char *pool_name, struct rte_pktmbuf_extmem **ext_mem) +{ + struct rte_pktmbuf_extmem *xmem; + unsigned int ext_num, zone_num, elt_num; + uint16_t elt_size; + + elt_size = RTE_ALIGN_CEIL(mbuf_sz, RTE_CACHE_LINE_SIZE); + elt_num = EXTBUF_ZONE_SIZE / elt_size; + zone_num = (nb_mbufs + elt_num - 1) / elt_num; + + xmem = malloc(sizeof(struct rte_pktmbuf_extmem) * zone_num); + if (xmem == NULL) { + TESTPMD_LOG(ERR, "Cannot allocate memory for " + "external buffer descriptors\n"); + *ext_mem = NULL; + return 0; + } + for (ext_num = 0; ext_num < zone_num; ext_num++) { + struct rte_pktmbuf_extmem *xseg = xmem + ext_num; + const struct rte_memzone *mz; + char mz_name[RTE_MEMZONE_NAMESIZE]; + int ret; + + ret = snprintf(mz_name, sizeof(mz_name), + RTE_MEMPOOL_MZ_FORMAT "_xb_%u", pool_name, ext_num); + if (ret < 0 || ret >= (int)sizeof(mz_name)) { + errno = ENAMETOOLONG; + ext_num = 0; + break; + } + mz = rte_memzone_reserve_aligned(mz_name, EXTBUF_ZONE_SIZE, + socket_id, + RTE_MEMZONE_IOVA_CONTIG | + RTE_MEMZONE_1GB | + RTE_MEMZONE_SIZE_HINT_ONLY, + EXTBUF_ZONE_SIZE); + if (mz == NULL) { + /* + * The caller exits on external buffer creation + * error, so there is no need to free memzones. + */ + errno = ENOMEM; + ext_num = 0; + break; + } + xseg->buf_ptr = mz->addr; + xseg->buf_iova = mz->iova; + xseg->buf_len = EXTBUF_ZONE_SIZE; + xseg->elt_size = elt_size; + } + if (ext_num == 0 && xmem != NULL) { + free(xmem); + xmem = NULL; + } + *ext_mem = xmem; + return ext_num; +} + /* * Configuration initialisation done once at init time. */ @@ -936,6 +997,26 @@ struct extmem_param { heap_socket); break; } + case MP_ALLOC_XBUF: + { + struct rte_pktmbuf_extmem *ext_mem; + unsigned int ext_num; + + ext_num = setup_extbuf(nb_mbuf, mbuf_seg_size, + socket_id, pool_name, &ext_mem); + if (ext_num == 0) + rte_exit(EXIT_FAILURE, + "Can't create pinned data buffers\n"); + + TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n", + rte_mbuf_best_mempool_ops()); + rte_mp = rte_pktmbuf_pool_create_extbuf + (pool_name, nb_mbuf, mb_mempool_cache, + 0, mbuf_seg_size, socket_id, + ext_mem, ext_num); + free(ext_mem); + break; + } default: { rte_exit(EXIT_FAILURE, "Invalid mempool creation mode\n"); diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 7cf48d0..3dd5fc7 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -76,8 +76,10 @@ enum { /**< allocate mempool natively, but populate using anonymous memory */ MP_ALLOC_XMEM, /**< allocate and populate mempool using anonymous memory */ - MP_ALLOC_XMEM_HUGE + MP_ALLOC_XMEM_HUGE, /**< allocate and populate mempool using anonymous hugepage memory */ + MP_ALLOC_XBUF + /**< allocate mempool natively, use rte_pktmbuf_pool_create_extbuf */ }; #ifdef RTE_TEST_PMD_RECORD_BURST_STATS diff --git a/app/test-pmd/txonly.c b/app/test-pmd/txonly.c index 3caf281..871cf6c 100644 --- a/app/test-pmd/txonly.c +++ b/app/test-pmd/txonly.c @@ -170,7 +170,8 @@ rte_pktmbuf_reset_headroom(pkt); pkt->data_len = tx_pkt_seg_lengths[0]; - pkt->ol_flags = ol_flags; + pkt->ol_flags &= EXT_ATTACHED_MBUF; + pkt->ol_flags |= ol_flags; pkt->vlan_tci = vlan_tci; pkt->vlan_tci_outer = vlan_tci_outer; pkt->l2_len = sizeof(struct rte_ether_hdr); From patchwork Mon Jan 20 17:23:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 64984 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8B2BDA0526; Mon, 20 Jan 2020 18:24:12 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 38D631C1A3; Mon, 20 Jan 2020 18:23:42 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 554771C135 for ; Mon, 20 Jan 2020 18:23:38 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from viacheslavo@mellanox.com) with ESMTPS (AES256-SHA encrypted); 20 Jan 2020 19:23:34 +0200 Received: from pegasus11.mtr.labs.mlnx (pegasus11.mtr.labs.mlnx [10.210.16.104]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 00KHNYSA011386; Mon, 20 Jan 2020 19:23:34 +0200 Received: from pegasus11.mtr.labs.mlnx (localhost [127.0.0.1]) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7) with ESMTP id 00KHNXS2002514; Mon, 20 Jan 2020 17:23:33 GMT Received: (from viacheslavo@localhost) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7/Submit) id 00KHNXYq002513; Mon, 20 Jan 2020 17:23:33 GMT X-Authentication-Warning: pegasus11.mtr.labs.mlnx: viacheslavo set sender to viacheslavo@mellanox.com using -f From: Viacheslav Ovsiienko To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, orika@mellanox.com, shahafs@mellanox.com, olivier.matz@6wind.com, stephen@networkplumber.org, thomas@mellanox.net Date: Mon, 20 Jan 2020 17:23:23 +0000 Message-Id: <1579541003-2399-6-git-send-email-viacheslavo@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> References: <20191118094938.192850-1-shahafs@mellanox.com> <1579541003-2399-1-git-send-email-viacheslavo@mellanox.com> Subject: [dpdk-dev] [PATCH v5 5/5] net/mlx5: allow use allocated mbuf with external buffer X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In the Rx datapath the flags in the newly allocated mbufs are all explicitly cleared but the EXT_ATTACHED_MBUF must be preserved. It would allow to use mbuf pools with pre-attached external data buffers. The vectorized rx_burst routines are updated in order to inherit the EXT_ATTACHED_MBUF from mbuf pool private RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF flag. Signed-off-by: Viacheslav Ovsiienko Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_rxq.c | 7 ++++++- drivers/net/mlx5/mlx5_rxtx.c | 2 +- drivers/net/mlx5/mlx5_rxtx.h | 2 +- drivers/net/mlx5/mlx5_rxtx_vec.h | 14 ++++---------- drivers/net/mlx5/mlx5_rxtx_vec_altivec.h | 5 ++--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 29 +++++++++++++++-------------- drivers/net/mlx5/mlx5_rxtx_vec_sse.h | 2 +- 7 files changed, 30 insertions(+), 31 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index c936a7f..4092cb7 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -225,6 +225,9 @@ if (mlx5_rxq_check_vec_support(&rxq_ctrl->rxq) > 0) { struct mlx5_rxq_data *rxq = &rxq_ctrl->rxq; struct rte_mbuf *mbuf_init = &rxq->fake_mbuf; + struct rte_pktmbuf_pool_private *priv = + (struct rte_pktmbuf_pool_private *) + rte_mempool_get_priv(rxq_ctrl->rxq.mp); int j; /* Initialize default rearm_data for vPMD. */ @@ -232,13 +235,15 @@ rte_mbuf_refcnt_set(mbuf_init, 1); mbuf_init->nb_segs = 1; mbuf_init->port = rxq->port_id; + if (priv->flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) + mbuf_init->ol_flags = EXT_ATTACHED_MBUF; /* * prevent compiler reordering: * rearm_data covers previous fields. */ rte_compiler_barrier(); rxq->mbuf_initializer = - *(uint64_t *)&mbuf_init->rearm_data; + *(rte_xmm_t *)&mbuf_init->rearm_data; /* Padding with a fake mbuf for vectorized Rx. */ for (j = 0; j < MLX5_VPMD_DESCS_PER_LOOP; ++j) (*rxq->elts)[elts_n + j] = &rxq->fake_mbuf; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 67cafd1..5e31f01 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -1337,7 +1337,7 @@ enum mlx5_txcmp_code { } pkt = seg; assert(len >= (rxq->crc_present << 2)); - pkt->ol_flags = 0; + pkt->ol_flags &= EXT_ATTACHED_MBUF; /* If compressed, take hash result from mini-CQE. */ rss_hash_res = rte_be_to_cpu_32(mcqe == NULL ? cqe->rx_hash_res : diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index b6a33c5..3f659d2 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -144,7 +144,7 @@ struct mlx5_rxq_data { struct mlx5_mprq_buf *mprq_repl; /* Stashed mbuf for replenish. */ uint16_t idx; /* Queue index. */ struct mlx5_rxq_stats stats; - uint64_t mbuf_initializer; /* Default rearm_data for vectorized Rx. */ + rte_xmm_t mbuf_initializer; /* Default rearm/flags for vectorized Rx. */ struct rte_mbuf fake_mbuf; /* elts padding for vectorized Rx. */ void *cq_uar; /* CQ user access region. */ uint32_t cqn; /* CQ number. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rxtx_vec.h index 85e0bd5..d8c07f2 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec.h @@ -97,18 +97,12 @@ void *buf_addr; /* - * Load the virtual address for Rx WQE. non-x86 processors - * (mostly RISC such as ARM and Power) are more vulnerable to - * load stall. For x86, reducing the number of instructions - * seems to matter most. + * In order to support the mbufs with external attached + * data buffer we should use the buf_addr pointer instead of + * rte_mbuf_buf_addr(). It touches the mbuf itself and may + * impact the performance. */ -#ifdef RTE_ARCH_X86_64 buf_addr = elts[i]->buf_addr; - assert(buf_addr == rte_mbuf_buf_addr(elts[i], rxq->mp)); -#else - buf_addr = rte_mbuf_buf_addr(elts[i], rxq->mp); - assert(buf_addr == elts[i]->buf_addr); -#endif wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + RTE_PKTMBUF_HEADROOM); /* If there's only one MR, no need to replace LKey in WQE. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h index 8e79883..9e5c6ee 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h @@ -344,9 +344,8 @@ PKT_RX_IP_CKSUM_GOOD | PKT_RX_L4_CKSUM_GOOD | PKT_RX_VLAN | PKT_RX_VLAN_STRIPPED}; const vector unsigned char mbuf_init = - (vector unsigned char)(vector unsigned long){ - *(__attribute__((__aligned__(8))) unsigned long *) - &rxq->mbuf_initializer, 0LL}; + (vector unsigned char)vec_vsx_ld + (0, (vector unsigned char *)&rxq->mbuf_initializer); const vector unsigned short rearm_sel_mask = (vector unsigned short){0, 0, 0, 0, 0xffff, 0xffff, 0, 0}; vector unsigned char rearm0, rearm1, rearm2, rearm3; diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h index 86785c7..332e9ac 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h @@ -264,8 +264,8 @@ const uint32x4_t cv_mask = vdupq_n_u32(PKT_RX_IP_CKSUM_GOOD | PKT_RX_L4_CKSUM_GOOD | PKT_RX_VLAN | PKT_RX_VLAN_STRIPPED); - const uint64x1_t mbuf_init = vld1_u64(&rxq->mbuf_initializer); - const uint64x1_t r32_mask = vcreate_u64(0xffffffff); + const uint64x2_t mbuf_init = vld1q_u64 + ((const uint64_t *)&rxq->mbuf_initializer); uint64x2_t rearm0, rearm1, rearm2, rearm3; uint8_t pt_idx0, pt_idx1, pt_idx2, pt_idx3; @@ -326,18 +326,19 @@ /* Merge to ol_flags. */ ol_flags = vorrq_u32(ol_flags, cv_flags); /* Merge mbuf_init and ol_flags, and store. */ - rearm0 = vcombine_u64(mbuf_init, - vshr_n_u64(vget_high_u64(vreinterpretq_u64_u32( - ol_flags)), 32)); - rearm1 = vcombine_u64(mbuf_init, - vand_u64(vget_high_u64(vreinterpretq_u64_u32( - ol_flags)), r32_mask)); - rearm2 = vcombine_u64(mbuf_init, - vshr_n_u64(vget_low_u64(vreinterpretq_u64_u32( - ol_flags)), 32)); - rearm3 = vcombine_u64(mbuf_init, - vand_u64(vget_low_u64(vreinterpretq_u64_u32( - ol_flags)), r32_mask)); + rearm0 = vreinterpretq_u64_u32(vsetq_lane_u32 + (vgetq_lane_u32(ol_flags, 3), + vreinterpretq_u32_u64(mbuf_init), 2)); + rearm1 = vreinterpretq_u64_u32(vsetq_lane_u32 + (vgetq_lane_u32(ol_flags, 2), + vreinterpretq_u32_u64(mbuf_init), 2)); + rearm2 = vreinterpretq_u64_u32(vsetq_lane_u32 + (vgetq_lane_u32(ol_flags, 1), + vreinterpretq_u32_u64(mbuf_init), 2)); + rearm3 = vreinterpretq_u64_u32(vsetq_lane_u32 + (vgetq_lane_u32(ol_flags, 0), + vreinterpretq_u32_u64(mbuf_init), 2)); + vst1q_u64((void *)&pkts[0]->rearm_data, rearm0); vst1q_u64((void *)&pkts[1]->rearm_data, rearm1); vst1q_u64((void *)&pkts[2]->rearm_data, rearm2); diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h index 35b7761..07d40d5 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h @@ -259,7 +259,7 @@ PKT_RX_IP_CKSUM_GOOD | PKT_RX_L4_CKSUM_GOOD | PKT_RX_VLAN | PKT_RX_VLAN_STRIPPED); const __m128i mbuf_init = - _mm_loadl_epi64((__m128i *)&rxq->mbuf_initializer); + _mm_load_si128((__m128i *)&rxq->mbuf_initializer); __m128i rearm0, rearm1, rearm2, rearm3; uint8_t pt_idx0, pt_idx1, pt_idx2, pt_idx3;