From patchwork Tue Oct 26 16:28:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102950 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 94F5DA0547; Tue, 26 Oct 2021 18:29:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 752B841137; Tue, 26 Oct 2021 18:29:22 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 2E02D4111F for ; Tue, 26 Oct 2021 18:29:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FSVJ5dvIlZGwRvt3dMQ92XKx2aXYlSaM5F8RwY0BCqI=; b=bDwfC5lIJGLIMHa7G6jajB1uIFXAK+MucRQ26iHEOj0atjwbj4CF6I115G8twu27xZspFv JvyJtwNlZ0lPNtQ8KcSm1xNCfO32pHDEiC7OvJwjW4vK8g0+tB1Zyq01OV173VZ5W6ZT6c ZydyedUNdsSgJIT/GyOmJf/OPt1TPZs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-456-O2PFLe1GPG2o08TH0UilTw-1; Tue, 26 Oct 2021 12:29:14 -0400 X-MC-Unique: O2PFLe1GPG2o08TH0UilTw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E31CE1808302; Tue, 26 Oct 2021 16:29:12 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id DEA05100E809; Tue, 26 Oct 2021 16:29:10 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:50 +0200 Message-Id: <20211026162904.482987-2-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 01/15] vhost: move async data in a dedicated structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch moves async-related metadata from vhost_virtqueue to a dedicated struct. It makes it clear which fields are async related, and also saves some memory when async feature is not in use. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.c | 130 +++++++++++++++++------------------------ lib/vhost/vhost.h | 53 ++++++++--------- lib/vhost/vhost_user.c | 4 +- lib/vhost/virtio_net.c | 114 +++++++++++++++++++----------------- 4 files changed, 142 insertions(+), 159 deletions(-) diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index 3b674ac320..452583f68d 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -340,19 +340,18 @@ cleanup_device(struct virtio_net *dev, int destroy) static void vhost_free_async_mem(struct vhost_virtqueue *vq) { - rte_free(vq->async_pkts_info); + if (!vq->async) + return; - rte_free(vq->async_buffers_packed); - vq->async_buffers_packed = NULL; - rte_free(vq->async_descs_split); - vq->async_descs_split = NULL; + rte_free(vq->async->pkts_info); - rte_free(vq->it_pool); - rte_free(vq->vec_pool); + rte_free(vq->async->buffers_packed); + vq->async->buffers_packed = NULL; + rte_free(vq->async->descs_split); + vq->async->descs_split = NULL; - vq->async_pkts_info = NULL; - vq->it_pool = NULL; - vq->vec_pool = NULL; + rte_free(vq->async); + vq->async = NULL; } void @@ -1632,77 +1631,63 @@ async_channel_register(int vid, uint16_t queue_id, { struct virtio_net *dev = get_device(vid); struct vhost_virtqueue *vq = dev->virtqueue[queue_id]; + struct vhost_async *async; + int node = vq->numa_node; - if (unlikely(vq->async_registered)) { + if (unlikely(vq->async)) { VHOST_LOG_CONFIG(ERR, - "async register failed: channel already registered " - "(vid %d, qid: %d)\n", vid, queue_id); + "async register failed: already registered (vid %d, qid: %d)\n", + vid, queue_id); return -1; } - vq->async_pkts_info = rte_malloc_socket(NULL, - vq->size * sizeof(struct async_inflight_info), - RTE_CACHE_LINE_SIZE, vq->numa_node); - if (!vq->async_pkts_info) { - vhost_free_async_mem(vq); - VHOST_LOG_CONFIG(ERR, - "async register failed: cannot allocate memory for async_pkts_info " - "(vid %d, qid: %d)\n", vid, queue_id); + async = rte_zmalloc_socket(NULL, sizeof(struct vhost_async), 0, node); + if (!async) { + VHOST_LOG_CONFIG(ERR, "failed to allocate async metadata (vid %d, qid: %d)\n", + vid, queue_id); return -1; } - vq->it_pool = rte_malloc_socket(NULL, - VHOST_MAX_ASYNC_IT * sizeof(struct rte_vhost_iov_iter), - RTE_CACHE_LINE_SIZE, vq->numa_node); - if (!vq->it_pool) { - vhost_free_async_mem(vq); - VHOST_LOG_CONFIG(ERR, - "async register failed: cannot allocate memory for it_pool " - "(vid %d, qid: %d)\n", vid, queue_id); - return -1; - } - - vq->vec_pool = rte_malloc_socket(NULL, - VHOST_MAX_ASYNC_VEC * sizeof(struct iovec), - RTE_CACHE_LINE_SIZE, vq->numa_node); - if (!vq->vec_pool) { - vhost_free_async_mem(vq); - VHOST_LOG_CONFIG(ERR, - "async register failed: cannot allocate memory for vec_pool " - "(vid %d, qid: %d)\n", vid, queue_id); - return -1; + async->pkts_info = rte_malloc_socket(NULL, vq->size * sizeof(struct async_inflight_info), + RTE_CACHE_LINE_SIZE, node); + if (!async->pkts_info) { + VHOST_LOG_CONFIG(ERR, "failed to allocate async_pkts_info (vid %d, qid: %d)\n", + vid, queue_id); + goto out_free_async; } if (vq_is_packed(dev)) { - vq->async_buffers_packed = rte_malloc_socket(NULL, - vq->size * sizeof(struct vring_used_elem_packed), - RTE_CACHE_LINE_SIZE, vq->numa_node); - if (!vq->async_buffers_packed) { - vhost_free_async_mem(vq); - VHOST_LOG_CONFIG(ERR, - "async register failed: cannot allocate memory for async buffers " - "(vid %d, qid: %d)\n", vid, queue_id); - return -1; + async->buffers_packed = rte_malloc_socket(NULL, + vq->size * sizeof(struct vring_used_elem_packed), + RTE_CACHE_LINE_SIZE, node); + if (!async->buffers_packed) { + VHOST_LOG_CONFIG(ERR, "failed to allocate async buffers (vid %d, qid: %d)\n", + vid, queue_id); + goto out_free_inflight; } } else { - vq->async_descs_split = rte_malloc_socket(NULL, - vq->size * sizeof(struct vring_used_elem), - RTE_CACHE_LINE_SIZE, vq->numa_node); - if (!vq->async_descs_split) { - vhost_free_async_mem(vq); - VHOST_LOG_CONFIG(ERR, - "async register failed: cannot allocate memory for async descs " - "(vid %d, qid: %d)\n", vid, queue_id); - return -1; + async->descs_split = rte_malloc_socket(NULL, + vq->size * sizeof(struct vring_used_elem), + RTE_CACHE_LINE_SIZE, node); + if (!async->descs_split) { + VHOST_LOG_CONFIG(ERR, "failed to allocate async descs (vid %d, qid: %d)\n", + vid, queue_id); + goto out_free_inflight; } } - vq->async_ops.check_completed_copies = ops->check_completed_copies; - vq->async_ops.transfer_data = ops->transfer_data; + async->ops.check_completed_copies = ops->check_completed_copies; + async->ops.transfer_data = ops->transfer_data; - vq->async_registered = true; + vq->async = async; return 0; +out_free_inflight: + rte_free(async->pkts_info); +out_free_async: + rte_free(async); + + return -1; } int @@ -1796,7 +1781,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id) ret = 0; - if (!vq->async_registered) + if (!vq->async) return ret; if (!rte_spinlock_trylock(&vq->access_lock)) { @@ -1805,7 +1790,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id) return -1; } - if (vq->async_pkts_inflight_n) { + if (vq->async->pkts_inflight_n) { VHOST_LOG_CONFIG(ERR, "Failed to unregister async channel. " "async inflight packets must be completed before unregistration.\n"); ret = -1; @@ -1813,11 +1798,6 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id) } vhost_free_async_mem(vq); - - vq->async_ops.transfer_data = NULL; - vq->async_ops.check_completed_copies = NULL; - vq->async_registered = false; - out: rte_spinlock_unlock(&vq->access_lock); @@ -1841,10 +1821,10 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id) if (vq == NULL) return -1; - if (!vq->async_registered) + if (!vq->async) return 0; - if (vq->async_pkts_inflight_n) { + if (vq->async->pkts_inflight_n) { VHOST_LOG_CONFIG(ERR, "Failed to unregister async channel. " "async inflight packets must be completed before unregistration.\n"); return -1; @@ -1852,10 +1832,6 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id) vhost_free_async_mem(vq); - vq->async_ops.transfer_data = NULL; - vq->async_ops.check_completed_copies = NULL; - vq->async_registered = false; - return 0; } @@ -1877,7 +1853,7 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id) if (vq == NULL) return ret; - if (!vq->async_registered) + if (!vq->async) return ret; if (!rte_spinlock_trylock(&vq->access_lock)) { @@ -1886,7 +1862,7 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id) return ret; } - ret = vq->async_pkts_inflight_n; + ret = vq->async->pkts_inflight_n; rte_spinlock_unlock(&vq->access_lock); return ret; diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 6c6a2da2c9..a556d61063 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -119,6 +119,32 @@ struct vring_used_elem_packed { uint32_t count; }; +struct vhost_async { + /* operation callbacks for DMA */ + struct rte_vhost_async_channel_ops ops; + + struct rte_vhost_iov_iter it_pool[VHOST_MAX_ASYNC_IT]; + struct iovec vec_pool[VHOST_MAX_ASYNC_VEC]; + + /* data transfer status */ + struct async_inflight_info *pkts_info; + uint16_t pkts_idx; + uint16_t pkts_inflight_n; + uint16_t last_pkts_n; + union { + struct vring_used_elem *descs_split; + struct vring_used_elem_packed *buffers_packed; + }; + union { + uint16_t desc_idx_split; + uint16_t buffer_idx_packed; + }; + union { + uint16_t last_desc_idx_split; + uint16_t last_buffer_idx_packed; + }; +}; + /** * Structure contains variables relevant to RX/TX virtqueues. */ @@ -193,32 +219,7 @@ struct vhost_virtqueue { struct rte_vhost_resubmit_info *resubmit_inflight; uint64_t global_counter; - /* operation callbacks for async dma */ - struct rte_vhost_async_channel_ops async_ops; - - struct rte_vhost_iov_iter *it_pool; - struct iovec *vec_pool; - - /* async data transfer status */ - struct async_inflight_info *async_pkts_info; - uint16_t async_pkts_idx; - uint16_t async_pkts_inflight_n; - uint16_t async_last_pkts_n; - union { - struct vring_used_elem *async_descs_split; - struct vring_used_elem_packed *async_buffers_packed; - }; - union { - uint16_t async_desc_idx_split; - uint16_t async_buffer_idx_packed; - }; - union { - uint16_t last_async_desc_idx_split; - uint16_t last_async_buffer_idx_packed; - }; - - /* vq async features */ - bool async_registered; + struct vhost_async *async; int notif_enable; #define VIRTIO_UNINITIALIZED_NOTIF (-1) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 720d1c1c9d..0bc23c8ae2 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2252,8 +2252,8 @@ vhost_user_set_vring_enable(struct virtio_net **pdev, "set queue enable: %d to qp idx: %d\n", enable, index); - if (enable && dev->virtqueue[index]->async_registered) { - if (dev->virtqueue[index]->async_pkts_inflight_n) { + if (enable && dev->virtqueue[index]->async) { + if (dev->virtqueue[index]->async->pkts_inflight_n) { VHOST_LOG_CONFIG(ERR, "failed to enable vring. " "async inflight packets must be completed first\n"); return RTE_VHOST_MSG_RESULT_ERR; diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index b6140a643b..d9ac85f829 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -1510,12 +1510,13 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, uint16_t num_buffers; uint16_t avail_head; - struct rte_vhost_iov_iter *it_pool = vq->it_pool; - struct iovec *vec_pool = vq->vec_pool; + struct vhost_async *async = vq->async; + struct rte_vhost_iov_iter *it_pool = async->it_pool; + struct iovec *vec_pool = async->vec_pool; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; struct iovec *src_iovec = vec_pool; struct iovec *dst_iovec = vec_pool + (VHOST_MAX_ASYNC_VEC >> 1); - struct async_inflight_info *pkts_info = vq->async_pkts_info; + struct async_inflight_info *pkts_info = async->pkts_info; uint32_t n_pkts = 0, pkt_err = 0; int32_t n_xfer; uint16_t segs_await = 0; @@ -1556,7 +1557,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, async_fill_desc(&tdes[pkt_burst_idx++], &it_pool[it_idx], &it_pool[it_idx + 1]); - slot_idx = (vq->async_pkts_idx + pkt_idx) & (vq->size - 1); + slot_idx = (async->pkts_idx + pkt_idx) & (vq->size - 1); pkts_info[slot_idx].descs = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; @@ -1574,7 +1575,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, if (unlikely(pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || ((VHOST_MAX_ASYNC_VEC >> 1) - segs_await < BUF_VECTOR_MAX))) { - n_xfer = vq->async_ops.transfer_data(dev->vid, + n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { n_pkts = n_xfer; @@ -1606,7 +1607,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, } if (pkt_burst_idx) { - n_xfer = vq->async_ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); + n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { n_pkts = n_xfer; } else { @@ -1638,15 +1639,15 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, /* keep used descriptors */ if (likely(vq->shadow_used_idx)) { - uint16_t to = vq->async_desc_idx_split & (vq->size - 1); + uint16_t to = async->desc_idx_split & (vq->size - 1); store_dma_desc_info_split(vq->shadow_used_split, - vq->async_descs_split, vq->size, 0, to, + async->descs_split, vq->size, 0, to, vq->shadow_used_idx); - vq->async_desc_idx_split += vq->shadow_used_idx; - vq->async_pkts_idx += pkt_idx; - vq->async_pkts_inflight_n += pkt_idx; + async->desc_idx_split += vq->shadow_used_idx; + async->pkts_idx += pkt_idx; + async->pkts_inflight_n += pkt_idx; vq->shadow_used_idx = 0; } @@ -1798,7 +1799,7 @@ dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx, { uint16_t descs_err = 0; uint16_t buffers_err = 0; - struct async_inflight_info *pkts_info = vq->async_pkts_info; + struct async_inflight_info *pkts_info = vq->async->pkts_info; *pkt_idx -= nr_err; /* calculate the sum of buffers and descs of DMA-error packets. */ @@ -1829,12 +1830,13 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, uint16_t num_buffers; uint16_t num_descs; - struct rte_vhost_iov_iter *it_pool = vq->it_pool; - struct iovec *vec_pool = vq->vec_pool; + struct vhost_async *async = vq->async; + struct rte_vhost_iov_iter *it_pool = async->it_pool; + struct iovec *vec_pool = async->vec_pool; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; struct iovec *src_iovec = vec_pool; struct iovec *dst_iovec = vec_pool + (VHOST_MAX_ASYNC_VEC >> 1); - struct async_inflight_info *pkts_info = vq->async_pkts_info; + struct async_inflight_info *pkts_info = async->pkts_info; uint32_t n_pkts = 0, pkt_err = 0; uint16_t slot_idx = 0; uint16_t segs_await = 0; @@ -1851,7 +1853,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, &it_pool[it_idx], &it_pool[it_idx + 1]) < 0)) break; - slot_idx = (vq->async_pkts_idx + pkt_idx) % vq->size; + slot_idx = (async->pkts_idx + pkt_idx) % vq->size; async_fill_desc(&tdes[pkt_burst_idx++], &it_pool[it_idx], &it_pool[it_idx + 1]); @@ -1873,7 +1875,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, */ if (unlikely(pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || ((VHOST_MAX_ASYNC_VEC >> 1) - segs_await < BUF_VECTOR_MAX))) { - n_xfer = vq->async_ops.transfer_data(dev->vid, + n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { n_pkts = n_xfer; @@ -1904,7 +1906,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, } while (pkt_idx < count); if (pkt_burst_idx) { - n_xfer = vq->async_ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); + n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { n_pkts = n_xfer; } else { @@ -1922,20 +1924,20 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, if (likely(vq->shadow_used_idx)) { /* keep used descriptors. */ - store_dma_desc_info_packed(vq->shadow_used_packed, vq->async_buffers_packed, - vq->size, 0, vq->async_buffer_idx_packed, + store_dma_desc_info_packed(vq->shadow_used_packed, async->buffers_packed, + vq->size, 0, async->buffer_idx_packed, vq->shadow_used_idx); - vq->async_buffer_idx_packed += vq->shadow_used_idx; - if (vq->async_buffer_idx_packed >= vq->size) - vq->async_buffer_idx_packed -= vq->size; + async->buffer_idx_packed += vq->shadow_used_idx; + if (async->buffer_idx_packed >= vq->size) + async->buffer_idx_packed -= vq->size; - vq->async_pkts_idx += pkt_idx; - if (vq->async_pkts_idx >= vq->size) - vq->async_pkts_idx -= vq->size; + async->pkts_idx += pkt_idx; + if (async->pkts_idx >= vq->size) + async->pkts_idx -= vq->size; vq->shadow_used_idx = 0; - vq->async_pkts_inflight_n += pkt_idx; + async->pkts_inflight_n += pkt_idx; } return pkt_idx; @@ -1944,28 +1946,29 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, static __rte_always_inline void write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs) { + struct vhost_async *async = vq->async; uint16_t nr_left = n_descs; uint16_t nr_copy; uint16_t to, from; do { - from = vq->last_async_desc_idx_split & (vq->size - 1); + from = async->last_desc_idx_split & (vq->size - 1); nr_copy = nr_left + from <= vq->size ? nr_left : vq->size - from; to = vq->last_used_idx & (vq->size - 1); if (to + nr_copy <= vq->size) { - rte_memcpy(&vq->used->ring[to], &vq->async_descs_split[from], + rte_memcpy(&vq->used->ring[to], &async->descs_split[from], nr_copy * sizeof(struct vring_used_elem)); } else { uint16_t size = vq->size - to; - rte_memcpy(&vq->used->ring[to], &vq->async_descs_split[from], + rte_memcpy(&vq->used->ring[to], &async->descs_split[from], size * sizeof(struct vring_used_elem)); - rte_memcpy(&vq->used->ring[0], &vq->async_descs_split[from + size], + rte_memcpy(&vq->used->ring[0], &async->descs_split[from + size], (nr_copy - size) * sizeof(struct vring_used_elem)); } - vq->last_async_desc_idx_split += nr_copy; + async->last_desc_idx_split += nr_copy; vq->last_used_idx += nr_copy; nr_left -= nr_copy; } while (nr_left > 0); @@ -1975,20 +1978,21 @@ static __rte_always_inline void write_back_completed_descs_packed(struct vhost_virtqueue *vq, uint16_t n_buffers) { + struct vhost_async *async = vq->async; uint16_t nr_left = n_buffers; uint16_t from, to; do { - from = vq->last_async_buffer_idx_packed; + from = async->last_buffer_idx_packed; to = (from + nr_left) % vq->size; if (to > from) { - vhost_update_used_packed(vq, vq->async_buffers_packed + from, to - from); - vq->last_async_buffer_idx_packed += nr_left; + vhost_update_used_packed(vq, async->buffers_packed + from, to - from); + async->last_buffer_idx_packed += nr_left; nr_left = 0; } else { - vhost_update_used_packed(vq, vq->async_buffers_packed + from, + vhost_update_used_packed(vq, async->buffers_packed + from, vq->size - from); - vq->last_async_buffer_idx_packed = 0; + async->last_buffer_idx_packed = 0; nr_left -= vq->size - from; } } while (nr_left > 0); @@ -1999,6 +2003,7 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, struct rte_mbuf **pkts, uint16_t count) { struct vhost_virtqueue *vq; + struct vhost_async *async; struct async_inflight_info *pkts_info; int32_t n_cpl; uint16_t n_pkts_cpl = 0, n_pkts_put = 0, n_descs = 0, n_buffers = 0; @@ -2006,15 +2011,16 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, uint16_t from, i; vq = dev->virtqueue[queue_id]; - pkts_idx = vq->async_pkts_idx % vq->size; - pkts_info = vq->async_pkts_info; + async = vq->async; + pkts_idx = async->pkts_idx % vq->size; + pkts_info = async->pkts_info; vq_size = vq->size; start_idx = virtio_dev_rx_async_get_info_idx(pkts_idx, - vq_size, vq->async_pkts_inflight_n); + vq_size, async->pkts_inflight_n); - if (count > vq->async_last_pkts_n) { - n_cpl = vq->async_ops.check_completed_copies(dev->vid, - queue_id, 0, count - vq->async_last_pkts_n); + if (count > async->last_pkts_n) { + n_cpl = async->ops.check_completed_copies(dev->vid, + queue_id, 0, count - async->last_pkts_n); if (likely(n_cpl >= 0)) { n_pkts_cpl = n_cpl; } else { @@ -2025,10 +2031,10 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, } } - n_pkts_cpl += vq->async_last_pkts_n; + n_pkts_cpl += async->last_pkts_n; n_pkts_put = RTE_MIN(n_pkts_cpl, count); if (unlikely(n_pkts_put == 0)) { - vq->async_last_pkts_n = n_pkts_cpl; + async->last_pkts_n = n_pkts_cpl; return 0; } @@ -2045,8 +2051,8 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, pkts[i] = pkts_info[from].mbuf; } } - vq->async_last_pkts_n = n_pkts_cpl - n_pkts_put; - vq->async_pkts_inflight_n -= n_pkts_put; + async->last_pkts_n = n_pkts_cpl - n_pkts_put; + async->pkts_inflight_n -= n_pkts_put; if (likely(vq->enabled && vq->access_ok)) { if (vq_is_packed(dev)) { @@ -2062,11 +2068,11 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, } } else { if (vq_is_packed(dev)) { - vq->last_async_buffer_idx_packed += n_buffers; - if (vq->last_async_buffer_idx_packed >= vq->size) - vq->last_async_buffer_idx_packed -= vq->size; + async->last_buffer_idx_packed += n_buffers; + if (async->last_buffer_idx_packed >= vq->size) + async->last_buffer_idx_packed -= vq->size; } else { - vq->last_async_desc_idx_split += n_descs; + async->last_desc_idx_split += n_descs; } } @@ -2093,7 +2099,7 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id, vq = dev->virtqueue[queue_id]; - if (unlikely(!vq->async_registered)) { + if (unlikely(!vq->async)) { VHOST_LOG_DATA(ERR, "(%d) %s: async not registered for queue id %d.\n", dev->vid, __func__, queue_id); return 0; @@ -2128,7 +2134,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id, vq = dev->virtqueue[queue_id]; - if (unlikely(!vq->async_registered)) { + if (unlikely(!vq->async)) { VHOST_LOG_DATA(ERR, "(%d) %s: async not registered for queue id %d.\n", dev->vid, __func__, queue_id); return 0; @@ -2157,7 +2163,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id, rte_spinlock_lock(&vq->access_lock); - if (unlikely(!vq->enabled || !vq->async_registered)) + if (unlikely(!vq->enabled || !vq->async)) goto out_access_unlock; if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) From patchwork Tue Oct 26 16:28:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102949 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5ECEEA0547; Tue, 26 Oct 2021 18:29:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F6E041122; Tue, 26 Oct 2021 18:29:21 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 112B34111E for ; Tue, 26 Oct 2021 18:29:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5PLW3TmEickxJaUcwp7+R48Z6pgp1IBNUKr38xk5Jhg=; b=IojArXjON+BjNqiVzett/Y5KQ60ceEcd2bB6GmyZs2G++7V0q6kcuePEJNT7y84/2KJpl3 qxg6nJ4VGPz5DSHtOeg7Mduuk2SXdRWMpwVyASbbx79cFy5G3XhahmJpeiPQSuBPcq9O6T gL8F4W6VwiCDgAD7yzw36jPXIYyv20M= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-126-X7I9IqpvNSKPEWs2zPhjFA-1; Tue, 26 Oct 2021 12:29:16 -0400 X-MC-Unique: X7I9IqpvNSKPEWs2zPhjFA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 10C541808302; Tue, 26 Oct 2021 16:29:15 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3F54210016FE; Tue, 26 Oct 2021 16:29:13 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:51 +0200 Message-Id: <20211026162904.482987-3-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 02/15] vhost: hide inflight async structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch moves async_inflight_info struct to internal header since it should not be part of the API. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/rte_vhost_async.h | 9 --------- lib/vhost/vhost.h | 9 +++++++++ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h index ad71555a7f..789b80f5a0 100644 --- a/lib/vhost/rte_vhost_async.h +++ b/lib/vhost/rte_vhost_async.h @@ -83,15 +83,6 @@ struct rte_vhost_async_channel_ops { uint16_t max_packets); }; -/** - * inflight async packet information - */ -struct async_inflight_info { - struct rte_mbuf *mbuf; - uint16_t descs; /* num of descs inflight */ - uint16_t nr_buffers; /* num of buffers inflight for packed ring */ -}; - /** * async channel features */ diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index a556d61063..8c1c33c852 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -119,6 +119,15 @@ struct vring_used_elem_packed { uint32_t count; }; +/** + * inflight async packet information + */ +struct async_inflight_info { + struct rte_mbuf *mbuf; + uint16_t descs; /* num of descs inflight */ + uint16_t nr_buffers; /* num of buffers inflight for packed ring */ +}; + struct vhost_async { /* operation callbacks for DMA */ struct rte_vhost_async_channel_ops ops; From patchwork Tue Oct 26 16:28:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102952 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9171CA0547; Tue, 26 Oct 2021 18:29:43 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A9B5D4114D; Tue, 26 Oct 2021 18:29:27 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 94C014113C for ; Tue, 26 Oct 2021 18:29:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265761; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AmXoYJyUMlKvIwqhbe721AhgZ8O2xEFvE8vTBa36eNo=; b=FLuytC1PzBwb/oMtz2kFP80hy4lyKJQKn1sV4VblR+g+uj709y/qP/aUM5+/8rlfprbUnB PC1edXBC5Gb+ncefEZ6ov7TAIFICBLdzoZRlquU94vojhW7I//Fiq10drk8sTJIrc7QVNG Vw87udEr6qTWL57EWmE2rkYW6xHwZE4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-349-6jVQcuTvNmaYBJkXByDzhQ-1; Tue, 26 Oct 2021 12:29:18 -0400 X-MC-Unique: 6jVQcuTvNmaYBJkXByDzhQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3156B1397B01; Tue, 26 Oct 2021 16:29:17 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 61ABC10016FE; Tue, 26 Oct 2021 16:29:15 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:52 +0200 Message-Id: <20211026162904.482987-4-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 03/15] vhost: simplify async IO vectors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" IO vectors implementation is unnecessarily complex, mixing source and destinations vectors in the same array. This patch declares two arrays, one for the source and one for the destination. It also get rid off seg_awaits variable in both packed and split implementation, which is the same as iovec_idx. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.h | 5 +++-- lib/vhost/virtio_net.c | 28 +++++++++++----------------- 2 files changed, 14 insertions(+), 19 deletions(-) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 8c1c33c852..686f468eff 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -49,7 +49,7 @@ #define MAX_PKT_BURST 32 #define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST * 2) -#define VHOST_MAX_ASYNC_VEC (BUF_VECTOR_MAX * 4) +#define VHOST_MAX_ASYNC_VEC (BUF_VECTOR_MAX * 2) #define PACKED_DESC_ENQUEUE_USED_FLAG(w) \ ((w) ? (VRING_DESC_F_AVAIL | VRING_DESC_F_USED | VRING_DESC_F_WRITE) : \ @@ -133,7 +133,8 @@ struct vhost_async { struct rte_vhost_async_channel_ops ops; struct rte_vhost_iov_iter it_pool[VHOST_MAX_ASYNC_IT]; - struct iovec vec_pool[VHOST_MAX_ASYNC_VEC]; + struct iovec src_iovec[VHOST_MAX_ASYNC_VEC]; + struct iovec dst_iovec[VHOST_MAX_ASYNC_VEC]; /* data transfer status */ struct async_inflight_info *pkts_info; diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index d9ac85f829..2a243701c0 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -1512,14 +1512,12 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_async *async = vq->async; struct rte_vhost_iov_iter *it_pool = async->it_pool; - struct iovec *vec_pool = async->vec_pool; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; - struct iovec *src_iovec = vec_pool; - struct iovec *dst_iovec = vec_pool + (VHOST_MAX_ASYNC_VEC >> 1); + struct iovec *src_iovec = async->src_iovec; + struct iovec *dst_iovec = async->dst_iovec; struct async_inflight_info *pkts_info = async->pkts_info; uint32_t n_pkts = 0, pkt_err = 0; int32_t n_xfer; - uint16_t segs_await = 0; uint16_t iovec_idx = 0, it_idx = 0, slot_idx = 0; /* @@ -1562,7 +1560,6 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, pkts_info[slot_idx].mbuf = pkts[pkt_idx]; iovec_idx += it_pool[it_idx].nr_segs; - segs_await += it_pool[it_idx].nr_segs; it_idx += 2; vq->last_avail_idx += num_buffers; @@ -1573,8 +1570,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, * - unused async iov number is less than max vhost vector */ if (unlikely(pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || - ((VHOST_MAX_ASYNC_VEC >> 1) - segs_await < - BUF_VECTOR_MAX))) { + (VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX))) { n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { @@ -1588,7 +1584,6 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, iovec_idx = 0; it_idx = 0; - segs_await = 0; if (unlikely(n_pkts < pkt_burst_idx)) { /* @@ -1745,8 +1740,11 @@ vhost_enqueue_async_packed(struct virtio_net *dev, if (unlikely(++tries > max_tries)) return -1; - if (unlikely(fill_vec_buf_packed(dev, vq, avail_idx, &desc_count, buf_vec, &nr_vec, - &buf_id, &len, VHOST_ACCESS_RW) < 0)) + if (unlikely(fill_vec_buf_packed(dev, vq, + avail_idx, &desc_count, + buf_vec, &nr_vec, + &buf_id, &len, + VHOST_ACCESS_RW) < 0)) return -1; len = RTE_MIN(len, size); @@ -1832,14 +1830,12 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_async *async = vq->async; struct rte_vhost_iov_iter *it_pool = async->it_pool; - struct iovec *vec_pool = async->vec_pool; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; - struct iovec *src_iovec = vec_pool; - struct iovec *dst_iovec = vec_pool + (VHOST_MAX_ASYNC_VEC >> 1); + struct iovec *src_iovec = async->src_iovec; + struct iovec *dst_iovec = async->dst_iovec; struct async_inflight_info *pkts_info = async->pkts_info; uint32_t n_pkts = 0, pkt_err = 0; uint16_t slot_idx = 0; - uint16_t segs_await = 0; uint16_t iovec_idx = 0, it_idx = 0; do { @@ -1861,7 +1857,6 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, pkts_info[slot_idx].nr_buffers = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; iovec_idx += it_pool[it_idx].nr_segs; - segs_await += it_pool[it_idx].nr_segs; it_idx += 2; pkt_idx++; @@ -1874,7 +1869,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, * - unused async iov number is less than max vhost vector */ if (unlikely(pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || - ((VHOST_MAX_ASYNC_VEC >> 1) - segs_await < BUF_VECTOR_MAX))) { + (VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX))) { n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { @@ -1888,7 +1883,6 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, iovec_idx = 0; it_idx = 0; - segs_await = 0; if (unlikely(n_pkts < pkt_burst_idx)) { /* From patchwork Tue Oct 26 16:28:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102951 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2BBDDA0547; Tue, 26 Oct 2021 18:29:37 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B37A941145; Tue, 26 Oct 2021 18:29:26 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 742D441136 for ; Tue, 26 Oct 2021 18:29:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265762; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wYivXFg4vyA5ZfGVDqBt4ojn3XbeNJfgwAN3puFXSNg=; b=U2hv8OFbxhP5n6kIGGpcpFrs1q38F/ZiMcXHJ0tWxoqwc3fYhVKSp3ki/cy5uq+ylD1/3f C4NCxx+o9TKMFPh/TXqXJIpD5kY0l48zk4651ItRO9lk7mMkTzqguYfRdcQNmDZHmW4xny 5am6Qqr3mYUXWS1zhz9QHiEAyXCvyyU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-10-_7k-vjrWMAGvjeLVIDte8g-1; Tue, 26 Oct 2021 12:29:20 -0400 X-MC-Unique: _7k-vjrWMAGvjeLVIDte8g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 734401397B00; Tue, 26 Oct 2021 16:29:19 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7F7FB100E809; Tue, 26 Oct 2021 16:29:17 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:53 +0200 Message-Id: <20211026162904.482987-5-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 04/15] vhost: simplify async IO vectors iterators X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch splits the iterator arrays in two, one for source and one for destination. The goal is make the code easier to understand. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.h | 5 +++-- lib/vhost/virtio_net.c | 24 ++++++++++++------------ 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 686f468eff..132c1b070e 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -48,7 +48,7 @@ #define MAX_PKT_BURST 32 -#define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST * 2) +#define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST) #define VHOST_MAX_ASYNC_VEC (BUF_VECTOR_MAX * 2) #define PACKED_DESC_ENQUEUE_USED_FLAG(w) \ @@ -132,7 +132,8 @@ struct vhost_async { /* operation callbacks for DMA */ struct rte_vhost_async_channel_ops ops; - struct rte_vhost_iov_iter it_pool[VHOST_MAX_ASYNC_IT]; + struct rte_vhost_iov_iter src_iov_iter[VHOST_MAX_ASYNC_IT]; + struct rte_vhost_iov_iter dst_iov_iter[VHOST_MAX_ASYNC_IT]; struct iovec src_iovec[VHOST_MAX_ASYNC_VEC]; struct iovec dst_iovec[VHOST_MAX_ASYNC_VEC]; diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index 2a243701c0..f0d923704a 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -1511,7 +1511,8 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, uint16_t avail_head; struct vhost_async *async = vq->async; - struct rte_vhost_iov_iter *it_pool = async->it_pool; + struct rte_vhost_iov_iter *src_iter = async->src_iov_iter; + struct rte_vhost_iov_iter *dst_iter = async->dst_iov_iter; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; struct iovec *src_iovec = async->src_iovec; struct iovec *dst_iovec = async->dst_iovec; @@ -1547,20 +1548,19 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, if (async_mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, num_buffers, &src_iovec[iovec_idx], &dst_iovec[iovec_idx], - &it_pool[it_idx], &it_pool[it_idx + 1]) < 0) { + &src_iter[it_idx], &dst_iter[it_idx]) < 0) { vq->shadow_used_idx -= num_buffers; break; } - async_fill_desc(&tdes[pkt_burst_idx++], &it_pool[it_idx], - &it_pool[it_idx + 1]); + async_fill_desc(&tdes[pkt_burst_idx++], &src_iter[it_idx], &dst_iter[it_idx]); slot_idx = (async->pkts_idx + pkt_idx) & (vq->size - 1); pkts_info[slot_idx].descs = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; - iovec_idx += it_pool[it_idx].nr_segs; - it_idx += 2; + iovec_idx += src_iter[it_idx].nr_segs; + it_idx++; vq->last_avail_idx += num_buffers; @@ -1829,7 +1829,8 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, uint16_t num_descs; struct vhost_async *async = vq->async; - struct rte_vhost_iov_iter *it_pool = async->it_pool; + struct rte_vhost_iov_iter *src_iter = async->src_iov_iter; + struct rte_vhost_iov_iter *dst_iter = async->dst_iov_iter; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; struct iovec *src_iovec = async->src_iovec; struct iovec *dst_iovec = async->dst_iovec; @@ -1846,18 +1847,17 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, if (unlikely(virtio_dev_rx_async_packed(dev, vq, pkts[pkt_idx], &num_descs, &num_buffers, &src_iovec[iovec_idx], &dst_iovec[iovec_idx], - &it_pool[it_idx], &it_pool[it_idx + 1]) < 0)) + &src_iter[it_idx], &dst_iter[it_idx]) < 0)) break; slot_idx = (async->pkts_idx + pkt_idx) % vq->size; - async_fill_desc(&tdes[pkt_burst_idx++], &it_pool[it_idx], - &it_pool[it_idx + 1]); + async_fill_desc(&tdes[pkt_burst_idx++], &src_iter[it_idx], &dst_iter[it_idx]); pkts_info[slot_idx].descs = num_descs; pkts_info[slot_idx].nr_buffers = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; - iovec_idx += it_pool[it_idx].nr_segs; - it_idx += 2; + iovec_idx += src_iter[it_idx].nr_segs; + it_idx++; pkt_idx++; remained--; From patchwork Tue Oct 26 16:28:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102953 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 38AB8A0547; Tue, 26 Oct 2021 18:29:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AFEF841151; Tue, 26 Oct 2021 18:29:28 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 65E99410D5 for ; Tue, 26 Oct 2021 18:29:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265763; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7FPF6OfKHy+z8s4JOR4WV/U5/+qfQmwHD2KThBsJfr0=; b=NGoNL/wnuyfdpk3aVhVfNBV/CYjA94FAftGOF/f1ZGQ/aiNiHmC5UjD4htdqTxcUv6kOap xih6l0htrpOECprLGxc2i5KDUgK57ZhKd38A/8kA+e9/gOMdZ1EEd5oy7Kt8K/Qk41ssLE /t5Wy/pAc7gHWHF/C5NpAZI1YW8VIpE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-58-rT2rybI6POqb9R-zc4dijA-1; Tue, 26 Oct 2021 12:29:22 -0400 X-MC-Unique: rT2rybI6POqb9R-zc4dijA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 785471808302; Tue, 26 Oct 2021 16:29:21 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id A882310016FE; Tue, 26 Oct 2021 16:29:19 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:54 +0200 Message-Id: <20211026162904.482987-6-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 05/15] vhost: remove async batch threshold X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Reaching the async batch threshold was one of the condition to trigger the DMA transfer. However, this condition was never met since the threshold value is 32, same as the MAX_PKT_BURST value. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/virtio_net.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index f0d923704a..c4a8b5276f 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -25,8 +25,6 @@ #define MAX_BATCH_LEN 256 -#define VHOST_ASYNC_BATCH_THRESHOLD 32 - static __rte_always_inline bool rxvq_is_mergeable(struct virtio_net *dev) { @@ -1565,12 +1563,10 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, vq->last_avail_idx += num_buffers; /* - * conditions to trigger async device transfer: - * - buffered packet number reaches transfer threshold + * condition to trigger async device transfer: * - unused async iov number is less than max vhost vector */ - if (unlikely(pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || - (VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX))) { + if (unlikely(VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX)) { n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { @@ -1864,12 +1860,10 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, vq_inc_last_avail_packed(vq, num_descs); /* - * conditions to trigger async device transfer: - * - buffered packet number reaches transfer threshold + * condition to trigger async device transfer: * - unused async iov number is less than max vhost vector */ - if (unlikely(pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || - (VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX))) { + if (unlikely(VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX)) { n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); if (likely(n_xfer >= 0)) { From patchwork Tue Oct 26 16:28:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102954 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DF11CA0547; Tue, 26 Oct 2021 18:29:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A009641156; Tue, 26 Oct 2021 18:29:31 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id E510141154 for ; Tue, 26 Oct 2021 18:29:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265768; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CnMZZ2nJFGdNLozQh7nF95upW+nK1beX7yJAj4y3is4=; b=W2vlM2X5aFRT4SnW6JVu+zhC3zfwnCQiT5UHgLyAx1wPTRAkHoc2emuBz4FN2Y4um43pgg UgrEIFyO/JH2a/9SW0q7oos4YCsFiicUR3CaQRtfiXNlBnuYKZ5yQuna5PRl5bBF3EdHfi x1XKauahKwu7rH/9LpQJP+sVkVuMTHw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-321-rJM98HarOo2oaYHYIjw63A-1; Tue, 26 Oct 2021 12:29:25 -0400 X-MC-Unique: rJM98HarOo2oaYHYIjw63A-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9D68436268; Tue, 26 Oct 2021 16:29:23 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id C6A8910023B8; Tue, 26 Oct 2021 16:29:21 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:55 +0200 Message-Id: <20211026162904.482987-7-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 06/15] vhost: introduce specific iovec structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch introduces rte_vhost_iovec struct that contains both source and destination addresses since we always have a 1:1 mapping between source and destination. While using the standard iovec struct might have seemed better, having to duplicate IO vectors and its iterators is memory inefficient and make the implementation more complex. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- examples/vhost/ioat.c | 24 ++++++------- lib/vhost/rte_vhost_async.h | 19 +++++++---- lib/vhost/vhost.h | 6 ++-- lib/vhost/virtio_net.c | 68 ++++++++++++++----------------------- 4 files changed, 52 insertions(+), 65 deletions(-) diff --git a/examples/vhost/ioat.c b/examples/vhost/ioat.c index 457f8171f0..dcbcf65e4e 100644 --- a/examples/vhost/ioat.c +++ b/examples/vhost/ioat.c @@ -129,33 +129,31 @@ ioat_transfer_data_cb(int vid, uint16_t queue_id, { uint32_t i_desc; uint16_t dev_id = dma_bind[vid].dmas[queue_id * 2 + VIRTIO_RXQ].dev_id; - struct rte_vhost_iov_iter *src = NULL; - struct rte_vhost_iov_iter *dst = NULL; + struct rte_vhost_iov_iter *iter = NULL; unsigned long i_seg; unsigned short mask = MAX_ENQUEUED_SIZE - 1; unsigned short write = cb_tracker[dev_id].next_write; if (!opaque_data) { for (i_desc = 0; i_desc < count; i_desc++) { - src = descs[i_desc].src; - dst = descs[i_desc].dst; + iter = descs[i_desc].iter; i_seg = 0; - if (cb_tracker[dev_id].ioat_space < src->nr_segs) + if (cb_tracker[dev_id].ioat_space < iter->nr_segs) break; - while (i_seg < src->nr_segs) { + while (i_seg < iter->nr_segs) { rte_ioat_enqueue_copy(dev_id, - (uintptr_t)(src->iov[i_seg].iov_base) - + src->offset, - (uintptr_t)(dst->iov[i_seg].iov_base) - + dst->offset, - src->iov[i_seg].iov_len, + (uintptr_t)(iter->iov[i_seg].src_addr) + + iter->offset, + (uintptr_t)(iter->iov[i_seg].dst_addr) + + iter->offset, + iter->iov[i_seg].len, 0, 0); i_seg++; } write &= mask; - cb_tracker[dev_id].size_track[write] = src->nr_segs; - cb_tracker[dev_id].ioat_space -= src->nr_segs; + cb_tracker[dev_id].size_track[write] = iter->nr_segs; + cb_tracker[dev_id].ioat_space -= iter->nr_segs; write++; } } else { diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h index 789b80f5a0..d7bb77bf90 100644 --- a/lib/vhost/rte_vhost_async.h +++ b/lib/vhost/rte_vhost_async.h @@ -7,6 +7,15 @@ #include "rte_vhost.h" +/** + * iovec + */ +struct rte_vhost_iovec { + void *src_addr; + void *dst_addr; + size_t len; +}; + /** * iovec iterator */ @@ -16,19 +25,17 @@ struct rte_vhost_iov_iter { /** total bytes of data in this iterator */ size_t count; /** pointer to the iovec array */ - struct iovec *iov; + struct rte_vhost_iovec *iov; /** number of iovec in this iterator */ unsigned long nr_segs; }; /** - * dma transfer descriptor pair + * dma transfer descriptor */ struct rte_vhost_async_desc { - /** source memory iov_iter */ - struct rte_vhost_iov_iter *src; - /** destination memory iov_iter */ - struct rte_vhost_iov_iter *dst; + /* memory iov_iter */ + struct rte_vhost_iov_iter *iter; }; /** diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 132c1b070e..e51e496ebe 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -132,10 +132,8 @@ struct vhost_async { /* operation callbacks for DMA */ struct rte_vhost_async_channel_ops ops; - struct rte_vhost_iov_iter src_iov_iter[VHOST_MAX_ASYNC_IT]; - struct rte_vhost_iov_iter dst_iov_iter[VHOST_MAX_ASYNC_IT]; - struct iovec src_iovec[VHOST_MAX_ASYNC_VEC]; - struct iovec dst_iovec[VHOST_MAX_ASYNC_VEC]; + struct rte_vhost_iov_iter iov_iter[VHOST_MAX_ASYNC_IT]; + struct rte_vhost_iovec iovec[VHOST_MAX_ASYNC_VEC]; /* data transfer status */ struct async_inflight_info *pkts_info; diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index c4a8b5276f..3c8be48ca7 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -925,15 +925,16 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } static __rte_always_inline void -async_fill_vec(struct iovec *v, void *base, size_t len) +async_fill_vec(struct rte_vhost_iovec *v, void *src, void *dst, size_t len) { - v->iov_base = base; - v->iov_len = len; + v->src_addr = src; + v->dst_addr = dst; + v->len = len; } static __rte_always_inline void async_fill_iter(struct rte_vhost_iov_iter *it, size_t count, - struct iovec *vec, unsigned long nr_seg) + struct rte_vhost_iovec *vec, unsigned long nr_seg) { it->offset = 0; it->count = count; @@ -948,20 +949,16 @@ async_fill_iter(struct rte_vhost_iov_iter *it, size_t count, } static __rte_always_inline void -async_fill_desc(struct rte_vhost_async_desc *desc, - struct rte_vhost_iov_iter *src, struct rte_vhost_iov_iter *dst) +async_fill_desc(struct rte_vhost_async_desc *desc, struct rte_vhost_iov_iter *iter) { - desc->src = src; - desc->dst = dst; + desc->iter = iter; } static __rte_always_inline int async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec, uint16_t nr_vec, uint16_t num_buffers, - struct iovec *src_iovec, struct iovec *dst_iovec, - struct rte_vhost_iov_iter *src_it, - struct rte_vhost_iov_iter *dst_it) + struct rte_vhost_iovec *iovec, struct rte_vhost_iov_iter *iter) { struct rte_mbuf *hdr_mbuf; struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; @@ -1075,11 +1072,9 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, goto out; } - async_fill_vec(src_iovec + tvec_idx, + async_fill_vec(iovec + tvec_idx, (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, - mbuf_offset), (size_t)mapped_len); - async_fill_vec(dst_iovec + tvec_idx, - hpa, (size_t)mapped_len); + mbuf_offset), hpa, (size_t)mapped_len); tlen += (uint32_t)mapped_len; cpy_len -= (uint32_t)mapped_len; @@ -1091,8 +1086,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } - async_fill_iter(src_it, tlen, src_iovec, tvec_idx); - async_fill_iter(dst_it, tlen, dst_iovec, tvec_idx); + async_fill_iter(iter, tlen, iovec, tvec_idx); out: return error; } @@ -1509,11 +1503,9 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, uint16_t avail_head; struct vhost_async *async = vq->async; - struct rte_vhost_iov_iter *src_iter = async->src_iov_iter; - struct rte_vhost_iov_iter *dst_iter = async->dst_iov_iter; + struct rte_vhost_iov_iter *iter = async->iov_iter; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; - struct iovec *src_iovec = async->src_iovec; - struct iovec *dst_iovec = async->dst_iovec; + struct rte_vhost_iovec *iovec = async->iovec; struct async_inflight_info *pkts_info = async->pkts_info; uint32_t n_pkts = 0, pkt_err = 0; int32_t n_xfer; @@ -1545,19 +1537,18 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, vq->last_avail_idx + num_buffers); if (async_mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, num_buffers, - &src_iovec[iovec_idx], &dst_iovec[iovec_idx], - &src_iter[it_idx], &dst_iter[it_idx]) < 0) { + &iovec[iovec_idx], &iter[it_idx]) < 0) { vq->shadow_used_idx -= num_buffers; break; } - async_fill_desc(&tdes[pkt_burst_idx++], &src_iter[it_idx], &dst_iter[it_idx]); + async_fill_desc(&tdes[pkt_burst_idx++], &iter[it_idx]); slot_idx = (async->pkts_idx + pkt_idx) & (vq->size - 1); pkts_info[slot_idx].descs = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; - iovec_idx += src_iter[it_idx].nr_segs; + iovec_idx += iter[it_idx].nr_segs; it_idx++; vq->last_avail_idx += num_buffers; @@ -1707,9 +1698,8 @@ vhost_enqueue_async_packed(struct virtio_net *dev, struct buf_vector *buf_vec, uint16_t *nr_descs, uint16_t *nr_buffers, - struct iovec *src_iovec, struct iovec *dst_iovec, - struct rte_vhost_iov_iter *src_it, - struct rte_vhost_iov_iter *dst_it) + struct rte_vhost_iovec *iovec, + struct rte_vhost_iov_iter *iter) { uint16_t nr_vec = 0; uint16_t avail_idx = vq->last_avail_idx; @@ -1757,8 +1747,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev, } if (unlikely(async_mbuf_to_desc(dev, vq, pkt, buf_vec, nr_vec, - *nr_buffers, src_iovec, dst_iovec, - src_it, dst_it) < 0)) + *nr_buffers, iovec, iter) < 0)) return -1; vhost_shadow_enqueue_packed(vq, buffer_len, buffer_buf_id, buffer_desc_count, *nr_buffers); @@ -1769,14 +1758,12 @@ vhost_enqueue_async_packed(struct virtio_net *dev, static __rte_always_inline int16_t virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers, - struct iovec *src_iovec, struct iovec *dst_iovec, - struct rte_vhost_iov_iter *src_it, struct rte_vhost_iov_iter *dst_it) + struct rte_vhost_iovec *iovec, struct rte_vhost_iov_iter *iter) { struct buf_vector buf_vec[BUF_VECTOR_MAX]; if (unlikely(vhost_enqueue_async_packed(dev, vq, pkt, buf_vec, nr_descs, nr_buffers, - src_iovec, dst_iovec, - src_it, dst_it) < 0)) { + iovec, iter) < 0)) { VHOST_LOG_DATA(DEBUG, "(%d) failed to get enough desc from vring\n", dev->vid); return -1; } @@ -1825,11 +1812,9 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, uint16_t num_descs; struct vhost_async *async = vq->async; - struct rte_vhost_iov_iter *src_iter = async->src_iov_iter; - struct rte_vhost_iov_iter *dst_iter = async->dst_iov_iter; + struct rte_vhost_iov_iter *iter = async->iov_iter; struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; - struct iovec *src_iovec = async->src_iovec; - struct iovec *dst_iovec = async->dst_iovec; + struct rte_vhost_iovec *iovec = async->iovec; struct async_inflight_info *pkts_info = async->pkts_info; uint32_t n_pkts = 0, pkt_err = 0; uint16_t slot_idx = 0; @@ -1842,17 +1827,16 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, num_descs = 0; if (unlikely(virtio_dev_rx_async_packed(dev, vq, pkts[pkt_idx], &num_descs, &num_buffers, - &src_iovec[iovec_idx], &dst_iovec[iovec_idx], - &src_iter[it_idx], &dst_iter[it_idx]) < 0)) + &iovec[iovec_idx], &iter[it_idx]) < 0)) break; slot_idx = (async->pkts_idx + pkt_idx) % vq->size; - async_fill_desc(&tdes[pkt_burst_idx++], &src_iter[it_idx], &dst_iter[it_idx]); + async_fill_desc(&tdes[pkt_burst_idx++], &iter[it_idx]); pkts_info[slot_idx].descs = num_descs; pkts_info[slot_idx].nr_buffers = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; - iovec_idx += src_iter[it_idx].nr_segs; + iovec_idx += iter[it_idx].nr_segs; it_idx++; pkt_idx++; From patchwork Tue Oct 26 16:28:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102955 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 004D4A0547; Tue, 26 Oct 2021 18:30:02 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A09F04115C; Tue, 26 Oct 2021 18:29:33 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id AC18A4115B for ; Tue, 26 Oct 2021 18:29:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VvpX2RUzxLBiFZOfMuppcLcJPnx79gcYGB8CX30nfCo=; b=MekvPd2kf6EGMo1++cnBBAiWDEPAoyZhYdLaSTVJV2IykW58HDZRF8Uu3XtHljLKIT1IiZ iFGbth5uuH3Lj1VRlkrpn0hAhLtaaw9744jdaxD0MwWRCbVmVVjUQFGI7QD040FvImVh2k 2HBPslQLuFoCrG+wclwR82wOtVdjgPU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-204-y_HsA303OuSZcO04_23aoA-1; Tue, 26 Oct 2021 12:29:27 -0400 X-MC-Unique: y_HsA303OuSZcO04_23aoA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BDCB48066F5; Tue, 26 Oct 2021 16:29:25 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id E9F9710016FE; Tue, 26 Oct 2021 16:29:23 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:56 +0200 Message-Id: <20211026162904.482987-8-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 07/15] vhost: remove useless fields in async iterator struct X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Offset and count fields are unused and so can be removed. The offset field was actually in the Vhost example, but in a way that does not make sense. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- examples/vhost/ioat.c | 6 ++---- lib/vhost/rte_vhost_async.h | 4 ---- lib/vhost/virtio_net.c | 19 ++++--------------- 3 files changed, 6 insertions(+), 23 deletions(-) diff --git a/examples/vhost/ioat.c b/examples/vhost/ioat.c index dcbcf65e4e..a8c588deff 100644 --- a/examples/vhost/ioat.c +++ b/examples/vhost/ioat.c @@ -142,10 +142,8 @@ ioat_transfer_data_cb(int vid, uint16_t queue_id, break; while (i_seg < iter->nr_segs) { rte_ioat_enqueue_copy(dev_id, - (uintptr_t)(iter->iov[i_seg].src_addr) - + iter->offset, - (uintptr_t)(iter->iov[i_seg].dst_addr) - + iter->offset, + (uintptr_t)(iter->iov[i_seg].src_addr), + (uintptr_t)(iter->iov[i_seg].dst_addr), iter->iov[i_seg].len, 0, 0); diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h index d7bb77bf90..4ea5cfab10 100644 --- a/lib/vhost/rte_vhost_async.h +++ b/lib/vhost/rte_vhost_async.h @@ -20,10 +20,6 @@ struct rte_vhost_iovec { * iovec iterator */ struct rte_vhost_iov_iter { - /** offset to the first byte of interesting data */ - size_t offset; - /** total bytes of data in this iterator */ - size_t count; /** pointer to the iovec array */ struct rte_vhost_iovec *iov; /** number of iovec in this iterator */ diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index 3c8be48ca7..29254a1b11 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -933,19 +933,10 @@ async_fill_vec(struct rte_vhost_iovec *v, void *src, void *dst, size_t len) } static __rte_always_inline void -async_fill_iter(struct rte_vhost_iov_iter *it, size_t count, - struct rte_vhost_iovec *vec, unsigned long nr_seg) +async_fill_iter(struct rte_vhost_iov_iter *it, struct rte_vhost_iovec *vec, unsigned long nr_seg) { - it->offset = 0; - it->count = count; - - if (count) { - it->iov = vec; - it->nr_segs = nr_seg; - } else { - it->iov = 0; - it->nr_segs = 0; - } + it->iov = vec; + it->nr_segs = nr_seg; } static __rte_always_inline void @@ -971,7 +962,6 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, uint32_t cpy_len, buf_len; int error = 0; - uint32_t tlen = 0; int tvec_idx = 0; void *hpa; @@ -1076,7 +1066,6 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, mbuf_offset), hpa, (size_t)mapped_len); - tlen += (uint32_t)mapped_len; cpy_len -= (uint32_t)mapped_len; mbuf_avail -= (uint32_t)mapped_len; mbuf_offset += (uint32_t)mapped_len; @@ -1086,7 +1075,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } - async_fill_iter(iter, tlen, iovec, tvec_idx); + async_fill_iter(iter, iovec, tvec_idx); out: return error; } From patchwork Tue Oct 26 16:28:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102956 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E90AFA0547; Tue, 26 Oct 2021 18:30:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EC3B641163; Tue, 26 Oct 2021 18:29:36 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 90D3041163 for ; Tue, 26 Oct 2021 18:29:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0SRsP45YBhh4B7hFINfaYCrcMF8yE+jJDAluKhn+MS0=; b=Ik+BQXxuP6kfCbvoabfh40k6omfp+G6HiUH03rr/GsiZuyctH/P69Zkwg4X7TfD4fE1KJa F7JrocnjYrvBv7XJlcSk5CAn1Sq1QNdw7RIMN/VY3MHnJrXx8SpxZqgg7eZW4jPX9p2D0Z hi8OuRLOaVXxcboxIVHYEEHGXYZRADA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-495-HbhiyQZSOSOaExdlgoiFQg-1; Tue, 26 Oct 2021 12:29:31 -0400 X-MC-Unique: HbhiyQZSOSOaExdlgoiFQg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7DDE21006AA2; Tue, 26 Oct 2021 16:29:30 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18051100E809; Tue, 26 Oct 2021 16:29:25 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:57 +0200 Message-Id: <20211026162904.482987-9-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 08/15] vhost: improve IO vector logic X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" IO vectors and their iterators arrays were part of the async metadata but not their indexes. In order to makes this more consistent, the patch adds the indexes to the async metadata. Doing that, we can avoid triggering DMA transfer within the loop as it IO vector index overflow is now prevented in the async_mbuf_to_desc() function. Note that previous detection mechanism was broken since the overflow already happened when detected, so OOB memory access would already have happened. With this changes done, virtio_dev_rx_async_submit_split() and virtio_dev_rx_async_submit_packed() can be further simplified. Signed-off-by: Maxime Coquelin --- lib/vhost/vhost.h | 2 + lib/vhost/virtio_net.c | 297 +++++++++++++++++++---------------------- 2 files changed, 137 insertions(+), 162 deletions(-) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index e51e496ebe..9afb75fa6e 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -134,6 +134,8 @@ struct vhost_async { struct rte_vhost_iov_iter iov_iter[VHOST_MAX_ASYNC_IT]; struct rte_vhost_iovec iovec[VHOST_MAX_ASYNC_VEC]; + uint16_t iter_idx; + uint16_t iovec_idx; /* data transfer status */ struct async_inflight_info *pkts_info; diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index 29254a1b11..a2044aae1b 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -924,33 +924,92 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, return error; } +static __rte_always_inline int +async_iter_initialize(struct vhost_async *async) +{ + struct rte_vhost_iov_iter *iter; + + if (unlikely(async->iovec_idx >= VHOST_MAX_ASYNC_VEC)) { + VHOST_LOG_DATA(ERR, "no more async iovec available\n"); + return -1; + } + + iter = async->iov_iter + async->iter_idx; + iter->iov = async->iovec + async->iovec_idx; + iter->nr_segs = 0; + + return 0; +} + +static __rte_always_inline int +async_iter_add_iovec(struct vhost_async *async, void *src, void *dst, size_t len) +{ + struct rte_vhost_iov_iter *iter; + struct rte_vhost_iovec *iovec; + + if (unlikely(async->iovec_idx >= VHOST_MAX_ASYNC_VEC)) { + static bool vhost_max_async_vec_log; + + if (!vhost_max_async_vec_log) { + VHOST_LOG_DATA(ERR, "no more async iovec available\n"); + vhost_max_async_vec_log = true; + } + + return -1; + } + + iter = async->iov_iter + async->iter_idx; + iovec = async->iovec + async->iovec_idx; + + iovec->src_addr = src; + iovec->dst_addr = dst; + iovec->len = len; + + iter->nr_segs++; + async->iovec_idx++; + + return 0; +} + static __rte_always_inline void -async_fill_vec(struct rte_vhost_iovec *v, void *src, void *dst, size_t len) +async_iter_finalize(struct vhost_async *async) { - v->src_addr = src; - v->dst_addr = dst; - v->len = len; + async->iter_idx++; } static __rte_always_inline void -async_fill_iter(struct rte_vhost_iov_iter *it, struct rte_vhost_iovec *vec, unsigned long nr_seg) +async_iter_cancel(struct vhost_async *async) { - it->iov = vec; - it->nr_segs = nr_seg; + struct rte_vhost_iov_iter *iter; + + iter = async->iov_iter + async->iter_idx; + async->iovec_idx -= iter->nr_segs; + iter->nr_segs = 0; + iter->iov = NULL; } static __rte_always_inline void -async_fill_desc(struct rte_vhost_async_desc *desc, struct rte_vhost_iov_iter *iter) +async_iter_reset(struct vhost_async *async) { - desc->iter = iter; + async->iter_idx = 0; + async->iovec_idx = 0; +} + +static __rte_always_inline void +async_fill_descs(struct vhost_async *async, struct rte_vhost_async_desc *descs) +{ + int i; + + for (i = 0; i < async->iter_idx; i++) + descs[i].iter = async->iov_iter + i; } static __rte_always_inline int async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec, - uint16_t nr_vec, uint16_t num_buffers, - struct rte_vhost_iovec *iovec, struct rte_vhost_iov_iter *iter) + uint16_t nr_vec, uint16_t num_buffers) { + struct vhost_async *async = vq->async; struct rte_mbuf *hdr_mbuf; struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; uint64_t buf_addr, buf_iova; @@ -960,24 +1019,18 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, uint32_t mbuf_offset, mbuf_avail; uint32_t buf_offset, buf_avail; uint32_t cpy_len, buf_len; - int error = 0; - int tvec_idx = 0; void *hpa; - if (unlikely(m == NULL)) { - error = -1; - goto out; - } + if (unlikely(m == NULL)) + return -1; buf_addr = buf_vec[vec_idx].buf_addr; buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; - if (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1)) { - error = -1; - goto out; - } + if (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1)) + return -1; hdr_mbuf = m; hdr_addr = buf_addr; @@ -1005,14 +1058,15 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, mbuf_avail = rte_pktmbuf_data_len(m); mbuf_offset = 0; + if (async_iter_initialize(async)) + return -1; + while (mbuf_avail != 0 || m->next != NULL) { /* done with current buf, get the next one */ if (buf_avail == 0) { vec_idx++; - if (unlikely(vec_idx >= nr_vec)) { - error = -1; - goto out; - } + if (unlikely(vec_idx >= nr_vec)) + goto error; buf_addr = buf_vec[vec_idx].buf_addr; buf_iova = buf_vec[vec_idx].buf_iova; @@ -1058,26 +1112,30 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, if (unlikely(!hpa)) { VHOST_LOG_DATA(ERR, "(%d) %s: failed to get hpa.\n", dev->vid, __func__); - error = -1; - goto out; + goto error; } - async_fill_vec(iovec + tvec_idx, - (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, - mbuf_offset), hpa, (size_t)mapped_len); + if (unlikely(async_iter_add_iovec(async, + (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, + mbuf_offset), + hpa, (size_t)mapped_len))) + goto error; cpy_len -= (uint32_t)mapped_len; mbuf_avail -= (uint32_t)mapped_len; mbuf_offset += (uint32_t)mapped_len; buf_avail -= (uint32_t)mapped_len; buf_offset += (uint32_t)mapped_len; - tvec_idx++; } } - async_fill_iter(iter, iovec, tvec_idx); -out: - return error; + async_iter_finalize(async); + + return 0; +error: + async_iter_cancel(async); + + return -1; } static __rte_always_inline int @@ -1487,18 +1545,16 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct rte_mbuf **pkts, uint32_t count) { struct buf_vector buf_vec[BUF_VECTOR_MAX]; - uint32_t pkt_idx = 0, pkt_burst_idx = 0; + uint32_t pkt_idx = 0; uint16_t num_buffers; uint16_t avail_head; struct vhost_async *async = vq->async; - struct rte_vhost_iov_iter *iter = async->iov_iter; - struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; - struct rte_vhost_iovec *iovec = async->iovec; + struct rte_vhost_async_desc async_descs[MAX_PKT_BURST]; struct async_inflight_info *pkts_info = async->pkts_info; - uint32_t n_pkts = 0, pkt_err = 0; + uint32_t pkt_err = 0; int32_t n_xfer; - uint16_t iovec_idx = 0, it_idx = 0, slot_idx = 0; + uint16_t slot_idx = 0; /* * The ordering between avail index and desc reads need to be enforced. @@ -1507,95 +1563,53 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]); + async_iter_reset(async); + for (pkt_idx = 0; pkt_idx < count; pkt_idx++) { uint32_t pkt_len = pkts[pkt_idx]->pkt_len + dev->vhost_hlen; uint16_t nr_vec = 0; - if (unlikely(reserve_avail_buf_split(dev, vq, - pkt_len, buf_vec, &num_buffers, - avail_head, &nr_vec) < 0)) { - VHOST_LOG_DATA(DEBUG, - "(%d) failed to get enough desc from vring\n", - dev->vid); + if (unlikely(reserve_avail_buf_split(dev, vq, pkt_len, buf_vec, + &num_buffers, avail_head, &nr_vec) < 0)) { + VHOST_LOG_DATA(DEBUG, "(%d) failed to get enough desc from vring\n", + dev->vid); vq->shadow_used_idx -= num_buffers; break; } VHOST_LOG_DATA(DEBUG, "(%d) current index %d | end index %d\n", - dev->vid, vq->last_avail_idx, - vq->last_avail_idx + num_buffers); + dev->vid, vq->last_avail_idx, vq->last_avail_idx + num_buffers); - if (async_mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, num_buffers, - &iovec[iovec_idx], &iter[it_idx]) < 0) { + if (async_mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, num_buffers) < 0) { vq->shadow_used_idx -= num_buffers; break; } - async_fill_desc(&tdes[pkt_burst_idx++], &iter[it_idx]); - slot_idx = (async->pkts_idx + pkt_idx) & (vq->size - 1); pkts_info[slot_idx].descs = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; - iovec_idx += iter[it_idx].nr_segs; - it_idx++; - vq->last_avail_idx += num_buffers; + } - /* - * condition to trigger async device transfer: - * - unused async iov number is less than max vhost vector - */ - if (unlikely(VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX)) { - n_xfer = async->ops.transfer_data(dev->vid, - queue_id, tdes, 0, pkt_burst_idx); - if (likely(n_xfer >= 0)) { - n_pkts = n_xfer; - } else { - VHOST_LOG_DATA(ERR, - "(%d) %s: failed to transfer data for queue id %d.\n", - dev->vid, __func__, queue_id); - n_pkts = 0; - } - - iovec_idx = 0; - it_idx = 0; - - if (unlikely(n_pkts < pkt_burst_idx)) { - /* - * log error packets number here and do actual - * error processing when applications poll - * completion - */ - pkt_err = pkt_burst_idx - n_pkts; - pkt_idx++; - pkt_burst_idx = 0; - break; - } + if (unlikely(pkt_idx == 0)) + return 0; - pkt_burst_idx = 0; - } - } + async_fill_descs(async, async_descs); - if (pkt_burst_idx) { - n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); - if (likely(n_xfer >= 0)) { - n_pkts = n_xfer; - } else { - VHOST_LOG_DATA(ERR, "(%d) %s: failed to transfer data for queue id %d.\n", + n_xfer = async->ops.transfer_data(dev->vid, queue_id, async_descs, 0, pkt_idx); + if (unlikely(n_xfer < 0)) { + VHOST_LOG_DATA(ERR, "(%d) %s: failed to transfer data for queue id %d.\n", dev->vid, __func__, queue_id); - n_pkts = 0; - } - - if (unlikely(n_pkts < pkt_burst_idx)) - pkt_err = pkt_burst_idx - n_pkts; + n_xfer = 0; } + pkt_err = pkt_idx - n_xfer; if (unlikely(pkt_err)) { uint16_t num_descs = 0; /* update number of completed packets */ - pkt_idx -= pkt_err; + pkt_idx = n_xfer; /* calculate the sum of descriptors to revert */ while (pkt_err-- > 0) { @@ -1686,9 +1700,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev, struct rte_mbuf *pkt, struct buf_vector *buf_vec, uint16_t *nr_descs, - uint16_t *nr_buffers, - struct rte_vhost_iovec *iovec, - struct rte_vhost_iov_iter *iter) + uint16_t *nr_buffers) { uint16_t nr_vec = 0; uint16_t avail_idx = vq->last_avail_idx; @@ -1736,7 +1748,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev, } if (unlikely(async_mbuf_to_desc(dev, vq, pkt, buf_vec, nr_vec, - *nr_buffers, iovec, iter) < 0)) + *nr_buffers) < 0)) return -1; vhost_shadow_enqueue_packed(vq, buffer_len, buffer_buf_id, buffer_desc_count, *nr_buffers); @@ -1746,13 +1758,12 @@ vhost_enqueue_async_packed(struct virtio_net *dev, static __rte_always_inline int16_t virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, - struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers, - struct rte_vhost_iovec *iovec, struct rte_vhost_iov_iter *iter) + struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers) { struct buf_vector buf_vec[BUF_VECTOR_MAX]; - if (unlikely(vhost_enqueue_async_packed(dev, vq, pkt, buf_vec, nr_descs, nr_buffers, - iovec, iter) < 0)) { + if (unlikely(vhost_enqueue_async_packed(dev, vq, pkt, buf_vec, + nr_descs, nr_buffers) < 0)) { VHOST_LOG_DATA(DEBUG, "(%d) failed to get enough desc from vring\n", dev->vid); return -1; } @@ -1794,20 +1805,17 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count) { - uint32_t pkt_idx = 0, pkt_burst_idx = 0; + uint32_t pkt_idx = 0; uint32_t remained = count; int32_t n_xfer; uint16_t num_buffers; uint16_t num_descs; struct vhost_async *async = vq->async; - struct rte_vhost_iov_iter *iter = async->iov_iter; - struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; - struct rte_vhost_iovec *iovec = async->iovec; + struct rte_vhost_async_desc async_descs[MAX_PKT_BURST]; struct async_inflight_info *pkts_info = async->pkts_info; - uint32_t n_pkts = 0, pkt_err = 0; + uint32_t pkt_err = 0; uint16_t slot_idx = 0; - uint16_t iovec_idx = 0, it_idx = 0; do { rte_prefetch0(&vq->desc_packed[vq->last_avail_idx]); @@ -1815,71 +1823,36 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, num_buffers = 0; num_descs = 0; if (unlikely(virtio_dev_rx_async_packed(dev, vq, pkts[pkt_idx], - &num_descs, &num_buffers, - &iovec[iovec_idx], &iter[it_idx]) < 0)) + &num_descs, &num_buffers) < 0)) break; slot_idx = (async->pkts_idx + pkt_idx) % vq->size; - async_fill_desc(&tdes[pkt_burst_idx++], &iter[it_idx]); pkts_info[slot_idx].descs = num_descs; pkts_info[slot_idx].nr_buffers = num_buffers; pkts_info[slot_idx].mbuf = pkts[pkt_idx]; - iovec_idx += iter[it_idx].nr_segs; - it_idx++; pkt_idx++; remained--; vq_inc_last_avail_packed(vq, num_descs); + } while (pkt_idx < count); - /* - * condition to trigger async device transfer: - * - unused async iov number is less than max vhost vector - */ - if (unlikely(VHOST_MAX_ASYNC_VEC - iovec_idx < BUF_VECTOR_MAX)) { - n_xfer = async->ops.transfer_data(dev->vid, - queue_id, tdes, 0, pkt_burst_idx); - if (likely(n_xfer >= 0)) { - n_pkts = n_xfer; - } else { - VHOST_LOG_DATA(ERR, - "(%d) %s: failed to transfer data for queue id %d.\n", - dev->vid, __func__, queue_id); - n_pkts = 0; - } - - iovec_idx = 0; - it_idx = 0; - - if (unlikely(n_pkts < pkt_burst_idx)) { - /* - * log error packets number here and do actual - * error processing when applications poll - * completion - */ - pkt_err = pkt_burst_idx - n_pkts; - pkt_burst_idx = 0; - break; - } + if (unlikely(pkt_idx == 0)) + return 0; - pkt_burst_idx = 0; - } - } while (pkt_idx < count); + async_fill_descs(async, async_descs); - if (pkt_burst_idx) { - n_xfer = async->ops.transfer_data(dev->vid, queue_id, tdes, 0, pkt_burst_idx); - if (likely(n_xfer >= 0)) { - n_pkts = n_xfer; - } else { - VHOST_LOG_DATA(ERR, "(%d) %s: failed to transfer data for queue id %d.\n", + n_xfer = async->ops.transfer_data(dev->vid, queue_id, async_descs, 0, pkt_idx); + if (unlikely(n_xfer < 0)) { + VHOST_LOG_DATA(ERR, "(%d) %s: failed to transfer data for queue id %d.\n", dev->vid, __func__, queue_id); - n_pkts = 0; - } - - if (unlikely(n_pkts < pkt_burst_idx)) - pkt_err = pkt_burst_idx - n_pkts; + n_xfer = 0; } + pkt_err = pkt_idx - n_xfer; + + async_iter_reset(async); + if (unlikely(pkt_err)) dma_error_handler_packed(vq, slot_idx, pkt_err, &pkt_idx); From patchwork Tue Oct 26 16:28:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102957 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 84C85A0547; Tue, 26 Oct 2021 18:30:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E59A14114F; Tue, 26 Oct 2021 18:29:41 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 95D8941169 for ; Tue, 26 Oct 2021 18:29:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cbIn/QfLIWc6JUHbVSRIYgRDFuzRl8w0JaK8zWtydUA=; b=Z3+w4+coz6f62heAjoI3la62f2UxFBd7Gz/YgWQxGNYDm/5i5VZxUnq/kkg/HvMyPDFX0Y 3qI7IB8rJqq0aNN3VXXbDAQYmoJvnKYoMEcnwKqTiEActqOGlWL/RtjhFIOplBKhqa5diK 7GaAt+3yyxH8BDxMQzEmfahVXVNHDMM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-389-BjKMVvHNMs6UQqA3wcfiQw-1; Tue, 26 Oct 2021 12:29:37 -0400 X-MC-Unique: BjKMVvHNMs6UQqA3wcfiQw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id ACB9B1006AA2; Tue, 26 Oct 2021 16:29:35 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id CCC05100E809; Tue, 26 Oct 2021 16:29:30 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:58 +0200 Message-Id: <20211026162904.482987-10-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 09/15] vhost: remove notion of async descriptor X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Now that IO vectors iterator have been simplified, the rte_vhost_async_desc struct only contains a pointer on the iterator array stored in the async metadata. This patch removes it, and pass directly the iterators array pointer to the transfer_data callback. Doing that, we avoid declaring the descriptor array in the stack, and also avoid the cost of filling it. Signed-off-by: Maxime Coquelin --- examples/vhost/ioat.c | 10 +++++----- examples/vhost/ioat.h | 4 ++-- lib/vhost/rte_vhost_async.h | 16 ++++------------ lib/vhost/virtio_net.c | 19 ++----------------- 4 files changed, 13 insertions(+), 36 deletions(-) diff --git a/examples/vhost/ioat.c b/examples/vhost/ioat.c index a8c588deff..9aeeb12fd9 100644 --- a/examples/vhost/ioat.c +++ b/examples/vhost/ioat.c @@ -124,10 +124,10 @@ open_ioat(const char *value) int32_t ioat_transfer_data_cb(int vid, uint16_t queue_id, - struct rte_vhost_async_desc *descs, + struct rte_vhost_iov_iter *iov_iter, struct rte_vhost_async_status *opaque_data, uint16_t count) { - uint32_t i_desc; + uint32_t i_iter; uint16_t dev_id = dma_bind[vid].dmas[queue_id * 2 + VIRTIO_RXQ].dev_id; struct rte_vhost_iov_iter *iter = NULL; unsigned long i_seg; @@ -135,8 +135,8 @@ ioat_transfer_data_cb(int vid, uint16_t queue_id, unsigned short write = cb_tracker[dev_id].next_write; if (!opaque_data) { - for (i_desc = 0; i_desc < count; i_desc++) { - iter = descs[i_desc].iter; + for (i_iter = 0; i_iter < count; i_iter++) { + iter = iov_iter + i_iter; i_seg = 0; if (cb_tracker[dev_id].ioat_space < iter->nr_segs) break; @@ -161,7 +161,7 @@ ioat_transfer_data_cb(int vid, uint16_t queue_id, /* ring the doorbell */ rte_ioat_perform_ops(dev_id); cb_tracker[dev_id].next_write = write; - return i_desc; + return i_iter; } int32_t diff --git a/examples/vhost/ioat.h b/examples/vhost/ioat.h index 62e163c585..d9bf717e8d 100644 --- a/examples/vhost/ioat.h +++ b/examples/vhost/ioat.h @@ -29,7 +29,7 @@ int open_ioat(const char *value); int32_t ioat_transfer_data_cb(int vid, uint16_t queue_id, - struct rte_vhost_async_desc *descs, + struct rte_vhost_iov_iter *iov_iter, struct rte_vhost_async_status *opaque_data, uint16_t count); int32_t @@ -44,7 +44,7 @@ static int open_ioat(const char *value __rte_unused) static int32_t ioat_transfer_data_cb(int vid __rte_unused, uint16_t queue_id __rte_unused, - struct rte_vhost_async_desc *descs __rte_unused, + struct rte_vhost_iov_iter *iov_iter __rte_unused, struct rte_vhost_async_status *opaque_data __rte_unused, uint16_t count __rte_unused) { diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h index 4ea5cfab10..a87ea6ba37 100644 --- a/lib/vhost/rte_vhost_async.h +++ b/lib/vhost/rte_vhost_async.h @@ -26,14 +26,6 @@ struct rte_vhost_iov_iter { unsigned long nr_segs; }; -/** - * dma transfer descriptor - */ -struct rte_vhost_async_desc { - /* memory iov_iter */ - struct rte_vhost_iov_iter *iter; -}; - /** * dma transfer status */ @@ -55,17 +47,17 @@ struct rte_vhost_async_channel_ops { * id of vhost device to perform data copies * @param queue_id * queue id to perform data copies - * @param descs - * an array of DMA transfer memory descriptors + * @param iov_iter + * an array of IOV iterators * @param opaque_data * opaque data pair sending to DMA engine * @param count * number of elements in the "descs" array * @return - * number of descs processed, negative value means error + * number of IOV iterators processed, negative value means error */ int32_t (*transfer_data)(int vid, uint16_t queue_id, - struct rte_vhost_async_desc *descs, + struct rte_vhost_iov_iter *iov_iter, struct rte_vhost_async_status *opaque_data, uint16_t count); /** diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index a2044aae1b..dfe864b3e9 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -995,15 +995,6 @@ async_iter_reset(struct vhost_async *async) async->iovec_idx = 0; } -static __rte_always_inline void -async_fill_descs(struct vhost_async *async, struct rte_vhost_async_desc *descs) -{ - int i; - - for (i = 0; i < async->iter_idx; i++) - descs[i].iter = async->iov_iter + i; -} - static __rte_always_inline int async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec, @@ -1550,7 +1541,6 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, uint16_t avail_head; struct vhost_async *async = vq->async; - struct rte_vhost_async_desc async_descs[MAX_PKT_BURST]; struct async_inflight_info *pkts_info = async->pkts_info; uint32_t pkt_err = 0; int32_t n_xfer; @@ -1595,9 +1585,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, if (unlikely(pkt_idx == 0)) return 0; - async_fill_descs(async, async_descs); - - n_xfer = async->ops.transfer_data(dev->vid, queue_id, async_descs, 0, pkt_idx); + n_xfer = async->ops.transfer_data(dev->vid, queue_id, async->iov_iter, 0, pkt_idx); if (unlikely(n_xfer < 0)) { VHOST_LOG_DATA(ERR, "(%d) %s: failed to transfer data for queue id %d.\n", dev->vid, __func__, queue_id); @@ -1812,7 +1800,6 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, uint16_t num_descs; struct vhost_async *async = vq->async; - struct rte_vhost_async_desc async_descs[MAX_PKT_BURST]; struct async_inflight_info *pkts_info = async->pkts_info; uint32_t pkt_err = 0; uint16_t slot_idx = 0; @@ -1840,9 +1827,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, if (unlikely(pkt_idx == 0)) return 0; - async_fill_descs(async, async_descs); - - n_xfer = async->ops.transfer_data(dev->vid, queue_id, async_descs, 0, pkt_idx); + n_xfer = async->ops.transfer_data(dev->vid, queue_id, async->iov_iter, 0, pkt_idx); if (unlikely(n_xfer < 0)) { VHOST_LOG_DATA(ERR, "(%d) %s: failed to transfer data for queue id %d.\n", dev->vid, __func__, queue_id); From patchwork Tue Oct 26 16:28:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102958 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9F4C4A0547; Tue, 26 Oct 2021 18:30:25 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 41D614115B; Tue, 26 Oct 2021 18:29:45 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 9831B41176 for ; Tue, 26 Oct 2021 18:29:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9GE1+vHZP24jGbmqE4FH+mJdx+Hk2H23J3AmM0P6ewY=; b=b9aU9Q9tWvgPmX8NhyIoiT5VjF34JFwSRo2FOZ4UFRpOeY0J9pKJ4t9gqkFpFQppS3djYs cihft3OUbYGRgW457MUK5LClArmikfAUggXONOwY4qTACY8qXRkqi0MU8fMTlgRbaWhvxK OaCG4k7nRM3n+pxAYQY6paDogvVKKR0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-457-8v8bCNZVPSmkkRTwwlhG6w-1; Tue, 26 Oct 2021 12:29:39 -0400 X-MC-Unique: 8v8bCNZVPSmkkRTwwlhG6w-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 51F4C36259; Tue, 26 Oct 2021 16:29:38 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 07CBE100E809; Tue, 26 Oct 2021 16:29:35 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:28:59 +0200 Message-Id: <20211026162904.482987-11-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 10/15] vhost: simplify async enqueue completion X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" vhost_poll_enqueue_completed() assumes some inflight packets could have been completed in a previous call but not returned to the application. But this is not the case, since check_completed_copies callback is never called with more than the current count as argument. In other words, async->last_pkts_n is always 0. Removing it greatly simplfies the function. Signed-off-by: Maxime Coquelin --- lib/vhost/vhost.h | 1 - lib/vhost/virtio_net.c | 76 ++++++++++++++++-------------------------- 2 files changed, 28 insertions(+), 49 deletions(-) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 9afb75fa6e..ae5bc257cc 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -141,7 +141,6 @@ struct vhost_async { struct async_inflight_info *pkts_info; uint16_t pkts_idx; uint16_t pkts_inflight_n; - uint16_t last_pkts_n; union { struct vring_used_elem *descs_split; struct vring_used_elem_packed *buffers_packed; diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index dfe864b3e9..3206b3f816 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -1619,7 +1619,11 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, vq->shadow_used_idx); async->desc_idx_split += vq->shadow_used_idx; + async->pkts_idx += pkt_idx; + if (async->pkts_idx >= vq->size) + async->pkts_idx -= vq->size; + async->pkts_inflight_n += pkt_idx; vq->shadow_used_idx = 0; } @@ -1921,68 +1925,44 @@ static __rte_always_inline uint16_t vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, struct rte_mbuf **pkts, uint16_t count) { - struct vhost_virtqueue *vq; - struct vhost_async *async; - struct async_inflight_info *pkts_info; + struct vhost_virtqueue *vq = dev->virtqueue[queue_id]; + struct vhost_async *async = vq->async; + struct async_inflight_info *pkts_info = async->pkts_info; int32_t n_cpl; - uint16_t n_pkts_cpl = 0, n_pkts_put = 0, n_descs = 0, n_buffers = 0; - uint16_t start_idx, pkts_idx, vq_size; - uint16_t from, i; + uint16_t n_descs = 0, n_buffers = 0; + uint16_t start_idx, from, i; - vq = dev->virtqueue[queue_id]; - async = vq->async; - pkts_idx = async->pkts_idx % vq->size; - pkts_info = async->pkts_info; - vq_size = vq->size; - start_idx = virtio_dev_rx_async_get_info_idx(pkts_idx, - vq_size, async->pkts_inflight_n); - - if (count > async->last_pkts_n) { - n_cpl = async->ops.check_completed_copies(dev->vid, - queue_id, 0, count - async->last_pkts_n); - if (likely(n_cpl >= 0)) { - n_pkts_cpl = n_cpl; - } else { - VHOST_LOG_DATA(ERR, - "(%d) %s: failed to check completed copies for queue id %d.\n", + start_idx = virtio_dev_rx_async_get_info_idx(async->pkts_idx, + vq->size, async->pkts_inflight_n); + + n_cpl = async->ops.check_completed_copies(dev->vid, queue_id, 0, count); + if (unlikely(n_cpl < 0)) { + VHOST_LOG_DATA(ERR, "(%d) %s: failed to check completed copies for queue id %d.\n", dev->vid, __func__, queue_id); - n_pkts_cpl = 0; - } + return 0; } - n_pkts_cpl += async->last_pkts_n; - n_pkts_put = RTE_MIN(n_pkts_cpl, count); - if (unlikely(n_pkts_put == 0)) { - async->last_pkts_n = n_pkts_cpl; + if (n_cpl == 0) return 0; - } - if (vq_is_packed(dev)) { - for (i = 0; i < n_pkts_put; i++) { - from = (start_idx + i) % vq_size; - n_buffers += pkts_info[from].nr_buffers; - pkts[i] = pkts_info[from].mbuf; - } - } else { - for (i = 0; i < n_pkts_put; i++) { - from = (start_idx + i) & (vq_size - 1); - n_descs += pkts_info[from].descs; - pkts[i] = pkts_info[from].mbuf; - } + for (i = 0; i < n_cpl; i++) { + from = (start_idx + i) % vq->size; + /* Only used with packed ring */ + n_buffers += pkts_info[from].nr_buffers; + /* Only used with split ring */ + n_descs += pkts_info[from].descs; + pkts[i] = pkts_info[from].mbuf; } - async->last_pkts_n = n_pkts_cpl - n_pkts_put; - async->pkts_inflight_n -= n_pkts_put; + + async->pkts_inflight_n -= n_cpl; if (likely(vq->enabled && vq->access_ok)) { if (vq_is_packed(dev)) { write_back_completed_descs_packed(vq, n_buffers); - vhost_vring_call_packed(dev, vq); } else { write_back_completed_descs_split(vq, n_descs); - - __atomic_add_fetch(&vq->used->idx, n_descs, - __ATOMIC_RELEASE); + __atomic_add_fetch(&vq->used->idx, n_descs, __ATOMIC_RELEASE); vhost_vring_call_split(dev, vq); } } else { @@ -1995,7 +1975,7 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, } } - return n_pkts_put; + return n_cpl; } uint16_t From patchwork Tue Oct 26 16:29:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102960 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 04388A0547; Tue, 26 Oct 2021 18:30:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 408394118A; Tue, 26 Oct 2021 18:29:48 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 5AF6F4117E for ; Tue, 26 Oct 2021 18:29:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265785; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2Q43kPtFtamHnnKtXzZGhGB7zN3B7EG3rCkH9rbespU=; b=TajI7K9CCgiDb1zB9/RXDs9DcAlxTIDw7IytMV/8i9auLERNeS6GCiHvxr6P3h8vfIujie FaovdYCWYRg0RAISaZezRhWHhH2qeAarxsTaXO4z6NX9Ta2dzdM0lndIaXYeRZNjDXfUdm 23gjm1tgPhbnJD16yt44JuD18lB+sYQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-25-qLXezuNJNASD7lco8FdPhA-1; Tue, 26 Oct 2021 12:29:41 -0400 X-MC-Unique: qLXezuNJNASD7lco8FdPhA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7A65A8066F3; Tue, 26 Oct 2021 16:29:40 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9DF63100EB3D; Tue, 26 Oct 2021 16:29:38 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:29:00 +0200 Message-Id: <20211026162904.482987-12-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 11/15] vhost: simplify getting the first in-flight index X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch reworks the function getting the index for the first packet in-flight. When this index turns out to be zero, let's use the simple path. Doing that avoid having to do a modulo with the virtqueue size. The patch also rename the function for better clarifty, and only pass the virtqueue metadata pointer, as all the needed information are stored there. Signed-off-by: Maxime Coquelin --- lib/vhost/virtio_net.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index 3206b3f816..2f76523e67 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -1490,11 +1490,14 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id, } static __rte_always_inline uint16_t -virtio_dev_rx_async_get_info_idx(uint16_t pkts_idx, - uint16_t vq_size, uint16_t n_inflight) +async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq) { - return pkts_idx > n_inflight ? (pkts_idx - n_inflight) : - (vq_size - n_inflight + pkts_idx) % vq_size; + struct vhost_async *async = vq->async; + + if (async->pkts_idx >= async->pkts_inflight_n) + return async->pkts_idx - async->pkts_inflight_n; + else + return vq->size - async->pkts_inflight_n + async->pkts_idx; } static __rte_always_inline void @@ -1932,9 +1935,6 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, uint16_t n_descs = 0, n_buffers = 0; uint16_t start_idx, from, i; - start_idx = virtio_dev_rx_async_get_info_idx(async->pkts_idx, - vq->size, async->pkts_inflight_n); - n_cpl = async->ops.check_completed_copies(dev->vid, queue_id, 0, count); if (unlikely(n_cpl < 0)) { VHOST_LOG_DATA(ERR, "(%d) %s: failed to check completed copies for queue id %d.\n", @@ -1945,6 +1945,8 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id, if (n_cpl == 0) return 0; + start_idx = async_get_first_inflight_pkt_idx(vq); + for (i = 0; i < n_cpl; i++) { from = (start_idx + i) % vq->size; /* Only used with packed ring */ From patchwork Tue Oct 26 16:29:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102959 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 67FCBA0547; Tue, 26 Oct 2021 18:30:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 40E464111E; Tue, 26 Oct 2021 18:29:47 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 045304117A for ; Tue, 26 Oct 2021 18:29:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265785; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z/dhlDR8hQXIV9WdxhkKIr4FYC0K1rdwEGX1TuG6Zsg=; b=TmFx0Kdeu0nKfGffvOpOn1Fj9dTzeyfLfjx3FBMFH4u+r8xMXqekE32++aHFfOV8Dct8QX Vgs+LSReXvsJjI5MgBLRDQoJaUam5Wl1xI1T6bc382l65Y1zqzkbQBoneq2lVavzMrQFsu aAggjGS3H5YJ79jCXz0uJvJ9BYn1KX0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-363-dSj8QMjqNleKofQjZzeT9g-1; Tue, 26 Oct 2021 12:29:44 -0400 X-MC-Unique: dSj8QMjqNleKofQjZzeT9g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9716B1808302; Tue, 26 Oct 2021 16:29:42 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id C75BE100E809; Tue, 26 Oct 2021 16:29:40 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:29:01 +0200 Message-Id: <20211026162904.482987-13-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 12/15] vhost: prepare async for mbuf to desc refactoring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch extracts the IO vectors filling from async_mbuf_to_desc() into a dedicated function as a preliminary step of merging copy_mubf_to_desc() and async_mbuf_to_desc(). Signed-off-by: Maxime Coquelin --- lib/vhost/virtio_net.c | 206 ++++++++++++++++++++++------------------- 1 file changed, 111 insertions(+), 95 deletions(-) diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index 2f76523e67..a901e29d4e 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -791,6 +791,109 @@ copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } +static __rte_always_inline int +async_iter_initialize(struct vhost_async *async) +{ + struct rte_vhost_iov_iter *iter; + + if (unlikely(async->iovec_idx >= VHOST_MAX_ASYNC_VEC)) { + VHOST_LOG_DATA(ERR, "no more async iovec available\n"); + return -1; + } + + iter = async->iov_iter + async->iter_idx; + iter->iov = async->iovec + async->iovec_idx; + iter->nr_segs = 0; + + return 0; +} + +static __rte_always_inline int +async_iter_add_iovec(struct vhost_async *async, void *src, void *dst, size_t len) +{ + struct rte_vhost_iov_iter *iter; + struct rte_vhost_iovec *iovec; + + if (unlikely(async->iovec_idx >= VHOST_MAX_ASYNC_VEC)) { + static bool vhost_max_async_vec_log; + + if (!vhost_max_async_vec_log) { + VHOST_LOG_DATA(ERR, "no more async iovec available\n"); + vhost_max_async_vec_log = true; + } + + return -1; + } + + iter = async->iov_iter + async->iter_idx; + iovec = async->iovec + async->iovec_idx; + + iovec->src_addr = src; + iovec->dst_addr = dst; + iovec->len = len; + + iter->nr_segs++; + async->iovec_idx++; + + return 0; +} + +static __rte_always_inline void +async_iter_finalize(struct vhost_async *async) +{ + async->iter_idx++; +} + +static __rte_always_inline void +async_iter_cancel(struct vhost_async *async) +{ + struct rte_vhost_iov_iter *iter; + + iter = async->iov_iter + async->iter_idx; + async->iovec_idx -= iter->nr_segs; + iter->nr_segs = 0; + iter->iov = NULL; +} + +static __rte_always_inline void +async_iter_reset(struct vhost_async *async) +{ + async->iter_idx = 0; + async->iovec_idx = 0; +} + +static __rte_always_inline int +async_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq, + struct rte_mbuf *m, uint32_t mbuf_offset, + uint64_t buf_iova, uint32_t cpy_len) +{ + struct vhost_async *async = vq->async; + uint64_t mapped_len; + uint32_t buf_offset = 0; + void *hpa; + + while (cpy_len) { + hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev, + buf_iova + buf_offset, cpy_len, &mapped_len); + if (unlikely(!hpa)) { + VHOST_LOG_DATA(ERR, "(%d) %s: failed to get hpa.\n", dev->vid, __func__); + return -1; + } + + if (unlikely(async_iter_add_iovec(async, + (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, + mbuf_offset), + hpa, (size_t)mapped_len))) + return -1; + + cpy_len -= (uint32_t)mapped_len; + mbuf_offset += (uint32_t)mapped_len; + buf_offset += (uint32_t)mapped_len; + } + + return 0; +} + static __rte_always_inline int copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec, @@ -924,77 +1027,6 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, return error; } -static __rte_always_inline int -async_iter_initialize(struct vhost_async *async) -{ - struct rte_vhost_iov_iter *iter; - - if (unlikely(async->iovec_idx >= VHOST_MAX_ASYNC_VEC)) { - VHOST_LOG_DATA(ERR, "no more async iovec available\n"); - return -1; - } - - iter = async->iov_iter + async->iter_idx; - iter->iov = async->iovec + async->iovec_idx; - iter->nr_segs = 0; - - return 0; -} - -static __rte_always_inline int -async_iter_add_iovec(struct vhost_async *async, void *src, void *dst, size_t len) -{ - struct rte_vhost_iov_iter *iter; - struct rte_vhost_iovec *iovec; - - if (unlikely(async->iovec_idx >= VHOST_MAX_ASYNC_VEC)) { - static bool vhost_max_async_vec_log; - - if (!vhost_max_async_vec_log) { - VHOST_LOG_DATA(ERR, "no more async iovec available\n"); - vhost_max_async_vec_log = true; - } - - return -1; - } - - iter = async->iov_iter + async->iter_idx; - iovec = async->iovec + async->iovec_idx; - - iovec->src_addr = src; - iovec->dst_addr = dst; - iovec->len = len; - - iter->nr_segs++; - async->iovec_idx++; - - return 0; -} - -static __rte_always_inline void -async_iter_finalize(struct vhost_async *async) -{ - async->iter_idx++; -} - -static __rte_always_inline void -async_iter_cancel(struct vhost_async *async) -{ - struct rte_vhost_iov_iter *iter; - - iter = async->iov_iter + async->iter_idx; - async->iovec_idx -= iter->nr_segs; - iter->nr_segs = 0; - iter->iov = NULL; -} - -static __rte_always_inline void -async_iter_reset(struct vhost_async *async) -{ - async->iter_idx = 0; - async->iovec_idx = 0; -} - static __rte_always_inline int async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec, @@ -1005,14 +1037,11 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; uint64_t buf_addr, buf_iova; uint64_t hdr_addr; - uint64_t mapped_len; uint32_t vec_idx = 0; uint32_t mbuf_offset, mbuf_avail; uint32_t buf_offset, buf_avail; uint32_t cpy_len, buf_len; - void *hpa; - if (unlikely(m == NULL)) return -1; @@ -1096,28 +1125,15 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len = RTE_MIN(buf_avail, mbuf_avail); - while (unlikely(cpy_len)) { - hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev, - buf_iova + buf_offset, - cpy_len, &mapped_len); - if (unlikely(!hpa)) { - VHOST_LOG_DATA(ERR, "(%d) %s: failed to get hpa.\n", - dev->vid, __func__); - goto error; - } - - if (unlikely(async_iter_add_iovec(async, - (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, - mbuf_offset), - hpa, (size_t)mapped_len))) - goto error; - - cpy_len -= (uint32_t)mapped_len; - mbuf_avail -= (uint32_t)mapped_len; - mbuf_offset += (uint32_t)mapped_len; - buf_avail -= (uint32_t)mapped_len; - buf_offset += (uint32_t)mapped_len; + if (async_mbuf_to_desc_seg(dev, vq, m, mbuf_offset, + buf_iova + buf_offset, cpy_len) < 0) { + goto error; } + + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; } async_iter_finalize(async); From patchwork Tue Oct 26 16:29:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102961 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7BFC9A0547; Tue, 26 Oct 2021 18:30:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4674C411A4; Tue, 26 Oct 2021 18:29:51 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id F177741183 for ; Tue, 26 Oct 2021 18:29:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rg54mzkwCFWXk5Kg15jftoPKnhAO1QQUMdXgj22HBmk=; b=IS2VxNgioAKNeXYANP+ddedIhFPUH5GTow6n8eV1yjWHjF3+3HENuD3cEyWdtrYgfcbHti vVtvuc6/kULZuFp2nrrYTsg6kc2+emJ9r428zNCZ8pdBl/T4JyWsVkMxrRsir/1APh81q5 eRn3zaosbitOlmB7JwadtKNo6lK2GwI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-263-pHcrZOLVO3C828apxpc0Lw-1; Tue, 26 Oct 2021 12:29:46 -0400 X-MC-Unique: pHcrZOLVO3C828apxpc0Lw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BF5F61397B00; Tue, 26 Oct 2021 16:29:44 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id E298410016FE; Tue, 26 Oct 2021 16:29:42 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:29:02 +0200 Message-Id: <20211026162904.482987-14-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 13/15] vhost: prepare sync for mbuf to desc refactoring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch extracts the descriptors buffers filling from copy_mbuf_to_desc() into a dedicated function as a preliminary step of merging copy_mubf_to_desc() and async_mbuf_to_desc(). Signed-off-by: Maxime Coquelin --- lib/vhost/virtio_net.c | 73 +++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 37 deletions(-) diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index a901e29d4e..e36468d34f 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -894,6 +894,30 @@ async_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq, return 0; } +static __rte_always_inline void +sync_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq, + struct rte_mbuf *m, uint32_t mbuf_offset, + uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len) +{ + struct batch_copy_elem *batch_copy = vq->batch_copy_elems; + + if (likely(cpy_len > MAX_BATCH_LEN || vq->batch_copy_nb_elems >= vq->size)) { + rte_memcpy((void *)((uintptr_t)(buf_addr)), + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), + cpy_len); + vhost_log_cache_write_iova(dev, vq, buf_iova, cpy_len); + PRINT_PACKET(dev, (uintptr_t)(buf_addr), cpy_len, 0); + } else { + batch_copy[vq->batch_copy_nb_elems].dst = + (void *)((uintptr_t)(buf_addr)); + batch_copy[vq->batch_copy_nb_elems].src = + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); + batch_copy[vq->batch_copy_nb_elems].log_addr = buf_iova; + batch_copy[vq->batch_copy_nb_elems].len = cpy_len; + vq->batch_copy_nb_elems++; + } +} + static __rte_always_inline int copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, struct rte_mbuf *m, struct buf_vector *buf_vec, @@ -906,23 +930,17 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, uint32_t cpy_len; uint64_t hdr_addr; struct rte_mbuf *hdr_mbuf; - struct batch_copy_elem *batch_copy = vq->batch_copy_elems; struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; - int error = 0; - if (unlikely(m == NULL)) { - error = -1; - goto out; - } + if (unlikely(m == NULL)) + return -1; buf_addr = buf_vec[vec_idx].buf_addr; buf_iova = buf_vec[vec_idx].buf_iova; buf_len = buf_vec[vec_idx].buf_len; - if (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1)) { - error = -1; - goto out; - } + if (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1)) + return -1; hdr_mbuf = m; hdr_addr = buf_addr; @@ -953,10 +971,8 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, /* done with current buf, get the next one */ if (buf_avail == 0) { vec_idx++; - if (unlikely(vec_idx >= nr_vec)) { - error = -1; - goto out; - } + if (unlikely(vec_idx >= nr_vec)) + goto error; buf_addr = buf_vec[vec_idx].buf_addr; buf_iova = buf_vec[vec_idx].buf_iova; @@ -995,26 +1011,9 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len = RTE_MIN(buf_avail, mbuf_avail); - if (likely(cpy_len > MAX_BATCH_LEN || - vq->batch_copy_nb_elems >= vq->size)) { - rte_memcpy((void *)((uintptr_t)(buf_addr + buf_offset)), - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), - cpy_len); - vhost_log_cache_write_iova(dev, vq, - buf_iova + buf_offset, - cpy_len); - PRINT_PACKET(dev, (uintptr_t)(buf_addr + buf_offset), - cpy_len, 0); - } else { - batch_copy[vq->batch_copy_nb_elems].dst = - (void *)((uintptr_t)(buf_addr + buf_offset)); - batch_copy[vq->batch_copy_nb_elems].src = - rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); - batch_copy[vq->batch_copy_nb_elems].log_addr = - buf_iova + buf_offset; - batch_copy[vq->batch_copy_nb_elems].len = cpy_len; - vq->batch_copy_nb_elems++; - } + sync_mbuf_to_desc_seg(dev, vq, m, mbuf_offset, + buf_addr + buf_offset, + buf_iova + buf_offset, cpy_len); mbuf_avail -= cpy_len; mbuf_offset += cpy_len; @@ -1022,9 +1021,9 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, buf_offset += cpy_len; } -out: - - return error; + return 0; +error: + return -1; } static __rte_always_inline int From patchwork Tue Oct 26 16:29:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102962 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2874DA0547; Tue, 26 Oct 2021 18:30:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3F9B441136; Tue, 26 Oct 2021 18:29:59 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id EFCC54111F for ; Tue, 26 Oct 2021 18:29:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265797; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hGLskTw3ndXkE59fCL3dwKUp36XIAFGuJR6yX4EdBAc=; b=ayonOhzSj8+90R6EFGU1xBB51N79T5+HlGqBet2LZ/YoLT7AvOoGr3vbHUXic3ThkQYUOJ eEp0tp+pIo0NwUbk1S2mEAUo0mLvIaAEam9pqvAavnfdjjN/WfNQnsI+W/Yvl7Hu9dHtzI ojU/0kD4eR+ktaOP2YHn4nYArQtgw/w= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-406-SaRD-gwSNtSBjfdNI-YAYQ-1; Tue, 26 Oct 2021 12:29:52 -0400 X-MC-Unique: SaRD-gwSNtSBjfdNI-YAYQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 064701397B04; Tue, 26 Oct 2021 16:29:51 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18A3810016FE; Tue, 26 Oct 2021 16:29:44 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:29:03 +0200 Message-Id: <20211026162904.482987-15-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 14/15] vhost: merge sync and async mbuf to desc filling X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patches merges copy_mbuf_to_desc() used by the sync path with async_mbuf_to_desc() used by the async path. Most of these complex functions are identical, so merging them will make the maintenance easier. In order not to degrade performance, the patch introduces a boolean function parameter to specify whether it is called in async context. This boolean is statically passed to this always-inlined function, so the compiler will optimize this out. Signed-off-by: Maxime Coquelin --- lib/vhost/virtio_net.c | 153 +++++++---------------------------------- 1 file changed, 26 insertions(+), 127 deletions(-) diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index e36468d34f..cef4bcf15c 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -919,9 +919,9 @@ sync_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq, } static __rte_always_inline int -copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, - struct rte_mbuf *m, struct buf_vector *buf_vec, - uint16_t nr_vec, uint16_t num_buffers) +mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, + struct rte_mbuf *m, struct buf_vector *buf_vec, + uint16_t nr_vec, uint16_t num_buffers, bool is_async) { uint32_t vec_idx = 0; uint32_t mbuf_offset, mbuf_avail; @@ -931,115 +931,7 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, uint64_t hdr_addr; struct rte_mbuf *hdr_mbuf; struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; - - if (unlikely(m == NULL)) - return -1; - - buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; - buf_len = buf_vec[vec_idx].buf_len; - - if (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1)) - return -1; - - hdr_mbuf = m; - hdr_addr = buf_addr; - if (unlikely(buf_len < dev->vhost_hlen)) { - memset(&tmp_hdr, 0, sizeof(struct virtio_net_hdr_mrg_rxbuf)); - hdr = &tmp_hdr; - } else - hdr = (struct virtio_net_hdr_mrg_rxbuf *)(uintptr_t)hdr_addr; - - VHOST_LOG_DATA(DEBUG, "(%d) RX: num merge buffers %d\n", - dev->vid, num_buffers); - - if (unlikely(buf_len < dev->vhost_hlen)) { - buf_offset = dev->vhost_hlen - buf_len; - vec_idx++; - buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; - buf_len = buf_vec[vec_idx].buf_len; - buf_avail = buf_len - buf_offset; - } else { - buf_offset = dev->vhost_hlen; - buf_avail = buf_len - dev->vhost_hlen; - } - - mbuf_avail = rte_pktmbuf_data_len(m); - mbuf_offset = 0; - while (mbuf_avail != 0 || m->next != NULL) { - /* done with current buf, get the next one */ - if (buf_avail == 0) { - vec_idx++; - if (unlikely(vec_idx >= nr_vec)) - goto error; - - buf_addr = buf_vec[vec_idx].buf_addr; - buf_iova = buf_vec[vec_idx].buf_iova; - buf_len = buf_vec[vec_idx].buf_len; - - buf_offset = 0; - buf_avail = buf_len; - } - - /* done with current mbuf, get the next one */ - if (mbuf_avail == 0) { - m = m->next; - - mbuf_offset = 0; - mbuf_avail = rte_pktmbuf_data_len(m); - } - - if (hdr_addr) { - virtio_enqueue_offload(hdr_mbuf, &hdr->hdr); - if (rxvq_is_mergeable(dev)) - ASSIGN_UNLESS_EQUAL(hdr->num_buffers, - num_buffers); - - if (unlikely(hdr == &tmp_hdr)) { - copy_vnet_hdr_to_desc(dev, vq, buf_vec, hdr); - } else { - PRINT_PACKET(dev, (uintptr_t)hdr_addr, - dev->vhost_hlen, 0); - vhost_log_cache_write_iova(dev, vq, - buf_vec[0].buf_iova, - dev->vhost_hlen); - } - - hdr_addr = 0; - } - - cpy_len = RTE_MIN(buf_avail, mbuf_avail); - - sync_mbuf_to_desc_seg(dev, vq, m, mbuf_offset, - buf_addr + buf_offset, - buf_iova + buf_offset, cpy_len); - - mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; - } - - return 0; -error: - return -1; -} - -static __rte_always_inline int -async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, - struct rte_mbuf *m, struct buf_vector *buf_vec, - uint16_t nr_vec, uint16_t num_buffers) -{ struct vhost_async *async = vq->async; - struct rte_mbuf *hdr_mbuf; - struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; - uint64_t buf_addr, buf_iova; - uint64_t hdr_addr; - uint32_t vec_idx = 0; - uint32_t mbuf_offset, mbuf_avail; - uint32_t buf_offset, buf_avail; - uint32_t cpy_len, buf_len; if (unlikely(m == NULL)) return -1; @@ -1077,8 +969,10 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, mbuf_avail = rte_pktmbuf_data_len(m); mbuf_offset = 0; - if (async_iter_initialize(async)) - return -1; + if (is_async) { + if (async_iter_initialize(async)) + return -1; + } while (mbuf_avail != 0 || m->next != NULL) { /* done with current buf, get the next one */ @@ -1092,7 +986,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, buf_len = buf_vec[vec_idx].buf_len; buf_offset = 0; - buf_avail = buf_len; + buf_avail = buf_len; } /* done with current mbuf, get the next one */ @@ -1100,7 +994,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, m = m->next; mbuf_offset = 0; - mbuf_avail = rte_pktmbuf_data_len(m); + mbuf_avail = rte_pktmbuf_data_len(m); } if (hdr_addr) { @@ -1124,9 +1018,14 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len = RTE_MIN(buf_avail, mbuf_avail); - if (async_mbuf_to_desc_seg(dev, vq, m, mbuf_offset, - buf_iova + buf_offset, cpy_len) < 0) { - goto error; + if (is_async) { + if (async_mbuf_to_desc_seg(dev, vq, m, mbuf_offset, + buf_iova + buf_offset, cpy_len) < 0) + goto error; + } else { + sync_mbuf_to_desc_seg(dev, vq, m, mbuf_offset, + buf_addr + buf_offset, + buf_iova + buf_offset, cpy_len); } mbuf_avail -= cpy_len; @@ -1135,11 +1034,13 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, buf_offset += cpy_len; } - async_iter_finalize(async); + if (is_async) + async_iter_finalize(async); return 0; error: - async_iter_cancel(async); + if (is_async) + async_iter_cancel(async); return -1; } @@ -1198,7 +1099,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev, avail_idx -= vq->size; } - if (copy_mbuf_to_desc(dev, vq, pkt, buf_vec, nr_vec, num_buffers) < 0) + if (mbuf_to_desc(dev, vq, pkt, buf_vec, nr_vec, num_buffers, false) < 0) return -1; vhost_shadow_enqueue_single_packed(dev, vq, buffer_len, buffer_buf_id, @@ -1242,9 +1143,8 @@ virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq, dev->vid, vq->last_avail_idx, vq->last_avail_idx + num_buffers); - if (copy_mbuf_to_desc(dev, vq, pkts[pkt_idx], - buf_vec, nr_vec, - num_buffers) < 0) { + if (mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, + num_buffers, false) < 0) { vq->shadow_used_idx -= num_buffers; break; } @@ -1588,7 +1488,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, VHOST_LOG_DATA(DEBUG, "(%d) current index %d | end index %d\n", dev->vid, vq->last_avail_idx, vq->last_avail_idx + num_buffers); - if (async_mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, num_buffers) < 0) { + if (mbuf_to_desc(dev, vq, pkts[pkt_idx], buf_vec, nr_vec, num_buffers, true) < 0) { vq->shadow_used_idx -= num_buffers; break; } @@ -1757,8 +1657,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev, avail_idx -= vq->size; } - if (unlikely(async_mbuf_to_desc(dev, vq, pkt, buf_vec, nr_vec, - *nr_buffers) < 0)) + if (unlikely(mbuf_to_desc(dev, vq, pkt, buf_vec, nr_vec, *nr_buffers, true) < 0)) return -1; vhost_shadow_enqueue_packed(vq, buffer_len, buffer_buf_id, buffer_desc_count, *nr_buffers); From patchwork Tue Oct 26 16:29:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 102963 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 24449A0547; Tue, 26 Oct 2021 18:31:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 732AA4115D; Tue, 26 Oct 2021 18:30:02 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 8DECC41136 for ; Tue, 26 Oct 2021 18:29:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1635265798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9C76bemecFzQQWa40wFXcUhVIQxzLQ0FmUnrJ/U0mfU=; b=fcj6JIruVYhOMVs1MGNxG3exIjzeQn1QhJamCXpCvfUnnBIj3ezHJBlES+YnhSeQg3JC8U tadIo3wkDT21goH4xOUhKgIJDJews1BaV9gvv7FILOCThdgt2pLrteUwC9wbH8NemglZBR l7bu20ZXacMbnygaRnnc1Y/Vwld6ynQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-123-Hy7rSwaKNFaa2MkvdFrzVw-1; Tue, 26 Oct 2021 12:29:55 -0400 X-MC-Unique: Hy7rSwaKNFaa2MkvdFrzVw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 363298066F4; Tue, 26 Oct 2021 16:29:53 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.37]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5BA7B100E809; Tue, 26 Oct 2021 16:29:51 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, jiayu.hu@intel.com, yuanx.wang@intel.com, wenwux.ma@intel.com, bruce.richardson@intel.com, john.mcnamara@intel.com Cc: Maxime Coquelin Date: Tue, 26 Oct 2021 18:29:04 +0200 Message-Id: <20211026162904.482987-16-maxime.coquelin@redhat.com> In-Reply-To: <20211026162904.482987-1-maxime.coquelin@redhat.com> References: <20211026162904.482987-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH v2 15/15] vhost: increase number of async IO vectors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch increases the number of IO vectors for the asynchronous data path from 512 to 2048. It has been reported during testing the startvation of IO vectors during iperf benchmark with 64KB packet size. As there are no direct relationship between VHOST_MAX_ASYNC_VEC and BUF_VECTOR_MAX, this patch also assign VHOST_MAX_ASYNC_VEC value directly instyead of being a multiple of BUF_VECTOR_MAX. Signed-off-by: Maxime Coquelin --- lib/vhost/vhost.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index ae5bc257cc..eaee59c0cc 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -49,7 +49,7 @@ #define MAX_PKT_BURST 32 #define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST) -#define VHOST_MAX_ASYNC_VEC (BUF_VECTOR_MAX * 2) +#define VHOST_MAX_ASYNC_VEC 2048 #define PACKED_DESC_ENQUEUE_USED_FLAG(w) \ ((w) ? (VRING_DESC_F_AVAIL | VRING_DESC_F_USED | VRING_DESC_F_WRITE) : \