From patchwork Thu May 25 16:15:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127440 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A3E4042B9D; Thu, 25 May 2023 18:16:07 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9372142B71; Thu, 25 May 2023 18:16:07 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 8E23442670 for ; Thu, 25 May 2023 18:16:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031363; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CZsJ8smmN1ebh9rwTIxBLIZEXRg2sSbjOTC1stycXWE=; b=dnUiExIvFwjvvcoN/sXi/sYKG2GxbsH4pgtqdj/RKWlxBQxQQ9A5Ds8nID7hvgyv5TBMaO vd03FOQwCFuHKEEkQzfRJtbLg2bHqNpEGvU0+VKg2UdsfTt3W2ZiePTITquObH5kvVx87T Jui6noSMlV67vyYgUK2NA0AA6JKH5lw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-588-hv8uJWEoNCCSqc5xH5GgSQ-1; Thu, 25 May 2023 12:15:59 -0400 X-MC-Unique: hv8uJWEoNCCSqc5xH5GgSQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 414B438035AA; Thu, 25 May 2023 16:15:59 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9117EC154D1; Thu, 25 May 2023 16:15:56 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin , stable@dpdk.org Subject: [PATCH v2 01/28] vhost: fix missing guest notif stat increment Date: Thu, 25 May 2023 18:15:23 +0200 Message-Id: <20230525161551.70002-2-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Guest notification counter was only incremented for split ring, this patch adds it also for packed ring. Fixes: 1ea74efd7fa4 ("vhost: add statistics for guest notification") Cc: stable@dpdk.org Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 8fdab13c70..8554ab4002 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -973,6 +973,8 @@ vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq) kick: if (kick) { eventfd_write(vq->callfd, (eventfd_t)1); + if (dev->flags & VIRTIO_DEV_STATS_ENABLED) + vq->stats.guest_notifications++; if (dev->notify_ops->guest_notified) dev->notify_ops->guest_notified(dev->vid); } From patchwork Thu May 25 16:15:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127441 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A09E42B9D; Thu, 25 May 2023 18:16:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0B37942D42; Thu, 25 May 2023 18:16:09 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 5DF74427EE for ; Thu, 25 May 2023 18:16:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031366; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EbXyXcsVp+6cHMD4j6aGPNclT/Ch3QN0nPlohmuhnTE=; b=brmPXsKls8SCgDD2j4ngeoEipaVgUF3vnImOdVC6seIaPNwgqIsQIz4e3Ttz+ilfWuuEbB Dx7lqEemQd5EU8mtWGnzPbxhmogmFcI90H+DstLXoNXFk2s4GqvKEJrCxwE75UekUwhOgN op1JIBuGtRjs6ZROmhUvQ2BNtqsBGBE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-321-LHeFq7EQOtOEZPQaDOkPcQ-1; Thu, 25 May 2023 12:16:02 -0400 X-MC-Unique: LHeFq7EQOtOEZPQaDOkPcQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4AB561C01516; Thu, 25 May 2023 16:16:02 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 92045C154D1; Thu, 25 May 2023 16:15:59 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin , stable@dpdk.org Subject: [PATCH v2 02/28] vhost: fix invalid call FD handling Date: Thu, 25 May 2023 18:15:24 +0200 Message-Id: <20230525161551.70002-3-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch fixes cases where IRQ injection is tried while the call FD is not valid, which should not happen. Fixes: b1cce26af1dc ("vhost: add notification for packed ring") Fixes: e37ff954405a ("vhost: support virtqueue interrupt/notification suppression") Cc: stable@dpdk.org Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 8554ab4002..40863f7bfd 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -902,9 +902,9 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq) "%s: used_event_idx=%d, old=%d, new=%d\n", __func__, vhost_used_event(vq), old, new); - if ((vhost_need_event(vhost_used_event(vq), new, old) && - (vq->callfd >= 0)) || - unlikely(!signalled_used_valid)) { + if ((vhost_need_event(vhost_used_event(vq), new, old) || + unlikely(!signalled_used_valid)) && + vq->callfd >= 0) { eventfd_write(vq->callfd, (eventfd_t) 1); if (dev->flags & VIRTIO_DEV_STATS_ENABLED) vq->stats.guest_notifications++; @@ -971,7 +971,7 @@ vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq) if (vhost_need_event(off, new, old)) kick = true; kick: - if (kick) { + if (kick && vq->callfd >= 0) { eventfd_write(vq->callfd, (eventfd_t)1); if (dev->flags & VIRTIO_DEV_STATS_ENABLED) vq->stats.guest_notifications++; From patchwork Thu May 25 16:15:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127443 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 440DF42B9D; Thu, 25 May 2023 18:16:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9EE7E42D4B; Thu, 25 May 2023 18:16:13 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 403D342D3B for ; Thu, 25 May 2023 18:16:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kdk7A4dPJ3aSyRCNhnDx2xvj/ieo1OEJTR+LhLYwHNo=; b=SsX/Ts51Ij2X1Tp+3gbrjVrLl1CqYidnz+GymrLUhb0SCe4wzgKOjbQS6mEQiH7nbQy58K PKTCTn06V3L88TfUEk9oXnhf3MuRs0uTVIQGE5zZ96trFVu5tFPl16cVeG3ZkCRL23Yy6Q 0HdaXYhwZ96p6wyQ2luRtJj+8VNUND4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-184-WTdK7zUnMsa8mIEFtO6r0w-1; Thu, 25 May 2023 12:16:05 -0400 X-MC-Unique: WTdK7zUnMsa8mIEFtO6r0w-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 46AED8030D1; Thu, 25 May 2023 16:16:05 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 92C67C154D1; Thu, 25 May 2023 16:16:02 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin , stable@dpdk.org Subject: [PATCH v2 03/28] vhost: fix IOTLB entries overlap check with previous entry Date: Thu, 25 May 2023 18:15:25 +0200 Message-Id: <20230525161551.70002-4-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Commit 22b6d0ac691a ("vhost: fix madvise IOTLB entries pages overlap check") fixed the check to ensure the entry to be removed does not overlap with the next one in the IOTLB cache before marking it as DONTDUMP with madvise(). This is not enough, because the same issue is present when comparing with the previous entry in the cache, where the end address of the previous entry should be used, not the start one. Fixes: dea092d0addb ("vhost: fix madvise arguments alignment") Cc: stable@dpdk.org Signed-off-by: Maxime Coquelin Acked-by: Mike Pattrick Reviewed-by: Chenbo Xia --- lib/vhost/iotlb.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index 3f45bc6061..870c8acb88 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -178,8 +178,8 @@ vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtque mask = ~(alignment - 1); /* Don't disable coredump if the previous node is in the same page */ - if (prev_node == NULL || - (node->uaddr & mask) != (prev_node->uaddr & mask)) { + if (prev_node == NULL || (node->uaddr & mask) != + ((prev_node->uaddr + prev_node->size - 1) & mask)) { next_node = RTE_TAILQ_NEXT(node, next); /* Don't disable coredump if the next node is in the same page */ if (next_node == NULL || ((node->uaddr + node->size - 1) & mask) != @@ -283,8 +283,8 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq mask = ~(alignment-1); /* Don't disable coredump if the previous node is in the same page */ - if (prev_node == NULL || - (node->uaddr & mask) != (prev_node->uaddr & mask)) { + if (prev_node == NULL || (node->uaddr & mask) != + ((prev_node->uaddr + prev_node->size - 1) & mask)) { next_node = RTE_TAILQ_NEXT(node, next); /* Don't disable coredump if the next node is in the same page */ if (next_node == NULL || ((node->uaddr + node->size - 1) & mask) != From patchwork Thu May 25 16:15:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127442 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 608A642B9D; Thu, 25 May 2023 18:16:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7911C42D36; Thu, 25 May 2023 18:16:12 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id A286B42D35 for ; Thu, 25 May 2023 18:16:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031370; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1dnndvsp7vpvr+HSx3wQz/htNvpXZQut92FiITrIDzI=; b=HnL8il1h7iVSjwM4iuFBGItvtEvr4tLwPMv7wgmchadSpNi8CkzFfnjkX+p+IWcPxgz6pT XLkuRejNSIKLttSqvVE9RVUlvJrPa62+MH0BxQesWMIMcCFGGv2KnD8sdcy4YGzPeoo82R fuFj2J0TgFj3GsVnXiLhljP4wtWsH3w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-609-wvQErC-jMnaQFvtktiwTLA-1; Thu, 25 May 2023 12:16:08 -0400 X-MC-Unique: wvQErC-jMnaQFvtktiwTLA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1390B858F15; Thu, 25 May 2023 16:16:08 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8C837C154D1; Thu, 25 May 2023 16:16:05 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 04/28] vhost: add helper of IOTLB entries coredump Date: Thu, 25 May 2023 18:15:26 +0200 Message-Id: <20230525161551.70002-5-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch reworks IOTLB code to extract madvise-related bits into dedicated helper. This refactoring improves code sharing. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/iotlb.c | 77 +++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 36 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index 870c8acb88..268352bf82 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -23,6 +23,34 @@ struct vhost_iotlb_entry { #define IOTLB_CACHE_SIZE 2048 +static void +vhost_user_iotlb_set_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node) +{ + uint64_t align; + + align = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); + + mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, true, align); +} + +static void +vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node, + struct vhost_iotlb_entry *prev, struct vhost_iotlb_entry *next) +{ + uint64_t align, mask; + + align = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); + mask = ~(align - 1); + + /* Don't disable coredump if the previous node is in the same page */ + if (prev == NULL || (node->uaddr & mask) != ((prev->uaddr + prev->size - 1) & mask)) { + /* Don't disable coredump if the next node is in the same page */ + if (next == NULL || + ((node->uaddr + node->size - 1) & mask) != (next->uaddr & mask)) + mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, false, align); + } +} + static struct vhost_iotlb_entry * vhost_user_iotlb_pool_get(struct vhost_virtqueue *vq) { @@ -149,8 +177,8 @@ vhost_user_iotlb_cache_remove_all(struct virtio_net *dev, struct vhost_virtqueue rte_rwlock_write_lock(&vq->iotlb_lock); RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) { - mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, false, - hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr)); + vhost_user_iotlb_set_dump(dev, node); + TAILQ_REMOVE(&vq->iotlb_list, node, next); vhost_user_iotlb_pool_put(vq, node); } @@ -164,7 +192,6 @@ static void vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtqueue *vq) { struct vhost_iotlb_entry *node, *temp_node, *prev_node = NULL; - uint64_t alignment, mask; int entry_idx; rte_rwlock_write_lock(&vq->iotlb_lock); @@ -173,20 +200,10 @@ vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtque RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) { if (!entry_idx) { - struct vhost_iotlb_entry *next_node; - alignment = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); - mask = ~(alignment - 1); - - /* Don't disable coredump if the previous node is in the same page */ - if (prev_node == NULL || (node->uaddr & mask) != - ((prev_node->uaddr + prev_node->size - 1) & mask)) { - next_node = RTE_TAILQ_NEXT(node, next); - /* Don't disable coredump if the next node is in the same page */ - if (next_node == NULL || ((node->uaddr + node->size - 1) & mask) != - (next_node->uaddr & mask)) - mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, - false, alignment); - } + struct vhost_iotlb_entry *next_node = RTE_TAILQ_NEXT(node, next); + + vhost_user_iotlb_clear_dump(dev, node, prev_node, next_node); + TAILQ_REMOVE(&vq->iotlb_list, node, next); vhost_user_iotlb_pool_put(vq, node); vq->iotlb_cache_nr--; @@ -240,16 +257,16 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq vhost_user_iotlb_pool_put(vq, new_node); goto unlock; } else if (node->iova > new_node->iova) { - mem_set_dump((void *)(uintptr_t)new_node->uaddr, new_node->size, true, - hua_to_alignment(dev->mem, (void *)(uintptr_t)new_node->uaddr)); + vhost_user_iotlb_set_dump(dev, new_node); + TAILQ_INSERT_BEFORE(node, new_node, next); vq->iotlb_cache_nr++; goto unlock; } } - mem_set_dump((void *)(uintptr_t)new_node->uaddr, new_node->size, true, - hua_to_alignment(dev->mem, (void *)(uintptr_t)new_node->uaddr)); + vhost_user_iotlb_set_dump(dev, new_node); + TAILQ_INSERT_TAIL(&vq->iotlb_list, new_node, next); vq->iotlb_cache_nr++; @@ -265,7 +282,6 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq uint64_t iova, uint64_t size) { struct vhost_iotlb_entry *node, *temp_node, *prev_node = NULL; - uint64_t alignment, mask; if (unlikely(!size)) return; @@ -278,20 +294,9 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq break; if (iova < node->iova + node->size) { - struct vhost_iotlb_entry *next_node; - alignment = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); - mask = ~(alignment-1); - - /* Don't disable coredump if the previous node is in the same page */ - if (prev_node == NULL || (node->uaddr & mask) != - ((prev_node->uaddr + prev_node->size - 1) & mask)) { - next_node = RTE_TAILQ_NEXT(node, next); - /* Don't disable coredump if the next node is in the same page */ - if (next_node == NULL || ((node->uaddr + node->size - 1) & mask) != - (next_node->uaddr & mask)) - mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, - false, alignment); - } + struct vhost_iotlb_entry *next_node = RTE_TAILQ_NEXT(node, next); + + vhost_user_iotlb_clear_dump(dev, node, prev_node, next_node); TAILQ_REMOVE(&vq->iotlb_list, node, next); vhost_user_iotlb_pool_put(vq, node); From patchwork Thu May 25 16:15:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127445 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D67F742B9D; Thu, 25 May 2023 18:16:48 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6EDF842D67; Thu, 25 May 2023 18:16:22 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 2E3F742D20 for ; Thu, 25 May 2023 18:16:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031378; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BwThloGfu9FYEVmri/LFa9Gkatx5ah2voGpH3XXEnrQ=; b=aNT1na1U22yyrglLLptvFYH5ncNvSaTE4xaTK9A6y6pCKAbH5DstuwXJc4sTCMQmPE7gr6 33myXFtavC7EODlmMQBxwKLLpRNpVdX0vkOtCSDCcnIKWHHHwG8DiOwbjgwVTYwLSgRzfo UBofahMQbJR0IWG760zi5RcSAEIWj/8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-232-1yvmKVWtM7a6iVcp6sV9fw-1; Thu, 25 May 2023 12:16:14 -0400 X-MC-Unique: 1yvmKVWtM7a6iVcp6sV9fw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D80E33802AD4; Thu, 25 May 2023 16:16:10 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 599F8C15612; Thu, 25 May 2023 16:16:08 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 05/28] vhost: add helper for IOTLB entries shared page check Date: Thu, 25 May 2023 18:15:27 +0200 Message-Id: <20230525161551.70002-6-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch introduces a helper to check whether two IOTLB entries share a page. Signed-off-by: Maxime Coquelin Acked-by: Mike Pattrick Reviewed-by: Chenbo Xia --- lib/vhost/iotlb.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index 268352bf82..59a2b2bbac 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -23,6 +23,23 @@ struct vhost_iotlb_entry { #define IOTLB_CACHE_SIZE 2048 +static bool +vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entry *b, + uint64_t align) +{ + uint64_t a_end, b_start; + + if (a == NULL || b == NULL) + return false; + + /* Assumes entry a lower than entry b */ + RTE_ASSERT(a->uaddr < b->uaddr); + a_end = RTE_ALIGN_CEIL(a->uaddr + a->size, align); + b_start = RTE_ALIGN_FLOOR(b->uaddr, align); + + return a_end > b_start; +} + static void vhost_user_iotlb_set_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node) { @@ -37,16 +54,14 @@ static void vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node, struct vhost_iotlb_entry *prev, struct vhost_iotlb_entry *next) { - uint64_t align, mask; + uint64_t align; align = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); - mask = ~(align - 1); /* Don't disable coredump if the previous node is in the same page */ - if (prev == NULL || (node->uaddr & mask) != ((prev->uaddr + prev->size - 1) & mask)) { + if (!vhost_user_iotlb_share_page(prev, node, align)) { /* Don't disable coredump if the next node is in the same page */ - if (next == NULL || - ((node->uaddr + node->size - 1) & mask) != (next->uaddr & mask)) + if (!vhost_user_iotlb_share_page(node, next, align)) mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, false, align); } } From patchwork Thu May 25 16:15:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127444 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52F1642B9D; Thu, 25 May 2023 18:16:40 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3CFCE42D59; Thu, 25 May 2023 18:16:21 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id DDF434282D for ; Thu, 25 May 2023 18:16:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031378; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NaYAOnGr3jmLE1GL843N20DwC5pSInNH70YXhq6TEzE=; b=BEhzstyezj/0gsNGZjTdzZfxzWLAbnfbJe1G2H/h7gwIhh5Da3ibIgrCzWPiSvmvzv09Nk nQmc/StNqTDlIphkk4i496dOwGNhoQq6bRL/3O96Bu1DaOwQsxsQjdeU3/6FdUfmtgOEMN I/S76rZsJYMGauNuRUC7FxrapNuVW2I= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-333-K2E_m2rjPPOidGWN-rZtmQ-1; Thu, 25 May 2023 12:16:14 -0400 X-MC-Unique: K2E_m2rjPPOidGWN-rZtmQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CEF371C08DB1; Thu, 25 May 2023 16:16:13 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2844FC0004B; Thu, 25 May 2023 16:16:10 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin , stable@dpdk.org Subject: [PATCH v2 06/28] vhost: don't dump unneeded pages with IOTLB Date: Thu, 25 May 2023 18:15:28 +0200 Message-Id: <20230525161551.70002-7-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On IOTLB entry removal, previous fixes took care of not marking pages shared with other IOTLB entries as DONTDUMP. However, if an IOTLB entry is spanned on multiple pages, the other pages were kept as DODUMP while they might not have been shared with other entries, increasing needlessly the coredump size. This patch addresses this issue by excluding only the shared pages from madvise's DONTDUMP. Fixes: dea092d0addb ("vhost: fix madvise arguments alignment") Cc: stable@dpdk.org Signed-off-by: Maxime Coquelin Acked-by: Mike Pattrick Reviewed-by: Chenbo Xia --- lib/vhost/iotlb.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index 59a2b2bbac..5c5200114e 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -54,16 +54,23 @@ static void vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node, struct vhost_iotlb_entry *prev, struct vhost_iotlb_entry *next) { - uint64_t align; + uint64_t align, start, end; + + start = node->uaddr; + end = node->uaddr + node->size; align = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); - /* Don't disable coredump if the previous node is in the same page */ - if (!vhost_user_iotlb_share_page(prev, node, align)) { - /* Don't disable coredump if the next node is in the same page */ - if (!vhost_user_iotlb_share_page(node, next, align)) - mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, false, align); - } + /* Skip first page if shared with previous entry. */ + if (vhost_user_iotlb_share_page(prev, node, align)) + start = RTE_ALIGN_CEIL(start, align); + + /* Skip last page if shared with next entry. */ + if (vhost_user_iotlb_share_page(node, next, align)) + end = RTE_ALIGN_FLOOR(end, align); + + if (end > start) + mem_set_dump((void *)(uintptr_t)start, end - start, false, align); } static struct vhost_iotlb_entry * From patchwork Thu May 25 16:15:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127446 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A0A342B9D; Thu, 25 May 2023 18:16:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 92B2542D62; Thu, 25 May 2023 18:16:23 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id E5AB942D55 for ; Thu, 25 May 2023 18:16:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=inOVRWgeh+2Kr6sJahuNG4IIkFwepYrfMmwDyNS6hrk=; b=gAruHCU7qwyYAKRMUgDNUUj6uoleygDS0P2yKXbKUnKLe1ULLQKHgxmEOaAJFuOrDpdn21 fiBUct2J5O95PYIxuRX0fquGXefGr/u+Cq6ijFi//Napxntl66CAd72azYrS8antCW8Igj 4uJNKBdeGDn8AfwYT0Tr+6t35R87LSM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-644-0Utr-m_MPPi-6Zvj-UZR2g-1; Thu, 25 May 2023 12:16:17 -0400 X-MC-Unique: 0Utr-m_MPPi-6Zvj-UZR2g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C8A8D2999B3A; Thu, 25 May 2023 16:16:16 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 21646C154D2; Thu, 25 May 2023 16:16:13 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 07/28] vhost: change to single IOTLB cache per device Date: Thu, 25 May 2023 18:15:29 +0200 Message-Id: <20230525161551.70002-8-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch simplifies IOTLB implementation and improves IOTLB memory consumption by having a single IOTLB cache per device, instead of having one per queue. In order to not impact performance, it keeps an IOTLB lock per virtqueue, so that there is no contention between multiple queue trying to acquire it. Signed-off-by: Maxime Coquelin --- lib/vhost/iotlb.c | 212 +++++++++++++++++++---------------------- lib/vhost/iotlb.h | 43 ++++++--- lib/vhost/vhost.c | 18 ++-- lib/vhost/vhost.h | 16 ++-- lib/vhost/vhost_user.c | 23 +++-- 5 files changed, 159 insertions(+), 153 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index 5c5200114e..f28483cb7a 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -74,86 +74,81 @@ vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *no } static struct vhost_iotlb_entry * -vhost_user_iotlb_pool_get(struct vhost_virtqueue *vq) +vhost_user_iotlb_pool_get(struct virtio_net *dev) { struct vhost_iotlb_entry *node; - rte_spinlock_lock(&vq->iotlb_free_lock); - node = SLIST_FIRST(&vq->iotlb_free_list); + rte_spinlock_lock(&dev->iotlb_free_lock); + node = SLIST_FIRST(&dev->iotlb_free_list); if (node != NULL) - SLIST_REMOVE_HEAD(&vq->iotlb_free_list, next_free); - rte_spinlock_unlock(&vq->iotlb_free_lock); + SLIST_REMOVE_HEAD(&dev->iotlb_free_list, next_free); + rte_spinlock_unlock(&dev->iotlb_free_lock); return node; } static void -vhost_user_iotlb_pool_put(struct vhost_virtqueue *vq, - struct vhost_iotlb_entry *node) +vhost_user_iotlb_pool_put(struct virtio_net *dev, struct vhost_iotlb_entry *node) { - rte_spinlock_lock(&vq->iotlb_free_lock); - SLIST_INSERT_HEAD(&vq->iotlb_free_list, node, next_free); - rte_spinlock_unlock(&vq->iotlb_free_lock); + rte_spinlock_lock(&dev->iotlb_free_lock); + SLIST_INSERT_HEAD(&dev->iotlb_free_list, node, next_free); + rte_spinlock_unlock(&dev->iotlb_free_lock); } static void -vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtqueue *vq); +vhost_user_iotlb_cache_random_evict(struct virtio_net *dev); static void -vhost_user_iotlb_pending_remove_all(struct vhost_virtqueue *vq) +vhost_user_iotlb_pending_remove_all(struct virtio_net *dev) { struct vhost_iotlb_entry *node, *temp_node; - rte_rwlock_write_lock(&vq->iotlb_pending_lock); + rte_rwlock_write_lock(&dev->iotlb_pending_lock); - RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_pending_list, next, temp_node) { - TAILQ_REMOVE(&vq->iotlb_pending_list, node, next); - vhost_user_iotlb_pool_put(vq, node); + RTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_pending_list, next, temp_node) { + TAILQ_REMOVE(&dev->iotlb_pending_list, node, next); + vhost_user_iotlb_pool_put(dev, node); } - rte_rwlock_write_unlock(&vq->iotlb_pending_lock); + rte_rwlock_write_unlock(&dev->iotlb_pending_lock); } bool -vhost_user_iotlb_pending_miss(struct vhost_virtqueue *vq, uint64_t iova, - uint8_t perm) +vhost_user_iotlb_pending_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm) { struct vhost_iotlb_entry *node; bool found = false; - rte_rwlock_read_lock(&vq->iotlb_pending_lock); + rte_rwlock_read_lock(&dev->iotlb_pending_lock); - TAILQ_FOREACH(node, &vq->iotlb_pending_list, next) { + TAILQ_FOREACH(node, &dev->iotlb_pending_list, next) { if ((node->iova == iova) && (node->perm == perm)) { found = true; break; } } - rte_rwlock_read_unlock(&vq->iotlb_pending_lock); + rte_rwlock_read_unlock(&dev->iotlb_pending_lock); return found; } void -vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue *vq, - uint64_t iova, uint8_t perm) +vhost_user_iotlb_pending_insert(struct virtio_net *dev, uint64_t iova, uint8_t perm) { struct vhost_iotlb_entry *node; - node = vhost_user_iotlb_pool_get(vq); + node = vhost_user_iotlb_pool_get(dev); if (node == NULL) { VHOST_LOG_CONFIG(dev->ifname, DEBUG, - "IOTLB pool for vq %"PRIu32" empty, clear entries for pending insertion\n", - vq->index); - if (!TAILQ_EMPTY(&vq->iotlb_pending_list)) - vhost_user_iotlb_pending_remove_all(vq); + "IOTLB pool empty, clear entries for pending insertion\n"); + if (!TAILQ_EMPTY(&dev->iotlb_pending_list)) + vhost_user_iotlb_pending_remove_all(dev); else - vhost_user_iotlb_cache_random_evict(dev, vq); - node = vhost_user_iotlb_pool_get(vq); + vhost_user_iotlb_cache_random_evict(dev); + node = vhost_user_iotlb_pool_get(dev); if (node == NULL) { VHOST_LOG_CONFIG(dev->ifname, ERR, - "IOTLB pool vq %"PRIu32" still empty, pending insertion failure\n", - vq->index); + "IOTLB pool still empty, pending insertion failure\n"); return; } } @@ -161,22 +156,21 @@ vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue * node->iova = iova; node->perm = perm; - rte_rwlock_write_lock(&vq->iotlb_pending_lock); + rte_rwlock_write_lock(&dev->iotlb_pending_lock); - TAILQ_INSERT_TAIL(&vq->iotlb_pending_list, node, next); + TAILQ_INSERT_TAIL(&dev->iotlb_pending_list, node, next); - rte_rwlock_write_unlock(&vq->iotlb_pending_lock); + rte_rwlock_write_unlock(&dev->iotlb_pending_lock); } void -vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq, - uint64_t iova, uint64_t size, uint8_t perm) +vhost_user_iotlb_pending_remove(struct virtio_net *dev, uint64_t iova, uint64_t size, uint8_t perm) { struct vhost_iotlb_entry *node, *temp_node; - rte_rwlock_write_lock(&vq->iotlb_pending_lock); + rte_rwlock_write_lock(&dev->iotlb_pending_lock); - RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_pending_list, next, + RTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_pending_list, next, temp_node) { if (node->iova < iova) continue; @@ -184,81 +178,78 @@ vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq, continue; if ((node->perm & perm) != node->perm) continue; - TAILQ_REMOVE(&vq->iotlb_pending_list, node, next); - vhost_user_iotlb_pool_put(vq, node); + TAILQ_REMOVE(&dev->iotlb_pending_list, node, next); + vhost_user_iotlb_pool_put(dev, node); } - rte_rwlock_write_unlock(&vq->iotlb_pending_lock); + rte_rwlock_write_unlock(&dev->iotlb_pending_lock); } static void -vhost_user_iotlb_cache_remove_all(struct virtio_net *dev, struct vhost_virtqueue *vq) +vhost_user_iotlb_cache_remove_all(struct virtio_net *dev) { struct vhost_iotlb_entry *node, *temp_node; - rte_rwlock_write_lock(&vq->iotlb_lock); + vhost_user_iotlb_wr_lock_all(dev); - RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) { + RTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) { vhost_user_iotlb_set_dump(dev, node); - TAILQ_REMOVE(&vq->iotlb_list, node, next); - vhost_user_iotlb_pool_put(vq, node); + TAILQ_REMOVE(&dev->iotlb_list, node, next); + vhost_user_iotlb_pool_put(dev, node); } - vq->iotlb_cache_nr = 0; + dev->iotlb_cache_nr = 0; - rte_rwlock_write_unlock(&vq->iotlb_lock); + vhost_user_iotlb_wr_unlock_all(dev); } static void -vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtqueue *vq) +vhost_user_iotlb_cache_random_evict(struct virtio_net *dev) { struct vhost_iotlb_entry *node, *temp_node, *prev_node = NULL; int entry_idx; - rte_rwlock_write_lock(&vq->iotlb_lock); + vhost_user_iotlb_wr_lock_all(dev); - entry_idx = rte_rand() % vq->iotlb_cache_nr; + entry_idx = rte_rand() % dev->iotlb_cache_nr; - RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) { + RTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) { if (!entry_idx) { struct vhost_iotlb_entry *next_node = RTE_TAILQ_NEXT(node, next); vhost_user_iotlb_clear_dump(dev, node, prev_node, next_node); - TAILQ_REMOVE(&vq->iotlb_list, node, next); - vhost_user_iotlb_pool_put(vq, node); - vq->iotlb_cache_nr--; + TAILQ_REMOVE(&dev->iotlb_list, node, next); + vhost_user_iotlb_pool_put(dev, node); + dev->iotlb_cache_nr--; break; } prev_node = node; entry_idx--; } - rte_rwlock_write_unlock(&vq->iotlb_lock); + vhost_user_iotlb_wr_unlock_all(dev); } void -vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq, - uint64_t iova, uint64_t uaddr, +vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr, uint64_t size, uint8_t perm) { struct vhost_iotlb_entry *node, *new_node; - new_node = vhost_user_iotlb_pool_get(vq); + new_node = vhost_user_iotlb_pool_get(dev); if (new_node == NULL) { VHOST_LOG_CONFIG(dev->ifname, DEBUG, - "IOTLB pool vq %"PRIu32" empty, clear entries for cache insertion\n", - vq->index); - if (!TAILQ_EMPTY(&vq->iotlb_list)) - vhost_user_iotlb_cache_random_evict(dev, vq); + "IOTLB pool empty, clear entries for cache insertion\n"); + if (!TAILQ_EMPTY(&dev->iotlb_list)) + vhost_user_iotlb_cache_random_evict(dev); else - vhost_user_iotlb_pending_remove_all(vq); - new_node = vhost_user_iotlb_pool_get(vq); + vhost_user_iotlb_pending_remove_all(dev); + new_node = vhost_user_iotlb_pool_get(dev); if (new_node == NULL) { VHOST_LOG_CONFIG(dev->ifname, ERR, - "IOTLB pool vq %"PRIu32" still empty, cache insertion failed\n", - vq->index); + "IOTLB pool still empty, cache insertion failed\n"); return; } } @@ -268,49 +259,47 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq new_node->size = size; new_node->perm = perm; - rte_rwlock_write_lock(&vq->iotlb_lock); + vhost_user_iotlb_wr_lock_all(dev); - TAILQ_FOREACH(node, &vq->iotlb_list, next) { + TAILQ_FOREACH(node, &dev->iotlb_list, next) { /* * Entries must be invalidated before being updated. * So if iova already in list, assume identical. */ if (node->iova == new_node->iova) { - vhost_user_iotlb_pool_put(vq, new_node); + vhost_user_iotlb_pool_put(dev, new_node); goto unlock; } else if (node->iova > new_node->iova) { vhost_user_iotlb_set_dump(dev, new_node); TAILQ_INSERT_BEFORE(node, new_node, next); - vq->iotlb_cache_nr++; + dev->iotlb_cache_nr++; goto unlock; } } vhost_user_iotlb_set_dump(dev, new_node); - TAILQ_INSERT_TAIL(&vq->iotlb_list, new_node, next); - vq->iotlb_cache_nr++; + TAILQ_INSERT_TAIL(&dev->iotlb_list, new_node, next); + dev->iotlb_cache_nr++; unlock: - vhost_user_iotlb_pending_remove(vq, iova, size, perm); - - rte_rwlock_write_unlock(&vq->iotlb_lock); + vhost_user_iotlb_pending_remove(dev, iova, size, perm); + vhost_user_iotlb_wr_unlock_all(dev); } void -vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq, - uint64_t iova, uint64_t size) +vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t size) { struct vhost_iotlb_entry *node, *temp_node, *prev_node = NULL; if (unlikely(!size)) return; - rte_rwlock_write_lock(&vq->iotlb_lock); + vhost_user_iotlb_wr_lock_all(dev); - RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) { + RTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) { /* Sorted list */ if (unlikely(iova + size < node->iova)) break; @@ -320,19 +309,19 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq vhost_user_iotlb_clear_dump(dev, node, prev_node, next_node); - TAILQ_REMOVE(&vq->iotlb_list, node, next); - vhost_user_iotlb_pool_put(vq, node); - vq->iotlb_cache_nr--; - } else + TAILQ_REMOVE(&dev->iotlb_list, node, next); + vhost_user_iotlb_pool_put(dev, node); + dev->iotlb_cache_nr--; + } else { prev_node = node; + } } - rte_rwlock_write_unlock(&vq->iotlb_lock); + vhost_user_iotlb_wr_unlock_all(dev); } uint64_t -vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova, - uint64_t *size, uint8_t perm) +vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova, uint64_t *size, uint8_t perm) { struct vhost_iotlb_entry *node; uint64_t offset, vva = 0, mapped = 0; @@ -340,7 +329,7 @@ vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova, if (unlikely(!*size)) goto out; - TAILQ_FOREACH(node, &vq->iotlb_list, next) { + TAILQ_FOREACH(node, &dev->iotlb_list, next) { /* List sorted by iova */ if (unlikely(iova < node->iova)) break; @@ -373,60 +362,57 @@ vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova, } void -vhost_user_iotlb_flush_all(struct virtio_net *dev, struct vhost_virtqueue *vq) +vhost_user_iotlb_flush_all(struct virtio_net *dev) { - vhost_user_iotlb_cache_remove_all(dev, vq); - vhost_user_iotlb_pending_remove_all(vq); + vhost_user_iotlb_cache_remove_all(dev); + vhost_user_iotlb_pending_remove_all(dev); } int -vhost_user_iotlb_init(struct virtio_net *dev, struct vhost_virtqueue *vq) +vhost_user_iotlb_init(struct virtio_net *dev) { unsigned int i; int socket = 0; - if (vq->iotlb_pool) { + if (dev->iotlb_pool) { /* * The cache has already been initialized, * just drop all cached and pending entries. */ - vhost_user_iotlb_flush_all(dev, vq); - rte_free(vq->iotlb_pool); + vhost_user_iotlb_flush_all(dev); + rte_free(dev->iotlb_pool); } #ifdef RTE_LIBRTE_VHOST_NUMA - if (get_mempolicy(&socket, NULL, 0, vq, MPOL_F_NODE | MPOL_F_ADDR) != 0) + if (get_mempolicy(&socket, NULL, 0, dev, MPOL_F_NODE | MPOL_F_ADDR) != 0) socket = 0; #endif - rte_spinlock_init(&vq->iotlb_free_lock); - rte_rwlock_init(&vq->iotlb_lock); - rte_rwlock_init(&vq->iotlb_pending_lock); + rte_spinlock_init(&dev->iotlb_free_lock); + rte_rwlock_init(&dev->iotlb_pending_lock); - SLIST_INIT(&vq->iotlb_free_list); - TAILQ_INIT(&vq->iotlb_list); - TAILQ_INIT(&vq->iotlb_pending_list); + SLIST_INIT(&dev->iotlb_free_list); + TAILQ_INIT(&dev->iotlb_list); + TAILQ_INIT(&dev->iotlb_pending_list); if (dev->flags & VIRTIO_DEV_SUPPORT_IOMMU) { - vq->iotlb_pool = rte_calloc_socket("iotlb", IOTLB_CACHE_SIZE, + dev->iotlb_pool = rte_calloc_socket("iotlb", IOTLB_CACHE_SIZE, sizeof(struct vhost_iotlb_entry), 0, socket); - if (!vq->iotlb_pool) { - VHOST_LOG_CONFIG(dev->ifname, ERR, - "Failed to create IOTLB cache pool for vq %"PRIu32"\n", - vq->index); + if (!dev->iotlb_pool) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to create IOTLB cache pool\n"); return -1; } for (i = 0; i < IOTLB_CACHE_SIZE; i++) - vhost_user_iotlb_pool_put(vq, &vq->iotlb_pool[i]); + vhost_user_iotlb_pool_put(dev, &dev->iotlb_pool[i]); } - vq->iotlb_cache_nr = 0; + dev->iotlb_cache_nr = 0; return 0; } void -vhost_user_iotlb_destroy(struct vhost_virtqueue *vq) +vhost_user_iotlb_destroy(struct virtio_net *dev) { - rte_free(vq->iotlb_pool); + rte_free(dev->iotlb_pool); } diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h index 73b5465b41..3490b9e6be 100644 --- a/lib/vhost/iotlb.h +++ b/lib/vhost/iotlb.h @@ -37,20 +37,37 @@ vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq) rte_rwlock_write_unlock(&vq->iotlb_lock); } -void vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq, - uint64_t iova, uint64_t uaddr, +static __rte_always_inline void +vhost_user_iotlb_wr_lock_all(struct virtio_net *dev) + __rte_no_thread_safety_analysis +{ + uint32_t i; + + for (i = 0; i < dev->nr_vring; i++) + rte_rwlock_write_lock(&dev->virtqueue[i]->iotlb_lock); +} + +static __rte_always_inline void +vhost_user_iotlb_wr_unlock_all(struct virtio_net *dev) + __rte_no_thread_safety_analysis +{ + uint32_t i; + + for (i = 0; i < dev->nr_vring; i++) + rte_rwlock_write_unlock(&dev->virtqueue[i]->iotlb_lock); +} + +void vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr, uint64_t size, uint8_t perm); -void vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq, - uint64_t iova, uint64_t size); -uint64_t vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova, +void vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t size); +uint64_t vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova, uint64_t *size, uint8_t perm); -bool vhost_user_iotlb_pending_miss(struct vhost_virtqueue *vq, uint64_t iova, - uint8_t perm); -void vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue *vq, - uint64_t iova, uint8_t perm); -void vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq, uint64_t iova, +bool vhost_user_iotlb_pending_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm); +void vhost_user_iotlb_pending_insert(struct virtio_net *dev, uint64_t iova, uint8_t perm); +void vhost_user_iotlb_pending_remove(struct virtio_net *dev, uint64_t iova, uint64_t size, uint8_t perm); -void vhost_user_iotlb_flush_all(struct virtio_net *dev, struct vhost_virtqueue *vq); -int vhost_user_iotlb_init(struct virtio_net *dev, struct vhost_virtqueue *vq); -void vhost_user_iotlb_destroy(struct vhost_virtqueue *vq); +void vhost_user_iotlb_flush_all(struct virtio_net *dev); +int vhost_user_iotlb_init(struct virtio_net *dev); +void vhost_user_iotlb_destroy(struct virtio_net *dev); + #endif /* _VHOST_IOTLB_H_ */ diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index ef37943817..d35075b96c 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -63,7 +63,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq, tmp_size = *size; - vva = vhost_user_iotlb_cache_find(vq, iova, &tmp_size, perm); + vva = vhost_user_iotlb_cache_find(dev, iova, &tmp_size, perm); if (tmp_size == *size) { if (dev->flags & VIRTIO_DEV_STATS_ENABLED) vq->stats.iotlb_hits++; @@ -75,7 +75,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq, iova += tmp_size; - if (!vhost_user_iotlb_pending_miss(vq, iova, perm)) { + if (!vhost_user_iotlb_pending_miss(dev, iova, perm)) { /* * iotlb_lock is read-locked for a full burst, * but it only protects the iotlb cache. @@ -85,12 +85,12 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq, */ vhost_user_iotlb_rd_unlock(vq); - vhost_user_iotlb_pending_insert(dev, vq, iova, perm); + vhost_user_iotlb_pending_insert(dev, iova, perm); if (vhost_user_iotlb_miss(dev, iova, perm)) { VHOST_LOG_DATA(dev->ifname, ERR, "IOTLB miss req failed for IOVA 0x%" PRIx64 "\n", iova); - vhost_user_iotlb_pending_remove(vq, iova, 1, perm); + vhost_user_iotlb_pending_remove(dev, iova, 1, perm); } vhost_user_iotlb_rd_lock(vq); @@ -397,7 +397,6 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq) vhost_free_async_mem(vq); rte_spinlock_unlock(&vq->access_lock); rte_free(vq->batch_copy_elems); - vhost_user_iotlb_destroy(vq); rte_free(vq->log_cache); rte_free(vq); } @@ -575,7 +574,7 @@ vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq } static void -init_vring_queue(struct virtio_net *dev, struct vhost_virtqueue *vq, +init_vring_queue(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq, uint32_t vring_idx) { int numa_node = SOCKET_ID_ANY; @@ -595,8 +594,6 @@ init_vring_queue(struct virtio_net *dev, struct vhost_virtqueue *vq, } #endif vq->numa_node = numa_node; - - vhost_user_iotlb_init(dev, vq); } static void @@ -631,6 +628,7 @@ alloc_vring_queue(struct virtio_net *dev, uint32_t vring_idx) dev->virtqueue[i] = vq; init_vring_queue(dev, vq, i); rte_spinlock_init(&vq->access_lock); + rte_rwlock_init(&vq->iotlb_lock); vq->avail_wrap_counter = 1; vq->used_wrap_counter = 1; vq->signalled_used_valid = false; @@ -795,6 +793,10 @@ vhost_setup_virtio_net(int vid, bool enable, bool compliant_ol_flags, bool stats dev->flags |= VIRTIO_DEV_SUPPORT_IOMMU; else dev->flags &= ~VIRTIO_DEV_SUPPORT_IOMMU; + + if (vhost_user_iotlb_init(dev) < 0) + VHOST_LOG_CONFIG("device", ERR, "failed to init IOTLB\n"); + } void diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 40863f7bfd..67cc4a2fdb 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -302,13 +302,6 @@ struct vhost_virtqueue { struct log_cache_entry *log_cache; rte_rwlock_t iotlb_lock; - rte_rwlock_t iotlb_pending_lock; - struct vhost_iotlb_entry *iotlb_pool; - TAILQ_HEAD(, vhost_iotlb_entry) iotlb_list; - TAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list; - int iotlb_cache_nr; - rte_spinlock_t iotlb_free_lock; - SLIST_HEAD(, vhost_iotlb_entry) iotlb_free_list; /* Used to notify the guest (trigger interrupt) */ int callfd; @@ -483,6 +476,15 @@ struct virtio_net { int extbuf; int linearbuf; struct vhost_virtqueue *virtqueue[VHOST_MAX_QUEUE_PAIRS * 2]; + + rte_rwlock_t iotlb_pending_lock; + struct vhost_iotlb_entry *iotlb_pool; + TAILQ_HEAD(, vhost_iotlb_entry) iotlb_list; + TAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list; + int iotlb_cache_nr; + rte_spinlock_t iotlb_free_lock; + SLIST_HEAD(, vhost_iotlb_entry) iotlb_free_list; + struct inflight_mem_info *inflight_info; #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ) char ifname[IF_NAME_SZ]; diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index d60e39b6bc..8d0f84348b 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -237,6 +237,8 @@ vhost_backend_cleanup(struct virtio_net *dev) } dev->postcopy_listening = 0; + + vhost_user_iotlb_destroy(dev); } static void @@ -539,7 +541,6 @@ numa_realloc(struct virtio_net **pdev, struct vhost_virtqueue **pvq) if (vq != dev->virtqueue[vq->index]) { VHOST_LOG_CONFIG(dev->ifname, INFO, "reallocated virtqueue on node %d\n", node); dev->virtqueue[vq->index] = vq; - vhost_user_iotlb_init(dev, vq); } if (vq_is_packed(dev)) { @@ -664,6 +665,8 @@ numa_realloc(struct virtio_net **pdev, struct vhost_virtqueue **pvq) return; } dev->guest_pages = gp; + + vhost_user_iotlb_init(dev); } #else static void @@ -1360,8 +1363,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev, /* Flush IOTLB cache as previous HVAs are now invalid */ if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) - for (i = 0; i < dev->nr_vring; i++) - vhost_user_iotlb_flush_all(dev, dev->virtqueue[i]); + vhost_user_iotlb_flush_all(dev); free_mem_region(dev); rte_free(dev->mem); @@ -2194,7 +2196,7 @@ vhost_user_get_vring_base(struct virtio_net **pdev, ctx->msg.size = sizeof(ctx->msg.payload.state); ctx->fd_num = 0; - vhost_user_iotlb_flush_all(dev, vq); + vhost_user_iotlb_flush_all(dev); vring_invalidate(dev, vq); @@ -2639,15 +2641,14 @@ vhost_user_iotlb_msg(struct virtio_net **pdev, if (!vva) return RTE_VHOST_MSG_RESULT_ERR; + vhost_user_iotlb_cache_insert(dev, imsg->iova, vva, len, imsg->perm); + for (i = 0; i < dev->nr_vring; i++) { struct vhost_virtqueue *vq = dev->virtqueue[i]; if (!vq) continue; - vhost_user_iotlb_cache_insert(dev, vq, imsg->iova, vva, - len, imsg->perm); - if (is_vring_iotlb(dev, vq, imsg)) { rte_spinlock_lock(&vq->access_lock); translate_ring_addresses(&dev, &vq); @@ -2657,15 +2658,14 @@ vhost_user_iotlb_msg(struct virtio_net **pdev, } break; case VHOST_IOTLB_INVALIDATE: + vhost_user_iotlb_cache_remove(dev, imsg->iova, imsg->size); + for (i = 0; i < dev->nr_vring; i++) { struct vhost_virtqueue *vq = dev->virtqueue[i]; if (!vq) continue; - vhost_user_iotlb_cache_remove(dev, vq, imsg->iova, - imsg->size); - if (is_vring_iotlb(dev, vq, imsg)) { rte_spinlock_lock(&vq->access_lock); vring_invalidate(dev, vq); @@ -2674,8 +2674,7 @@ vhost_user_iotlb_msg(struct virtio_net **pdev, } break; default: - VHOST_LOG_CONFIG(dev->ifname, ERR, - "invalid IOTLB message type (%d)\n", + VHOST_LOG_CONFIG(dev->ifname, ERR, "invalid IOTLB message type (%d)\n", imsg->type); return RTE_VHOST_MSG_RESULT_ERR; } From patchwork Thu May 25 16:15:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127447 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AB28C42B9D; Thu, 25 May 2023 18:17:06 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1368142D84; Thu, 25 May 2023 18:16:26 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 0480642D6D for ; Thu, 25 May 2023 18:16:23 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031383; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9czbGkIafZvcd40//Ui+57ZvmoAvFB4nCHC2HULjqiI=; b=FYuOe0LOKJX25x2CPQKIHw5F9O9DaNPKXFY0iIPLxjPmI8lfMsGMVhg9/zaQuPhC8emeGY DzvkkGGyS87ww/G/iKwmWyrix664DUXN8piB259X9NLAOd8hwWG9afldgKelmWnR6Z6Soc F4wsfpCWs0LJib87U6nP+H1hRWjXw+Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-371-7aGdgEy8NqyxyovJpiyUAQ-1; Thu, 25 May 2023 12:16:20 -0400 X-MC-Unique: 7aGdgEy8NqyxyovJpiyUAQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A17968030D2; Thu, 25 May 2023 16:16:19 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2472FC154D1; Thu, 25 May 2023 16:16:16 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 08/28] vhost: add offset field to IOTLB entries Date: Thu, 25 May 2023 18:15:30 +0200 Message-Id: <20230525161551.70002-9-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch is a preliminary work to prepare for VDUSE support, for which we need to keep track of the mmaped base address and offset in order to be able to unmap it later when IOTLB entry is invalidated. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/iotlb.c | 30 ++++++++++++++++++------------ lib/vhost/iotlb.h | 2 +- lib/vhost/vhost_user.c | 2 +- 3 files changed, 20 insertions(+), 14 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index f28483cb7a..14d143366b 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -17,6 +17,7 @@ struct vhost_iotlb_entry { uint64_t iova; uint64_t uaddr; + uint64_t uoffset; uint64_t size; uint8_t perm; }; @@ -27,15 +28,18 @@ static bool vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entry *b, uint64_t align) { - uint64_t a_end, b_start; + uint64_t a_start, a_end, b_start; if (a == NULL || b == NULL) return false; + a_start = a->uaddr + a->uoffset; + b_start = b->uaddr + b->uoffset; + /* Assumes entry a lower than entry b */ - RTE_ASSERT(a->uaddr < b->uaddr); - a_end = RTE_ALIGN_CEIL(a->uaddr + a->size, align); - b_start = RTE_ALIGN_FLOOR(b->uaddr, align); + RTE_ASSERT(a_start < b_start); + a_end = RTE_ALIGN_CEIL(a_start + a->size, align); + b_start = RTE_ALIGN_FLOOR(b_start, align); return a_end > b_start; } @@ -43,11 +47,12 @@ vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entr static void vhost_user_iotlb_set_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node) { - uint64_t align; + uint64_t align, start; - align = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); + start = node->uaddr + node->uoffset; + align = hua_to_alignment(dev->mem, (void *)(uintptr_t)start); - mem_set_dump((void *)(uintptr_t)node->uaddr, node->size, true, align); + mem_set_dump((void *)(uintptr_t)start, node->size, true, align); } static void @@ -56,10 +61,10 @@ vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *no { uint64_t align, start, end; - start = node->uaddr; - end = node->uaddr + node->size; + start = node->uaddr + node->uoffset; + end = start + node->size; - align = hua_to_alignment(dev->mem, (void *)(uintptr_t)node->uaddr); + align = hua_to_alignment(dev->mem, (void *)(uintptr_t)start); /* Skip first page if shared with previous entry. */ if (vhost_user_iotlb_share_page(prev, node, align)) @@ -234,7 +239,7 @@ vhost_user_iotlb_cache_random_evict(struct virtio_net *dev) void vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr, - uint64_t size, uint8_t perm) + uint64_t uoffset, uint64_t size, uint8_t perm) { struct vhost_iotlb_entry *node, *new_node; @@ -256,6 +261,7 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t ua new_node->iova = iova; new_node->uaddr = uaddr; + new_node->uoffset = uoffset; new_node->size = size; new_node->perm = perm; @@ -344,7 +350,7 @@ vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova, uint64_t *siz offset = iova - node->iova; if (!vva) - vva = node->uaddr + offset; + vva = node->uaddr + node->uoffset + offset; mapped += node->size - offset; iova = node->iova + node->size; diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h index 3490b9e6be..bee36c5903 100644 --- a/lib/vhost/iotlb.h +++ b/lib/vhost/iotlb.h @@ -58,7 +58,7 @@ vhost_user_iotlb_wr_unlock_all(struct virtio_net *dev) } void vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr, - uint64_t size, uint8_t perm); + uint64_t uoffset, uint64_t size, uint8_t perm); void vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t size); uint64_t vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova, uint64_t *size, uint8_t perm); diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 8d0f84348b..222ccbf819 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2641,7 +2641,7 @@ vhost_user_iotlb_msg(struct virtio_net **pdev, if (!vva) return RTE_VHOST_MSG_RESULT_ERR; - vhost_user_iotlb_cache_insert(dev, imsg->iova, vva, len, imsg->perm); + vhost_user_iotlb_cache_insert(dev, imsg->iova, vva, 0, len, imsg->perm); for (i = 0; i < dev->nr_vring; i++) { struct vhost_virtqueue *vq = dev->virtqueue[i]; From patchwork Thu May 25 16:15:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127448 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B162542B9D; Thu, 25 May 2023 18:17:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 455BB42D39; Thu, 25 May 2023 18:16:28 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 98E4342D38 for ; Thu, 25 May 2023 18:16:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031385; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7rPsrxAWe2pXPEMqG+q1gSuIdMFC09KIX9SIQ8ZwNSU=; b=WKs7TYE0ZtuEFgLOa4veX0YGxMm+Vk86dkVkoxodjeNCNsK2qjARrfXNzYPK1aAZgHOI/7 5khF8EkSHLulQqJNbWY6oHqW1c75Y2f1Ts3DSAwtWwu6IHaBI9X0Rl/IGshp+FECKwgxtP ZDFBUIoXl21BomCmETu3Xz9Altswamw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-626-mYbCYPZDNCqjy7QyAKyC6g-1; Thu, 25 May 2023 12:16:23 -0400 X-MC-Unique: mYbCYPZDNCqjy7QyAKyC6g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3B29F85A5BB; Thu, 25 May 2023 16:16:23 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id E81DBC154D2; Thu, 25 May 2023 16:16:19 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 09/28] vhost: add page size info to IOTLB entry Date: Thu, 25 May 2023 18:15:31 +0200 Message-Id: <20230525161551.70002-10-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org VDUSE will close the file descriptor after having mapped the shared memory, so it will not be possible to get the page size afterwards. This patch adds an new page_shift field to the IOTLB entry, so that the information will be passed at IOTLB cache insertion time. The information is stored as a bit shift value so that IOTLB entry keeps fitting in a single cacheline. Signed-off-by: Maxime Coquelin --- lib/vhost/iotlb.c | 46 ++++++++++++++++++++---------------------- lib/vhost/iotlb.h | 2 +- lib/vhost/vhost.h | 1 - lib/vhost/vhost_user.c | 8 +++++--- 4 files changed, 28 insertions(+), 29 deletions(-) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index 14d143366b..a23008909f 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -19,14 +19,14 @@ struct vhost_iotlb_entry { uint64_t uaddr; uint64_t uoffset; uint64_t size; + uint8_t page_shift; uint8_t perm; }; #define IOTLB_CACHE_SIZE 2048 static bool -vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entry *b, - uint64_t align) +vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entry *b) { uint64_t a_start, a_end, b_start; @@ -38,44 +38,41 @@ vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entr /* Assumes entry a lower than entry b */ RTE_ASSERT(a_start < b_start); - a_end = RTE_ALIGN_CEIL(a_start + a->size, align); - b_start = RTE_ALIGN_FLOOR(b_start, align); + a_end = RTE_ALIGN_CEIL(a_start + a->size, RTE_BIT64(a->page_shift)); + b_start = RTE_ALIGN_FLOOR(b_start, RTE_BIT64(b->page_shift)); return a_end > b_start; } static void -vhost_user_iotlb_set_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node) +vhost_user_iotlb_set_dump(struct vhost_iotlb_entry *node) { - uint64_t align, start; + uint64_t start; start = node->uaddr + node->uoffset; - align = hua_to_alignment(dev->mem, (void *)(uintptr_t)start); - - mem_set_dump((void *)(uintptr_t)start, node->size, true, align); + mem_set_dump((void *)(uintptr_t)start, node->size, true, RTE_BIT64(node->page_shift)); } static void -vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *node, +vhost_user_iotlb_clear_dump(struct vhost_iotlb_entry *node, struct vhost_iotlb_entry *prev, struct vhost_iotlb_entry *next) { - uint64_t align, start, end; + uint64_t start, end; start = node->uaddr + node->uoffset; end = start + node->size; - align = hua_to_alignment(dev->mem, (void *)(uintptr_t)start); - /* Skip first page if shared with previous entry. */ - if (vhost_user_iotlb_share_page(prev, node, align)) - start = RTE_ALIGN_CEIL(start, align); + if (vhost_user_iotlb_share_page(prev, node)) + start = RTE_ALIGN_CEIL(start, RTE_BIT64(node->page_shift)); /* Skip last page if shared with next entry. */ - if (vhost_user_iotlb_share_page(node, next, align)) - end = RTE_ALIGN_FLOOR(end, align); + if (vhost_user_iotlb_share_page(node, next)) + end = RTE_ALIGN_FLOOR(end, RTE_BIT64(node->page_shift)); if (end > start) - mem_set_dump((void *)(uintptr_t)start, end - start, false, align); + mem_set_dump((void *)(uintptr_t)start, end - start, false, + RTE_BIT64(node->page_shift)); } static struct vhost_iotlb_entry * @@ -198,7 +195,7 @@ vhost_user_iotlb_cache_remove_all(struct virtio_net *dev) vhost_user_iotlb_wr_lock_all(dev); RTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) { - vhost_user_iotlb_set_dump(dev, node); + vhost_user_iotlb_set_dump(node); TAILQ_REMOVE(&dev->iotlb_list, node, next); vhost_user_iotlb_pool_put(dev, node); @@ -223,7 +220,7 @@ vhost_user_iotlb_cache_random_evict(struct virtio_net *dev) if (!entry_idx) { struct vhost_iotlb_entry *next_node = RTE_TAILQ_NEXT(node, next); - vhost_user_iotlb_clear_dump(dev, node, prev_node, next_node); + vhost_user_iotlb_clear_dump(node, prev_node, next_node); TAILQ_REMOVE(&dev->iotlb_list, node, next); vhost_user_iotlb_pool_put(dev, node); @@ -239,7 +236,7 @@ vhost_user_iotlb_cache_random_evict(struct virtio_net *dev) void vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr, - uint64_t uoffset, uint64_t size, uint8_t perm) + uint64_t uoffset, uint64_t size, uint64_t page_size, uint8_t perm) { struct vhost_iotlb_entry *node, *new_node; @@ -263,6 +260,7 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t ua new_node->uaddr = uaddr; new_node->uoffset = uoffset; new_node->size = size; + new_node->page_shift = __builtin_ctzll(page_size); new_node->perm = perm; vhost_user_iotlb_wr_lock_all(dev); @@ -276,7 +274,7 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t ua vhost_user_iotlb_pool_put(dev, new_node); goto unlock; } else if (node->iova > new_node->iova) { - vhost_user_iotlb_set_dump(dev, new_node); + vhost_user_iotlb_set_dump(new_node); TAILQ_INSERT_BEFORE(node, new_node, next); dev->iotlb_cache_nr++; @@ -284,7 +282,7 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t ua } } - vhost_user_iotlb_set_dump(dev, new_node); + vhost_user_iotlb_set_dump(new_node); TAILQ_INSERT_TAIL(&dev->iotlb_list, new_node, next); dev->iotlb_cache_nr++; @@ -313,7 +311,7 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t si if (iova < node->iova + node->size) { struct vhost_iotlb_entry *next_node = RTE_TAILQ_NEXT(node, next); - vhost_user_iotlb_clear_dump(dev, node, prev_node, next_node); + vhost_user_iotlb_clear_dump(node, prev_node, next_node); TAILQ_REMOVE(&dev->iotlb_list, node, next); vhost_user_iotlb_pool_put(dev, node); diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h index bee36c5903..81ca04df21 100644 --- a/lib/vhost/iotlb.h +++ b/lib/vhost/iotlb.h @@ -58,7 +58,7 @@ vhost_user_iotlb_wr_unlock_all(struct virtio_net *dev) } void vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr, - uint64_t uoffset, uint64_t size, uint8_t perm); + uint64_t uoffset, uint64_t size, uint64_t page_size, uint8_t perm); void vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t size); uint64_t vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova, uint64_t *size, uint8_t perm); diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 67cc4a2fdb..4ace5ab081 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -1016,6 +1016,5 @@ mbuf_is_consumed(struct rte_mbuf *m) return true; } -uint64_t hua_to_alignment(struct rte_vhost_memory *mem, void *ptr); void mem_set_dump(void *ptr, size_t size, bool enable, uint64_t alignment); #endif /* _VHOST_NET_CDEV_H_ */ diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 222ccbf819..11b265c1ba 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -743,7 +743,7 @@ log_addr_to_gpa(struct virtio_net *dev, struct vhost_virtqueue *vq) return log_gpa; } -uint64_t +static uint64_t hua_to_alignment(struct rte_vhost_memory *mem, void *ptr) { struct rte_vhost_mem_region *r; @@ -2632,7 +2632,7 @@ vhost_user_iotlb_msg(struct virtio_net **pdev, struct virtio_net *dev = *pdev; struct vhost_iotlb_msg *imsg = &ctx->msg.payload.iotlb; uint16_t i; - uint64_t vva, len; + uint64_t vva, len, pg_sz; switch (imsg->type) { case VHOST_IOTLB_UPDATE: @@ -2641,7 +2641,9 @@ vhost_user_iotlb_msg(struct virtio_net **pdev, if (!vva) return RTE_VHOST_MSG_RESULT_ERR; - vhost_user_iotlb_cache_insert(dev, imsg->iova, vva, 0, len, imsg->perm); + pg_sz = hua_to_alignment(dev->mem, (void *)(uintptr_t)vva); + + vhost_user_iotlb_cache_insert(dev, imsg->iova, vva, 0, len, pg_sz, imsg->perm); for (i = 0; i < dev->nr_vring; i++) { struct vhost_virtqueue *vq = dev->virtqueue[i]; From patchwork Thu May 25 16:15:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127449 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9B8DE42B9D; Thu, 25 May 2023 18:17:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6785E42D48; Thu, 25 May 2023 18:16:31 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 06FDB42D3F for ; Thu, 25 May 2023 18:16:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031389; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tXiJeF5mzGiVVFEgPVYt9RbCHIhRdJ6LiGjeUyYofFg=; b=H3IAOqjTiL3uB021Xzgr/u6xe0PyZOyBaXiv4o8RvpTAbt07KS/kt2H61mzNAtDwS5CNWN pGVpsH0aRid7fRDE/ylLVwTg8A7TNOy+r6ONxeovczkvmaAcJXfs38FR14vlpV8rrMYA8H 7A7sEnNnreldAAgE4bSvZdTswYInW4o= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-114-9MIK4a8pMCeFP-tLIhYNqw-1; Thu, 25 May 2023 12:16:26 -0400 X-MC-Unique: 9MIK4a8pMCeFP-tLIhYNqw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 11D793C13506; Thu, 25 May 2023 16:16:26 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 81A3AC154D1; Thu, 25 May 2023 16:16:23 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 10/28] vhost: retry translating IOVA after IOTLB miss Date: Thu, 25 May 2023 18:15:32 +0200 Message-Id: <20230525161551.70002-11-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Vhost-user backend IOTLB misses and updates are asynchronous, so IOVA address translation function just fails after having sent an IOTLB miss update if needed entry was not in the IOTLB cache. This is not the case for VDUSE, for which the needed IOTLB update is returned directly when sending an IOTLB miss. This patch retry again finding the needed entry in the IOTLB cache after having sent an IOTLB miss. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index d35075b96c..4f16307e4d 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -96,6 +96,12 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq, vhost_user_iotlb_rd_lock(vq); } + tmp_size = *size; + /* Retry in case of VDUSE, as it is synchronous */ + vva = vhost_user_iotlb_cache_find(dev, iova, &tmp_size, perm); + if (tmp_size == *size) + return vva; + return 0; } From patchwork Thu May 25 16:15:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127450 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0ADA142B9D; Thu, 25 May 2023 18:17:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 840CE42D8B; Thu, 25 May 2023 18:16:32 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 58ACC42D3F for ; Thu, 25 May 2023 18:16:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031390; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MYi8SuNYB5W0FggCVE13lfadWUIWXGxk61DFoAMO/AM=; b=ciFNmCvr/Y9LCxjBPhMcO5n/rPzSBy99fLSzYdMPqbDX8iMT6JL1OD7rBqSAoiIKDIZPqB 9nFKtVMCt7uptDbcoNUYRBeG7fc+BC7oWuWKWUpyjnx6OZH+C44kF83MkRj78j3xXInoBA 6tEnWRtyPx1CoG5afFpJ2YU9UWbN4u4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-571-Hh41n_t8NXaDw6vOzMmMeg-1; Thu, 25 May 2023 12:16:29 -0400 X-MC-Unique: Hh41n_t8NXaDw6vOzMmMeg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EAF0485A5A8; Thu, 25 May 2023 16:16:28 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 62354C154D2; Thu, 25 May 2023 16:16:26 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 11/28] vhost: introduce backend ops Date: Thu, 25 May 2023 18:15:33 +0200 Message-Id: <20230525161551.70002-12-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch introduces backend ops struct, that will enable calling backend specifics callbacks (Vhost-user, VDUSE), in shared code. This is an empty shell for now, it will be filled in later patches. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/socket.c | 2 +- lib/vhost/vhost.c | 8 +++++++- lib/vhost/vhost.h | 10 +++++++++- lib/vhost/vhost_user.c | 8 ++++++++ lib/vhost/vhost_user.h | 1 + 5 files changed, 26 insertions(+), 3 deletions(-) diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index 669c322e12..ba54263824 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -221,7 +221,7 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) return; } - vid = vhost_new_device(); + vid = vhost_user_new_device(); if (vid == -1) { goto err; } diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index 4f16307e4d..41f212315e 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -676,11 +676,16 @@ reset_device(struct virtio_net *dev) * there is a new virtio device being attached). */ int -vhost_new_device(void) +vhost_new_device(struct vhost_backend_ops *ops) { struct virtio_net *dev; int i; + if (ops == NULL) { + VHOST_LOG_CONFIG("device", ERR, "missing backend ops.\n"); + return -1; + } + pthread_mutex_lock(&vhost_dev_lock); for (i = 0; i < RTE_MAX_VHOST_DEVICE; i++) { if (vhost_devices[i] == NULL) @@ -708,6 +713,7 @@ vhost_new_device(void) dev->backend_req_fd = -1; dev->postcopy_ufd = -1; rte_spinlock_init(&dev->backend_req_lock); + dev->backend_ops = ops; return i; } diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 4ace5ab081..cc5c707205 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -89,6 +89,12 @@ for (iter = val; iter < num; iter++) #endif +/** + * Structure that contains backend-specific ops. + */ +struct vhost_backend_ops { +}; + /** * Structure contains buffer address, length and descriptor index * from vring to do scatter RX. @@ -513,6 +519,8 @@ struct virtio_net { void *extern_data; /* pre and post vhost user message handlers for the device */ struct rte_vhost_user_extern_ops extern_ops; + + struct vhost_backend_ops *backend_ops; } __rte_cache_aligned; static inline void @@ -812,7 +820,7 @@ get_device(int vid) return dev; } -int vhost_new_device(void); +int vhost_new_device(struct vhost_backend_ops *ops); void cleanup_device(struct virtio_net *dev, int destroy); void reset_device(struct virtio_net *dev); void vhost_destroy_device(int); diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 11b265c1ba..7655082c4b 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3464,3 +3464,11 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) return ret; } + +static struct vhost_backend_ops vhost_user_backend_ops; + +int +vhost_user_new_device(void) +{ + return vhost_new_device(&vhost_user_backend_ops); +} diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index a0987a58f9..61456049c8 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -185,5 +185,6 @@ int vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm); int read_fd_message(char *ifname, int sockfd, char *buf, int buflen, int *fds, int max_fds, int *fd_num); int send_fd_message(char *ifname, int sockfd, char *buf, int buflen, int *fds, int fd_num); +int vhost_user_new_device(void); #endif From patchwork Thu May 25 16:15:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127452 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4014942B9D; Thu, 25 May 2023 18:17:42 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BA97242D46; Thu, 25 May 2023 18:16:39 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 787C342D38 for ; Thu, 25 May 2023 18:16:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031397; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d9CWwzgpw5ZKeUXnBGcoNvCEpEuSEOQoC3TIcinzHiM=; b=XMEjIU5CsYzAXaKOQ7J9/MQmznRcQuVosL0lItgQu8NnEhHr5hVy9g8HjsL5kSDOWi4Sj5 HUf+TsaKELmbrTrJFoSNVSPz10GDUropYmyxotCjoyABurWwGtDj8HiZWnwzCptRtxPonb aRCX4hqe1VXXMtH7emAHZwNz8DX3QmM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-288-b8_rEwaXMGifGh0Zv8h1yA-1; Thu, 25 May 2023 12:16:32 -0400 X-MC-Unique: b8_rEwaXMGifGh0Zv8h1yA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B1434101A53A; Thu, 25 May 2023 16:16:31 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3C495C154D1; Thu, 25 May 2023 16:16:29 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 12/28] vhost: add IOTLB cache entry removal callback Date: Thu, 25 May 2023 18:15:34 +0200 Message-Id: <20230525161551.70002-13-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org VDUSE will need to munmap() the IOTLB entry on removal from the cache, as it performs mmap() before insertion. This patch introduces a callback that VDUSE layer will implement to achieve this. Signed-off-by: Maxime Coquelin --- lib/vhost/iotlb.c | 12 ++++++++++++ lib/vhost/vhost.h | 3 +++ 2 files changed, 15 insertions(+) diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c index a23008909f..6dca0ba7d0 100644 --- a/lib/vhost/iotlb.c +++ b/lib/vhost/iotlb.c @@ -25,6 +25,15 @@ struct vhost_iotlb_entry { #define IOTLB_CACHE_SIZE 2048 +static void +vhost_user_iotlb_remove_notify(struct virtio_net *dev, struct vhost_iotlb_entry *entry) +{ + if (dev->backend_ops->iotlb_remove_notify == NULL) + return; + + dev->backend_ops->iotlb_remove_notify(entry->uaddr, entry->uoffset, entry->size); +} + static bool vhost_user_iotlb_share_page(struct vhost_iotlb_entry *a, struct vhost_iotlb_entry *b) { @@ -198,6 +207,7 @@ vhost_user_iotlb_cache_remove_all(struct virtio_net *dev) vhost_user_iotlb_set_dump(node); TAILQ_REMOVE(&dev->iotlb_list, node, next); + vhost_user_iotlb_remove_notify(dev, node); vhost_user_iotlb_pool_put(dev, node); } @@ -223,6 +233,7 @@ vhost_user_iotlb_cache_random_evict(struct virtio_net *dev) vhost_user_iotlb_clear_dump(node, prev_node, next_node); TAILQ_REMOVE(&dev->iotlb_list, node, next); + vhost_user_iotlb_remove_notify(dev, node); vhost_user_iotlb_pool_put(dev, node); dev->iotlb_cache_nr--; break; @@ -314,6 +325,7 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t si vhost_user_iotlb_clear_dump(node, prev_node, next_node); TAILQ_REMOVE(&dev->iotlb_list, node, next); + vhost_user_iotlb_remove_notify(dev, node); vhost_user_iotlb_pool_put(dev, node); dev->iotlb_cache_nr--; } else { diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index cc5c707205..f37e0b83b8 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -89,10 +89,13 @@ for (iter = val; iter < num; iter++) #endif +typedef void (*vhost_iotlb_remove_notify)(uint64_t addr, uint64_t off, uint64_t size); + /** * Structure that contains backend-specific ops. */ struct vhost_backend_ops { + vhost_iotlb_remove_notify iotlb_remove_notify; }; /** From patchwork Thu May 25 16:15:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127451 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DDBE842B9D; Thu, 25 May 2023 18:17:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 98F6C4114B; Thu, 25 May 2023 18:16:38 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 2E78342D93 for ; Thu, 25 May 2023 18:16:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4lCj+YpzHo1VLWHA8dmAAygLOjFMUAOVyYKHSxcuLFk=; b=XCn6v2ygpSjvJ8S9J4aX2omXcs5sEzDn1e9X7TUHb26vXwa8VP2b1HUwdfTktkHqfJWmp/ 8jX8NF978k+6arFBuVwv7IlbUQf/A1tcQdSy1ntJZ9+++vrWUrpv76dxP5/lhKEm8cvYFh EpLsQeHEBBGXornOKvI2V/6zxDk7sho= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-319-UIY_2gAJMQODfzplOolTyg-1; Thu, 25 May 2023 12:16:35 -0400 X-MC-Unique: UIY_2gAJMQODfzplOolTyg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 816752999B34; Thu, 25 May 2023 16:16:34 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 02F7DC154D1; Thu, 25 May 2023 16:16:31 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 13/28] vhost: add helper for IOTLB misses Date: Thu, 25 May 2023 18:15:35 +0200 Message-Id: <20230525161551.70002-14-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds a helper for sending IOTLB misses as VDUSE will use an ioctl while Vhost-user use a dedicated Vhost-user backend request. Signed-off-by: Maxime Coquelin --- lib/vhost/vhost.c | 13 ++++++++++++- lib/vhost/vhost.h | 4 ++++ lib/vhost/vhost_user.c | 6 ++++-- lib/vhost/vhost_user.h | 1 - 4 files changed, 20 insertions(+), 4 deletions(-) diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index 41f212315e..790eb06b28 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -52,6 +52,12 @@ static const struct vhost_vq_stats_name_off vhost_vq_stat_strings[] = { #define VHOST_NB_VQ_STATS RTE_DIM(vhost_vq_stat_strings) +static int +vhost_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm) +{ + return dev->backend_ops->iotlb_miss(dev, iova, perm); +} + uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq, uint64_t iova, uint64_t *size, uint8_t perm) @@ -86,7 +92,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq, vhost_user_iotlb_rd_unlock(vq); vhost_user_iotlb_pending_insert(dev, iova, perm); - if (vhost_user_iotlb_miss(dev, iova, perm)) { + if (vhost_iotlb_miss(dev, iova, perm)) { VHOST_LOG_DATA(dev->ifname, ERR, "IOTLB miss req failed for IOVA 0x%" PRIx64 "\n", iova); @@ -686,6 +692,11 @@ vhost_new_device(struct vhost_backend_ops *ops) return -1; } + if (ops->iotlb_miss == NULL) { + VHOST_LOG_CONFIG("device", ERR, "missing IOTLB miss backend op.\n"); + return -1; + } + pthread_mutex_lock(&vhost_dev_lock); for (i = 0; i < RTE_MAX_VHOST_DEVICE; i++) { if (vhost_devices[i] == NULL) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index f37e0b83b8..ee7640e901 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -89,13 +89,17 @@ for (iter = val; iter < num; iter++) #endif +struct virtio_net; typedef void (*vhost_iotlb_remove_notify)(uint64_t addr, uint64_t off, uint64_t size); +typedef int (*vhost_iotlb_miss_cb)(struct virtio_net *dev, uint64_t iova, uint8_t perm); + /** * Structure that contains backend-specific ops. */ struct vhost_backend_ops { vhost_iotlb_remove_notify iotlb_remove_notify; + vhost_iotlb_miss_cb iotlb_miss; }; /** diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 7655082c4b..972559a2b5 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3305,7 +3305,7 @@ vhost_user_msg_handler(int vid, int fd) return ret; } -int +static int vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm) { int ret; @@ -3465,7 +3465,9 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) return ret; } -static struct vhost_backend_ops vhost_user_backend_ops; +static struct vhost_backend_ops vhost_user_backend_ops = { + .iotlb_miss = vhost_user_iotlb_miss, +}; int vhost_user_new_device(void) diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index 61456049c8..1ffeca92f3 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -179,7 +179,6 @@ struct __rte_packed vhu_msg_context { /* vhost_user.c */ int vhost_user_msg_handler(int vid, int fd); -int vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm); /* socket.c */ int read_fd_message(char *ifname, int sockfd, char *buf, int buflen, int *fds, int max_fds, From patchwork Thu May 25 16:15:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127455 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7EC7F42B9D; Thu, 25 May 2023 18:18:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5327842DAA; Thu, 25 May 2023 18:16:49 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id F017B42D94 for ; Thu, 25 May 2023 18:16:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DDS6ctomOODlHarhe/GqAOXaMXknHLXO7EDtrMhO6RY=; b=KOKjHS/SlqWT2h5fbLn+8W2sUk74BccvB3+UDop1mDJoWnz0RVqE76Ew15QCiZvs3MQq0f 5OHwbnL8M7pjm84Zp9+6i+0DgLf7DGvtxmbPponpFQJqPjFgNt6M8zNDwsPagYP4N3mUBx ul0USi5PpQ4/nBmG+ikoPGWg6qmuwLA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-648-5jUu-n7XMu-ceqUNKRe9fA-1; Thu, 25 May 2023 12:16:44 -0400 X-MC-Unique: 5jUu-n7XMu-ceqUNKRe9fA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4F7593802143; Thu, 25 May 2023 16:16:37 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id C791AC154D1; Thu, 25 May 2023 16:16:34 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 14/28] vhost: add helper for interrupt injection Date: Thu, 25 May 2023 18:15:36 +0200 Message-Id: <20230525161551.70002-15-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Vhost-user uses eventfd to inject IRQs, but VDUSE uses an ioctl. This patch prepares vhost_vring_call_split() and vhost_vring_call_packed() to support VDUSE by introducing a new helper. It also adds a new counter for guest notification failures, which could happen in case of uninitialized call file descriptor for example. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost.c | 6 +++++ lib/vhost/vhost.h | 54 +++++++++++++++++++++++------------------- lib/vhost/vhost_user.c | 10 ++++++++ 3 files changed, 46 insertions(+), 24 deletions(-) diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index 790eb06b28..c07028f2b3 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -44,6 +44,7 @@ static const struct vhost_vq_stats_name_off vhost_vq_stat_strings[] = { {"size_1024_1518_packets", offsetof(struct vhost_virtqueue, stats.size_bins[6])}, {"size_1519_max_packets", offsetof(struct vhost_virtqueue, stats.size_bins[7])}, {"guest_notifications", offsetof(struct vhost_virtqueue, stats.guest_notifications)}, + {"guest_notifications_error", offsetof(struct vhost_virtqueue, stats.guest_notifications_error)}, {"iotlb_hits", offsetof(struct vhost_virtqueue, stats.iotlb_hits)}, {"iotlb_misses", offsetof(struct vhost_virtqueue, stats.iotlb_misses)}, {"inflight_submitted", offsetof(struct vhost_virtqueue, stats.inflight_submitted)}, @@ -697,6 +698,11 @@ vhost_new_device(struct vhost_backend_ops *ops) return -1; } + if (ops->inject_irq == NULL) { + VHOST_LOG_CONFIG("device", ERR, "missing IRQ injection backend op.\n"); + return -1; + } + pthread_mutex_lock(&vhost_dev_lock); for (i = 0; i < RTE_MAX_VHOST_DEVICE; i++) { if (vhost_devices[i] == NULL) diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index ee7640e901..8f0875b4e2 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -90,16 +90,20 @@ #endif struct virtio_net; +struct vhost_virtqueue; + typedef void (*vhost_iotlb_remove_notify)(uint64_t addr, uint64_t off, uint64_t size); typedef int (*vhost_iotlb_miss_cb)(struct virtio_net *dev, uint64_t iova, uint8_t perm); +typedef int (*vhost_vring_inject_irq_cb)(struct virtio_net *dev, struct vhost_virtqueue *vq); /** * Structure that contains backend-specific ops. */ struct vhost_backend_ops { vhost_iotlb_remove_notify iotlb_remove_notify; vhost_iotlb_miss_cb iotlb_miss; + vhost_vring_inject_irq_cb inject_irq; }; /** @@ -149,6 +153,7 @@ struct virtqueue_stats { /* Size bins in array as RFC 2819, undersized [0], 64 [1], etc */ uint64_t size_bins[8]; uint64_t guest_notifications; + uint64_t guest_notifications_error; uint64_t iotlb_hits; uint64_t iotlb_misses; uint64_t inflight_submitted; @@ -900,6 +905,24 @@ vhost_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old) return (uint16_t)(new_idx - event_idx - 1) < (uint16_t)(new_idx - old); } +static __rte_always_inline void +vhost_vring_inject_irq(struct virtio_net *dev, struct vhost_virtqueue *vq) +{ + int ret; + + ret = dev->backend_ops->inject_irq(dev, vq); + if (ret) { + if (dev->flags & VIRTIO_DEV_STATS_ENABLED) + vq->stats.guest_notifications_error++; + return; + } + + if (dev->flags & VIRTIO_DEV_STATS_ENABLED) + vq->stats.guest_notifications++; + if (dev->notify_ops->guest_notified) + dev->notify_ops->guest_notified(dev->vid); +} + static __rte_always_inline void vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq) { @@ -919,25 +942,13 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq) "%s: used_event_idx=%d, old=%d, new=%d\n", __func__, vhost_used_event(vq), old, new); - if ((vhost_need_event(vhost_used_event(vq), new, old) || - unlikely(!signalled_used_valid)) && - vq->callfd >= 0) { - eventfd_write(vq->callfd, (eventfd_t) 1); - if (dev->flags & VIRTIO_DEV_STATS_ENABLED) - vq->stats.guest_notifications++; - if (dev->notify_ops->guest_notified) - dev->notify_ops->guest_notified(dev->vid); - } + if (vhost_need_event(vhost_used_event(vq), new, old) || + unlikely(!signalled_used_valid)) + vhost_vring_inject_irq(dev, vq); } else { /* Kick the guest if necessary. */ - if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT) - && (vq->callfd >= 0)) { - eventfd_write(vq->callfd, (eventfd_t)1); - if (dev->flags & VIRTIO_DEV_STATS_ENABLED) - vq->stats.guest_notifications++; - if (dev->notify_ops->guest_notified) - dev->notify_ops->guest_notified(dev->vid); - } + if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)) + vhost_vring_inject_irq(dev, vq); } } @@ -988,13 +999,8 @@ vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq) if (vhost_need_event(off, new, old)) kick = true; kick: - if (kick && vq->callfd >= 0) { - eventfd_write(vq->callfd, (eventfd_t)1); - if (dev->flags & VIRTIO_DEV_STATS_ENABLED) - vq->stats.guest_notifications++; - if (dev->notify_ops->guest_notified) - dev->notify_ops->guest_notified(dev->vid); - } + if (kick) + vhost_vring_inject_irq(dev, vq); } static __rte_always_inline void diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 972559a2b5..1362cf16e5 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3465,8 +3465,18 @@ int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable) return ret; } +static int +vhost_user_inject_irq(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq) +{ + if (vq->callfd < 0) + return -1; + + return eventfd_write(vq->callfd, (eventfd_t)1); +} + static struct vhost_backend_ops vhost_user_backend_ops = { .iotlb_miss = vhost_user_iotlb_miss, + .inject_irq = vhost_user_inject_irq, }; int From patchwork Thu May 25 16:15:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127453 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0CFAB42B9D; Thu, 25 May 2023 18:17:51 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 25499427EE; Thu, 25 May 2023 18:16:44 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 386F3427EE for ; Thu, 25 May 2023 18:16:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031401; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K32mk7gApwAlvzb+vadMAjtMjjh55AszdlfdxKSofKU=; b=dH84FKqcX2SM9RZJKIDHLCLl0MZNcHiqv7/2ZZFAlOOFzVxPXQztlTjirJdMfI10TpQXcj +snrvyuFg1hqlOjgXzy0I5TRdw+KCY8q79fps5OedKGqHm1etOp9pwp+G0q3go0Kt+4C+r I23EEfxiNoRm2xsUgsu95kOLrewvKvs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-445-NYtV5Wq_NJ-jCC1KEOlj0A-1; Thu, 25 May 2023 12:16:40 -0400 X-MC-Unique: NYtV5Wq_NJ-jCC1KEOlj0A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1ED0385A5BD; Thu, 25 May 2023 16:16:40 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 95DC1C154D1; Thu, 25 May 2023 16:16:37 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 15/28] vhost: add API to set max queue pairs Date: Thu, 25 May 2023 18:15:37 +0200 Message-Id: <20230525161551.70002-16-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch introduces a new rte_vhost_driver_set_max_queues API as preliminary work for multiqueue support with VDUSE. Indeed, with VDUSE we need to pre-allocate the vrings at device creation time, so we need such API not to allocate the 128 queue pairs supported by the Vhost library. Calling the API is optional, 128 queue pairs remaining the default. Signed-off-by: Maxime Coquelin --- doc/guides/prog_guide/vhost_lib.rst | 4 +++ doc/guides/rel_notes/release_23_07.rst | 5 ++++ lib/vhost/rte_vhost.h | 17 ++++++++++++ lib/vhost/socket.c | 36 ++++++++++++++++++++++++-- lib/vhost/version.map | 3 +++ 5 files changed, 63 insertions(+), 2 deletions(-) diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst index e8bb8c9b7b..cd4b109139 100644 --- a/doc/guides/prog_guide/vhost_lib.rst +++ b/doc/guides/prog_guide/vhost_lib.rst @@ -334,6 +334,10 @@ The following is an overview of some key Vhost API functions: Clean DMA vChannel finished to use. After this function is called, the specified DMA vChannel should no longer be used by the Vhost library. +* ``rte_vhost_driver_set_max_queue_num(path, max_queue_pairs)`` + + Set the maximum number of queue pairs supported by the device. + Vhost-user Implementations -------------------------- diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst index a9b1293689..fa889a5ee7 100644 --- a/doc/guides/rel_notes/release_23_07.rst +++ b/doc/guides/rel_notes/release_23_07.rst @@ -55,6 +55,11 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added Vhost API to set maximum queue pairs supported + + Introduced ``rte_vhost_driver_set_max_queue_num()`` to be able to limit the + maximum number of supported queue pairs, required for VDUSE support. + Removed Items ------------- diff --git a/lib/vhost/rte_vhost.h b/lib/vhost/rte_vhost.h index 58a5d4be92..44cbfcb469 100644 --- a/lib/vhost/rte_vhost.h +++ b/lib/vhost/rte_vhost.h @@ -588,6 +588,23 @@ rte_vhost_driver_get_protocol_features(const char *path, int rte_vhost_driver_get_queue_num(const char *path, uint32_t *queue_num); +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice. + * + * Set the maximum number of queue pairs supported by the device. + * + * @param path + * The vhost-user socket file path + * @param max_queue_pairs + * The maximum number of queue pairs + * @return + * 0 on success, -1 on failure + */ +__rte_experimental +int +rte_vhost_driver_set_max_queue_num(const char *path, uint32_t max_queue_pairs); + /** * Get the feature bits after negotiation * diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index ba54263824..e95c3ffeac 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -56,6 +56,8 @@ struct vhost_user_socket { uint64_t protocol_features; + uint32_t max_queue_pairs; + struct rte_vdpa_device *vdpa_dev; struct rte_vhost_device_ops const *notify_ops; @@ -821,7 +823,7 @@ rte_vhost_driver_get_queue_num(const char *path, uint32_t *queue_num) vdpa_dev = vsocket->vdpa_dev; if (!vdpa_dev) { - *queue_num = VHOST_MAX_QUEUE_PAIRS; + *queue_num = vsocket->max_queue_pairs; goto unlock_exit; } @@ -831,7 +833,36 @@ rte_vhost_driver_get_queue_num(const char *path, uint32_t *queue_num) goto unlock_exit; } - *queue_num = RTE_MIN((uint32_t)VHOST_MAX_QUEUE_PAIRS, vdpa_queue_num); + *queue_num = RTE_MIN(vsocket->max_queue_pairs, vdpa_queue_num); + +unlock_exit: + pthread_mutex_unlock(&vhost_user.mutex); + return ret; +} + +int +rte_vhost_driver_set_max_queue_num(const char *path, uint32_t max_queue_pairs) +{ + struct vhost_user_socket *vsocket; + int ret = 0; + + VHOST_LOG_CONFIG(path, INFO, "Setting max queue pairs to %u\n", max_queue_pairs); + + if (max_queue_pairs > VHOST_MAX_QUEUE_PAIRS) { + VHOST_LOG_CONFIG(path, ERR, "Library only supports up to %u queue pairs\n", + VHOST_MAX_QUEUE_PAIRS); + return -1; + } + + pthread_mutex_lock(&vhost_user.mutex); + vsocket = find_vhost_user_socket(path); + if (!vsocket) { + VHOST_LOG_CONFIG(path, ERR, "socket file is not registered yet.\n"); + ret = -1; + goto unlock_exit; + } + + vsocket->max_queue_pairs = max_queue_pairs; unlock_exit: pthread_mutex_unlock(&vhost_user.mutex); @@ -890,6 +921,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags) goto out_free; } vsocket->vdpa_dev = NULL; + vsocket->max_queue_pairs = VHOST_MAX_QUEUE_PAIRS; vsocket->extbuf = flags & RTE_VHOST_USER_EXTBUF_SUPPORT; vsocket->linearbuf = flags & RTE_VHOST_USER_LINEARBUF_SUPPORT; vsocket->async_copy = flags & RTE_VHOST_USER_ASYNC_COPY; diff --git a/lib/vhost/version.map b/lib/vhost/version.map index d322a4a888..dffb126aa8 100644 --- a/lib/vhost/version.map +++ b/lib/vhost/version.map @@ -98,6 +98,9 @@ EXPERIMENTAL { # added in 22.11 rte_vhost_async_dma_unconfigure; rte_vhost_vring_call_nonblock; + + # added in 23.07 + rte_vhost_driver_set_max_queue_num; }; INTERNAL { From patchwork Thu May 25 16:15:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127454 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 24CB042B9D; Thu, 25 May 2023 18:17:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 301CD42D9B; Thu, 25 May 2023 18:16:48 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id E211A42D82 for ; Thu, 25 May 2023 18:16:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031406; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HKy6hSXmK5VsHBV/QWhIAznNiUc9xtI8d5uBiU8cXMs=; b=c+xa4WgA01DFPwWn+qXHkQPUCz9T8X26fEZX2fE6XA+tmZgvBKzc1QEn+a64w578Acht7H D+KqvH+tQstOj9Bt1LUIz1WohdgZ2hb7Db+6PwBty8ATykhpQYM9MO+Nd/KNZwx7E6GGYu hsz9hnr3Y9DEjzfjhlo5CKiEgBCY0QQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-114-8fhXxRZ5NgCTKBYOjiJcRA-1; Thu, 25 May 2023 12:16:43 -0400 X-MC-Unique: 8fhXxRZ5NgCTKBYOjiJcRA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D69B785A5BE; Thu, 25 May 2023 16:16:42 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 64925C154D1; Thu, 25 May 2023 16:16:40 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 16/28] net/vhost: use API to set max queue pairs Date: Thu, 25 May 2023 18:15:38 +0200 Message-Id: <20230525161551.70002-17-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In order to support multiqueue with VDUSE, we need to be able to limit the maximum number of queue pairs, to avoid unnecessary memory consumption since the maximum number of queue pairs need to be allocated at device creation time, as opposed to Vhost-user which allocate only when the frontend initialize them. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- drivers/net/vhost/rte_eth_vhost.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c index 62ef955ebc..8d37ec9775 100644 --- a/drivers/net/vhost/rte_eth_vhost.c +++ b/drivers/net/vhost/rte_eth_vhost.c @@ -1013,6 +1013,9 @@ vhost_driver_setup(struct rte_eth_dev *eth_dev) goto drv_unreg; } + if (rte_vhost_driver_set_max_queue_num(internal->iface_name, internal->max_queues)) + goto drv_unreg; + if (rte_vhost_driver_callback_register(internal->iface_name, &vhost_ops) < 0) { VHOST_LOG(ERR, "Can't register callbacks\n"); From patchwork Thu May 25 16:15:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127456 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0793442B9D; Thu, 25 May 2023 18:18:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 76BF142DAF; Thu, 25 May 2023 18:16:50 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 25C0F42B71 for ; Thu, 25 May 2023 18:16:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031407; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WKWaGmvhABJCZWkpMNjtZbLe8hk/zxK8w9uil+4XUOw=; b=JSHmTCxwgzdsjICH6m1Ls8kCrO4CDGiTdeV7YCWA+4t4R09k9wSz1ndBPREPUfZ1c+WJlA Wtz+PBFz+3TCSwcuoiLHX05hdXwnJ5E8x9DbZGW7TzCfa1FjZ4+y+A14lX970xRM5YOG0H Om1koqaUqYticDdg0WuXhFlMi4nVWOw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-436-rywJVzjtNR-1NHRe8GHNcA-1; Thu, 25 May 2023 12:16:46 -0400 X-MC-Unique: rywJVzjtNR-1NHRe8GHNcA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DFD31185A797; Thu, 25 May 2023 16:16:45 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 33746C154D1; Thu, 25 May 2023 16:16:43 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 17/28] vhost: add control virtqueue support Date: Thu, 25 May 2023 18:15:39 +0200 Message-Id: <20230525161551.70002-18-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In order to support multi-queue with VDUSE, having control queue support is required. This patch adds control queue implementation, it will be used later when adding VDUSE support. Only split ring layout is supported for now, packed ring support will be added later. Signed-off-by: Maxime Coquelin --- lib/vhost/meson.build | 1 + lib/vhost/vhost.h | 2 + lib/vhost/virtio_net_ctrl.c | 286 ++++++++++++++++++++++++++++++++++++ lib/vhost/virtio_net_ctrl.h | 10 ++ 4 files changed, 299 insertions(+) create mode 100644 lib/vhost/virtio_net_ctrl.c create mode 100644 lib/vhost/virtio_net_ctrl.h diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build index 0d1abf6283..83c8482c9e 100644 --- a/lib/vhost/meson.build +++ b/lib/vhost/meson.build @@ -27,6 +27,7 @@ sources = files( 'vhost_crypto.c', 'vhost_user.c', 'virtio_net.c', + 'virtio_net_ctrl.c', ) headers = files( 'rte_vdpa.h', diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 8f0875b4e2..76663aed24 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -525,6 +525,8 @@ struct virtio_net { int postcopy_ufd; int postcopy_listening; + struct vhost_virtqueue *cvq; + struct rte_vdpa_device *vdpa_dev; /* context data for the external message handlers */ diff --git a/lib/vhost/virtio_net_ctrl.c b/lib/vhost/virtio_net_ctrl.c new file mode 100644 index 0000000000..f4b8d5f7cc --- /dev/null +++ b/lib/vhost/virtio_net_ctrl.c @@ -0,0 +1,286 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Red Hat, Inc. + */ + +#include +#include +#include + +#include "iotlb.h" +#include "vhost.h" +#include "virtio_net_ctrl.h" + +struct virtio_net_ctrl { + uint8_t class; + uint8_t command; + uint8_t command_data[]; +}; + +struct virtio_net_ctrl_elem { + struct virtio_net_ctrl *ctrl_req; + uint16_t head_idx; + uint16_t n_descs; + uint8_t *desc_ack; +}; + +static int +virtio_net_ctrl_pop(struct virtio_net *dev, struct vhost_virtqueue *cvq, + struct virtio_net_ctrl_elem *ctrl_elem) + __rte_shared_locks_required(&cvq->iotlb_lock) +{ + uint16_t avail_idx, desc_idx, n_descs = 0; + uint64_t desc_len, desc_addr, desc_iova, data_len = 0; + uint8_t *ctrl_req; + struct vring_desc *descs; + + avail_idx = __atomic_load_n(&cvq->avail->idx, __ATOMIC_ACQUIRE); + if (avail_idx == cvq->last_avail_idx) { + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "Control queue empty\n"); + return 0; + } + + desc_idx = cvq->avail->ring[cvq->last_avail_idx]; + if (desc_idx >= cvq->size) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Out of range desc index, dropping\n"); + goto err; + } + + ctrl_elem->head_idx = desc_idx; + + if (cvq->desc[desc_idx].flags & VRING_DESC_F_INDIRECT) { + desc_len = cvq->desc[desc_idx].len; + desc_iova = cvq->desc[desc_idx].addr; + + descs = (struct vring_desc *)(uintptr_t)vhost_iova_to_vva(dev, cvq, + desc_iova, &desc_len, VHOST_ACCESS_RO); + if (!descs || desc_len != cvq->desc[desc_idx].len) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to map ctrl indirect descs\n"); + goto err; + } + + desc_idx = 0; + } else { + descs = cvq->desc; + } + + while (1) { + desc_len = descs[desc_idx].len; + desc_iova = descs[desc_idx].addr; + + n_descs++; + + if (descs[desc_idx].flags & VRING_DESC_F_WRITE) { + if (ctrl_elem->desc_ack) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Unexpected ctrl chain layout\n"); + goto err; + } + + if (desc_len != sizeof(uint8_t)) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Invalid ack size for ctrl req, dropping\n"); + goto err; + } + + ctrl_elem->desc_ack = (uint8_t *)(uintptr_t)vhost_iova_to_vva(dev, cvq, + desc_iova, &desc_len, VHOST_ACCESS_WO); + if (!ctrl_elem->desc_ack || desc_len != sizeof(uint8_t)) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Failed to map ctrl ack descriptor\n"); + goto err; + } + } else { + if (ctrl_elem->desc_ack) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Unexpected ctrl chain layout\n"); + goto err; + } + + data_len += desc_len; + } + + if (!(descs[desc_idx].flags & VRING_DESC_F_NEXT)) + break; + + desc_idx = descs[desc_idx].next; + } + + desc_idx = ctrl_elem->head_idx; + + if (cvq->desc[desc_idx].flags & VRING_DESC_F_INDIRECT) + ctrl_elem->n_descs = 1; + else + ctrl_elem->n_descs = n_descs; + + if (!ctrl_elem->desc_ack) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Missing ctrl ack descriptor\n"); + goto err; + } + + if (data_len < sizeof(ctrl_elem->ctrl_req->class) + sizeof(ctrl_elem->ctrl_req->command)) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Invalid control header size\n"); + goto err; + } + + ctrl_elem->ctrl_req = malloc(data_len); + if (!ctrl_elem->ctrl_req) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to alloc ctrl request\n"); + goto err; + } + + ctrl_req = (uint8_t *)ctrl_elem->ctrl_req; + + if (cvq->desc[desc_idx].flags & VRING_DESC_F_INDIRECT) { + desc_len = cvq->desc[desc_idx].len; + desc_iova = cvq->desc[desc_idx].addr; + + descs = (struct vring_desc *)(uintptr_t)vhost_iova_to_vva(dev, cvq, + desc_iova, &desc_len, VHOST_ACCESS_RO); + if (!descs || desc_len != cvq->desc[desc_idx].len) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to map ctrl indirect descs\n"); + goto free_err; + } + + desc_idx = 0; + } else { + descs = cvq->desc; + } + + while (!(descs[desc_idx].flags & VRING_DESC_F_WRITE)) { + desc_len = descs[desc_idx].len; + desc_iova = descs[desc_idx].addr; + + desc_addr = vhost_iova_to_vva(dev, cvq, desc_iova, &desc_len, VHOST_ACCESS_RO); + if (!desc_addr || desc_len < descs[desc_idx].len) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to map ctrl descriptor\n"); + goto free_err; + } + + memcpy(ctrl_req, (void *)(uintptr_t)desc_addr, desc_len); + ctrl_req += desc_len; + + if (!(descs[desc_idx].flags & VRING_DESC_F_NEXT)) + break; + + desc_idx = descs[desc_idx].next; + } + + cvq->last_avail_idx++; + if (cvq->last_avail_idx >= cvq->size) + cvq->last_avail_idx -= cvq->size; + + if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) + vhost_avail_event(cvq) = cvq->last_avail_idx; + + return 1; + +free_err: + free(ctrl_elem->ctrl_req); +err: + cvq->last_avail_idx++; + if (cvq->last_avail_idx >= cvq->size) + cvq->last_avail_idx -= cvq->size; + + if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) + vhost_avail_event(cvq) = cvq->last_avail_idx; + + return -1; +} + +static uint8_t +virtio_net_ctrl_handle_req(struct virtio_net *dev, struct virtio_net_ctrl *ctrl_req) +{ + uint8_t ret = VIRTIO_NET_ERR; + + if (ctrl_req->class == VIRTIO_NET_CTRL_MQ && + ctrl_req->command == VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET) { + uint16_t queue_pairs; + uint32_t i; + + queue_pairs = *(uint16_t *)(uintptr_t)ctrl_req->command_data; + VHOST_LOG_CONFIG(dev->ifname, INFO, "Ctrl req: MQ %u queue pairs\n", queue_pairs); + ret = VIRTIO_NET_OK; + + for (i = 0; i < dev->nr_vring; i++) { + struct vhost_virtqueue *vq = dev->virtqueue[i]; + bool enable; + + if (vq == dev->cvq) + continue; + + if (i < queue_pairs * 2) + enable = true; + else + enable = false; + + vq->enabled = enable; + if (dev->notify_ops->vring_state_changed) + dev->notify_ops->vring_state_changed(dev->vid, i, enable); + } + } + + return ret; +} + +static int +virtio_net_ctrl_push(struct virtio_net *dev, struct virtio_net_ctrl_elem *ctrl_elem) +{ + struct vhost_virtqueue *cvq = dev->cvq; + struct vring_used_elem *used_elem; + + used_elem = &cvq->used->ring[cvq->last_used_idx]; + used_elem->id = ctrl_elem->head_idx; + used_elem->len = ctrl_elem->n_descs; + + cvq->last_used_idx++; + if (cvq->last_used_idx >= cvq->size) + cvq->last_used_idx -= cvq->size; + + __atomic_store_n(&cvq->used->idx, cvq->last_used_idx, __ATOMIC_RELEASE); + + vhost_vring_call_split(dev, dev->cvq); + + free(ctrl_elem->ctrl_req); + + return 0; +} + +int +virtio_net_ctrl_handle(struct virtio_net *dev) +{ + int ret = 0; + + if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Packed ring not supported yet\n"); + return -1; + } + + if (!dev->cvq) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "missing control queue\n"); + return -1; + } + + rte_spinlock_lock(&dev->cvq->access_lock); + vhost_user_iotlb_rd_lock(dev->cvq); + + while (1) { + struct virtio_net_ctrl_elem ctrl_elem; + + memset(&ctrl_elem, 0, sizeof(struct virtio_net_ctrl_elem)); + + ret = virtio_net_ctrl_pop(dev, dev->cvq, &ctrl_elem); + if (ret <= 0) + break; + + *ctrl_elem.desc_ack = virtio_net_ctrl_handle_req(dev, ctrl_elem.ctrl_req); + + ret = virtio_net_ctrl_push(dev, &ctrl_elem); + if (ret < 0) + break; + } + + vhost_user_iotlb_rd_unlock(dev->cvq); + rte_spinlock_unlock(&dev->cvq->access_lock); + + return ret; +} diff --git a/lib/vhost/virtio_net_ctrl.h b/lib/vhost/virtio_net_ctrl.h new file mode 100644 index 0000000000..9a90f4b9da --- /dev/null +++ b/lib/vhost/virtio_net_ctrl.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Red Hat, Inc. + */ + +#ifndef _VIRTIO_NET_CTRL_H +#define _VIRTIO_NET_CTRL_H + +int virtio_net_ctrl_handle(struct virtio_net *dev); + +#endif From patchwork Thu May 25 16:15:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127457 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 042EE42B9D; Thu, 25 May 2023 18:18:18 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EBF2242D44; Thu, 25 May 2023 18:16:54 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 1404C42D44 for ; Thu, 25 May 2023 18:16:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5KxuuND75VeyavOEM34r53sQPcQmaDbybOCq+LZIwV8=; b=aIEBuhgs+J2z20LHmUerHYA/Shd9F3U/8tNpwlQo86LYHJf4SXCxPyxp7Z/QxafjngdEiy XSlW1Kzi4Azrab26pRDOVGoIX7hc4Dgv65dtJo0WlY6JyIh4CsSAo2DsIGClc7ZvzFbH1o Nez1yIMbdw3VAFAX/tYgIH1AbnsT1xM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-470-BEWFTQCYNjOQERWzqLEjGg-1; Thu, 25 May 2023 12:16:49 -0400 X-MC-Unique: BEWFTQCYNjOQERWzqLEjGg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DBAA4101A590; Thu, 25 May 2023 16:16:48 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5DE56C154D1; Thu, 25 May 2023 16:16:46 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 18/28] vhost: add VDUSE device creation and destruction Date: Thu, 25 May 2023 18:15:40 +0200 Message-Id: <20230525161551.70002-19-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds initial support for VDUSE, which includes the device creation and destruction. It does not include the virtqueues configuration, so this is not functionnal at this point. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/meson.build | 4 + lib/vhost/socket.c | 34 ++++--- lib/vhost/vduse.c | 201 ++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vduse.h | 33 +++++++ lib/vhost/vhost.h | 2 + 5 files changed, 262 insertions(+), 12 deletions(-) create mode 100644 lib/vhost/vduse.c create mode 100644 lib/vhost/vduse.h diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build index 83c8482c9e..e63072d7a1 100644 --- a/lib/vhost/meson.build +++ b/lib/vhost/meson.build @@ -29,6 +29,10 @@ sources = files( 'virtio_net.c', 'virtio_net_ctrl.c', ) +if cc.has_header('linux/vduse.h') + sources += files('vduse.c') + cflags += '-DVHOST_HAS_VDUSE' +endif headers = files( 'rte_vdpa.h', 'rte_vhost.h', diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index e95c3ffeac..a8a1c4cd2b 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -18,6 +18,7 @@ #include #include "fd_man.h" +#include "vduse.h" #include "vhost.h" #include "vhost_user.h" @@ -35,6 +36,7 @@ struct vhost_user_socket { int socket_fd; struct sockaddr_un un; bool is_server; + bool is_vduse; bool reconnect; bool iommu_support; bool use_builtin_virtio_net; @@ -992,18 +994,21 @@ rte_vhost_driver_register(const char *path, uint64_t flags) #endif } - if ((flags & RTE_VHOST_USER_CLIENT) != 0) { - vsocket->reconnect = !(flags & RTE_VHOST_USER_NO_RECONNECT); - if (vsocket->reconnect && reconn_tid == 0) { - if (vhost_user_reconnect_init() != 0) - goto out_mutex; - } + if (!strncmp("/dev/vduse/", path, strlen("/dev/vduse/"))) { + vsocket->is_vduse = true; } else { - vsocket->is_server = true; - } - ret = create_unix_socket(vsocket); - if (ret < 0) { - goto out_mutex; + if ((flags & RTE_VHOST_USER_CLIENT) != 0) { + vsocket->reconnect = !(flags & RTE_VHOST_USER_NO_RECONNECT); + if (vsocket->reconnect && reconn_tid == 0) { + if (vhost_user_reconnect_init() != 0) + goto out_mutex; + } + } else { + vsocket->is_server = true; + } + ret = create_unix_socket(vsocket); + if (ret < 0) + goto out_mutex; } vhost_user.vsockets[vhost_user.vsocket_cnt++] = vsocket; @@ -1068,7 +1073,9 @@ rte_vhost_driver_unregister(const char *path) if (strcmp(vsocket->path, path)) continue; - if (vsocket->is_server) { + if (vsocket->is_vduse) { + vduse_device_destroy(path); + } else if (vsocket->is_server) { /* * If r/wcb is executing, release vhost_user's * mutex lock, and try again since the r/wcb @@ -1171,6 +1178,9 @@ rte_vhost_driver_start(const char *path) if (!vsocket) return -1; + if (vsocket->is_vduse) + return vduse_device_create(path); + if (fdset_tid == 0) { /** * create a pipe which will be waited by poll and notified to diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c new file mode 100644 index 0000000000..d67818bfb5 --- /dev/null +++ b/lib/vhost/vduse.c @@ -0,0 +1,201 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Red Hat, Inc. + */ + +#include +#include +#include +#include + + +#include +#include + +#include +#include + +#include + +#include "vduse.h" +#include "vhost.h" + +#define VHOST_VDUSE_API_VERSION 0 +#define VDUSE_CTRL_PATH "/dev/vduse/control" + +#define VDUSE_NET_SUPPORTED_FEATURES ((1ULL << VIRTIO_NET_F_MRG_RXBUF) | \ + (1ULL << VIRTIO_F_ANY_LAYOUT) | \ + (1ULL << VIRTIO_F_VERSION_1) | \ + (1ULL << VIRTIO_NET_F_GSO) | \ + (1ULL << VIRTIO_NET_F_HOST_TSO4) | \ + (1ULL << VIRTIO_NET_F_HOST_TSO6) | \ + (1ULL << VIRTIO_NET_F_HOST_UFO) | \ + (1ULL << VIRTIO_NET_F_HOST_ECN) | \ + (1ULL << VIRTIO_NET_F_CSUM) | \ + (1ULL << VIRTIO_NET_F_GUEST_CSUM) | \ + (1ULL << VIRTIO_NET_F_GUEST_TSO4) | \ + (1ULL << VIRTIO_NET_F_GUEST_TSO6) | \ + (1ULL << VIRTIO_NET_F_GUEST_UFO) | \ + (1ULL << VIRTIO_NET_F_GUEST_ECN) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_F_IN_ORDER) | \ + (1ULL << VIRTIO_F_IOMMU_PLATFORM)) + +static struct vhost_backend_ops vduse_backend_ops = { +}; + +int +vduse_device_create(const char *path) +{ + int control_fd, dev_fd, vid, ret; + uint32_t i; + struct virtio_net *dev; + uint64_t ver = VHOST_VDUSE_API_VERSION; + struct vduse_dev_config *dev_config = NULL; + const char *name = path + strlen("/dev/vduse/"); + + control_fd = open(VDUSE_CTRL_PATH, O_RDWR); + if (control_fd < 0) { + VHOST_LOG_CONFIG(name, ERR, "Failed to open %s: %s\n", + VDUSE_CTRL_PATH, strerror(errno)); + return -1; + } + + if (ioctl(control_fd, VDUSE_SET_API_VERSION, &ver)) { + VHOST_LOG_CONFIG(name, ERR, "Failed to set API version: %" PRIu64 ": %s\n", + ver, strerror(errno)); + ret = -1; + goto out_ctrl_close; + } + + dev_config = malloc(offsetof(struct vduse_dev_config, config)); + if (!dev_config) { + VHOST_LOG_CONFIG(name, ERR, "Failed to allocate VDUSE config\n"); + ret = -1; + goto out_ctrl_close; + } + + memset(dev_config, 0, sizeof(struct vduse_dev_config)); + + strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1); + dev_config->device_id = VIRTIO_ID_NET; + dev_config->vendor_id = 0; + dev_config->features = VDUSE_NET_SUPPORTED_FEATURES; + dev_config->vq_num = 2; + dev_config->vq_align = sysconf(_SC_PAGE_SIZE); + dev_config->config_size = 0; + + ret = ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); + if (ret < 0) { + VHOST_LOG_CONFIG(name, ERR, "Failed to create VDUSE device: %s\n", + strerror(errno)); + goto out_free; + } + + dev_fd = open(path, O_RDWR); + if (dev_fd < 0) { + VHOST_LOG_CONFIG(name, ERR, "Failed to open device %s: %s\n", + path, strerror(errno)); + ret = -1; + goto out_dev_close; + } + + ret = fcntl(dev_fd, F_SETFL, O_NONBLOCK); + if (ret < 0) { + VHOST_LOG_CONFIG(name, ERR, "Failed to set chardev as non-blocking: %s\n", + strerror(errno)); + goto out_dev_close; + } + + vid = vhost_new_device(&vduse_backend_ops); + if (vid < 0) { + VHOST_LOG_CONFIG(name, ERR, "Failed to create new Vhost device\n"); + ret = -1; + goto out_dev_close; + } + + dev = get_device(vid); + if (!dev) { + ret = -1; + goto out_dev_close; + } + + strncpy(dev->ifname, path, IF_NAME_SZ - 1); + dev->vduse_ctrl_fd = control_fd; + dev->vduse_dev_fd = dev_fd; + vhost_setup_virtio_net(dev->vid, true, true, true, true); + + for (i = 0; i < 2; i++) { + struct vduse_vq_config vq_cfg = { 0 }; + + ret = alloc_vring_queue(dev, i); + if (ret) { + VHOST_LOG_CONFIG(name, ERR, "Failed to alloc vring %d metadata\n", i); + goto out_dev_destroy; + } + + vq_cfg.index = i; + vq_cfg.max_size = 1024; + + ret = ioctl(dev->vduse_dev_fd, VDUSE_VQ_SETUP, &vq_cfg); + if (ret) { + VHOST_LOG_CONFIG(name, ERR, "Failed to set-up VQ %d\n", i); + goto out_dev_destroy; + } + } + + free(dev_config); + + return 0; + +out_dev_destroy: + vhost_destroy_device(vid); +out_dev_close: + if (dev_fd >= 0) + close(dev_fd); + ioctl(control_fd, VDUSE_DESTROY_DEV, name); +out_free: + free(dev_config); +out_ctrl_close: + close(control_fd); + + return ret; +} + +int +vduse_device_destroy(const char *path) +{ + const char *name = path + strlen("/dev/vduse/"); + struct virtio_net *dev; + int vid, ret; + + for (vid = 0; vid < RTE_MAX_VHOST_DEVICE; vid++) { + dev = vhost_devices[vid]; + + if (dev == NULL) + continue; + + if (!strcmp(path, dev->ifname)) + break; + } + + if (vid == RTE_MAX_VHOST_DEVICE) + return -1; + + if (dev->vduse_dev_fd >= 0) { + close(dev->vduse_dev_fd); + dev->vduse_dev_fd = -1; + } + + if (dev->vduse_ctrl_fd >= 0) { + ret = ioctl(dev->vduse_ctrl_fd, VDUSE_DESTROY_DEV, name); + if (ret) + VHOST_LOG_CONFIG(name, ERR, "Failed to destroy VDUSE device: %s\n", + strerror(errno)); + close(dev->vduse_ctrl_fd); + dev->vduse_ctrl_fd = -1; + } + + vhost_destroy_device(vid); + + return 0; +} diff --git a/lib/vhost/vduse.h b/lib/vhost/vduse.h new file mode 100644 index 0000000000..a15e5d4c16 --- /dev/null +++ b/lib/vhost/vduse.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright (c) 2023 Red Hat, Inc. + */ + +#ifndef _VDUSE_H +#define _VDUSE_H + +#include "vhost.h" + +#ifdef VHOST_HAS_VDUSE + +int vduse_device_create(const char *path); +int vduse_device_destroy(const char *path); + +#else + +static inline int +vduse_device_create(const char *path) +{ + VHOST_LOG_CONFIG(path, ERR, "VDUSE support disabled at build time\n"); + return -1; +} + +static inline int +vduse_device_destroy(const char *path) +{ + VHOST_LOG_CONFIG(path, ERR, "VDUSE support disabled at build time\n"); + return -1; +} + +#endif /* VHOST_HAS_VDUSE */ + +#endif /* _VDUSE_H */ diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 76663aed24..c8f2a0d43a 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -524,6 +524,8 @@ struct virtio_net { int postcopy_ufd; int postcopy_listening; + int vduse_ctrl_fd; + int vduse_dev_fd; struct vhost_virtqueue *cvq; From patchwork Thu May 25 16:15:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127458 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 43DB042B9D; Thu, 25 May 2023 18:18:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0C95642DB9; Thu, 25 May 2023 18:16:57 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id ADF1640FAE for ; Thu, 25 May 2023 18:16:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031415; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bCjb9bPrkWjcMRuH9nhbBJ1yUcyb/M8upG8Hsux6BOI=; b=dqts7ZzPXreoIBPG1CaqTLsW/D6obwA/K8DFufuzIYa9QJI1i/IzLNcaZjPQodH1D9Kx+v S+RcHBn383E+7pc/i05nidCOP0I91V5nJrKDAqjDMCFDJNJ/vQStqLbsfhApoOySIdmMZj EZrtF8jzhaLHCR943EAro+e9TPVl1ek= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-539-PwasmqE2Pka-jrQOUqWBpQ-1; Thu, 25 May 2023 12:16:52 -0400 X-MC-Unique: PwasmqE2Pka-jrQOUqWBpQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AB9D52999B34; Thu, 25 May 2023 16:16:51 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2DADEC154D1; Thu, 25 May 2023 16:16:49 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 19/28] vhost: add VDUSE callback for IOTLB miss Date: Thu, 25 May 2023 18:15:41 +0200 Message-Id: <20230525161551.70002-20-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch implements the VDUSE callback for IOTLB misses, which is done by using the VDUSE VDUSE_IOTLB_GET_FD ioctl. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index d67818bfb5..f72c7bf6ab 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -13,9 +13,11 @@ #include #include +#include #include +#include "iotlb.h" #include "vduse.h" #include "vhost.h" @@ -40,7 +42,63 @@ (1ULL << VIRTIO_F_IN_ORDER) | \ (1ULL << VIRTIO_F_IOMMU_PLATFORM)) +static int +vduse_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm __rte_unused) +{ + struct vduse_iotlb_entry entry; + uint64_t size, page_size; + struct stat stat; + void *mmap_addr; + int fd, ret; + + entry.start = iova; + entry.last = iova + 1; + + ret = ioctl(dev->vduse_dev_fd, VDUSE_IOTLB_GET_FD, &entry); + if (ret < 0) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to get IOTLB entry for 0x%" PRIx64 "\n", + iova); + return -1; + } + + fd = ret; + + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "New IOTLB entry:\n"); + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "\tIOVA: %" PRIx64 " - %" PRIx64 "\n", + (uint64_t)entry.start, (uint64_t)entry.last); + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "\toffset: %" PRIx64 "\n", (uint64_t)entry.offset); + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "\tfd: %d\n", fd); + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "\tperm: %x\n", entry.perm); + + size = entry.last - entry.start + 1; + mmap_addr = mmap(0, size + entry.offset, entry.perm, MAP_SHARED, fd, 0); + if (!mmap_addr) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Failed to mmap IOTLB entry for 0x%" PRIx64 "\n", iova); + ret = -1; + goto close_fd; + } + + ret = fstat(fd, &stat); + if (ret < 0) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to get page size.\n"); + munmap(mmap_addr, entry.offset + size); + goto close_fd; + } + page_size = (uint64_t)stat.st_blksize; + + vhost_user_iotlb_cache_insert(dev, entry.start, (uint64_t)(uintptr_t)mmap_addr, + entry.offset, size, page_size, entry.perm); + + ret = 0; +close_fd: + close(fd); + + return ret; +} + static struct vhost_backend_ops vduse_backend_ops = { + .iotlb_miss = vduse_iotlb_miss, }; int From patchwork Thu May 25 16:15:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127459 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 052BE42B9D; Thu, 25 May 2023 18:18:30 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1871A42D5A; Thu, 25 May 2023 18:16:59 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 79EF540FAE for ; Thu, 25 May 2023 18:16:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y+TEPirlb+wDds0kd2o148PP5zMX1aP0VmLytrR1aKU=; b=dtQuDCRKAabZP+pIyx8/IF8i7ULl4tDT2DaHwSkfY5l5NL2ZIcFmQqEQ4y2MSB/Jlb4cNh SimPnQPPDO/d6wDKekIK5e3v/RvOEzDcNpcMKKHJbxMOWrQ6Z1sejO8K7wazuQseS4suGq UzDc8+zmEYLaBfQncPihaEEmKxNHDik= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-647-Em19X1gnP66vnVpLha3jiA-1; Thu, 25 May 2023 12:16:54 -0400 X-MC-Unique: Em19X1gnP66vnVpLha3jiA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7D3F73C13509; Thu, 25 May 2023 16:16:54 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id F1844C154D1; Thu, 25 May 2023 16:16:51 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 20/28] vhost: add VDUSE callback for IOTLB entry removal Date: Thu, 25 May 2023 18:15:42 +0200 Message-Id: <20230525161551.70002-21-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch implements the VDUSE callback for IOTLB entry removal, where it unmaps the pages from the invalidated IOTLB entry. Signed-off-by: Maxime Coquelin --- lib/vhost/vduse.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index f72c7bf6ab..58c1b384a8 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -42,6 +42,12 @@ (1ULL << VIRTIO_F_IN_ORDER) | \ (1ULL << VIRTIO_F_IOMMU_PLATFORM)) +static void +vduse_iotlb_remove_notify(uint64_t addr, uint64_t offset, uint64_t size) +{ + munmap((void *)(uintptr_t)addr, offset + size); +} + static int vduse_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm __rte_unused) { @@ -99,6 +105,7 @@ vduse_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm __rte_unuse static struct vhost_backend_ops vduse_backend_ops = { .iotlb_miss = vduse_iotlb_miss, + .iotlb_remove_notify = vduse_iotlb_remove_notify, }; int From patchwork Thu May 25 16:15:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127461 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 74DDC42B9D; Thu, 25 May 2023 18:18:39 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 02E2C42F82; Thu, 25 May 2023 18:17:08 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id EA58742F86 for ; Thu, 25 May 2023 18:17:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031424; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aYGAMnOnshRq1wFFVMK1JfUyFWNr5tDK5GwYNj+Lrw8=; b=I8huzH5I1rpIZfrpB3ymxLVq+cgesNQUdfXiNmptDp18Jd2XnE+Pi27Fi5qexCZyI8Nlyr rUaZWjpmOgrWfio4VpJhumGO9zt0ZzD8IEh+niG8WiksZVt72vhFN18qgdB5JCPki8ZNWK osFJ68dcGWbSSpRxkuioZEtV8lGXIzo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-648-yMNzP6TrPD-soGLsdgwH5Q-1; Thu, 25 May 2023 12:16:57 -0400 X-MC-Unique: yMNzP6TrPD-soGLsdgwH5Q-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 529DD185A7A9; Thu, 25 May 2023 16:16:57 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id CD1D4C154D1; Thu, 25 May 2023 16:16:54 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 21/28] vhost: add VDUSE callback for IRQ injection Date: Thu, 25 May 2023 18:15:43 +0200 Message-Id: <20230525161551.70002-22-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch implements the VDUSE callback for kicking virtqueues. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index 58c1b384a8..d39e39b9dc 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -42,6 +42,12 @@ (1ULL << VIRTIO_F_IN_ORDER) | \ (1ULL << VIRTIO_F_IOMMU_PLATFORM)) +static int +vduse_inject_irq(struct virtio_net *dev, struct vhost_virtqueue *vq) +{ + return ioctl(dev->vduse_dev_fd, VDUSE_VQ_INJECT_IRQ, &vq->index); +} + static void vduse_iotlb_remove_notify(uint64_t addr, uint64_t offset, uint64_t size) { @@ -106,6 +112,7 @@ vduse_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm __rte_unuse static struct vhost_backend_ops vduse_backend_ops = { .iotlb_miss = vduse_iotlb_miss, .iotlb_remove_notify = vduse_iotlb_remove_notify, + .inject_irq = vduse_inject_irq, }; int From patchwork Thu May 25 16:15:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127460 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AEB4642B9D; Thu, 25 May 2023 18:18:34 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1762C42D64; Thu, 25 May 2023 18:17:05 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 0EC4F42D36 for ; Thu, 25 May 2023 18:17:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031423; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/NaxkaDcGVWBez8isP823L/8Qgqwaypg58E+DhHMgoE=; b=ECdAOw/4KNHrzp0mB/JNTTBYoRcvs69tlER78E4XC2WHMvW28nYcQJ2HhQGszA5q9Q8Q5Z 65KrGNh6nXBD+Bci4Qmcs5cEHvF7jxTNE5j+iR5kmiXiTeH4YZ7jKSySnaIAnjH1kb5eIg 4WCm26rVcpIJ8bTVoX3fY50SlGTM1ME= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-96--xCf7P4hONSKl2QQy-Vg2g-1; Thu, 25 May 2023 12:17:00 -0400 X-MC-Unique: -xCf7P4hONSKl2QQy-Vg2g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2C745185A795; Thu, 25 May 2023 16:17:00 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id A3189C154D3; Thu, 25 May 2023 16:16:57 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 22/28] vhost: add VDUSE events handler Date: Thu, 25 May 2023 18:15:44 +0200 Message-Id: <20230525161551.70002-23-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch makes use of Vhost lib's FD manager to install a handler for VDUSE events occurring on the VDUSE device FD. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index d39e39b9dc..92c515cff2 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -17,6 +17,7 @@ #include +#include "fd_man.h" #include "iotlb.h" #include "vduse.h" #include "vhost.h" @@ -42,6 +43,31 @@ (1ULL << VIRTIO_F_IN_ORDER) | \ (1ULL << VIRTIO_F_IOMMU_PLATFORM)) +struct vduse { + struct fdset fdset; +}; + +static struct vduse vduse = { + .fdset = { + .fd = { [0 ... MAX_FDS - 1] = {-1, NULL, NULL, NULL, 0} }, + .fd_mutex = PTHREAD_MUTEX_INITIALIZER, + .fd_pooling_mutex = PTHREAD_MUTEX_INITIALIZER, + .num = 0 + }, +}; + +static bool vduse_events_thread; + +static const char * const vduse_reqs_str[] = { + "VDUSE_GET_VQ_STATE", + "VDUSE_SET_STATUS", + "VDUSE_UPDATE_IOTLB", +}; + +#define vduse_req_id_to_str(id) \ + (id < RTE_DIM(vduse_reqs_str) ? \ + vduse_reqs_str[id] : "Unknown") + static int vduse_inject_irq(struct virtio_net *dev, struct vhost_virtqueue *vq) { @@ -115,16 +141,80 @@ static struct vhost_backend_ops vduse_backend_ops = { .inject_irq = vduse_inject_irq, }; +static void +vduse_events_handler(int fd, void *arg, int *remove __rte_unused) +{ + struct virtio_net *dev = arg; + struct vduse_dev_request req; + struct vduse_dev_response resp; + int ret; + + memset(&resp, 0, sizeof(resp)); + + ret = read(fd, &req, sizeof(req)); + if (ret < 0) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to read request: %s\n", + strerror(errno)); + return; + } else if (ret < (int)sizeof(req)) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Incomplete to read request %d\n", ret); + return; + } + + VHOST_LOG_CONFIG(dev->ifname, INFO, "New request: %s (%u)\n", + vduse_req_id_to_str(req.type), req.type); + + switch (req.type) { + default: + resp.result = VDUSE_REQ_RESULT_FAILED; + break; + } + + resp.request_id = req.request_id; + + ret = write(dev->vduse_dev_fd, &resp, sizeof(resp)); + if (ret != sizeof(resp)) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to write response %s\n", + strerror(errno)); + return; + } + VHOST_LOG_CONFIG(dev->ifname, INFO, "Request %s (%u) handled successfully\n", + vduse_req_id_to_str(req.type), req.type); +} + int vduse_device_create(const char *path) { int control_fd, dev_fd, vid, ret; + pthread_t fdset_tid; uint32_t i; struct virtio_net *dev; uint64_t ver = VHOST_VDUSE_API_VERSION; struct vduse_dev_config *dev_config = NULL; const char *name = path + strlen("/dev/vduse/"); + /* If first device, create events dispatcher thread */ + if (vduse_events_thread == false) { + /** + * create a pipe which will be waited by poll and notified to + * rebuild the wait list of poll. + */ + if (fdset_pipe_init(&vduse.fdset) < 0) { + VHOST_LOG_CONFIG(path, ERR, "failed to create pipe for vduse fdset\n"); + return -1; + } + + ret = rte_ctrl_thread_create(&fdset_tid, "vduse-events", NULL, + fdset_event_dispatch, &vduse.fdset); + if (ret != 0) { + VHOST_LOG_CONFIG(path, ERR, "failed to create vduse fdset handling thread\n"); + fdset_pipe_uninit(&vduse.fdset); + return -1; + } + + vduse_events_thread = true; + } + control_fd = open(VDUSE_CTRL_PATH, O_RDWR); if (control_fd < 0) { VHOST_LOG_CONFIG(name, ERR, "Failed to open %s: %s\n", @@ -215,6 +305,14 @@ vduse_device_create(const char *path) } } + ret = fdset_add(&vduse.fdset, dev->vduse_dev_fd, vduse_events_handler, NULL, dev); + if (ret) { + VHOST_LOG_CONFIG(name, ERR, "Failed to add fd %d to vduse fdset\n", + dev->vduse_dev_fd); + goto out_dev_destroy; + } + fdset_pipe_notify(&vduse.fdset); + free(dev_config); return 0; @@ -253,6 +351,9 @@ vduse_device_destroy(const char *path) if (vid == RTE_MAX_VHOST_DEVICE) return -1; + fdset_del(&vduse.fdset, dev->vduse_dev_fd); + fdset_pipe_notify(&vduse.fdset); + if (dev->vduse_dev_fd >= 0) { close(dev->vduse_dev_fd); dev->vduse_dev_fd = -1; From patchwork Thu May 25 16:15:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127462 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CFD4342B9D; Thu, 25 May 2023 18:18:45 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4C48042F9C; Thu, 25 May 2023 18:17:10 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 530A042F82 for ; Thu, 25 May 2023 18:17:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031426; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/mMbfPtlgp+8d/URc01bG4SThKEEoWRRLfDp81aBQ+U=; b=CkXexOXXI5vwPE5C5PHXGUiAjhVgRgrf3vkgi/jLP+tuNKHd4Ysr2qSW+RSrUhAgJd81Ou AfrmipMUy95T4Pi6OsqyXocWK5/qJaWETsIRHCKv0AmrNCdSrr7WMihGK42Ex8qba9LdFi kzkaaSnrDBWRi3cI9aSX9XDmKz4Sppg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-440-mRHXfVkRNxCfWcbbDHpQNg-1; Thu, 25 May 2023 12:17:03 -0400 X-MC-Unique: mRHXfVkRNxCfWcbbDHpQNg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F0D502999B36; Thu, 25 May 2023 16:17:02 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 72605C154D1; Thu, 25 May 2023 16:17:00 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 23/28] vhost: add support for virtqueue state get event Date: Thu, 25 May 2023 18:15:45 +0200 Message-Id: <20230525161551.70002-24-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds support for VDUSE_GET_VQ_STATE event handling, which consists in providing the backend last available index for the specified virtqueue. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index 92c515cff2..7e36c50b6c 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -147,6 +147,7 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) struct virtio_net *dev = arg; struct vduse_dev_request req; struct vduse_dev_response resp; + struct vhost_virtqueue *vq; int ret; memset(&resp, 0, sizeof(resp)); @@ -165,6 +166,13 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) vduse_req_id_to_str(req.type), req.type); switch (req.type) { + case VDUSE_GET_VQ_STATE: + vq = dev->virtqueue[req.vq_state.index]; + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tvq index: %u, avail_index: %u\n", + req.vq_state.index, vq->last_avail_idx); + resp.vq_state.split.avail_index = vq->last_avail_idx; + resp.result = VDUSE_REQ_RESULT_OK; + break; default: resp.result = VDUSE_REQ_RESULT_FAILED; break; From patchwork Thu May 25 16:15:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127463 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DCB9642B9D; Thu, 25 May 2023 18:18:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 475FE42F8E; Thu, 25 May 2023 18:17:12 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id EB03742F98 for ; Thu, 25 May 2023 18:17:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+AtyqhGEYrXQHh7YOVU0PAAJ8QpP4RzlHjniUfhJWNo=; b=W/HrncQjS8nr5rXJewK1QZM/dSz7uL7iWupNINes/0iXOBtmjXge+ColM5LM8If2qe8zD2 qT8RU2X457EmjPyubIfZglKm9iuB6T99o0+/ZULiua9erdtpvQdVyPY/L/TqRuwzk38IiU pjfpTwKN38nzKJb+fmYjBbH+FAJMS08= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-350-icVwSbUGMuC22m63jkbuBw-1; Thu, 25 May 2023 12:17:06 -0400 X-MC-Unique: icVwSbUGMuC22m63jkbuBw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF374101B051; Thu, 25 May 2023 16:17:05 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 42F91C154D1; Thu, 25 May 2023 16:17:03 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 24/28] vhost: add support for VDUSE status set event Date: Thu, 25 May 2023 18:15:46 +0200 Message-Id: <20230525161551.70002-25-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds support for VDUSE_SET_STATUS event handling, which consists in updating the Virtio device status set by the Virtio driver. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index 7e36c50b6c..3bf65d4b8b 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -173,6 +173,12 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) resp.vq_state.split.avail_index = vq->last_avail_idx; resp.result = VDUSE_REQ_RESULT_OK; break; + case VDUSE_SET_STATUS: + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tnew status: 0x%08x\n", + req.s.status); + dev->status = req.s.status; + resp.result = VDUSE_REQ_RESULT_OK; + break; default: resp.result = VDUSE_REQ_RESULT_FAILED; break; From patchwork Thu May 25 16:15:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127464 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8F86B42B9D; Thu, 25 May 2023 18:18:55 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3C50342FA0; Thu, 25 May 2023 18:17:15 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 172B542F90 for ; Thu, 25 May 2023 18:17:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031433; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hZIc54wQsoQo2oDzD8tMk/y1rjat5yw4yAF15DwczB8=; b=CpC0m+4+M39uc/CXjdaJdAeRTnNY1LtzqmHSLA/2xyAHe+686jVj+iV194E6Ta00W1UKFP CLrHRbczlFM5w7fH3ODmNC8HY5o+6JZIFCAK7NCTl+yDE/ODK3Qr3eemfYw44fHTDv0WV+ QjXTjdZwSm2ytIFr09d8oZEfuWLq2wQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-665-2xqZowQ-PD2WK-iRMZwMcw-1; Thu, 25 May 2023 12:17:10 -0400 X-MC-Unique: 2xqZowQ-PD2WK-iRMZwMcw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9A625811E93; Thu, 25 May 2023 16:17:08 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1B9EDC154D2; Thu, 25 May 2023 16:17:05 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 25/28] vhost: add support for VDUSE IOTLB update event Date: Thu, 25 May 2023 18:15:47 +0200 Message-Id: <20230525161551.70002-26-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds support for VDUSE_UPDATE_IOTLB event handling, which consists in invaliding IOTLB entries for the range specified in the request. Signed-off-by: Maxime Coquelin --- lib/vhost/vduse.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index 3bf65d4b8b..110654ec68 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -179,6 +179,13 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) dev->status = req.s.status; resp.result = VDUSE_REQ_RESULT_OK; break; + case VDUSE_UPDATE_IOTLB: + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tIOVA range: %" PRIx64 " - %" PRIx64 "\n", + (uint64_t)req.iova.start, (uint64_t)req.iova.last); + vhost_user_iotlb_cache_remove(dev, req.iova.start, + req.iova.last - req.iova.start + 1); + resp.result = VDUSE_REQ_RESULT_OK; + break; default: resp.result = VDUSE_REQ_RESULT_FAILED; break; From patchwork Thu May 25 16:15:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127465 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A0FFE42B9D; Thu, 25 May 2023 18:19:00 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4ED5C42FAE; Thu, 25 May 2023 18:17:18 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 5CCA442FA4 for ; Thu, 25 May 2023 18:17:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YwlzYHwt7IM+hV5pg2MDq7egkCHBIQihzJGivnXO/ZE=; b=PZefgMBAvlPf9f4XYaxrZEJmYyAAmEba0MNV3KAldLOs8fljN3u3z/aTHTl4rQv3zul+GR JRrtWCTDWfmvotWf81N58ld5ezHHqysmsz+cxPIqFIvR4Dzv4TPwsG8eEuJGkeKxrIPVJa SUoUnEbUU7Byxvbg/RIbYbOnn44Ioho= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-42-jH9sHQemPMePJ04Hl-Y6Xg-1; Thu, 25 May 2023 12:17:11 -0400 X-MC-Unique: jH9sHQemPMePJ04Hl-Y6Xg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 72B732999B36; Thu, 25 May 2023 16:17:11 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA925C0448E; Thu, 25 May 2023 16:17:08 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 26/28] vhost: add VDUSE device startup Date: Thu, 25 May 2023 18:15:48 +0200 Message-Id: <20230525161551.70002-27-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds the device and its virtqueues initialization once the Virtio driver has set the DRIVER_OK in the Virtio status register. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index 110654ec68..a10dc24d38 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -141,6 +141,128 @@ static struct vhost_backend_ops vduse_backend_ops = { .inject_irq = vduse_inject_irq, }; +static void +vduse_vring_setup(struct virtio_net *dev, unsigned int index) +{ + struct vhost_virtqueue *vq = dev->virtqueue[index]; + struct vhost_vring_addr *ra = &vq->ring_addrs; + struct vduse_vq_info vq_info; + struct vduse_vq_eventfd vq_efd; + int ret; + + vq_info.index = index; + ret = ioctl(dev->vduse_dev_fd, VDUSE_VQ_GET_INFO, &vq_info); + if (ret) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to get VQ %u info: %s\n", + index, strerror(errno)); + return; + } + + VHOST_LOG_CONFIG(dev->ifname, INFO, "VQ %u info:\n", index); + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tnum: %u\n", vq_info.num); + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tdesc_addr: %llx\n", vq_info.desc_addr); + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tdriver_addr: %llx\n", vq_info.driver_addr); + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tdevice_addr: %llx\n", vq_info.device_addr); + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tavail_idx: %u\n", vq_info.split.avail_index); + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tready: %u\n", vq_info.ready); + + vq->last_avail_idx = vq_info.split.avail_index; + vq->size = vq_info.num; + vq->ready = true; + vq->enabled = vq_info.ready; + ra->desc_user_addr = vq_info.desc_addr; + ra->avail_user_addr = vq_info.driver_addr; + ra->used_user_addr = vq_info.device_addr; + + vq->kickfd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (vq->kickfd < 0) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to init kickfd for VQ %u: %s\n", + index, strerror(errno)); + vq->kickfd = VIRTIO_INVALID_EVENTFD; + return; + } + VHOST_LOG_CONFIG(dev->ifname, INFO, "\tkick fd: %d\n", vq->kickfd); + + vq->shadow_used_split = rte_malloc_socket(NULL, + vq->size * sizeof(struct vring_used_elem), + RTE_CACHE_LINE_SIZE, 0); + vq->batch_copy_elems = rte_malloc_socket(NULL, + vq->size * sizeof(struct batch_copy_elem), + RTE_CACHE_LINE_SIZE, 0); + + vhost_user_iotlb_rd_lock(vq); + if (vring_translate(dev, vq)) + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to translate vring %d addresses\n", + index); + + if (vhost_enable_guest_notification(dev, vq, 0)) + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Failed to disable guest notifications on vring %d\n", + index); + vhost_user_iotlb_rd_unlock(vq); + + vq_efd.index = index; + vq_efd.fd = vq->kickfd; + + ret = ioctl(dev->vduse_dev_fd, VDUSE_VQ_SETUP_KICKFD, &vq_efd); + if (ret) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to setup kickfd for VQ %u: %s\n", + index, strerror(errno)); + close(vq->kickfd); + vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD; + return; + } +} + +static void +vduse_device_start(struct virtio_net *dev) +{ + unsigned int i, ret; + + VHOST_LOG_CONFIG(dev->ifname, INFO, "Starting device...\n"); + + dev->notify_ops = vhost_driver_callback_get(dev->ifname); + if (!dev->notify_ops) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Failed to get callback ops for driver\n"); + return; + } + + ret = ioctl(dev->vduse_dev_fd, VDUSE_DEV_GET_FEATURES, &dev->features); + if (ret) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to get features: %s\n", + strerror(errno)); + return; + } + + VHOST_LOG_CONFIG(dev->ifname, INFO, "Negotiated Virtio features: 0x%" PRIx64 "\n", + dev->features); + + if (dev->features & + ((1ULL << VIRTIO_NET_F_MRG_RXBUF) | + (1ULL << VIRTIO_F_VERSION_1) | + (1ULL << VIRTIO_F_RING_PACKED))) { + dev->vhost_hlen = sizeof(struct virtio_net_hdr_mrg_rxbuf); + } else { + dev->vhost_hlen = sizeof(struct virtio_net_hdr); + } + + for (i = 0; i < dev->nr_vring; i++) + vduse_vring_setup(dev, i); + + dev->flags |= VIRTIO_DEV_READY; + + if (dev->notify_ops->new_device(dev->vid) == 0) + dev->flags |= VIRTIO_DEV_RUNNING; + + for (i = 0; i < dev->nr_vring; i++) { + struct vhost_virtqueue *vq = dev->virtqueue[i]; + + if (dev->notify_ops->vring_state_changed) + dev->notify_ops->vring_state_changed(dev->vid, i, vq->enabled); + } +} + static void vduse_events_handler(int fd, void *arg, int *remove __rte_unused) { @@ -177,6 +299,10 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) VHOST_LOG_CONFIG(dev->ifname, INFO, "\tnew status: 0x%08x\n", req.s.status); dev->status = req.s.status; + + if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) + vduse_device_start(dev); + resp.result = VDUSE_REQ_RESULT_OK; break; case VDUSE_UPDATE_IOTLB: From patchwork Thu May 25 16:15:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127466 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F34F142B9D; Thu, 25 May 2023 18:19:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 49EF542FB0; Thu, 25 May 2023 18:17:19 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 3ADD142FAA for ; Thu, 25 May 2023 18:17:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GB/AU+Vbvek3rhMkceixizYLiCXsskLFM3Hzutrpcc4=; b=NQLjDkMNSzP4Z0WlZnhTbRf6O0L+kAiiZbDwUd93SpXVhBa2SDAnwI7B1mvDb6XNiU57eZ KgQXroHNjl5WqnlPkrRaf9N18eHiiHG6O6xplnhyvyQj1YyaJrHUK0eQ4IkBkY3VJJKkbC EG91UcSyBQfRBUCm6QNNJq4xybot2LY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-34-DkbiVqNqOvyYD71oqojxPg-1; Thu, 25 May 2023 12:17:14 -0400 X-MC-Unique: DkbiVqNqOvyYD71oqojxPg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 418FA811E8D; Thu, 25 May 2023 16:17:14 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id B87B7C154D1; Thu, 25 May 2023 16:17:11 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH v2 27/28] vhost: add multiqueue support to VDUSE Date: Thu, 25 May 2023 18:15:49 +0200 Message-Id: <20230525161551.70002-28-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch enables control queue support in order to support multiqueue. Signed-off-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vduse.c | 83 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 76 insertions(+), 7 deletions(-) diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index a10dc24d38..699cfed9e3 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -21,6 +21,7 @@ #include "iotlb.h" #include "vduse.h" #include "vhost.h" +#include "virtio_net_ctrl.h" #define VHOST_VDUSE_API_VERSION 0 #define VDUSE_CTRL_PATH "/dev/vduse/control" @@ -41,7 +42,9 @@ (1ULL << VIRTIO_NET_F_GUEST_ECN) | \ (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ (1ULL << VIRTIO_F_IN_ORDER) | \ - (1ULL << VIRTIO_F_IOMMU_PLATFORM)) + (1ULL << VIRTIO_F_IOMMU_PLATFORM) | \ + (1ULL << VIRTIO_NET_F_CTRL_VQ) | \ + (1ULL << VIRTIO_NET_F_MQ)) struct vduse { struct fdset fdset; @@ -141,6 +144,25 @@ static struct vhost_backend_ops vduse_backend_ops = { .inject_irq = vduse_inject_irq, }; +static void +vduse_control_queue_event(int fd, void *arg, int *remove __rte_unused) +{ + struct virtio_net *dev = arg; + uint64_t buf; + int ret; + + ret = read(fd, &buf, sizeof(buf)); + if (ret < 0) { + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to read control queue event: %s\n", + strerror(errno)); + return; + } + + VHOST_LOG_CONFIG(dev->ifname, DEBUG, "Control queue kicked\n"); + if (virtio_net_ctrl_handle(dev)) + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to handle ctrl request\n"); +} + static void vduse_vring_setup(struct virtio_net *dev, unsigned int index) { @@ -212,6 +234,22 @@ vduse_vring_setup(struct virtio_net *dev, unsigned int index) vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD; return; } + + if (vq == dev->cvq) { + ret = fdset_add(&vduse.fdset, vq->kickfd, vduse_control_queue_event, NULL, dev); + if (ret) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "Failed to setup kickfd handler for VQ %u: %s\n", + index, strerror(errno)); + vq_efd.fd = VDUSE_EVENTFD_DEASSIGN; + ioctl(dev->vduse_dev_fd, VDUSE_VQ_SETUP_KICKFD, &vq_efd); + close(vq->kickfd); + vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD; + } + fdset_pipe_notify(&vduse.fdset); + vhost_enable_guest_notification(dev, vq, 1); + VHOST_LOG_CONFIG(dev->ifname, INFO, "Ctrl queue event handler installed\n"); + } } static void @@ -258,6 +296,9 @@ vduse_device_start(struct virtio_net *dev) for (i = 0; i < dev->nr_vring; i++) { struct vhost_virtqueue *vq = dev->virtqueue[i]; + if (vq == dev->cvq) + continue; + if (dev->notify_ops->vring_state_changed) dev->notify_ops->vring_state_changed(dev->vid, i, vq->enabled); } @@ -334,9 +375,11 @@ vduse_device_create(const char *path) { int control_fd, dev_fd, vid, ret; pthread_t fdset_tid; - uint32_t i; + uint32_t i, max_queue_pairs, total_queues; struct virtio_net *dev; + struct virtio_net_config vnet_config = { 0 }; uint64_t ver = VHOST_VDUSE_API_VERSION; + uint64_t features = VDUSE_NET_SUPPORTED_FEATURES; struct vduse_dev_config *dev_config = NULL; const char *name = path + strlen("/dev/vduse/"); @@ -376,22 +419,39 @@ vduse_device_create(const char *path) goto out_ctrl_close; } - dev_config = malloc(offsetof(struct vduse_dev_config, config)); + dev_config = malloc(offsetof(struct vduse_dev_config, config) + + sizeof(vnet_config)); if (!dev_config) { VHOST_LOG_CONFIG(name, ERR, "Failed to allocate VDUSE config\n"); ret = -1; goto out_ctrl_close; } + ret = rte_vhost_driver_get_queue_num(path, &max_queue_pairs); + if (ret < 0) { + VHOST_LOG_CONFIG(name, ERR, "Failed to get max queue pairs\n"); + goto out_free; + } + + VHOST_LOG_CONFIG(path, INFO, "VDUSE max queue pairs: %u\n", max_queue_pairs); + total_queues = max_queue_pairs * 2; + + if (max_queue_pairs == 1) + features &= ~(RTE_BIT64(VIRTIO_NET_F_CTRL_VQ) | RTE_BIT64(VIRTIO_NET_F_MQ)); + else + total_queues += 1; /* Includes ctrl queue */ + + vnet_config.max_virtqueue_pairs = max_queue_pairs; memset(dev_config, 0, sizeof(struct vduse_dev_config)); strncpy(dev_config->name, name, VDUSE_NAME_MAX - 1); dev_config->device_id = VIRTIO_ID_NET; dev_config->vendor_id = 0; - dev_config->features = VDUSE_NET_SUPPORTED_FEATURES; - dev_config->vq_num = 2; + dev_config->features = features; + dev_config->vq_num = total_queues; dev_config->vq_align = sysconf(_SC_PAGE_SIZE); - dev_config->config_size = 0; + dev_config->config_size = sizeof(struct virtio_net_config); + memcpy(dev_config->config, &vnet_config, sizeof(vnet_config)); ret = ioctl(control_fd, VDUSE_CREATE_DEV, dev_config); if (ret < 0) { @@ -433,7 +493,7 @@ vduse_device_create(const char *path) dev->vduse_dev_fd = dev_fd; vhost_setup_virtio_net(dev->vid, true, true, true, true); - for (i = 0; i < 2; i++) { + for (i = 0; i < total_queues; i++) { struct vduse_vq_config vq_cfg = { 0 }; ret = alloc_vring_queue(dev, i); @@ -452,6 +512,8 @@ vduse_device_create(const char *path) } } + dev->cvq = dev->virtqueue[max_queue_pairs * 2]; + ret = fdset_add(&vduse.fdset, dev->vduse_dev_fd, vduse_events_handler, NULL, dev); if (ret) { VHOST_LOG_CONFIG(name, ERR, "Failed to add fd %d to vduse fdset\n", @@ -498,6 +560,13 @@ vduse_device_destroy(const char *path) if (vid == RTE_MAX_VHOST_DEVICE) return -1; + if (dev->cvq && dev->cvq->kickfd >= 0) { + fdset_del(&vduse.fdset, dev->cvq->kickfd); + fdset_pipe_notify(&vduse.fdset); + close(dev->cvq->kickfd); + dev->cvq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD; + } + fdset_del(&vduse.fdset, dev->vduse_dev_fd); fdset_pipe_notify(&vduse.fdset); From patchwork Thu May 25 16:15:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 127467 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DB01E42B9D; Thu, 25 May 2023 18:19:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 70D1642D98; Thu, 25 May 2023 18:17:22 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 061B442FB2 for ; Thu, 25 May 2023 18:17:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685031440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EeQ5GZJUghyhR8DpGMRGpQcu1hkqaGtpbWqcIorQNU8=; b=IW+TlBXmN8kr4RU4Mjn3sTE5ZyaqF3qmbVIPKMAfaBUjl8h23+aiPxtz1vcgzoGUFpuU+r 91jszBgnB8Bd7kpWJ+2h31QaRW6IaK9S2B2PyUjPlPvJz4MtuPgSZXfOMC9PqnzY9wFjyT p2CEZ3xVzegHbNL4Mzmsbb3SVxukNYU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-298-XVkEvN2TP_y8mc1n1UJl8A-1; Thu, 25 May 2023 12:17:17 -0400 X-MC-Unique: XVkEvN2TP_y8mc1n1UJl8A-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0FDCF802A55; Thu, 25 May 2023 16:17:17 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.23]) by smtp.corp.redhat.com (Postfix) with ESMTP id 87373C154D1; Thu, 25 May 2023 16:17:14 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com, cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com, eperezma@redhat.com, amorenoz@redhat.com, lulu@redhat.com Cc: Maxime Coquelin Subject: [PATCH 28/28] vhost: Add VDUSE device stop Date: Thu, 25 May 2023 18:15:50 +0200 Message-Id: <20230525161551.70002-29-maxime.coquelin@redhat.com> In-Reply-To: <20230525161551.70002-1-maxime.coquelin@redhat.com> References: <20230525161551.70002-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds VDUSE device stop and cleanup of its virtqueues. Signed-off-by: Maxime Coquelin --- doc/guides/rel_notes/release_23_07.rst | 6 +++ lib/vhost/vduse.c | 72 +++++++++++++++++++++++--- 2 files changed, 70 insertions(+), 8 deletions(-) diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst index fa889a5ee7..66ba9e25dd 100644 --- a/doc/guides/rel_notes/release_23_07.rst +++ b/doc/guides/rel_notes/release_23_07.rst @@ -60,6 +60,12 @@ New Features Introduced ``rte_vhost_driver_set_max_queue_num()`` to be able to limit the maximum number of supported queue pairs, required for VDUSE support. +* **Added VDUSE support into Vhost library + + VDUSE aims at implementing vDPA devices in userspace. It can be used as an + alternative to Vhost-user when using Vhost-vDPA, but also enable providing a + virtio-net netdev to the host when using Virtio-vDPA driver. + Removed Items ------------- diff --git a/lib/vhost/vduse.c b/lib/vhost/vduse.c index 699cfed9e3..f421b1cf4c 100644 --- a/lib/vhost/vduse.c +++ b/lib/vhost/vduse.c @@ -252,6 +252,44 @@ vduse_vring_setup(struct virtio_net *dev, unsigned int index) } } +static void +vduse_vring_cleanup(struct virtio_net *dev, unsigned int index) +{ + struct vhost_virtqueue *vq = dev->virtqueue[index]; + struct vduse_vq_eventfd vq_efd; + int ret; + + if (vq == dev->cvq && vq->kickfd >= 0) { + fdset_del(&vduse.fdset, vq->kickfd); + fdset_pipe_notify(&vduse.fdset); + } + + vq_efd.index = index; + vq_efd.fd = VDUSE_EVENTFD_DEASSIGN; + + ret = ioctl(dev->vduse_dev_fd, VDUSE_VQ_SETUP_KICKFD, &vq_efd); + if (ret) + VHOST_LOG_CONFIG(dev->ifname, ERR, "Failed to cleanup kickfd for VQ %u: %s\n", + index, strerror(errno)); + + close(vq->kickfd); + vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD; + + vring_invalidate(dev, vq); + + rte_free(vq->batch_copy_elems); + vq->batch_copy_elems = NULL; + + rte_free(vq->shadow_used_split); + vq->shadow_used_split = NULL; + + vq->enabled = false; + vq->ready = false; + vq->size = 0; + vq->last_used_idx = 0; + vq->last_avail_idx = 0; +} + static void vduse_device_start(struct virtio_net *dev) { @@ -304,6 +342,23 @@ vduse_device_start(struct virtio_net *dev) } } +static void +vduse_device_stop(struct virtio_net *dev) +{ + unsigned int i; + + VHOST_LOG_CONFIG(dev->ifname, INFO, "Stopping device...\n"); + + vhost_destroy_device_notify(dev); + + dev->flags &= ~VIRTIO_DEV_READY; + + for (i = 0; i < dev->nr_vring; i++) + vduse_vring_cleanup(dev, i); + + vhost_user_iotlb_flush_all(dev); +} + static void vduse_events_handler(int fd, void *arg, int *remove __rte_unused) { @@ -311,6 +366,7 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) struct vduse_dev_request req; struct vduse_dev_response resp; struct vhost_virtqueue *vq; + uint8_t old_status; int ret; memset(&resp, 0, sizeof(resp)); @@ -339,10 +395,15 @@ vduse_events_handler(int fd, void *arg, int *remove __rte_unused) case VDUSE_SET_STATUS: VHOST_LOG_CONFIG(dev->ifname, INFO, "\tnew status: 0x%08x\n", req.s.status); + old_status = dev->status; dev->status = req.s.status; - if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) - vduse_device_start(dev); + if ((old_status ^ dev->status) & VIRTIO_DEVICE_STATUS_DRIVER_OK) { + if (dev->status & VIRTIO_DEVICE_STATUS_DRIVER_OK) + vduse_device_start(dev); + else + vduse_device_stop(dev); + } resp.result = VDUSE_REQ_RESULT_OK; break; @@ -560,12 +621,7 @@ vduse_device_destroy(const char *path) if (vid == RTE_MAX_VHOST_DEVICE) return -1; - if (dev->cvq && dev->cvq->kickfd >= 0) { - fdset_del(&vduse.fdset, dev->cvq->kickfd); - fdset_pipe_notify(&vduse.fdset); - close(dev->cvq->kickfd); - dev->cvq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD; - } + vduse_device_stop(dev); fdset_del(&vduse.fdset, dev->vduse_dev_fd); fdset_pipe_notify(&vduse.fdset);