From patchwork Thu Feb 9 09:16:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 123520 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 485CC41C4D; Thu, 9 Feb 2023 10:17:19 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 31AA040DF8; Thu, 9 Feb 2023 10:17:19 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 1A5FC4067B for ; Thu, 9 Feb 2023 10:17:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675934237; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=UgxUrzaLCUr9tQA0/encekZmUuZ5dKkWYsnz5hXn5RA=; b=BpahCC5AoLSPxP+jX07nUFW4PZ4P4Uya9wCc2HQHR0k0+diY1MOSxLoIe8db6w165lE9vq p1Vm5oRn/TOGlpYEZoWOJRmE4zzrehcwuynu+iOf6ZviOF2gV+w0cTxHAXDqtEgqEtWyIC kFQ/BMXfhpnrMwWIUYAqUTE6Qr2OPlY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-651-XlwoXiQcOOyriEFQk-gQnQ-1; Thu, 09 Feb 2023 04:17:14 -0500 X-MC-Unique: XlwoXiQcOOyriEFQk-gQnQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 328CD8028B0; Thu, 9 Feb 2023 09:17:14 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.26]) by smtp.corp.redhat.com (Postfix) with ESMTP id CBA72492C3F; Thu, 9 Feb 2023 09:17:12 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, chenbo.xia@intel.com, david.marchand@redhat.com, eperezma@redhat.com, stephen@networkplumber.org Cc: Maxime Coquelin Subject: [PATCH 00/21] Add control queue & MQ support to Virtio-user vDPA Date: Thu, 9 Feb 2023 10:16:49 +0100 Message-Id: <20230209091710.485512-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This series introduces control queue support for Vhost-vDPA backend. This is a requirement to support multiqueue, but be usefull for other features like RSS for example. Since the Virtio-user layer of the Virtio PMD must handle some control messages, like the number of queue pairs to be used by the device, a shadow control queue is created at Virtio-user layer. Control messages from the regular Virtio control queue are still dequeues and handled if needed by the Virtio-user layer, and are then forwarded to the shadow control queue so that the physical vDPA device can handle them. This model is similar to the one adopted by the QEMU project. In order to avoid code duplication, virtqueue allocation and control queue message sending has been factored out of the Virtio layer to be reusable by the Virtio-user layer. Finally, in order to support vDPA hardware which may support large number of queues, last patch removes the 8 queue pairs limitation by dynamically allocating vring metadata. The series has been tested with Nvidia Cx-6 DX NIC with up to 16 queue pairs: # echo 0 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs # echo 0 > /sys/bus/pci/devices/0000\:3b\:00.1/sriov_numvfs # modprobe vhost_vdpa # modprobe mlx5_vdpa # echo 1 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs # echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind # devlink dev eswitch set pci/0000:3b:00.0 mode switchdev # echo 0000:3b:00.2 >/sys/bus/pci/drivers/mlx5_core/bind # vdpa dev add name vdpa0 mgmtdev pci/0000:3b:00.2 mac 00:11:22:33:44:03 max_vqp 16 # ulimit -l unlimited # dpdk-testpmd -l 0,2,4,6 --socket-mem 1024,0 --vdev 'virtio_user0,path=/dev/vhost-vdpa-0' --no-pci -n 3 -- --nb-cores=3 -i --rxq=16 --txq=16 Changes in v3: ============== - Trivial code simplifications (Eugenio) Changes in v2: ============== - Fix double spaces (Chenbo) - Get rid of uneeded gotos (Stephen) - Only allocate packed ring metadata if supported (Chenbo) - Rebased on top of main Maxime Coquelin (21): net/virtio: move CVQ code into a dedicated file net/virtio: introduce notify callback for control queue net/virtio: virtqueue headers alloc refactoring net/virtio: remove port ID info from Rx queue net/virtio: remove unused fields in Tx queue struct net/virtio: remove unused queue ID field in Rx queue net/virtio: remove unused Port ID in control queue net/virtio: move vring memzone to virtqueue struct net/virtio: refactor indirect desc headers init net/virtio: alloc Rx SW ring only if vectorized path net/virtio: extract virtqueue init from virtio queue init net/virtio-user: fix device starting failure handling net/virtio-user: simplify queues setup net/virtio-user: use proper type for number of queue pairs net/virtio-user: get max number of queue pairs from device net/virtio-user: allocate shadow control queue net/virtio-user: send shadow virtqueue info to the backend net/virtio-user: add new callback to enable control queue net/virtio-user: forward control messages to shadow queue net/virtio-user: advertize control VQ support with vDPA net/virtio-user: remove max queues limitation drivers/net/virtio/meson.build | 1 + drivers/net/virtio/virtio.h | 6 - drivers/net/virtio/virtio_cvq.c | 229 +++++++++ drivers/net/virtio/virtio_cvq.h | 127 +++++ drivers/net/virtio/virtio_ethdev.c | 472 +----------------- drivers/net/virtio/virtio_rxtx.c | 47 +- drivers/net/virtio/virtio_rxtx.h | 31 +- drivers/net/virtio/virtio_rxtx_packed.c | 3 +- drivers/net/virtio/virtio_rxtx_simple.c | 3 +- drivers/net/virtio/virtio_rxtx_simple.h | 7 +- .../net/virtio/virtio_rxtx_simple_altivec.c | 4 +- drivers/net/virtio/virtio_rxtx_simple_neon.c | 4 +- drivers/net/virtio/virtio_rxtx_simple_sse.c | 4 +- drivers/net/virtio/virtio_user/vhost.h | 1 + drivers/net/virtio/virtio_user/vhost_vdpa.c | 16 +- .../net/virtio/virtio_user/virtio_user_dev.c | 305 +++++++++-- .../net/virtio/virtio_user/virtio_user_dev.h | 30 +- drivers/net/virtio/virtio_user_ethdev.c | 49 +- drivers/net/virtio/virtqueue.c | 346 ++++++++++++- drivers/net/virtio/virtqueue.h | 127 +---- 20 files changed, 1066 insertions(+), 746 deletions(-) create mode 100644 drivers/net/virtio/virtio_cvq.c create mode 100644 drivers/net/virtio/virtio_cvq.h