From patchwork Fri May 27 16:26:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 111983 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6EEB6A0545; Fri, 27 May 2022 18:27:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1BB23410DC; Fri, 27 May 2022 18:27:22 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mails.dpdk.org (Postfix) with ESMTP id EBD6C40E78 for ; Fri, 27 May 2022 18:27:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653668840; x=1685204840; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=WGonMCxKDZNdqdt/qo/rSzMpoAdySWZtuWvIJs/ETiw=; b=GPj9u1GCUh+1G2abfD1l5h4zvUJscyq0TWDNQdyGJBCsbSPqWPoRkjFa 2KoVUz2qi4WLsYa60w/GCbfoQO8JWvFv2Qkz+L/PqBIiPqYjLw/kunsG+ eoxiKOOQ/aMYOeQ2m9FR6WA+XuvDJNMANqjjV6efhpHtSImbv7E3kFSTd QGvMUmibfzGPhpF3SB48kS4XWDO4awgw4sMaAUPNRVsjoHjqj5xePG0Ch 9R7pWsah4d3aGbETD+hRpgzqyfSvuHf2gck1/NO5FZ7grp1O70j1r3NPO SFFfnWX52J+7kSh7TLEez5HPxM1b4hJp2mRZhjphVhCF7NPtkrsONyMtT w==; X-IronPort-AV: E=McAfee;i="6400,9594,10360"; a="272091266" X-IronPort-AV: E=Sophos;i="5.91,256,1647327600"; d="scan'208";a="272091266" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 May 2022 09:27:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,256,1647327600"; d="scan'208";a="705192862" Received: from silpixa00401385.ir.intel.com (HELO silpixa00401385.ger.corp.intel.com) ([10.237.222.171]) by orsmga004.jf.intel.com with ESMTP; 27 May 2022 09:27:10 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Maxime Coquelin , Chenbo Xia , Bruce Richardson Subject: [PATCH 1/2] doc/howto: rework section on virtio-user as exception path Date: Fri, 27 May 2022 17:26:58 +0100 Message-Id: <20220527162659.129022-1-bruce.richardson@intel.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch extensively reworks the howto guide on using virtio-user for exception packets. Changes include: * rename "exceptional path" to "exception path" * remove references to uio and just reference vfio-pci * simplify testpmd command-lines, giving a basic usage example first before adding on detail about checksum or TSO parameters * give a complete working example showing traffic flowing through the whole system from a testpmd loopback using the created TAP netdev * replace use of "ifconfig" with Linux standard "ip" command * other general rewording. Signed-off-by: Bruce Richardson --- .../howto/virtio_user_as_exceptional_path.rst | 159 +++++++++++------- 1 file changed, 100 insertions(+), 59 deletions(-) diff --git a/doc/guides/howto/virtio_user_as_exceptional_path.rst b/doc/guides/howto/virtio_user_as_exceptional_path.rst index ec021af399..0709eeff10 100644 --- a/doc/guides/howto/virtio_user_as_exceptional_path.rst +++ b/doc/guides/howto/virtio_user_as_exceptional_path.rst @@ -3,116 +3,157 @@ .. _virtio_user_as_exceptional_path: -Virtio_user as Exceptional Path -=============================== +Virtio_user as Exception Path +============================= -The virtual device, virtio-user, was originally introduced with vhost-user -backend, as a high performance solution for IPC (Inter-Process Communication) +.. note:: + + This solution is only applicable to Linux systems. + +The virtual device, virtio-user, was originally introduced with the vhost-user +backend as a high performance solution for IPC (Inter-Process Communication) and user space container networking. -Virtio_user with vhost-kernel backend is a solution for exceptional path, -such as KNI which exchanges packets with kernel networking stack. This -solution is very promising in: +Beyond this originally intended use, virtio-user can be used in conjunction with the vhost-kernel +backend as a solution for dealing with exception path packets which need to be injected into the +Linux kernel for processing there. +In this regard, virtio-user and vhost in kernel space are an alternative to DPDK KNI for +transferring packets between a DPDK packet processing application and the kernel stack. + +This solution has a number of advantages over alternatives such as KNI: * Maintenance All kernel modules needed by this solution, vhost and vhost-net (kernel), - are upstreamed and extensively used kernel module. + are upstreamed and extensively used. * Features - vhost-net is born to be a networking solution, which has lots of networking - related features, like multi queue, tso, multi-seg mbuf, etc. + vhost-net is designed to be a networking solution, and, as such, has lots of networking + related features, such as multi queue support, TSO, multi-segment buffer support, etc. * Performance - similar to KNI, this solution would use one or more kthreads to - send/receive packets to/from user space DPDK applications, which has little - impact on user space polling thread (except that it might enter into kernel - space to wake up those kthreads if necessary). + similar to KNI, this solution would uses one or more kthreads to + send/receive packets to/from user space DPDK applications, which minimises the impact + on the polling DPDK threads. -The overview of an application using virtio-user as exceptional path is shown +The overview of an application using virtio-user as exception path is shown in :numref:`figure_virtio_user_as_exceptional_path`. .. _figure_virtio_user_as_exceptional_path: .. figure:: img/virtio_user_as_exceptional_path.* - Overview of a DPDK app using virtio-user as exceptional path + Overview of a DPDK app using virtio-user as exception path + +Example Usage With Testpmd +--------------------------- -Sample Usage ------------- +.. note:: + + These instruction assume that the vhost/vhost-net kernel modules are available and have already + been loaded into the running kernel. + It also assumes that the DPDK virtio driver has not been disabled in the DPDK build. -As a prerequisite, the vhost/vhost-net kernel CONFIG should be chosen before -compiling the kernel and those kernel modules should be inserted. +To run a simple test of virtio-user as exception path using testpmd: -#. Compile DPDK and bind a physical NIC to igb_uio/uio_pci_generic/vfio-pci. +#. Compile DPDK and bind a NIC to vfio-pci as documented in :ref:`linux_gsg_linux_drivers`. - This physical NIC is for communicating with outside. + This physical NIC is for communicating with the outside world, + and serves as a packet source in this example. -#. Run testpmd. +#. Run testpmd to forward packets from NIC to kernel, + passing in a suitable list of logical cores to run on (``-l``.parameter), + and optionally the PCI address of the physical NIC to use (``-a`` parameter). + The virtio-user device for interfacing to the kernel is specified via a ``-vdev`` argument, + taking the parameters described below. .. code-block:: console - $(testpmd) -l 2-3 -n 4 \ - --vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024 \ - -- -i --tx-offloads=0x0000002c --enable-lro \ - --txd=1024 --rxd=1024 + /path/to/dpdk-testpmd -l -a \ + --vdev=virtio_user0,path=/dev/vhost-net,queues=1,queue_size=1024 - This command runs testpmd with two ports, one physical NIC to communicate - with outside, and one virtio-user to communicate with kernel. + * ``path`` -* ``--enable-lro`` + The path to the kernel vhost-net device. - This is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and - VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from kernel can be - transmitted to DPDK application and further TSOed by physical NIC. + * ``queue_size`` -* ``queue_size`` + 256 by default. To avoid shortage of descriptors, we can increase it to 1024. - 256 by default. To avoid shortage of descriptors, we can increase it to 1024. + * ``queues`` -* ``queues`` + Number of virt-queues. Each queue will be served by a kthread. - Number of multi-queues. Each queue will be served by a kthread. For example: +#. Once testpmd is running, a new network interface - called ``tap0`` by default - + will be present on the system. + This should be configured with an IP address and then enabled for use: .. code-block:: console - $(testpmd) -l 2-3 -n 4 \ - --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 \ - -- -i --tx-offloads=0x0000002c --enable-lro \ - --txq=2 --rxq=2 --txd=1024 --rxd=1024 + ip addr add 192.168.1.1/24 dev tap0 + ip link set dev tap0 up -#. Enable Rx checksum offloads in testpmd: +#. To observe packet forwarding through the kernel, + a second testpmd instance can be run on the system, + taking packets from the kernel using an ``af_packet`` socket on the ``tap0`` interface. - .. code-block:: console + .. code-block:: console - (testpmd) port stop 0 - (testpmd) port config 0 rx_offload tcp_cksum on - (testpmd) port config 0 rx_offload udp_cksum on - (testpmd) port start 0 + /path/to/dpdk-testpmd -l --vdev=net_af_packet0,iface=tap0 --in-memory --no-pci -#. Start testpmd: + When running this instance, + we can use ``--in-memory`` flag to avoid hugepage naming conflicts with the previous instance, + and we also use ``--no-pci`` flag to only use the ``af_packet`` interface for all traffic forwarding. - .. code-block:: console +#. Running traffic into the system through the NIC should see that traffic returned back again, + having been forwarded through both testpmd instances. + This can be confirmed by checking the testpmd statistics on testpmd exit. - (testpmd) start +For more advanced use of virtio-user with testpmd in this scenario, +some other more advanced options may also be used. +For example: -#. Configure IP address and start tap: +* ``--tx-offloads=0x02c`` + + This testpmd option enables TX offloads for UDP and TCP checksum on transmit, + as well as TCP TSO support. + The list of the offload flag values can be seen in header `rte_ethdev.h + `_. + +* ``--enable-lro`` + + This testpmd option is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and + VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from the kernel can be + transmitted to the DPDK application and further TSOed by physical NIC. + If unsupported by the physical NIC, errors may be reported by testpmd with this option. + + +* Enabling Rx checksum offloads for physical port: + + Within testpmd, you can enable and disable offloads on a per-port basis, + rather than enabling them for both ports. + For the physical NIC, it may be desirable to enable checksum offload on packet RX. + This may be done as below, if testpmd is run with ``-i`` flag for interactive mode. .. code-block:: console - ifconfig tap0 1.1.1.1/24 up + testpmd> port stop 0 + testpmd> port config 0 rx_offload tcp_cksum on + testpmd> port config 0 rx_offload udp_cksum on + testpmd> port start 0 -.. note:: +* Multiple queue support - The tap device will be named tap0, tap1, etc, by kernel. + Better performance may be achived by using multiple queues, + so that multiple kernel threads are handling the traffic on the kernel side. + For example, to use 2 queues on both NIC and virtio ports, + while also enabling TX offloads and LRO support: -Then, all traffic from physical NIC can be forwarded into kernel stack, and all -traffic on the tap0 can be sent out from physical NIC. + .. code-block:: console -Limitations ------------ + /path/to/dpdk-testpmd --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 -- \ + -i --tx-offloads=0x002c --enable-lro --txq=2 --rxq=2 --txd=1024 --rxd=1024 -This solution is only available on Linux systems. From patchwork Fri May 27 16:26:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 111984 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE583A0545; Fri, 27 May 2022 18:27:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 355A14282A; Fri, 27 May 2022 18:27:24 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mails.dpdk.org (Postfix) with ESMTP id E24D140E78 for ; Fri, 27 May 2022 18:27:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653668841; x=1685204841; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zX4uj8IGsM/mMGH7s39KFkLVPV+vOetK1nqRMTZDP6I=; b=Oi0GwyUQE15r+SZ/KlEb6pwoBxo0f19N+iem9m9ILLFtbFbPza1Y/i+f Rck/xjQONw2gZzeSuTjzuU7UWBoFhwLlXKmVm4plHD9LZdXur5fADGA56 DJqu6YntZ6vsSODo8uWgVp222sjcEvJbF7G2RL14HdvnCRzftSabyNmv2 MMGTUBoLbyQO5IQs/DYmOPZo1pUnxwznbaIgqL2t0ayA/rDENC4PPzL8t IFZyHYzttzCMNoUn4OeTUPfPsSlYcwk1i73F1YXez44M/JOQL8OQT0DHe sIE0VAFRv4uxENZEIYgA+HFODqj2BttZp4/j8Tcz51kpsQTq7RGJdId/7 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10360"; a="272091276" X-IronPort-AV: E=Sophos;i="5.91,256,1647327600"; d="scan'208";a="272091276" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 May 2022 09:27:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,256,1647327600"; d="scan'208";a="705192878" Received: from silpixa00401385.ir.intel.com (HELO silpixa00401385.ger.corp.intel.com) ([10.237.222.171]) by orsmga004.jf.intel.com with ESMTP; 27 May 2022 09:27:15 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Maxime Coquelin , Chenbo Xia , Bruce Richardson Subject: [PATCH 2/2] doc/howto: add code example to virtio-user exception path doc Date: Fri, 27 May 2022 17:26:59 +0100 Message-Id: <20220527162659.129022-2-bruce.richardson@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220527162659.129022-1-bruce.richardson@intel.com> References: <20220527162659.129022-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The HOWTO guide for using virtio-user as an exception path to the kernel only provided an example of how testpmd may be used for that purpose. However, a real application wanting to use virtio-user as exception path would likely want to create such devices from code within the app itself. Therefore, we update the doc with instructions and a code snippet showing how this may be done. Signed-off-by: Bruce Richardson --- .../howto/virtio_user_as_exceptional_path.rst | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/doc/guides/howto/virtio_user_as_exceptional_path.rst b/doc/guides/howto/virtio_user_as_exceptional_path.rst index 0709eeff10..f4bd909458 100644 --- a/doc/guides/howto/virtio_user_as_exceptional_path.rst +++ b/doc/guides/howto/virtio_user_as_exceptional_path.rst @@ -157,3 +157,58 @@ For example: /path/to/dpdk-testpmd --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 -- \ -i --tx-offloads=0x002c --enable-lro --txq=2 --rxq=2 --txd=1024 --rxd=1024 + +Creating Virtio-User Ports within an Application +------------------------------------------------ + +To use virtio-user ports within an application, +it is not necessary to explicitly initialize those ports using EAL arguments at startup. +Instead, one can use the generic EAL API +`rte_eal_hotplug_add `_ +function to create a new instance at startup. +For example, to create a basic virtio-user port, the following code could be used: + +.. code-block:: C + + rte_eal_hotplug_add("vdev", "virtio_user0", "path=/dev/vhost-net"); + +A fuller code example is shown below, where a virtio-user port, and hence kernel netdev, +is created for each NIC port discovered by DPDK. +Each virtio-user port is given the MAC address of its matching physical port +(assuming app was run without vdev args on commandline, so all ports auto-discovered are HW ones). +These new virtio-user netdevs will appear in the kernel port listings as ``virtio_user0``, +``virtio_user1``, etc., +based on the names passed in as ``iface=`` via the ``portargs`` parameter. + +.. code-block:: C + + nb_ports = rte_eth_dev_count_avail(); + + /* Create a vhost_user port for each physical port */ + unsigned port_count = 0; + RTE_ETH_FOREACH_DEV(portid) { + char portname[32]; + char portargs[256]; + struct rte_ether_addr addr = {0}; + + /* don't create virtio_user ports for other virtio_user ports */ + if (++port_count > nb_ports) + break; + + /* get mac address of physical port to use as mac of virtio_user port */ + rte_eth_macaddr_get(portid, &addr); + + /* set the name and arguments */ + snprintf(portname, sizeof(portname), "virtio_user%u", portid); + snprintf(portargs, sizeof(portargs), + "path=/dev/vhost-net,queues=1,queue_size=%u,iface=%s,mac=" RTE_ETHER_ADDR_PRT_FMT, + RX_RING_SIZE, portname, RTE_ETHER_ADDR_BYTES(&addr)); + + /* add the vdev for virtio_user */ + if (rte_eal_hotplug_add("vdev", portname, portargs) < 0) + rte_exit(EXIT_FAILURE, "Cannot create paired port for port %u\n", portid); + + } + +Once these virtio-user ports have been created in the loop, all ports, both physical and virtual, +may be initialized and used as normal in the application.