Hi,
> -----Original Message-----
> From: Ye, Xiaolong
> Sent: Tuesday, September 25, 2018 8:07 PM
> To: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Bie,
> Tiwei <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; Rami Rosen
> <roszenrami@gmail.com>; Wang, Haiyue <haiyue.wang@intel.com>; Ye,
> Xiaolong <xiaolong.ye@intel.com>
> Subject: [PATCH v5 2/2] examples/vdpa: introduce a new sample for vDPA
>
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
>
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
> MAINTAINERS | 2 +
> doc/guides/rel_notes/release_18_11.rst | 8 +
> doc/guides/sample_app_ug/index.rst | 1 +
> doc/guides/sample_app_ug/vdpa.rst | 118 +++++++
> examples/Makefile | 2 +-
> examples/vdpa/Makefile | 32 ++
> examples/vdpa/main.c | 466 +++++++++++++++++++++++++
> examples/vdpa/meson.build | 16 +
> 8 files changed, 644 insertions(+), 1 deletion(-)
> create mode 100644 doc/guides/sample_app_ug/vdpa.rst
> create mode 100644 examples/vdpa/Makefile
> create mode 100644 examples/vdpa/main.c
> create mode 100644 examples/vdpa/meson.build
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
> F: examples/vhost_scsi/
> F: doc/guides/sample_app_ug/vhost_scsi.rst
> F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
>
> Vhost PMD
> M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst
> b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
> SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
> vdev_netvsc, tap, and failsafe drivers combination.
>
> +* **Add a new sample for vDPA**
> +
> + The vdpa sample application creates vhost-user sockets by using the
> + vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> + virtio ring compatible devices to serve virtio driver directly to enable
> + datapath acceleration. As vDPA driver can help to set up vhost datapath,
> + this application doesn't need to launch dedicated worker threads for vhost
> + enqueue/dequeue operations.
>
> API Changes
> -----------
> diff --git a/doc/guides/sample_app_ug/index.rst
> b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
> vhost
> vhost_scsi
> vhost_crypto
> + vdpa
> netmap_compatibility
> ip_pipeline
> test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst
> b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..a089393c0
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,118 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> + ./vdpa [EAL options] -- [--client] [--interactive|-i] or [--iface
> SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU
> needs
> + to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> + /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
> + (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> + internal cmds are supported:
> +
> + 1. help: show help message
> + 2. list: list all available vdpa devices
> + 3. create: create a new vdpa port with socket file and vdpa device address
> + 4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> + ./vdpa --log-level=9 -c 0x2 -n 4 --socket-mem 1024,1024 \
> + -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> + -- --interactive
DPDK has log level 1-8, setting the log-level=9 looks strange.
Here you can add description about the whitelisted device, since they are
virtio ring compatible devices.
> +
> +.. note::
> + We need to bind vfio-pci to VFs before running vdpa sample.
> +
> + * modprobe vfio-pci
> + * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> + vdpa> list
> + device id device address queue num supported features
> + 0 0000:06:00.3 1 0x5572362272
> + 1 0000:06:00.4 1 0x5572362272
> +
> + vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> + vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> + qemu-system-x86_64 -cpu host -enable-kvm \
> + <snip>
> + -mem-prealloc \
> + -chardev socket,id=char0,path=<socket_file created in above steps> \
> + -netdev type=vhost-user,id=vdpa,chardev=char0 \
> + -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
In order to leverage EPT, add parameter "page-per-vq=on" for the virtio-net-pci deivce.
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> + Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost
> backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume
> A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM
> on A,
> + in migration-listen mode:
> +
> +.. code-block:: console
> +
> + B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> + A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> + A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
> DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
> endif
> DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
> ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
> DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
> endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +
> +endif
> diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
> new file mode 100644
> index 000000000..7acda67c4
> --- /dev/null
> +++ b/examples/vdpa/main.c
> @@ -0,0 +1,466 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#include <getopt.h>
> +#include <signal.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#include <rte_ethdev.h>
> +#include <rte_malloc.h>
> +#include <rte_vhost.h>
> +#include <rte_vdpa.h>
> +#include <rte_pci.h>
> +
> +#include <cmdline_parse.h>
> +#include <cmdline_socket.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +
> +#define MAX_PATH_LEN 128
> +#define MAX_VDPA_SAMPLE_PORTS 1024
> +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
> +
[...]
> +};
> +
> +/* *** List all available vdpa devices *** */
> +struct cmd_list_result {
> + cmdline_fixed_string_t action;
> +};
> +
> +static void cmd_list_vdpa_devices_parsed(
> + __attribute__((unused)) void *parsed_result,
> + struct cmdline *cl,
> + __attribute__((unused)) void *data)
> +{
> + int did;
> + uint32_t queue_num;
> + uint64_t features;
> + struct rte_vdpa_device *vdev;
> + struct rte_pci_addr addr;
> +
> + cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported
> features\n");
> + for (did = 0; did < dev_total; did++) {
> + vdev = rte_vdpa_get_device(did);
> + if (!vdev)
> + continue;
> + if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
> + RTE_LOG(ERR, VDPA,
> + "failed to get vdpa queue number "
> + "for device id %d.\n", did);
> + continue;
> + }
> + if (vdev->ops->get_features(did, &features) < 0) {
> + RTE_LOG(ERR, VDPA,
> + "failed to get vdpa features "
> + "for device id %d.\n", did);
> + continue;
> + }
> + addr = vdev->addr.pci_addr;
> + cmdline_printf(cl,
> + "%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIu64
> "\n", did,
s/PRIu64/PRIx64
> + addr.domain, addr.bus, addr.devid,
> + addr.function, queue_num, features);
> + }
> +}
BRs,
Xiao
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
F: examples/vhost_scsi/
F: doc/guides/sample_app_ug/vhost_scsi.rst
F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
Vhost PMD
M: Maxime Coquelin <maxime.coquelin@redhat.com>
@@ -67,6 +67,14 @@ New Features
SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
vdev_netvsc, tap, and failsafe drivers combination.
+* **Add a new sample for vDPA**
+
+ The vdpa sample application creates vhost-user sockets by using the
+ vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+ virtio ring compatible devices to serve virtio driver directly to enable
+ datapath acceleration. As vDPA driver can help to set up vhost datapath,
+ this application doesn't need to launch dedicated worker threads for vhost
+ enqueue/dequeue operations.
API Changes
-----------
@@ -45,6 +45,7 @@ Sample Applications User Guides
vhost
vhost_scsi
vhost_crypto
+ vdpa
netmap_compatibility
ip_pipeline
test_pipeline
new file mode 100644
@@ -0,0 +1,118 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+ ./vdpa [EAL options] -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --client means running vdpa app in client mode, in the client mode, QEMU needs
+ to run as the server mode and take charge of socket file creation.
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+ /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+ (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+ internal cmds are supported:
+
+ 1. help: show help message
+ 2. list: list all available vdpa devices
+ 3. create: create a new vdpa port with socket file and vdpa device address
+ 4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+ ./vdpa --log-level=9 -c 0x2 -n 4 --socket-mem 1024,1024 \
+ -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
+ -- --interactive
+
+.. note::
+ We need to bind vfio-pci to VFs before running vdpa sample.
+
+ * modprobe vfio-pci
+ * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+ vdpa> list
+ device id device address queue num supported features
+ 0 0000:06:00.3 1 0x5572362272
+ 1 0000:06:00.4 1 0x5572362272
+
+ vdpa> create /tmp/vdpa-socket0 0000:06:00.3
+ vdpa> create /tmp/vdpa-socket1 0000:06:00.4
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+ qemu-system-x86_64 -cpu host -enable-kvm \
+ <snip>
+ -mem-prealloc \
+ -chardev socket,id=char0,path=<socket_file created in above steps> \
+ -netdev type=vhost-user,id=vdpa,chardev=char0 \
+ -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+ Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+ in migration-listen mode:
+
+.. code-block:: console
+
+ B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+ A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+ A: (qemu) info migrate
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
endif
DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
endif
new file mode 100644
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
new file mode 100644
@@ -0,0 +1,466 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+ char ifname[MAX_PATH_LEN];
+ int did;
+ int vid;
+ uint64_t flags;
+};
+
+struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+static int client_mode;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+ printf("Usage: %s [EAL options] -- "
+ " --interactive|-i: run in interactive mode.\n"
+ " --iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+ " --client: register a vhost-user socket as client mode.\n",
+ prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+ static const char *short_option = "i";
+ static struct option long_option[] = {
+ {"iface", required_argument, NULL, 0},
+ {"interactive", no_argument, &interactive, 1},
+ {"client", no_argument, &client_mode, 1},
+ {NULL, 0, 0, 0},
+ };
+ int opt, idx;
+ char *prgname = argv[0];
+
+ while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+ != EOF) {
+ switch (opt) {
+ case 'i':
+ printf("Interactive-mode selected\n");
+ interactive = 1;
+ break;
+ /* long options */
+ case 0:
+ if (strncmp(long_option[idx].name, "iface",
+ MAX_PATH_LEN) == 0) {
+ strncpy(iface, optarg, MAX_PATH_LEN);
+ printf("iface %s\n", iface);
+ }
+ if (!strcmp(long_option[idx].name, "interactive")) {
+ printf("Interactive-mode selected\n");
+ interactive = 1;
+ }
+ break;
+
+ default:
+ vdpa_usage(prgname);
+ return -1;
+ }
+ }
+
+ if (iface[0] == '\0' && interactive == 0) {
+ vdpa_usage(prgname);
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+data_init(void)
+{
+ dev_total = rte_vdpa_get_device_num();
+ if (dev_total <= 0) {
+ printf("No available vdpa device found\n");
+ return -1;
+ }
+ devcnt = 0;
+ interactive = 0;
+ memset(iface, 0, sizeof(iface));
+ memset(vports, 0, sizeof(vports));
+ return 0;
+}
+
+static int
+new_device(int vid)
+{
+ char ifname[MAX_PATH_LEN];
+ int i;
+
+ rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+ for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+ if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+ printf("\nnew port %s, did: %d\n",
+ ifname, vports[i].did);
+ vports[i].vid = vid;
+ break;
+ }
+ }
+
+ if (i >= MAX_VDPA_SAMPLE_PORTS)
+ return -1;
+
+ return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+ char ifname[MAX_PATH_LEN];
+ int i;
+
+ rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+ for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+ if (strcmp(ifname, vports[i].ifname) == 0) {
+ printf("\ndestroy port %s, did: %d\n",
+ ifname, vports[i].did);
+ break;
+ }
+ }
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+ .new_device = new_device,
+ .destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+ int ret;
+ char *socket_path = vport->ifname;
+ int did = vport->did;
+
+ if (client_mode)
+ vport->flags |= RTE_VHOST_USER_CLIENT;
+
+ if (access(socket_path, F_OK) != -1 && !client_mode) {
+ RTE_LOG(ERR, VDPA,
+ "%s exists, please remove it or specify another file and try again.\n",
+ socket_path);
+ return -1;
+ }
+ ret = rte_vhost_driver_register(socket_path, vport->flags);
+ if (ret != 0)
+ rte_exit(EXIT_FAILURE,
+ "register driver failed: %s\n",
+ socket_path);
+
+ ret = rte_vhost_driver_callback_register(socket_path,
+ &vdpa_sample_devops);
+ if (ret != 0)
+ rte_exit(EXIT_FAILURE,
+ "register driver ops failed: %s\n",
+ socket_path);
+
+ ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+ if (ret != 0)
+ rte_exit(EXIT_FAILURE,
+ "attach vdpa device failed: %s\n",
+ socket_path);
+
+ if (rte_vhost_driver_start(socket_path) < 0)
+ rte_exit(EXIT_FAILURE,
+ "start vhost driver failed: %s\n",
+ socket_path);
+ return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+ int ret;
+ char *socket_path = vport->ifname;
+
+ ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+ if (ret != 0)
+ RTE_LOG(ERR, VDPA,
+ "detach vdpa device failed: %s\n",
+ socket_path);
+
+ ret = rte_vhost_driver_unregister(socket_path);
+ if (ret != 0)
+ RTE_LOG(ERR, VDPA,
+ "Fail to unregister vhost driver for %s.\n",
+ socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+ int i;
+ for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+ if (vports[i].ifname[0] != '\0')
+ close_vdpa(&vports[i]);
+ }
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\nSignal %d received, preparing to exit...\n", signum);
+ vdpa_sample_quit();
+ exit(0);
+ }
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+ cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+ struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ cmdline_printf(
+ cl,
+ "\n"
+ "The following commands are currently available:\n\n"
+ "Control:\n"
+ " help : Show interactive instructions.\n"
+ " list : list all available vdpa devices.\n"
+ " create <socket file> <vdev addr> : create a new vdpa port.\n"
+ " quit : exit vdpa sample app.\n"
+ );
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+ TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+ .f = cmd_help_parsed,
+ .data = NULL,
+ .help_str = "show help",
+ .tokens = {
+ (void *)&cmd_help_help,
+ NULL,
+ },
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+ cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+ __attribute__((unused)) void *parsed_result,
+ struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ int did;
+ uint32_t queue_num;
+ uint64_t features;
+ struct rte_vdpa_device *vdev;
+ struct rte_pci_addr addr;
+
+ cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+ for (did = 0; did < dev_total; did++) {
+ vdev = rte_vdpa_get_device(did);
+ if (!vdev)
+ continue;
+ if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+ RTE_LOG(ERR, VDPA,
+ "failed to get vdpa queue number "
+ "for device id %d.\n", did);
+ continue;
+ }
+ if (vdev->ops->get_features(did, &features) < 0) {
+ RTE_LOG(ERR, VDPA,
+ "failed to get vdpa features "
+ "for device id %d.\n", did);
+ continue;
+ }
+ addr = vdev->addr.pci_addr;
+ cmdline_printf(cl,
+ "%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIu64 "\n", did,
+ addr.domain, addr.bus, addr.devid,
+ addr.function, queue_num, features);
+ }
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+ TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+ .f = cmd_list_vdpa_devices_parsed,
+ .data = NULL,
+ .help_str = "list all available vdpa devices",
+ .tokens = {
+ (void *)&cmd_action_list,
+ NULL,
+ },
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+ cmdline_fixed_string_t action;
+ cmdline_fixed_string_t socket_path;
+ cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+ struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ int did;
+ struct cmd_create_result *res = parsed_result;
+ struct rte_vdpa_dev_addr addr;
+
+ strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+ if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+ cmdline_printf(cl, "Unable to parse the given bdf.\n");
+ return;
+ }
+ addr.type = PCI_ADDR;
+ did = rte_vdpa_find_device_id(&addr);
+ if (did < 0) {
+ cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+ return;
+ }
+
+ vports[devcnt].did = did;
+
+ if (start_vdpa(&vports[devcnt]) == 0)
+ devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+ TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+ TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+ TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+ .f = cmd_create_vdpa_port_parsed,
+ .data = NULL,
+ .help_str = "create a new vdpa port",
+ .tokens = {
+ (void *)&cmd_action_create,
+ (void *)&cmd_socket_path,
+ (void *)&cmd_bdf,
+ NULL,
+ },
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+ cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+ struct cmdline *cl,
+ __attribute__((unused)) void *data)
+{
+ vdpa_sample_quit();
+ cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+ TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+ .f = cmd_quit_parsed,
+ .data = NULL,
+ .help_str = "quit: exit application",
+ .tokens = {
+ (void *)&cmd_quit_quit,
+ NULL,
+ },
+};
+cmdline_parse_ctx_t main_ctx[] = {
+ (cmdline_parse_inst_t *)&cmd_help,
+ (cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+ (cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+ (cmdline_parse_inst_t *)&cmd_quit,
+ NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+ char ch;
+ int i;
+ int ret;
+ struct cmdline *cl;
+
+ ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "eal init failed\n");
+ argc -= ret;
+ argv += ret;
+
+ data_init();
+
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ ret = parse_args(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+ if (interactive == 1) {
+ cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+ if (cl == NULL)
+ rte_panic("Cannot create cmdline instance\n");
+ cmdline_interact(cl);
+ cmdline_stdin_exit(cl);
+ } else {
+ for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+ i++) {
+ vports[i].did = i;
+ snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+ iface, i);
+
+ start_vdpa(&vports[i]);
+ }
+
+ printf("enter \'q\' to quit\n");
+ while (scanf("%c", &ch)) {
+ if (ch == 'q')
+ break;
+ while (ch != '\n') {
+ if (scanf("%c", &ch))
+ printf("%c", ch);
+ }
+ printf("enter \'q\' to quit\n");
+ }
+ vdpa_sample_quit();
+ }
+
+ return 0;
+}
new file mode 100644
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+ build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+ 'main.c'
+)
\ No newline at end of file