Message ID | 20201218073851.93609-1-chenbo.xia@intel.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6FF1CA09F6; Fri, 18 Dec 2020 08:54:11 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B6DF4CA2C; Fri, 18 Dec 2020 08:54:09 +0100 (CET) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id C4126CA28 for <dev@dpdk.org>; Fri, 18 Dec 2020 08:54:07 +0100 (CET) IronPort-SDR: Rdi0LfbCvpgA8sSNlVnV72BsIu9qD7ipzvI1RodpTtzBzzynTdqWzhlckHGOFt7zBcbxbhe4HC iFzKUpAUaOeQ== X-IronPort-AV: E=McAfee;i="6000,8403,9838"; a="260126129" X-IronPort-AV: E=Sophos;i="5.78,429,1599548400"; d="scan'208";a="260126129" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Dec 2020 23:54:05 -0800 IronPort-SDR: aRLOcast32SWd1BWIoUP/BU2T3BuSQviF8PmQmYsAjA8WcAfaNRUS8Oqh+fxXrW+EgPLu3UKB/ dDkglBIIDRKQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,429,1599548400"; d="scan'208";a="340270266" Received: from npg-dpdk-virtio-xiachenbo-nw.sh.intel.com ([10.67.119.123]) by fmsmga008.fm.intel.com with ESMTP; 17 Dec 2020 23:54:03 -0800 From: Chenbo Xia <chenbo.xia@intel.com> To: dev@dpdk.org, thomas@monjalon.net, david.marchand@redhat.com Cc: stephen@networkplumber.org, cunming.liang@intel.com, xiuchun.lu@intel.com, miao.li@intel.com, jingjing.wu@intel.com Date: Fri, 18 Dec 2020 15:38:42 +0800 Message-Id: <20201218073851.93609-1-chenbo.xia@intel.com> X-Mailer: git-send-email 2.17.1 Subject: [dpdk-dev] [PATCH 0/9] Introduce vfio-user library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
Introduce vfio-user library
|
|
Message
Chenbo Xia
Dec. 18, 2020, 7:38 a.m. UTC
This series enables DPDK to be an alternative I/O device emulation library of building virtualized devices in separate processes outside QEMU. It introduces a new library for device emulation (librte_vfio_user). *librte_vfio_user* library is an implementation of VFIO-over-socket[1] (also known as vfio-user) which is a protocol that allows a device to be virtualized in a separate process outside of QEMU. Background & Motivation ----------------------- The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate IO services from QEMU[2]. Vfio-user essentially implements the VFIO device model presented to the user process by a set of messages over a unix-domain socket. The main difference between application using vfio-user and application using vfio kernel module is that device manipulation is based on socket messages for vfio-user but system calls for vfio kernel module. The vfio-user devices consist of a generic VFIO device type, living in QEMU, which is called the client[3], and the core device implementation (emulated device), living outside of QEMU, which is called the server. With emulated devices removed from QEMU enabled by vfio-user implementation, other places should be introduced to accommodate virtualized/emulated device. This series introduces vfio-user support in DPDK to enable DPDK as one of the living places for emulated device except QEMU. This series introduce the server and client implementation of vfio-user protocol. The server plays the role as emulated devices and the client is the device consumer. With this implementation, DPDK will be enabled to be both device provider and consumer. Design overview --------------- +--------------+ +--------------+ | +----------+ | | +----------+ | | | Generic | | | | Emulated | | | | vfio-dev | | | | device | | | +----------+ | | +----|-----+ | | +----------+ | | +----|-----+ | | | vfio-user| | | | vfio-user| | | | client | |<--->| | server | | | +----------+ | | +----------+ | | QEMU/DPDK | | DPDK | +--------------+ +--------------+ - Generic vfio-dev. It is the generic vfio framework in vfio applications like QEMU or DPDK. Applications can keep the most of vfio device management and plug in a vfio-user device type. Note that in current implementation, we have not yet integrated client vfio-user into kernel vfio in DPDK but it is viable and good to do so. - vfio-user client. For DPDK, it is part of librte_vfio_user implementation to provide ways to manipulate a vfio-user based emulated devices. This manipulation is very similar with kernel vfio (i.e., syscalls like ioctl, mmap and pread/pwrite). It is a base for vfio-user device consumer. - vfio-user server. It is server part of librte_vfio_user. It provides ways to emulate your own device. A device provider could only care about device layout that VFIO defines but does not need to know how it communicates with vfio-user client. - Emulated device. It is emulated device of any type (e.g., network, crypto and etc.). References ---------- [1]: https://patchew.org/QEMU/20201130161229.23164-1-thanos.makatos@nutanix.com/ [2]: https://wiki.qemu.org/Features/MultiProcessQEMU [3]: https://github.com/oracle/qemu/tree/vfio-user-v0.2 Chenbo Xia (9): lib: introduce vfio-user library vfio_user: implement lifecycle related APIs vfio_user: implement device and region related APIs vfio_user: implement DMA table and socket address API vfio_user: implement interrupt related APIs vfio_user: add client APIs of device attach/detach vfio_user: add client APIs of DMA/IRQ/region test/vfio_user: introduce functional test doc: add vfio-user library guide MAINTAINERS | 4 + app/test/meson.build | 4 + app/test/test_vfio_user.c | 646 ++++++++++ doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/vfio_user_lib.rst | 215 ++++ doc/guides/rel_notes/release_21_02.rst | 11 + lib/librte_vfio_user/meson.build | 11 + lib/librte_vfio_user/rte_vfio_user.h | 426 +++++++ lib/librte_vfio_user/version.map | 26 + lib/librte_vfio_user/vfio_user_base.c | 217 ++++ lib/librte_vfio_user/vfio_user_base.h | 109 ++ lib/librte_vfio_user/vfio_user_client.c | 691 ++++++++++ lib/librte_vfio_user/vfio_user_client.h | 25 + lib/librte_vfio_user/vfio_user_server.c | 1553 +++++++++++++++++++++++ lib/librte_vfio_user/vfio_user_server.h | 66 + lib/meson.build | 1 + 16 files changed, 4006 insertions(+) create mode 100644 app/test/test_vfio_user.c create mode 100644 doc/guides/prog_guide/vfio_user_lib.rst create mode 100644 lib/librte_vfio_user/meson.build create mode 100644 lib/librte_vfio_user/rte_vfio_user.h create mode 100644 lib/librte_vfio_user/version.map create mode 100644 lib/librte_vfio_user/vfio_user_base.c create mode 100644 lib/librte_vfio_user/vfio_user_base.h create mode 100644 lib/librte_vfio_user/vfio_user_client.c create mode 100644 lib/librte_vfio_user/vfio_user_client.h create mode 100644 lib/librte_vfio_user/vfio_user_server.c create mode 100644 lib/librte_vfio_user/vfio_user_server.h
Comments
Hello, On Fri, Dec 18, 2020 at 8:54 AM Chenbo Xia <chenbo.xia@intel.com> wrote: > *librte_vfio_user* library is an implementation of VFIO-over-socket[1] (also > known as vfio-user) which is a protocol that allows a device to be virtualized > in a separate process outside of QEMU. What is the status of the specification on QEMU side? Integrating an implementation in DPDK is premature until we have an agreed specification.
> Hello, > > On Fri, Dec 18, 2020 at 8:54 AM Chenbo Xia <chenbo.xia@intel.com> wrote: > > *librte_vfio_user* library is an implementation of VFIO-over-socket[1] > (also > > known as vfio-user) which is a protocol that allows a device to be virtualized > > in a separate process outside of QEMU. > > What is the status of the specification on QEMU side? > Integrating an implementation in DPDK is premature until we have an > agreed specification. We're in the process of reviewing the specification, the latest version (v7) is here: https://www.mail-archive.com/qemu-devel@nongnu.org/msg763207.html. We haven't had any reviews yet for that revision, IMO we're getting close. FYI John Johnson is implementing the relevant changes in multiprocess QEMU. John Levon, Swapnil, and I are implementing the server part in libvfio-user (formerly known as MUSER). We also have a mailing list now: https://lists.nongnu.org/mailman/listinfo/libvfio-user-devel. We've been working on integrating the two parts. Finally, Changpeng is implementing an NVMe controller in SPDK. We're trying to make it work with multiprocess QEMU and libvfio-user, we're very close.
On Fri, 18 Dec 2020 14:07:20 +0000 Thanos Makatos <thanos.makatos@nutanix.com> wrote: > > Hello, > > > > On Fri, Dec 18, 2020 at 8:54 AM Chenbo Xia <chenbo.xia@intel.com> wrote: > > > *librte_vfio_user* library is an implementation of VFIO-over-socket[1] > > (also > > > known as vfio-user) which is a protocol that allows a device to be virtualized > > > in a separate process outside of QEMU. > > > > What is the status of the specification on QEMU side? > > Integrating an implementation in DPDK is premature until we have an > > agreed specification. > > We're in the process of reviewing the specification, the latest version (v7) is here: https://www.mail-archive.com/qemu-devel@nongnu.org/msg763207.html. We haven't had any reviews yet for that revision, IMO we're getting close. > > FYI John Johnson is implementing the relevant changes in multiprocess QEMU. John Levon, Swapnil, and I are implementing the server part in libvfio-user (formerly known as MUSER). We also have a mailing list now: https://lists.nongnu.org/mailman/listinfo/libvfio-user-devel. We've been working on integrating the two parts. > > Finally, Changpeng is implementing an NVMe controller in SPDK. We're trying to make it work with multiprocess QEMU and libvfio-user, we're very close. > > Any status update? It has been 3 years. If not ready will change this patchset to state "awaiting upstream". I.e not ready until QEMU is ready
Hi Stephen, > -----Original Message----- > From: Stephen Hemminger <stephen@networkplumber.org> > Sent: Friday, June 30, 2023 12:10 AM > To: Thanos Makatos <thanos.makatos@nutanix.com> > Cc: David Marchand <david.marchand@redhat.com>; Xia, Chenbo > <chenbo.xia@intel.com>; dev <dev@dpdk.org>; Thomas Monjalon > <thomas@monjalon.net>; Liang, Cunming <cunming.liang@intel.com>; Lu, > Xiuchun <xiuchun.lu@intel.com>; Li, Miao <miao.li@intel.com>; Wu, Jingjing > <jingjing.wu@intel.com>; john.g.johnson@oracle.com; Stefan Hajnoczi > <stefanha@redhat.com>; Marc-Andre Lureau <marcandre.lureau@redhat.com>; > Maxime Coquelin <maxime.coquelin@redhat.com>; Swapnil Ingle > <swapnil.ingle@nutanix.com>; John Levon <john.levon@nutanix.com>; Liu, > Changpeng <changpeng.liu@intel.com>; Jagannathan Raman > <jag.raman@oracle.com>; Elena Ufimtseva <elena.ufimtseva@oracle.com> > Subject: Re: [dpdk-dev] [PATCH 0/9] Introduce vfio-user library > > On Fri, 18 Dec 2020 14:07:20 +0000 > Thanos Makatos <thanos.makatos@nutanix.com> wrote: > > > > Hello, > > > > > > On Fri, Dec 18, 2020 at 8:54 AM Chenbo Xia <chenbo.xia@intel.com> > wrote: > > > > *librte_vfio_user* library is an implementation of VFIO-over- > socket[1] > > > (also > > > > known as vfio-user) which is a protocol that allows a device to be > virtualized > > > > in a separate process outside of QEMU. > > > > > > What is the status of the specification on QEMU side? > > > Integrating an implementation in DPDK is premature until we have an > > > agreed specification. > > > > We're in the process of reviewing the specification, the latest version > (v7) is here: https://www.mail-archive.com/qemu- > devel@nongnu.org/msg763207.html. We haven't had any reviews yet for that > revision, IMO we're getting close. > > > > FYI John Johnson is implementing the relevant changes in multiprocess > QEMU. John Levon, Swapnil, and I are implementing the server part in > libvfio-user (formerly known as MUSER). We also have a mailing list now: > https://lists.nongnu.org/mailman/listinfo/libvfio-user-devel. We've been > working on integrating the two parts. > > > > Finally, Changpeng is implementing an NVMe controller in SPDK. We're > trying to make it work with multiprocess QEMU and libvfio-user, we're very > close. > > > > > > Any status update? It has been 3 years. > If not ready will change this patchset to state "awaiting upstream". > I.e not ready until QEMU is ready There is already a vfio-user lib in open source: https://github.com/nutanix/libvfio-user So for this lib, we may just use this libvfio-user instead of introducing new one in DPDK, but we will still submit patches about emulating devices in DPDK. Thanks, Chenbo