[v6,00/10] vhost: support inflight share memory protocol feature

  This patches introduces two new messages VHOST_USER_GET_INFLIGHT_FD
and VHOST_USER_SET_INFLIGHT_FD to support transferring a shared
buffer between qemu and backend for backend reconnection.
Now It can both support split and packed ring. The example code show
how these APIs work. The test has passed.

How to test the example:
1, Qemu need a patch.
https://patchwork.kernel.org/patch/10861411/
it also needs some manual modifications:
we should confirm that before we send get inflight we have already
sent the set features but it seems Qemu didn't do like this. So we
manually revise this, we can add below code in vhost_dev_get_inflight
    int r;

    r = vhost_dev_set_features(dev, dev->log_enabled);
    if (r < 0) {
        return -errno;
    }
before get_inflight_fd.
2, Guest OS version >= 5.0
3, run the example
4, run the qemu with vhost-user-blk-pci.
 eg:
        -chardev socket,id=spdk_vhost_blk0,reconnect=1,path=xxxx\
        -device vhost-user-blk-pci,ring_packed=on,chardev=spdk_vhost_blk0,num-queues=1\
5, run fio in the guest
6, kill the example and run again.
7, the fio in the guest should continue run without errors.

JinYu (10):
  vhost: add the inflight description
  vhost: add packed ring
  vhost: add the inflight structure
  vhost: add two new messages to support a shared buffer
  vhost: checkout and cleanup the resubmit inflight information
  vhost: add the APIs to operate inflight ring
  vhost: add APIs for user getting inflight ring
  vhost: fix vring functions to support packed ring
  vhost: add an API for judging vq format
  vhost: add vhost-user-blk example which support inflight

 examples/vhost_blk/Makefile           |   67 ++
 examples/vhost_blk/blk.c              |  125 +++
 examples/vhost_blk/blk_spec.h         |   95 ++
 examples/vhost_blk/meson.build        |   20 +
 examples/vhost_blk/vhost_blk.c        | 1313 +++++++++++++++++++++++++
 examples/vhost_blk/vhost_blk.h        |  116 +++
 examples/vhost_blk/vhost_blk_compat.c |  195 ++++
 lib/librte_vhost/rte_vhost.h          |  250 ++++-
 lib/librte_vhost/vhost.c              |  396 +++++++-
 lib/librte_vhost/vhost.h              |   16 +
 lib/librte_vhost/vhost_user.c         |  439 +++++++++
 lib/librte_vhost/vhost_user.h         |   12 +-
 12 files changed, 3032 insertions(+), 12 deletions(-)
 create mode 100644 examples/vhost_blk/Makefile
 create mode 100644 examples/vhost_blk/blk.c
 create mode 100644 examples/vhost_blk/blk_spec.h
 create mode 100644 examples/vhost_blk/meson.build
 create mode 100644 examples/vhost_blk/vhost_blk.c
 create mode 100644 examples/vhost_blk/vhost_blk.h
 create mode 100644 examples/vhost_blk/vhost_blk_compat.c
  

Hi Jin,

I see in this patch set there are several callings rte_compiler_barrier() which is sufficient for x86 but not for aarch64/ppc and other weaker memory model architectures.
EAL has defined coherent memory io barriers(I think this should be the right barrier) for mutli-arch support. Could you change these barriers to rte_cio_r/wmb barriers?
http://git.dpdk.org/dpdk/tree/lib/librte_eal/common/include/generic/rte_atomic.h#n110

Best Regards,
Gavin

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of JinYu
> Sent: Thursday, August 29, 2019 10:12 PM
> To: dev@dpdk.org
> Cc: changpeng.liu@intel.com; maxime.coquelin@redhat.com;
> tiwei.bie@intel.com; zhihong.wang@intel.com; JinYu <jin.yu@intel.com>
> Subject: [dpdk-dev] [PATCH v6 00/10] vhost: support inflight share memory
> protocol feature
>
> This patches introduces two new messages
> VHOST_USER_GET_INFLIGHT_FD
> and VHOST_USER_SET_INFLIGHT_FD to support transferring a shared
> buffer between qemu and backend for backend reconnection.
> Now It can both support split and packed ring. The example code show
> how these APIs work. The test has passed.
>
> How to test the example:
> 1, Qemu need a patch.
> https://patchwork.kernel.org/patch/10861411/
> it also needs some manual modifications:
> we should confirm that before we send get inflight we have already
> sent the set features but it seems Qemu didn't do like this. So we
> manually revise this, we can add below code in vhost_dev_get_inflight
>     int r;
>
>     r = vhost_dev_set_features(dev, dev->log_enabled);
>     if (r < 0) {
>         return -errno;
>     }
> before get_inflight_fd.
> 2, Guest OS version >= 5.0
> 3, run the example
> 4, run the qemu with vhost-user-blk-pci.
>  eg:
>         -chardev socket,id=spdk_vhost_blk0,reconnect=1,path=xxxx\
>         -device vhost-user-blk-
> pci,ring_packed=on,chardev=spdk_vhost_blk0,num-queues=1\
> 5, run fio in the guest
> 6, kill the example and run again.
> 7, the fio in the guest should continue run without errors.
>
> JinYu (10):
>   vhost: add the inflight description
>   vhost: add packed ring
>   vhost: add the inflight structure
>   vhost: add two new messages to support a shared buffer
>   vhost: checkout and cleanup the resubmit inflight information
>   vhost: add the APIs to operate inflight ring
>   vhost: add APIs for user getting inflight ring
>   vhost: fix vring functions to support packed ring
>   vhost: add an API for judging vq format
>   vhost: add vhost-user-blk example which support inflight
>
>  examples/vhost_blk/Makefile           |   67 ++
>  examples/vhost_blk/blk.c              |  125 +++
>  examples/vhost_blk/blk_spec.h         |   95 ++
>  examples/vhost_blk/meson.build        |   20 +
>  examples/vhost_blk/vhost_blk.c        | 1313 +++++++++++++++++++++++++
>  examples/vhost_blk/vhost_blk.h        |  116 +++
>  examples/vhost_blk/vhost_blk_compat.c |  195 ++++
>  lib/librte_vhost/rte_vhost.h          |  250 ++++-
>  lib/librte_vhost/vhost.c              |  396 +++++++-
>  lib/librte_vhost/vhost.h              |   16 +
>  lib/librte_vhost/vhost_user.c         |  439 +++++++++
>  lib/librte_vhost/vhost_user.h         |   12 +-
>  12 files changed, 3032 insertions(+), 12 deletions(-)
>  create mode 100644 examples/vhost_blk/Makefile
>  create mode 100644 examples/vhost_blk/blk.c
>  create mode 100644 examples/vhost_blk/blk_spec.h
>  create mode 100644 examples/vhost_blk/meson.build
>  create mode 100644 examples/vhost_blk/vhost_blk.c
>  create mode 100644 examples/vhost_blk/vhost_blk.h
>  create mode 100644 examples/vhost_blk/vhost_blk_compat.c
>
> --
> 2.17.2

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
  

On 8/29/19 4:12 PM, JinYu wrote:
> This patches introduces two new messages VHOST_USER_GET_INFLIGHT_FD
> and VHOST_USER_SET_INFLIGHT_FD to support transferring a shared
> buffer between qemu and backend for backend reconnection.
> Now It can both support split and packed ring. The example code show
> how these APIs work. The test has passed.
> 
> How to test the example:
> 1, Qemu need a patch.
> https://patchwork.kernel.org/patch/10861411/
> it also needs some manual modifications:
> we should confirm that before we send get inflight we have already
> sent the set features but it seems Qemu didn't do like this. So we
> manually revise this, we can add below code in vhost_dev_get_inflight
>     int r;
> 
>     r = vhost_dev_set_features(dev, dev->log_enabled);
>     if (r < 0) {
>         return -errno;
>     }

Are you working on fixing that upstream?
What happens if we don't do these modifications and apply your series?
Would it be considered as a regression?
  

> -----Original Message-----
> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> Sent: Saturday, September 7, 2019 12:36 AM
> To: Yu, Jin <jin.yu@intel.com>; dev@dpdk.org
> Cc: Liu, Changpeng <changpeng.liu@intel.com>; Bie, Tiwei
> <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
> Subject: Re: [PATCH v6 00/10] vhost: support inflight share memory protocol
> feature
> 
> 
> 
> On 8/29/19 4:12 PM, JinYu wrote:
> > This patches introduces two new messages
> VHOST_USER_GET_INFLIGHT_FD
> > and VHOST_USER_SET_INFLIGHT_FD to support transferring a shared
> buffer
> > between qemu and backend for backend reconnection.
> > Now It can both support split and packed ring. The example code show
> > how these APIs work. The test has passed.
> >
> > How to test the example:
> > 1, Qemu need a patch.
> > https://patchwork.kernel.org/patch/10861411/
> > it also needs some manual modifications:
> > we should confirm that before we send get inflight we have already
> > sent the set features but it seems Qemu didn't do like this. So we
> > manually revise this, we can add below code in vhost_dev_get_inflight
> >     int r;
> >
> >     r = vhost_dev_set_features(dev, dev->log_enabled);
> >     if (r < 0) {
> >         return -errno;
> >     }
> 
> Are you working on fixing that upstream?
> What happens if we don't do these modifications and apply your series?
> Would it be considered as a regression?
Hi  Maxime,
	Actually I didn't fix the upstream.
	Without these modifications, get_inflight_fd and set_inflight_fd can't confirm the dev packet format.
	So the inflight ring format may always be split.
Thanks.