Cover Detail
Show a cover letter.
GET /api/covers/43822/?format=api
{ "id": 43822, "url": "http://patchwork.dpdk.org/api/covers/43822/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/cover/20180823165157.30001-1-maxime.coquelin@redhat.com/", "project": { "id": 1, "url": "http://patchwork.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20180823165157.30001-1-maxime.coquelin@redhat.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20180823165157.30001-1-maxime.coquelin@redhat.com", "date": "2018-08-23T16:51:47", "name": "[RFC,00/10] vhost: add postcopy live-migration support", "submitter": { "id": 512, "url": "http://patchwork.dpdk.org/api/people/512/?format=api", "name": "Maxime Coquelin", "email": "maxime.coquelin@redhat.com" }, "mbox": "http://patchwork.dpdk.org/project/dpdk/cover/20180823165157.30001-1-maxime.coquelin@redhat.com/mbox/", "series": [ { "id": 1039, "url": "http://patchwork.dpdk.org/api/series/1039/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/list/?series=1039", "date": "2018-08-23T16:51:47", "name": "vhost: add postcopy live-migration support", "version": 1, "mbox": "http://patchwork.dpdk.org/series/1039/mbox/" } ], "comments": "http://patchwork.dpdk.org/api/covers/43822/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id CC7952C38;\n\tThu, 23 Aug 2018 18:52:08 +0200 (CEST)", "from mx1.redhat.com (mx3-rdu2.redhat.com [66.187.233.73])\n\tby dpdk.org (Postfix) with ESMTP id D545998\n\tfor <dev@dpdk.org>; Thu, 23 Aug 2018 18:52:07 +0200 (CEST)", "from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id C70DA40241DD;\n\tThu, 23 Aug 2018 16:52:06 +0000 (UTC)", "from localhost.localdomain (ovpn-112-46.ams2.redhat.com\n\t[10.36.112.46])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 745812166BA1;\n\tThu, 23 Aug 2018 16:52:01 +0000 (UTC)" ], "From": "Maxime Coquelin <maxime.coquelin@redhat.com>", "To": "dev@dpdk.org, tiwei.bie@intel.com, zhihong.wang@intel.com,\n\tjfreimann@redhat.com", "Cc": "dgilbert@redhat.com,\n\tMaxime Coquelin <maxime.coquelin@redhat.com>", "Date": "Thu, 23 Aug 2018 18:51:47 +0200", "Message-Id": "<20180823165157.30001-1-maxime.coquelin@redhat.com>", "X-Scanned-By": "MIMEDefang 2.78 on 10.11.54.6", "X-Greylist": [ "Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.11.55.7]); Thu, 23 Aug 2018 16:52:06 +0000 (UTC)", "inspected by milter-greylist-4.5.16 (mx1.redhat.com\n\t[10.11.55.7]); \n\tThu, 23 Aug 2018 16:52:06 +0000 (UTC) for IP:'10.11.54.6'\n\tDOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com'\n\tHELO:'smtp.corp.redhat.com' FROM:'maxime.coquelin@redhat.com' RCPT:''" ], "Subject": "[dpdk-dev] [RFC 00/10] vhost: add postcopy live-migration support", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "This RFC adds support for postcopy live-migration.\n\nWith classic live-migration, the VM runs on source while its\ncontent is being migrated to destination. When pages already\nmigrated to destination are dirtied by the source, they get\ncopied until both source and destination memory converge.\nAt that time, the source is stopped and destination is\nstarted.\n\nWith postcopy live-migration, the VM is started on destination\nbefore all the memory has been migrated. When the VM tries to\naccess a page that haven't been migrated yet, a pagefault is\ntriggered, handled by userfaultfd which pauses the thread.\nA Qemu thread in charge of postcopy request the source for\nthe missing page. Once received and mapped, the paused thread\ngets resumed.\n\nUserfaultfd supports handling faults from a different process,\nand Qemu supports postcopy with vhost-user backends since\nv2.12.\n\nOne problem encountered with classic live-migration for VMs\nrelying on vhost-user backends is that when the traffic is\nhigh (e.g. PVP), it happens that it never converges as\npages gets dirtied at a faster rate than they are copied\nto the destination.\nIt is expected this problem sould be solved with using\npostcopy, as rings memory and buffers will be copied once,\nwhen destination will pagefault on them.\n\nMy current test bench is limited, so I could test above\nscenario. I just ran with flooding the guest using testpmd's\ntxonly forwarding mode, with either having the virtio-net\nbinded to its kernel or DPDK driver in guest. In my setup,\nmigration is done on the same machine. Results are average\nof 5 runs.\n\nA. Flooding virtio-net kernel driver (~3Mpps):\n1. Classic live-migration:\n - Total time: 12592ms\n - Downtime: 53ms\n\n2. Postcopy live-migration:\n - Total time: 2324ms\n - Downtime: 48ms\n\nB. Flooding Virtio PMD (~15Mpps):\n1. Classic live-migration:\n - Total time: 22101ms\n - Downtime: 35ms\n\n2. Postcopy live-migration:\n - Total time: 2995ms\n - Downtime: 47ms\n\nNote that the Total time reported by Qemu is really steady\naccross runs, whereas the Downtime is very unsteady.\n\nOne problem remaining to be fixed is the memory locking.\nIndeed, userfaultfd requires that the memory registered is\nneither mmapped with MAP_POPULATE attribute nor mlocked.\nFor the former, the series address this by not advertizing\npostcopy feature if dequeue zero-copy, which requires it, is\nenabled.\nFor the latter, this is really problematic because vhost-user\nbackend is a library so it cannot prevent the application\nto call mlockall(), as opposed to Qemu. When using testpmd,\none has just to append --no-mlockall to the command-line,\nbut missing it results in non-trivial warnings in Qemu logs.\n\nSteps to test postcopy:\n1. Run DPDK's Testpmd application on source:\n./install/bin/testpmd -m 512 --file-prefix=src -l 0,2 -n 4 \\\n --vdev 'net_vhost0,iface=/tmp/vu-src' -- --portmask=1 -i \\\n --rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \\\n --no-mlockall\n\n2. Run DPDK's Testpmd application on destination:\n./install/bin/testpmd -m 512 --file-prefix=dst -l 0,2 -n 4 \\\n --vdev 'net_vhost0,iface=/tmp/vu-dst' -- --portmask=1 -i \\\n --rxq=1 --txq=1 --nb-cores=1 --eth-peer=0,52:54:00:11:22:12 \\\n --no-mlockall\n\n3. Launch VM on source:\n./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \\\n -object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \\\n -numa node,memdev=mem -mem-prealloc \\\n -chardev socket,id=char0,path=/tmp/vu-src \\\n -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \\\n -device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \\\n -net none -vnc :0 -monitor stdio\n\n4. Launch VM on destination:\n./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 3G -smp 2 -cpu host \\\n -object memory-backend-file,id=mem,size=3G,mem-path=/dev/shm,share=on \\\n -numa node,memdev=mem -mem-prealloc \\\n -chardev socket,id=char0,path=/tmp/vu-dst \\\n -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \\\n -device virtio-net-pci,netdev=mynet1 /home/virt/rhel7.6-1-clone.qcow2 \\\n -net none -vnc :1 -monitor stdio -incoming tcp::8888\n\n5. In both testpmd prompts, start flooding the virtio-net device:\ntestpmd> set fwd txonly\ntestpmd> start\n\n6. In destination's Qemu monitor, enable postcopy:\n(qemu) migrate_set_capability postcopy-ram on\n\n7. In source's Qemu monitor, enable postcopy and launch migration:\n(qemu) migrate_set_capability postcopy-ram on\n(qemu) migrate -d tcp:0:8888\n(qemu) migrate_start_postcopy\n\nMaxime Coquelin (10):\n vhost: define postcopy protocol flag\n vhost: add number of fds to vhost-user messages and use it\n vhost: enable fds passing when sending vhost-user messages\n vhost: introduce postcopy's advise message\n vhost: add support for postcopy's listen message\n vhost: register new regions with userfaultfd\n vhost: avoid useless VhostUserMemory copy\n vhost: send userfault range addresses back to qemu\n vhost: add support to postcopy's end request\n vhost: enable postcopy protocol feature\n\n lib/librte_vhost/rte_vhost.h | 4 +\n lib/librte_vhost/socket.c | 21 +++-\n lib/librte_vhost/vhost.h | 3 +\n lib/librte_vhost/vhost_user.c | 201 +++++++++++++++++++++++++++++-----\n lib/librte_vhost/vhost_user.h | 12 +-\n 5 files changed, 206 insertions(+), 35 deletions(-)" }