diff mbox series

[v4] doc: add iavf live migration guide

Message ID 20230707060841.70826-1-lingyu.liu@intel.com (mailing list archive)
State Superseded, archived
Delegated to: Qi Zhang
Headers
Series [v4] doc: add iavf live migration guide |

Checks

Context Check Description
ci/checkpatch warning coding style issues
ci/loongarch-compilation success Compilation OK
ci/loongarch-unit-testing success Unit Testing PASS
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-aarch-unit-testing success Testing PASS
ci/iol-unit-testing success Testing PASS
ci/github-robot: build success github build: passed
ci/intel-Functional success Functional PASS
ci/iol-x86_64-compile-testing success Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-broadcom-Performance success Performance Testing PASS
ci/iol-testing warning Testing issues
ci/iol-x86_64-unit-testing success Testing PASS
ci/iol-aarch64-compile-testing success Testing PASS
ci/iol-broadcom-Functional success Functional Testing PASS
ci/iol-intel-Functional success Functional Testing PASS

Commit Message

Liu, Lingyu July 7, 2023, 6:08 a.m. UTC 
  Add iavf live migration steps based on KVM VFIO migration.

Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
---
v2: Fixed CI.
    Added brief introduction about live migration.
    Clarified this is iavf feature.

v3: Added intro and link about vfio live migration.
    Added description about kernel boot parameters.
    Changed to use sysfs to bind device to driver.
    Noted for running dpdk-testpmd.
    Highlighted KVM vfio migration.

v4: Removed ice kernel driver version.
---
 doc/guides/nics/intel_vf.rst           | 113 +++++++++++++++++++++++++
 doc/guides/rel_notes/release_23_07.rst |   3 +
 2 files changed, 116 insertions(+)

Comments

Cao, Yahui July 7, 2023, 6:21 a.m. UTC | #1 
On 7/7/2023 2:08 PM, Lingyu Liu wrote:
> Add iavf live migration steps based on KVM VFIO migration.
>
> Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
> ---
> v2: Fixed CI.
>      Added brief introduction about live migration.
>      Clarified this is iavf feature.
>
> v3: Added intro and link about vfio live migration.
>      Added description about kernel boot parameters.
>      Changed to use sysfs to bind device to driver.
>      Noted for running dpdk-testpmd.
>      Highlighted KVM vfio migration.
>
> v4: Removed ice kernel driver version.
> ---
>   doc/guides/nics/intel_vf.rst           | 113 +++++++++++++++++++++++++
>   doc/guides/rel_notes/release_23_07.rst |   3 +
>   2 files changed, 116 insertions(+)
>
> diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst
> index d365dbc185..85b3b4fc24 100644
> --- a/doc/guides/nics/intel_vf.rst
> +++ b/doc/guides/nics/intel_vf.rst
> @@ -622,6 +622,119 @@ which belongs to the destination VF on the VM.
>      Inter-VM Communication
>   
>   
> +Live Migrating a VM running DPDK
> +--------------------------------
> +
> +Live migration refers to the process of moving a running virtual machine (VM) or application
> +between different physical machines without disconnecting the client or application
> +(see https://en.wikipedia.org/wiki/Live_migration for more information).
> +
> +VFIO device migration refers to migrating a VM which have VFIO device pass-through
> +(see https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html for more information).
> +
> +This part describes stpes to migrate a VM which has a iavf device pass through.
> +
> +The following describes a target environment:
> +
> +*   Host Operating System: Ubuntu 20.04.5
> +
> +*   Guest Operating System: Ubuntu 20.04.5
> +
> +*   Linux Kernel Version: 5.15.0-72-generic
> +
> +*   Target Applications: dpdk-testpmd
> +
> +*   Ice Kernel Driver: Download from `<https://www.intel.com/content/www/us/en/download/19630>`_
> +
> +*   Qemu Version: 7.2
> +
> +The setup procedure is as follows:
> +
> +#.  Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**.
> +
> +#.  While booting the Host OS kernel, pass the intel_iommu=on kernel command line argument using GRUB.
> +
> +#.  In the Host OS
> +
> +    Install the ice driver and migration driver:
> +
> +    .. code-block:: console
> +
> +        insmod ice.ko
> +        insmod ice-vfio-pci.ko
> +
> +    Create 2 VFs and bind them to vfio pci driver:
> +
> +    .. code-block:: console
> +
> +        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
> +        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
> +        echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind
> +        echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> +        echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind
> +        echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> +
> +    .. note::
> +
> +        The command above creates two vfs for device 0000:ca:00.1:
> +
> +    .. code-block:: console
> +
> +        0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
> +        0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
> +
> +#.  Now, start the migration source Virtual Machine by running the following command:
> +
> +    .. code-block:: console
> +
> +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio
> +
> +    .. note::
> +        The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach a vfio PCI device
> +        to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed,
> +        x-enable-migration=true indicates that this VF supports migration. Dirty page tracking
> +        is not supported, so set x-pre-copy-dirty-page-tracking=off.
> +
> +#.  In VM, install iavf driver and vfio-pci driver
> +
> +    .. code-block:: console
> +
> +        insmod iavf.ko
> +        modprobe vfio enable_unsafe_noiommu_mode=1
> +        moodprobe vfio-pci
> +
> +#.  Bind net device to vfio-pci driver and launch dpdk-testpmd
> +
> +    .. code-block:: console
> +
> +        dpdk-testpmd -l 0-1 -- -i
> +        testpmd> set txpkts 64
> +        testpmd> start tx_first
> +
> +    .. note::
> +        Please ensure dpdk-testpmd to run independently of ssh console.
> +        Suggest to put it in a background process like tmux/screen so that
> +        migration will not causing ssh console exit and dpdk-testpmd killed.
> +
> +#. Start the migration destination Virtual Machine
> +
> +    .. code-block:: console
> +
> +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5556-:22 -monitor stdio -incoming tcp:127.0.0.1:4444
> +
> + #. Start migration by issuing the command in qemu console
> +
> +    .. code-block:: console
> +
> +        migrate -d tcp:127.0.0.1:4444
> +
> +#. Log in the destination VM, and dpdk-testpmd is not interrupt
> +
> +.. note::
> +
> +        Host kernel version above 5.17 is currently not supported due to kAPI change.
> +
> +
>   Windows Support
>   ---------------
>   
> diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
> index 4459144140..794a86eba3 100644
> --- a/doc/guides/rel_notes/release_23_07.rst
> +++ b/doc/guides/rel_notes/release_23_07.rst
> @@ -200,6 +200,9 @@ New Features
>   
>     Enhanced the GRO library to support TCP packets over IPv6 network.
>   
> +* **Updated Intel iavf driver.**
> +
> +  Added the document for iavf driver live migration based on KVM vfio migration.
>   
>   Removed Items
>   -------------


Reviewed-by: Yahui Cao <yahui.cao@intel.com>
Qi Zhang July 7, 2023, 6:41 a.m. UTC | #2 
> -----Original Message-----
> From: Cao, Yahui <yahui.cao@intel.com>
> Sent: Friday, July 7, 2023 2:22 PM
> To: Liu, Lingyu <lingyu.liu@intel.com>; dev@dpdk.org; Zhang, Qi Z
> <qi.z.zhang@intel.com>; Xing, Beilei <beilei.xing@intel.com>
> Subject: Re: [PATCH v4] doc: add iavf live migration guide
> 
> 
> On 7/7/2023 2:08 PM, Lingyu Liu wrote:
> > Add iavf live migration steps based on KVM VFIO migration.
> >
> > Signed-off-by: Lingyu Liu <lingyu.liu@intel.com>
> > ---
> > v2: Fixed CI.
> >      Added brief introduction about live migration.
> >      Clarified this is iavf feature.
> >
> > v3: Added intro and link about vfio live migration.
> >      Added description about kernel boot parameters.
> >      Changed to use sysfs to bind device to driver.
> >      Noted for running dpdk-testpmd.
> >      Highlighted KVM vfio migration.
> >
> > v4: Removed ice kernel driver version.
> > ---
> >   doc/guides/nics/intel_vf.rst           | 113 +++++++++++++++++++++++++
> >   doc/guides/rel_notes/release_23_07.rst |   3 +
> >   2 files changed, 116 insertions(+)
> >
> > diff --git a/doc/guides/nics/intel_vf.rst
> > b/doc/guides/nics/intel_vf.rst index d365dbc185..85b3b4fc24 100644
> > --- a/doc/guides/nics/intel_vf.rst
> > +++ b/doc/guides/nics/intel_vf.rst
> > @@ -622,6 +622,119 @@ which belongs to the destination VF on the VM.
> >      Inter-VM Communication
> >
> >
> > +Live Migrating a VM running DPDK
> > +--------------------------------
> > +
> > +Live migration refers to the process of moving a running virtual
> > +machine (VM) or application between different physical machines
> > +without disconnecting the client or application (see
> https://en.wikipedia.org/wiki/Live_migration for more information).
> > +
> > +VFIO device migration refers to migrating a VM which have VFIO device
> > +pass-through (see https://qemu.readthedocs.io/en/latest/devel/vfio-
> migration.html for more information).
> > +
> > +This part describes stpes to migrate a VM which has a iavf device pass
> through.
> > +
> > +The following describes a target environment:
> > +
> > +*   Host Operating System: Ubuntu 20.04.5
> > +
> > +*   Guest Operating System: Ubuntu 20.04.5
> > +
> > +*   Linux Kernel Version: 5.15.0-72-generic
> > +
> > +*   Target Applications: dpdk-testpmd
> > +
> > +*   Ice Kernel Driver: Download from
> `<https://www.intel.com/content/www/us/en/download/19630>`_
> > +
> > +*   Qemu Version: 7.2
> > +
> > +The setup procedure is as follows:
> > +
> > +#.  Before booting the Host OS, open **BIOS setup** and enable **Intel®
> VT features**.
> > +
> > +#.  While booting the Host OS kernel, pass the intel_iommu=on kernel
> command line argument using GRUB.
> > +
> > +#.  In the Host OS
> > +
> > +    Install the ice driver and migration driver:
> > +
> > +    .. code-block:: console
> > +
> > +        insmod ice.ko
> > +        insmod ice-vfio-pci.ko
> > +
> > +    Create 2 VFs and bind them to vfio pci driver:
> > +
> > +    .. code-block:: console
> > +
> > +        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
> > +        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
> > +        echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind
> > +        echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> > +        echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind
> > +        echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> > +
> > +    .. note::
> > +
> > +        The command above creates two vfs for device 0000:ca:00.1:
> > +
> > +    .. code-block:: console
> > +
> > +        0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-
> pci unused=iavf
> > +        0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if=
> > + drv=ice-vfio-pci unused=iavf
> > +
> > +#.  Now, start the migration source Virtual Machine by running the
> following command:
> > +
> > +    .. code-block:: console
> > +
> > +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu
> > + host -m 4G -smp 1 -device
> > + vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-
> > + page-tracking=off -drive file=ubuntu-2004.qcow2 -nic
> > + user,hostfwd=tcp::5555-:22 -monitor stdio
> > +
> > +    .. note::
> > +        The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach
> a vfio PCI device
> > +        to a Virtual Machine and the respective (Bus:Device.Function) numbers
> should be passed,
> > +        x-enable-migration=true indicates that this VF supports migration. Dirty
> page tracking
> > +        is not supported, so set x-pre-copy-dirty-page-tracking=off.
> > +
> > +#.  In VM, install iavf driver and vfio-pci driver
> > +
> > +    .. code-block:: console
> > +
> > +        insmod iavf.ko
> > +        modprobe vfio enable_unsafe_noiommu_mode=1
> > +        moodprobe vfio-pci
> > +
> > +#.  Bind net device to vfio-pci driver and launch dpdk-testpmd
> > +
> > +    .. code-block:: console
> > +
> > +        dpdk-testpmd -l 0-1 -- -i
> > +        testpmd> set txpkts 64
> > +        testpmd> start tx_first
> > +
> > +    .. note::
> > +        Please ensure dpdk-testpmd to run independently of ssh console.
> > +        Suggest to put it in a background process like tmux/screen so that
> > +        migration will not causing ssh console exit and dpdk-testpmd killed.
> > +
> > +#. Start the migration destination Virtual Machine
> > +
> > +    .. code-block:: console
> > +
> > +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu
> > + host -m 4G -smp 1 -device
> > + vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-
> > + page-tracking=off -drive file=ubuntu-2004.qcow2 -nic
> > + user,hostfwd=tcp::5556-:22 -monitor stdio -incoming
> > + tcp:127.0.0.1:4444
> > +
> > + #. Start migration by issuing the command in qemu console
> > +
> > +    .. code-block:: console
> > +
> > +        migrate -d tcp:127.0.0.1:4444
> > +
> > +#. Log in the destination VM, and dpdk-testpmd is not interrupt
> > +
> > +.. note::
> > +
> > +        Host kernel version above 5.17 is currently not supported due to kAPI
> change.
> > +
> > +
> >   Windows Support
> >   ---------------
> >
> > diff --git a/doc/guides/rel_notes/release_23_07.rst
> > b/doc/guides/rel_notes/release_23_07.rst
> > index 4459144140..794a86eba3 100644
> > --- a/doc/guides/rel_notes/release_23_07.rst
> > +++ b/doc/guides/rel_notes/release_23_07.rst
> > @@ -200,6 +200,9 @@ New Features
> >
> >     Enhanced the GRO library to support TCP packets over IPv6 network.
> >
> > +* **Updated Intel iavf driver.**
> > +
> > +  Added the document for iavf driver live migration based on KVM vfio
> migration.
> >
> >   Removed Items
> >   -------------
> 
> 
> Reviewed-by: Yahui Cao <yahui.cao@intel.com>‘

Applied to dpdk-next-net-intel.

Thanks
Qi
Thomas Monjalon July 11, 2023, 7:54 p.m. UTC | #3 
Hello,

I don't pull this patch in -rc3 because I feel some improvements should be done.


> On 7/7/2023 2:08 PM, Lingyu Liu wrote:
> > +        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
> > +        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
> > +        echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind
> > +        echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> > +        echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind
> > +        echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind

Why not using dpdk-devbind.py?

> > +
> > +    .. note::
> > +
> > +        The command above creates two vfs for device 0000:ca:00.1:

vfs should be VFs

> > +
> > +    .. code-block:: console
> > +
> > +        0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
> > +        0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
> > +
> > +#.  Now, start the migration source Virtual Machine by running the following command:
> > +
> > +    .. code-block:: console
> > +
> > +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio

No need the full qemu command with your local details here.


> > --- a/doc/guides/rel_notes/release_23_07.rst
> > +++ b/doc/guides/rel_notes/release_23_07.rst
> > @@ -200,6 +200,9 @@ New Features
> >   
> >     Enhanced the GRO library to support TCP packets over IPv6 network.
> >   
> > +* **Updated Intel iavf driver.**
> > +
> > +  Added the document for iavf driver live migration based on KVM vfio migration.

Not sure about the importance of this note.
Liu, Lingyu July 27, 2023, 8:34 a.m. UTC | #4 
On 7/12/2023 3:54 AM, Thomas Monjalon wrote:
> Hello,
>
> I don't pull this patch in -rc3 because I feel some improvements should be done.
>
Hi Thomas, thanks for your review. I am sorry for the late response.

>> On 7/7/2023 2:08 PM, Lingyu Liu wrote:
>>> +        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
>>> +        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
>>> +        echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind
>>> +        echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind
>>> +        echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind
>>> +        echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> Why not using dpdk-devbind.py?

dpdk-devbind.py doesn't support ice-vfio-pci driver yet.

>>> +
>>> +    .. note::
>>> +
>>> +        The command above creates two vfs for device 0000:ca:00.1:
> vfs should be VFs

Thanks. Will change in next version.

>>> +
>>> +    .. code-block:: console
>>> +
>>> +        0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
>>> +        0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
>>> +
>>> +#.  Now, start the migration source Virtual Machine by running the following command:
>>> +
>>> +    .. code-block:: console
>>> +
>>> +        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio
> No need the full qemu command with your local details here.

Thanks. Will change.

>
>>> --- a/doc/guides/rel_notes/release_23_07.rst
>>> +++ b/doc/guides/rel_notes/release_23_07.rst
>>> @@ -200,6 +200,9 @@ New Features
>>>    
>>>      Enhanced the GRO library to support TCP packets over IPv6 network.
>>>    
>>> +* **Updated Intel iavf driver.**
>>> +
>>> +  Added the document for iavf driver live migration based on KVM vfio migration.
> Not sure about the importance of this note.

It shows that AVF PMD live migration feature has been supported and 
lists the steps for users.

>
>
Thomas Monjalon July 27, 2023, 10:42 a.m. UTC | #5 
27/07/2023 10:34, Liu, Lingyu:
> 
> On 7/12/2023 3:54 AM, Thomas Monjalon wrote:
> > Hello,
> >
> > I don't pull this patch in -rc3 because I feel some improvements should be done.
> >
> Hi Thomas, thanks for your review. I am sorry for the late response.
> 
> >> On 7/7/2023 2:08 PM, Lingyu Liu wrote:
> >>> +        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
> >>> +        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
> >>> +        echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind
> >>> +        echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> >>> +        echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind
> >>> +        echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind
> > Why not using dpdk-devbind.py?
> 
> dpdk-devbind.py doesn't support ice-vfio-pci driver yet.

So it should be a requirement.
Please work on dpdk-devbind.py
diff mbox series

Patch

diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst
index d365dbc185..85b3b4fc24 100644
--- a/doc/guides/nics/intel_vf.rst
+++ b/doc/guides/nics/intel_vf.rst
@@ -622,6 +622,119 @@  which belongs to the destination VF on the VM.
    Inter-VM Communication
 
 
+Live Migrating a VM running DPDK
+--------------------------------
+
+Live migration refers to the process of moving a running virtual machine (VM) or application
+between different physical machines without disconnecting the client or application
+(see https://en.wikipedia.org/wiki/Live_migration for more information).
+
+VFIO device migration refers to migrating a VM which have VFIO device pass-through
+(see https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html for more information).
+
+This part describes stpes to migrate a VM which has a iavf device pass through.
+
+The following describes a target environment:
+
+*   Host Operating System: Ubuntu 20.04.5
+
+*   Guest Operating System: Ubuntu 20.04.5
+
+*   Linux Kernel Version: 5.15.0-72-generic
+
+*   Target Applications: dpdk-testpmd
+
+*   Ice Kernel Driver: Download from `<https://www.intel.com/content/www/us/en/download/19630>`_
+
+*   Qemu Version: 7.2
+
+The setup procedure is as follows:
+
+#.  Before booting the Host OS, open **BIOS setup** and enable **Intel® VT features**.
+
+#.  While booting the Host OS kernel, pass the intel_iommu=on kernel command line argument using GRUB.
+
+#.  In the Host OS
+
+    Install the ice driver and migration driver:
+
+    .. code-block:: console
+
+        insmod ice.ko
+        insmod ice-vfio-pci.ko
+
+    Create 2 VFs and bind them to vfio pci driver:
+
+    .. code-block:: console
+
+        echo 2 > /sys/bus/pci/devices/0000:ca:00.1/sriov_numvfs
+        echo "8086 1889" > /sys/bus/pci/drivers/ice-vfio-pci/new_id
+        echo 0000:ca:11.0 > /sys/bus/pci/devices/0000:ca:11.0/driver/unbind
+        echo 0000:ca:11.0 > /sys/bus/pci/drivers/ice-vfio-pci/bind
+        echo 0000:ca:11.1 > /sys/bus/pci/devices/0000:ca:11.1/driver/unbind
+        echo 0000:ca:11.1 > /sys/bus/pci/drivers/ice-vfio-pci/bind
+
+    .. note::
+
+        The command above creates two vfs for device 0000:ca:00.1:
+
+    .. code-block:: console
+
+        0000:ca:11.0 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
+        0000:ca:11.1 'Ethernet Adaptive Virtual Function 1889' if= drv=ice-vfio-pci unused=iavf
+
+#.  Now, start the migration source Virtual Machine by running the following command:
+
+    .. code-block:: console
+
+        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.0,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5555-:22 -monitor stdio
+
+    .. note::
+        The vfio-pci,host=0000:ca:11.0 value indicates that you want to attach a vfio PCI device
+        to a Virtual Machine and the respective (Bus:Device.Function) numbers should be passed,
+        x-enable-migration=true indicates that this VF supports migration. Dirty page tracking
+        is not supported, so set x-pre-copy-dirty-page-tracking=off.
+
+#.  In VM, install iavf driver and vfio-pci driver
+
+    .. code-block:: console
+
+        insmod iavf.ko
+        modprobe vfio enable_unsafe_noiommu_mode=1
+        moodprobe vfio-pci
+
+#.  Bind net device to vfio-pci driver and launch dpdk-testpmd
+
+    .. code-block:: console
+
+        dpdk-testpmd -l 0-1 -- -i
+        testpmd> set txpkts 64
+        testpmd> start tx_first
+
+    .. note::
+        Please ensure dpdk-testpmd to run independently of ssh console.
+        Suggest to put it in a background process like tmux/screen so that
+        migration will not causing ssh console exit and dpdk-testpmd killed.
+
+#. Start the migration destination Virtual Machine
+
+    .. code-block:: console
+
+        qemu/build/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -cpu host -m 4G -smp 1 -device vfio-pci,host=0000:ca:11.1,x-enable-migration=true,x-pre-copy-dirty-page-tracking=off -drive file=ubuntu-2004.qcow2 -nic user,hostfwd=tcp::5556-:22 -monitor stdio -incoming tcp:127.0.0.1:4444
+
+ #. Start migration by issuing the command in qemu console
+
+    .. code-block:: console
+
+        migrate -d tcp:127.0.0.1:4444
+
+#. Log in the destination VM, and dpdk-testpmd is not interrupt
+
+.. note::
+
+        Host kernel version above 5.17 is currently not supported due to kAPI change.
+
+
 Windows Support
 ---------------
 
diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst
index 4459144140..794a86eba3 100644
--- a/doc/guides/rel_notes/release_23_07.rst
+++ b/doc/guides/rel_notes/release_23_07.rst
@@ -200,6 +200,9 @@  New Features
 
   Enhanced the GRO library to support TCP packets over IPv6 network.
 
+* **Updated Intel iavf driver.**
+
+  Added the document for iavf driver live migration based on KVM vfio migration.
 
 Removed Items
 -------------