doc: update NFP documentation with Corigine information

  From: Walter Heymans <walter.heymans@corigine.com>

The NFP PMD documentation is updated to include information about
Corigine and their new vendor device ID.

Outdated information regarding the use of the PMD is also updated.

While making major changes to the document, the maximum number of
characters per line is updated to 80 characters to improve the
readability in raw format.

Signed-off-by: Walter Heymans <walter.heymans@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
---
 doc/guides/nics/nfp.rst | 168 +++++++++++++++++++++-------------------
 1 file changed, 90 insertions(+), 78 deletions(-)
  

On 2/3/2023 8:08 AM, Chaoyong He wrote:
> From: Walter Heymans <walter.heymans@corigine.com>
> 
> The NFP PMD documentation is updated to include information about
> Corigine and their new vendor device ID.
> 
> Outdated information regarding the use of the PMD is also updated.
> 
> While making major changes to the document, the maximum number of
> characters per line is updated to 80 characters to improve the
> readability in raw format.
> 

There are three groups of changes done to documentation as explained in
three paragraphs above.

To help review, is it possible to separate this patch into three
patches? Later they can be squashed and merged as a single patch.
But as it is, easy to miss content changes among formatting changes.

(You can include simple grammar updates (that doesn't change either
content or Corigine related information) to formatting update patch)


> Signed-off-by: Walter Heymans <walter.heymans@corigine.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> ---
>  doc/guides/nics/nfp.rst | 168 +++++++++++++++++++++-------------------
>  1 file changed, 90 insertions(+), 78 deletions(-)
> 
> diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst
> index a085d7d9ae..6fea280411 100644
> --- a/doc/guides/nics/nfp.rst
> +++ b/doc/guides/nics/nfp.rst
> @@ -1,35 +1,34 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2015-2017 Netronome Systems, Inc. All rights reserved.
> -    All rights reserved.
> +    Copyright(c) 2021 Corigine, Inc. All rights reserved.
>  
>  NFP poll mode driver library
>  ============================
>  
> -Netronome's sixth generation of flow processors pack 216 programmable
> -cores and over 100 hardware accelerators that uniquely combine packet,
> -flow, security and content processing in a single device that scales
> +Netronome and Corigine's sixth generation of flow processors pack 216
> +programmable cores and over 100 hardware accelerators that uniquely combine
> +packet, flow, security and content processing in a single device that scales
>  up to 400-Gb/s.
>  
> -This document explains how to use DPDK with the Netronome Poll Mode
> -Driver (PMD) supporting Netronome's Network Flow Processor 6xxx
> -(NFP-6xxx), Netronome's Network Flow Processor 4xxx (NFP-4xxx) and
> -Netronome's Network Flow Processor 38xx (NFP-38xx).
> +This document explains how to use DPDK with the Network Flow Processor (NFP)
> +Poll Mode Driver (PMD) supporting Netronome and Corigine's NFP-6xxx, NFP-4xxx
> +and NFP-38xx product lines.
>  
> -NFP is a SRIOV capable device and the PMD supports the physical
> -function (PF) and the virtual functions (VFs).
> +NFP is a SR-IOV capable device and the PMD supports the physical function (PF)
> +and the virtual functions (VFs).
>  
>  Dependencies
>  ------------
>  
> -Before using the Netronome's DPDK PMD some NFP configuration,
> -which is not related to DPDK, is required. The system requires
> -installation of **Netronome's BSP (Board Support Package)** along
> -with a specific NFP firmware application. Netronome's NSP ABI
> -version should be 0.20 or higher.
> +Before using the NFP DPDK PMD some NFP configuration, which is not related to
> +DPDK, is required. The system requires installation of
> +**NFP-BSP (Board Support Package)** along with a specific NFP firmware
> +application. The NSP ABI version should be 0.20 or higher.
>  
> -If you have a NFP device you should already have the code and
> -documentation for this configuration. Contact
> -**support@netronome.com** to obtain the latest available firmware.
> +If you have a NFP device you should already have the documentation to perform
> +this configuration. Contact **support@netronome.com** (for Netronome products)
> +or **smartnic-support@corigine.com** (for Corigine products) to obtain the
> +latest available firmware.
>  
>  The NFP Linux netdev kernel driver for VFs has been a part of the
>  vanilla kernel since kernel version 4.5, and support for the PF
> @@ -44,11 +43,11 @@ Linux kernel driver.
>  Building the software
>  ---------------------
>  
> -Netronome's PMD code is provided in the **drivers/net/nfp** directory.
> -Although NFP PMD has Netronome´s BSP dependencies, it is possible to
> -compile it along with other DPDK PMDs even if no BSP was installed previously.
> -Of course, a DPDK app will require such a BSP installed for using the
> -NFP PMD, along with a specific NFP firmware application.
> +The NFP PMD code is provided in the **drivers/net/nfp** directory. Although
> +NFP PMD has BSP dependencies, it is possible to compile it along with other
> +DPDK PMDs even if no BSP was installed previously. Of course, a DPDK app will
> +require such a BSP installed for using the NFP PMD, along with a specific NFP
> +firmware application.
>  
>  Once the DPDK is built all the DPDK apps and examples include support for
>  the NFP PMD.
> @@ -57,27 +56,20 @@ the NFP PMD.
>  Driver compilation and testing
>  ------------------------------
>  
> -Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
> -for details.
> +Refer to the document
> +:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` for details.
>  
>  Using the PF
>  ------------
>  
> -NFP PMD supports using the NFP PF as another DPDK port, but it does not
> -have any functionality for controlling VFs. In fact, it is not possible to use
> -the PMD with the VFs if the PF is being used by DPDK, that is, with the NFP PF
> -bound to ``igb_uio`` or ``vfio-pci`` kernel drivers. Future DPDK versions will
> -have a PMD able to work with the PF and VFs at the same time and with the PF
> -implementing VF management along with other PF-only functionalities/offloads.
> -

Why this paragraph is removed? Is it because it is not correct anymore,
or just because of document organization change.

>  The PMD PF has extra work to do which will delay the DPDK app initialization
> -like uploading the firmware and configure the Link state properly when starting or
> -stopping a PF port. Since DPDK 18.05 the firmware upload happens when
> +like uploading the firmware and configure the Link state properly when starting
> +or stopping a PF port. Since DPDK 18.05 the firmware upload happens when
>  a PF is initialized, which was not always true with older DPDK versions.
>  
> -Depending on the Netronome product installed in the system, firmware files
> -should be available under ``/lib/firmware/netronome``. DPDK PMD supporting the
> -PF looks for a firmware file in this order:
> +Depending on the product installed in the system, firmware files should be
> +available under ``/lib/firmware/netronome``. DPDK PMD supporting the PF looks
> +for a firmware file in this order:
>  
>  	1) First try to find a firmware image specific for this device using the
>  	   NFP serial number:
> @@ -92,18 +84,21 @@ PF looks for a firmware file in this order:
>  
>  		nic_AMDA0099-0001_2x25.nffw
>  
> -Netronome's software packages install firmware files under ``/lib/firmware/netronome``
> -to support all the Netronome's SmartNICs and different firmware applications.
> -This is usually done using file names based on SmartNIC type and media and with a
> -directory per firmware application. Options 1 and 2 for firmware filenames allow
> -more than one SmartNIC, same type of SmartNIC or different ones, and to upload a
> -different firmware to each SmartNIC.
> +Netronome and Corigine's software packages install firmware files under
> +``/lib/firmware/netronome`` to support all the SmartNICs and different firmware
> +applications. This is usually done using file names based on SmartNIC type and
> +media and with a directory per firmware application. Options 1 and 2 for
> +firmware filenames allow more than one SmartNIC, same type of SmartNIC or
> +different ones, and to upload a different firmware to each SmartNIC.
>  
>     .. Note::
> -      Currently the NFP PMD supports using the PF with Agilio Firmware with NFD3
> -      and Agilio Firmware with NFDk. See https://help.netronome.com/support/solutions
> +      Currently the NFP PMD supports using the PF with Agilio Firmware with
> +      NFD3 and Agilio Firmware with NFDk. See
> +      `Netronome Support <https://help.netronome.com/support/solutions>`_.
>        for more information on the various firmwares supported by the Netronome
> -      Agilio CX smartNIC.
> +      Agilio SmartNICs range, or
> +      `Corigine Support <https://www.corigine.com/productsOverviewList-30.html>`_.
> +      for more information about Corigine's range.
>  
>  PF multiport support
>  --------------------
> @@ -118,7 +113,7 @@ this particular configuration requires the PMD to create ports in a special way,
>  although once they are created, DPDK apps should be able to use them as normal
>  PCI ports.
>  
> -NFP ports belonging to same PF can be seen inside PMD initialization with a
> +NFP ports belonging to the same PF can be seen inside PMD initialization with a
>  suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID
>  0000:03:00.0 and four ports is seen by the PMD code as:
>  
> @@ -137,50 +132,67 @@ suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID
>  PF multiprocess support
>  -----------------------
>  
> -Due to how the driver needs to access the NFP through a CPP interface, which implies
> -to use specific registers inside the chip, the number of secondary processes with PF
> -ports is limited to only one.
> +Due to how the driver needs to access the NFP through a CPP interface, which
> +implies to use specific registers inside the chip, the number of secondary
> +processes with PF ports is limited to only one.
>  
> -This limitation will be solved in future versions but having basic multiprocess support
> -is important for allowing development and debugging through the PF using a secondary
> -process which will create a CPP bridge for user space tools accessing the NFP.
> +This limitation will be solved in future versions, but having basic
> +multiprocess support is important for allowing development and debugging
> +through the PF using a secondary process, which will create a CPP bridge
> +for user space tools accessing the NFP.
>  
>  
>  System configuration
>  --------------------
>  
>  #. **Enable SR-IOV on the NFP device:** The current NFP PMD supports the PF and
> -   the VFs on a NFP device. However, it is not possible to work with both at the
> -   same time because the VFs require the PF being bound to the NFP PF Linux
> -   netdev driver.  Make sure you are working with a kernel with NFP PF support or
> -   get the drivers from the above Github repository and follow the instructions
> -   for building and installing it.
> +   the VFs on a NFP device. However, it is not possible to work with both at
> +   the same time when using the netdev NFP Linux netdev driver.

Old and new text doesn't say same thing.
Old one says: "For DPDK to support VF, PF needs to bound to kernel driver.:

Is this changed, or just wording mistake?


>     It is possible
> +   to bind the PF to the ``vfio-pci`` kernel module, and create VFs afterwards.
> +   This requires loading the ``vfio-pci`` module with the following parameters:
> +
> +   .. code-block:: console
> +
> +      modprobe vfio-pci enable_sriov=1 disable_idle_d3=1
> +
> +   VFs need to be enabled before they can be used with the PMD. Before enabling
> +   the VFs it is useful to obtain information about the current NFP PCI device
> +   detected by the system. This can be done on Netronome SmartNICs using:
> +
> +   .. code-block:: console
> +
> +      lspci -d 19ee:
>  

What I understand is, to support VF by DPDK two things are required:
1) Ability to create VFs, this can be done both by using device's kernel
driver or 'vfio-pci'
2) PF driver should support managing VFs.

Above lines document about item (1) and how 'vfio-pci' is used for it.

But old documentation mentions about item (2) is missing, why that part
removed, isn't it valid anymore? I mean is "PF -> kernel, VF -> DPDK"
combination supported now?


> -   VFs need to be enabled before they can be used with the PMD.
> -   Before enabling the VFs it is useful to obtain information about the
> -   current NFP PCI device detected by the system:
> +   and on Corigine SmartNICs using:
>  
>     .. code-block:: console
>  
> -      lspci -d19ee:
> +      lspci -d 1da8:
>  
> -   Now, for example, configure two virtual functions on a NFP-6xxx device
> +   Now, for example, to configure two virtual functions on a NFP device
>     whose PCI system identity is "0000:03:00.0":
>  
>     .. code-block:: console
>  
>        echo 2 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs
>  
> -   The result of this command may be shown using lspci again:
> +   The result of this command may be shown using lspci again on Netronome
> +   SmartNICs:
> +
> +   .. code-block:: console
> +
> +      lspci -d 19ee: -k
> +
> +   and on Corigine SmartNICs:
>  
>     .. code-block:: console
>  
> -      lspci -d19ee: -k
> +      lspci -d 1da8: -k
>  
>     Two new PCI devices should appear in the output of the above command. The
> -   -k option shows the device driver, if any, that devices are bound to.
> -   Depending on the modules loaded at this point the new PCI devices may be
> -   bound to nfp_netvf driver.
> +   -k option shows the device driver, if any, that the devices are bound to.
> +   Depending on the modules loaded, at this point the new PCI devices may be
> +   bound to the ``nfp`` kernel driver or ``vfio-pci``.
>  
>  
>  Flow offload
> @@ -193,13 +205,13 @@ The flower firmware application requires the PMD running two services:
>  
>  	* PF vNIC service: handling the feedback traffic.
>  	* ctrl vNIC service: communicate between PMD and firmware through
> -	  control message.
> +	  control messages.
>  
>  To achieve the offload of flow, the representor ports are exposed to OVS.
> -The flower firmware application support representor port for VF and physical
> -port. There will always exist a representor port for each physical port,
> -and the number of the representor port for VF is specified by the user through
> -parameter.
> +The flower firmware application supports VF, PF, and physical port representor
> +ports. 

Again old document and new one is not saying same thing, is it intentional?

Old one says: "Having representor ports for both VF and PF is supported."

New one says: "FW supports representor port, VF and PF."

> There will always exist a representor port for a PF and each physical
> +port. The number of the representor ports for VFs are specified by the user
> +through a parameter.
>  
>  In the Rx direction, the flower firmware application will prepend the input
>  port information into metadata for each packet which can't offloaded. The PF
> @@ -207,12 +219,12 @@ vNIC service will keep polling packets from the firmware, and multiplex them
>  to the corresponding representor port.
>  
>  In the Tx direction, the representor port will prepend the output port
> -information into metadata for each packet, and then send it to firmware through
> -PF vNIC.
> +information into metadata for each packet, and then send it to the firmware
> +through the PF vNIC.
>  
> -The ctrl vNIC service handling various control message, like the creation and
> -configuration of representor port, the pattern and action of flow rules, the
> -statistics of flow rules, and so on.
> +The ctrl vNIC service handles various control messages, for example, the
> +creation and configuration of a representor port, the pattern and action of
> +flow rules, the statistics of flow rules, etc.
>  
>  Metadata Format
>  ---------------
  

Hello Ferruh,

Thanks for your feedback.

On 2023-02-15 13:37:05 +0000, Ferruh Yigit wrote:
> On 2/3/2023 8:08 AM, Chaoyong He wrote:
> > From: Walter Heymans <walter.heymans@corigine.com>
> > 
> > The NFP PMD documentation is updated to include information about
> > Corigine and their new vendor device ID.
> > 
> > Outdated information regarding the use of the PMD is also updated.
> > 
> > While making major changes to the document, the maximum number of
> > characters per line is updated to 80 characters to improve the
> > readability in raw format.
> > 
> 
> There are three groups of changes done to documentation as explained in
> three paragraphs above.
> 
> To help review, is it possible to separate this patch into three
> patches? Later they can be squashed and merged as a single patch.
> But as it is, easy to miss content changes among formatting changes.
> 
> (You can include simple grammar updates (that doesn't change either
> content or Corigine related information) to formatting update patch)

We will break this patch in to three as you suggest, address the 
comments below and post a v2.

> 
> 
> > Signed-off-by: Walter Heymans <walter.heymans@corigine.com>
> > Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > ---
> >  doc/guides/nics/nfp.rst | 168 +++++++++++++++++++++-------------------
> >  1 file changed, 90 insertions(+), 78 deletions(-)
> > 
> > diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst
> > index a085d7d9ae..6fea280411 100644
> > --- a/doc/guides/nics/nfp.rst
> > +++ b/doc/guides/nics/nfp.rst
> > @@ -1,35 +1,34 @@
> >  ..  SPDX-License-Identifier: BSD-3-Clause
> >      Copyright(c) 2015-2017 Netronome Systems, Inc. All rights reserved.
> > -    All rights reserved.
> > +    Copyright(c) 2021 Corigine, Inc. All rights reserved.
> >  
> >  NFP poll mode driver library
> >  ============================
> >  
> > -Netronome's sixth generation of flow processors pack 216 programmable
> > -cores and over 100 hardware accelerators that uniquely combine packet,
> > -flow, security and content processing in a single device that scales
> > +Netronome and Corigine's sixth generation of flow processors pack 216
> > +programmable cores and over 100 hardware accelerators that uniquely combine
> > +packet, flow, security and content processing in a single device that scales
> >  up to 400-Gb/s.
> >  
> > -This document explains how to use DPDK with the Netronome Poll Mode
> > -Driver (PMD) supporting Netronome's Network Flow Processor 6xxx
> > -(NFP-6xxx), Netronome's Network Flow Processor 4xxx (NFP-4xxx) and
> > -Netronome's Network Flow Processor 38xx (NFP-38xx).
> > +This document explains how to use DPDK with the Network Flow Processor (NFP)
> > +Poll Mode Driver (PMD) supporting Netronome and Corigine's NFP-6xxx, NFP-4xxx
> > +and NFP-38xx product lines.
> >  
> > -NFP is a SRIOV capable device and the PMD supports the physical
> > -function (PF) and the virtual functions (VFs).
> > +NFP is a SR-IOV capable device and the PMD supports the physical function (PF)
> > +and the virtual functions (VFs).
> >  
> >  Dependencies
> >  ------------
> >  
> > -Before using the Netronome's DPDK PMD some NFP configuration,
> > -which is not related to DPDK, is required. The system requires
> > -installation of **Netronome's BSP (Board Support Package)** along
> > -with a specific NFP firmware application. Netronome's NSP ABI
> > -version should be 0.20 or higher.
> > +Before using the NFP DPDK PMD some NFP configuration, which is not related to
> > +DPDK, is required. The system requires installation of
> > +**NFP-BSP (Board Support Package)** along with a specific NFP firmware
> > +application. The NSP ABI version should be 0.20 or higher.
> >  
> > -If you have a NFP device you should already have the code and
> > -documentation for this configuration. Contact
> > -**support@netronome.com** to obtain the latest available firmware.
> > +If you have a NFP device you should already have the documentation to perform
> > +this configuration. Contact **support@netronome.com** (for Netronome products)
> > +or **smartnic-support@corigine.com** (for Corigine products) to obtain the
> > +latest available firmware.
> >  
> >  The NFP Linux netdev kernel driver for VFs has been a part of the
> >  vanilla kernel since kernel version 4.5, and support for the PF
> > @@ -44,11 +43,11 @@ Linux kernel driver.
> >  Building the software
> >  ---------------------
> >  
> > -Netronome's PMD code is provided in the **drivers/net/nfp** directory.
> > -Although NFP PMD has Netronome´s BSP dependencies, it is possible to
> > -compile it along with other DPDK PMDs even if no BSP was installed previously.
> > -Of course, a DPDK app will require such a BSP installed for using the
> > -NFP PMD, along with a specific NFP firmware application.
> > +The NFP PMD code is provided in the **drivers/net/nfp** directory. Although
> > +NFP PMD has BSP dependencies, it is possible to compile it along with other
> > +DPDK PMDs even if no BSP was installed previously. Of course, a DPDK app will
> > +require such a BSP installed for using the NFP PMD, along with a specific NFP
> > +firmware application.
> >  
> >  Once the DPDK is built all the DPDK apps and examples include support for
> >  the NFP PMD.
> > @@ -57,27 +56,20 @@ the NFP PMD.
> >  Driver compilation and testing
> >  ------------------------------
> >  
> > -Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
> > -for details.
> > +Refer to the document
> > +:ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` for details.
> >  
> >  Using the PF
> >  ------------
> >  
> > -NFP PMD supports using the NFP PF as another DPDK port, but it does not
> > -have any functionality for controlling VFs. In fact, it is not possible to use
> > -the PMD with the VFs if the PF is being used by DPDK, that is, with the NFP PF
> > -bound to ``igb_uio`` or ``vfio-pci`` kernel drivers. Future DPDK versions will
> > -have a PMD able to work with the PF and VFs at the same time and with the PF
> > -implementing VF management along with other PF-only functionalities/offloads.
> > -
> 
> Why this paragraph is removed? Is it because it is not correct anymore,
> or just because of document organization change.
> 
> >  The PMD PF has extra work to do which will delay the DPDK app initialization
> > -like uploading the firmware and configure the Link state properly when starting or
> > -stopping a PF port. Since DPDK 18.05 the firmware upload happens when
> > +like uploading the firmware and configure the Link state properly when starting
> > +or stopping a PF port. Since DPDK 18.05 the firmware upload happens when
> >  a PF is initialized, which was not always true with older DPDK versions.
> >  
> > -Depending on the Netronome product installed in the system, firmware files
> > -should be available under ``/lib/firmware/netronome``. DPDK PMD supporting the
> > -PF looks for a firmware file in this order:
> > +Depending on the product installed in the system, firmware files should be
> > +available under ``/lib/firmware/netronome``. DPDK PMD supporting the PF looks
> > +for a firmware file in this order:
> >  
> >  	1) First try to find a firmware image specific for this device using the
> >  	   NFP serial number:
> > @@ -92,18 +84,21 @@ PF looks for a firmware file in this order:
> >  
> >  		nic_AMDA0099-0001_2x25.nffw
> >  
> > -Netronome's software packages install firmware files under ``/lib/firmware/netronome``
> > -to support all the Netronome's SmartNICs and different firmware applications.
> > -This is usually done using file names based on SmartNIC type and media and with a
> > -directory per firmware application. Options 1 and 2 for firmware filenames allow
> > -more than one SmartNIC, same type of SmartNIC or different ones, and to upload a
> > -different firmware to each SmartNIC.
> > +Netronome and Corigine's software packages install firmware files under
> > +``/lib/firmware/netronome`` to support all the SmartNICs and different firmware
> > +applications. This is usually done using file names based on SmartNIC type and
> > +media and with a directory per firmware application. Options 1 and 2 for
> > +firmware filenames allow more than one SmartNIC, same type of SmartNIC or
> > +different ones, and to upload a different firmware to each SmartNIC.
> >  
> >     .. Note::
> > -      Currently the NFP PMD supports using the PF with Agilio Firmware with NFD3
> > -      and Agilio Firmware with NFDk. See https://help.netronome.com/support/solutions
> > +      Currently the NFP PMD supports using the PF with Agilio Firmware with
> > +      NFD3 and Agilio Firmware with NFDk. See
> > +      `Netronome Support <https://help.netronome.com/support/solutions>`_.
> >        for more information on the various firmwares supported by the Netronome
> > -      Agilio CX smartNIC.
> > +      Agilio SmartNICs range, or
> > +      `Corigine Support <https://www.corigine.com/productsOverviewList-30.html>`_.
> > +      for more information about Corigine's range.
> >  
> >  PF multiport support
> >  --------------------
> > @@ -118,7 +113,7 @@ this particular configuration requires the PMD to create ports in a special way,
> >  although once they are created, DPDK apps should be able to use them as normal
> >  PCI ports.
> >  
> > -NFP ports belonging to same PF can be seen inside PMD initialization with a
> > +NFP ports belonging to the same PF can be seen inside PMD initialization with a
> >  suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID
> >  0000:03:00.0 and four ports is seen by the PMD code as:
> >  
> > @@ -137,50 +132,67 @@ suffix added to the PCI ID: wwww:xx:yy.z_portn. For example, a PF with PCI ID
> >  PF multiprocess support
> >  -----------------------
> >  
> > -Due to how the driver needs to access the NFP through a CPP interface, which implies
> > -to use specific registers inside the chip, the number of secondary processes with PF
> > -ports is limited to only one.
> > +Due to how the driver needs to access the NFP through a CPP interface, which
> > +implies to use specific registers inside the chip, the number of secondary
> > +processes with PF ports is limited to only one.
> >  
> > -This limitation will be solved in future versions but having basic multiprocess support
> > -is important for allowing development and debugging through the PF using a secondary
> > -process which will create a CPP bridge for user space tools accessing the NFP.
> > +This limitation will be solved in future versions, but having basic
> > +multiprocess support is important for allowing development and debugging
> > +through the PF using a secondary process, which will create a CPP bridge
> > +for user space tools accessing the NFP.
> >  
> >  
> >  System configuration
> >  --------------------
> >  
> >  #. **Enable SR-IOV on the NFP device:** The current NFP PMD supports the PF and
> > -   the VFs on a NFP device. However, it is not possible to work with both at the
> > -   same time because the VFs require the PF being bound to the NFP PF Linux
> > -   netdev driver.  Make sure you are working with a kernel with NFP PF support or
> > -   get the drivers from the above Github repository and follow the instructions
> > -   for building and installing it.
> > +   the VFs on a NFP device. However, it is not possible to work with both at
> > +   the same time when using the netdev NFP Linux netdev driver.
> 
> Old and new text doesn't say same thing.
> Old one says: "For DPDK to support VF, PF needs to bound to kernel driver.:
> 
> Is this changed, or just wording mistake?
> 
> 
> >     It is possible
> > +   to bind the PF to the ``vfio-pci`` kernel module, and create VFs afterwards.
> > +   This requires loading the ``vfio-pci`` module with the following parameters:
> > +
> > +   .. code-block:: console
> > +
> > +      modprobe vfio-pci enable_sriov=1 disable_idle_d3=1
> > +
> > +   VFs need to be enabled before they can be used with the PMD. Before enabling
> > +   the VFs it is useful to obtain information about the current NFP PCI device
> > +   detected by the system. This can be done on Netronome SmartNICs using:
> > +
> > +   .. code-block:: console
> > +
> > +      lspci -d 19ee:
> >  
> 
> What I understand is, to support VF by DPDK two things are required:
> 1) Ability to create VFs, this can be done both by using device's kernel
> driver or 'vfio-pci'
> 2) PF driver should support managing VFs.
> 
> Above lines document about item (1) and how 'vfio-pci' is used for it.
> 
> But old documentation mentions about item (2) is missing, why that part
> removed, isn't it valid anymore? I mean is "PF -> kernel, VF -> DPDK"
> combination supported now?
> 
> 
> > -   VFs need to be enabled before they can be used with the PMD.
> > -   Before enabling the VFs it is useful to obtain information about the
> > -   current NFP PCI device detected by the system:
> > +   and on Corigine SmartNICs using:
> >  
> >     .. code-block:: console
> >  
> > -      lspci -d19ee:
> > +      lspci -d 1da8:
> >  
> > -   Now, for example, configure two virtual functions on a NFP-6xxx device
> > +   Now, for example, to configure two virtual functions on a NFP device
> >     whose PCI system identity is "0000:03:00.0":
> >  
> >     .. code-block:: console
> >  
> >        echo 2 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs
> >  
> > -   The result of this command may be shown using lspci again:
> > +   The result of this command may be shown using lspci again on Netronome
> > +   SmartNICs:
> > +
> > +   .. code-block:: console
> > +
> > +      lspci -d 19ee: -k
> > +
> > +   and on Corigine SmartNICs:
> >  
> >     .. code-block:: console
> >  
> > -      lspci -d19ee: -k
> > +      lspci -d 1da8: -k
> >  
> >     Two new PCI devices should appear in the output of the above command. The
> > -   -k option shows the device driver, if any, that devices are bound to.
> > -   Depending on the modules loaded at this point the new PCI devices may be
> > -   bound to nfp_netvf driver.
> > +   -k option shows the device driver, if any, that the devices are bound to.
> > +   Depending on the modules loaded, at this point the new PCI devices may be
> > +   bound to the ``nfp`` kernel driver or ``vfio-pci``.
> >  
> >  
> >  Flow offload
> > @@ -193,13 +205,13 @@ The flower firmware application requires the PMD running two services:
> >  
> >  	* PF vNIC service: handling the feedback traffic.
> >  	* ctrl vNIC service: communicate between PMD and firmware through
> > -	  control message.
> > +	  control messages.
> >  
> >  To achieve the offload of flow, the representor ports are exposed to OVS.
> > -The flower firmware application support representor port for VF and physical
> > -port. There will always exist a representor port for each physical port,
> > -and the number of the representor port for VF is specified by the user through
> > -parameter.
> > +The flower firmware application supports VF, PF, and physical port representor
> > +ports. 
> 
> Again old document and new one is not saying same thing, is it intentional?
> 
> Old one says: "Having representor ports for both VF and PF is supported."
> 
> New one says: "FW supports representor port, VF and PF."
> 
> > There will always exist a representor port for a PF and each physical
> > +port. The number of the representor ports for VFs are specified by the user
> > +through a parameter.
> >  
> >  In the Rx direction, the flower firmware application will prepend the input
> >  port information into metadata for each packet which can't offloaded. The PF
> > @@ -207,12 +219,12 @@ vNIC service will keep polling packets from the firmware, and multiplex them
> >  to the corresponding representor port.
> >  
> >  In the Tx direction, the representor port will prepend the output port
> > -information into metadata for each packet, and then send it to firmware through
> > -PF vNIC.
> > +information into metadata for each packet, and then send it to the firmware
> > +through the PF vNIC.
> >  
> > -The ctrl vNIC service handling various control message, like the creation and
> > -configuration of representor port, the pattern and action of flow rules, the
> > -statistics of flow rules, and so on.
> > +The ctrl vNIC service handles various control messages, for example, the
> > +creation and configuration of a representor port, the pattern and action of
> > +flow rules, the statistics of flow rules, etc.
> >  
> >  Metadata Format
> >  ---------------
>