[v4,0/6] Flow entites behavior on port restart

  It is unspecified whether flow rules and indirect actions are kept
when a port is stopped, possibly reconfigured, and started again.
Vendors approach the topic differently, e.g. mlx5 and i40e PMD
disagree in whether flow rules can be kept, and mlx5 PMD would keep
indirect actions. In the end, applications are greatly affected
by whatever contract there is and need to know it.

Applications may wish to restart the port to reconfigure it,
e.g. switch offloads or even modify queues.
Keeping rte_flow entities enables application improvements:
1. Since keeping the rules across restart comes with the ability
   to create rules before the device is started. This allows
   to have all the rules created at the moment of start,
   so that there is no time frame when traffic is coming already,
   but the rules are not yet created (restored).
2. When a rule or an indirect action has some associated state,
   such as a counter, application saves the need to keep
   additional state in order to cope with information loss
   if such an entity would be destroyed.

It is proposed to advertise capabilities of keeping flow rules
and indirect actions (as a special case of shared object)
using a combination of ethdev info and rte_flow calls.
Then a bug is fixed in mlx5 PMD that prevented indirect RSS action
from being kept, and the driver starts advertising the new capability.

Prior discussions:
1) http://inbox.dpdk.org/dev/20210727073121.895620-1-dkozlyuk@nvidia.com/
2) http://inbox.dpdk.org/dev/20210901085516.3647814-1-dkozlyuk@nvidia.com/

v4:  1. Fix rebase conflicts (CI).
     2. State rule behavior when a port is not started or stopped (Ori).
     3. Improve wording on rule features, add examples (Andrew).
     4. State that rules/actions that cannot be kept while other can be
        must be destroyed by the application (Andrew/Ori).
     5. Add rationale to the cover letter (Andrew).

Dmitry Kozlyuk (6):
  ethdev: add capability to keep flow rules on restart
  ethdev: add capability to keep shared objects on restart
  net: advertise no support for keeping flow rules
  net/mlx5: discover max flow priority using DevX
  net/mlx5: create drop queue using DevX
  net/mlx5: preserve indirect actions on restart

 doc/guides/prog_guide/rte_flow.rst      |  57 +++++
 drivers/net/bnxt/bnxt_ethdev.c          |   1 +
 drivers/net/bnxt/bnxt_reps.c            |   1 +
 drivers/net/cnxk/cnxk_ethdev_ops.c      |   1 +
 drivers/net/cxgbe/cxgbe_ethdev.c        |   2 +
 drivers/net/dpaa2/dpaa2_ethdev.c        |   1 +
 drivers/net/e1000/em_ethdev.c           |   2 +
 drivers/net/e1000/igb_ethdev.c          |   1 +
 drivers/net/enic/enic_ethdev.c          |   1 +
 drivers/net/failsafe/failsafe_ops.c     |   1 +
 drivers/net/hinic/hinic_pmd_ethdev.c    |   2 +
 drivers/net/hns3/hns3_ethdev.c          |   1 +
 drivers/net/hns3/hns3_ethdev_vf.c       |   1 +
 drivers/net/i40e/i40e_ethdev.c          |   1 +
 drivers/net/i40e/i40e_vf_representor.c  |   2 +
 drivers/net/iavf/iavf_ethdev.c          |   1 +
 drivers/net/ice/ice_dcf_ethdev.c        |   1 +
 drivers/net/igc/igc_ethdev.c            |   1 +
 drivers/net/ipn3ke/ipn3ke_representor.c |   1 +
 drivers/net/mlx5/linux/mlx5_os.c        |   5 -
 drivers/net/mlx5/mlx5_devx.c            | 211 ++++++++++++++---
 drivers/net/mlx5/mlx5_ethdev.c          |   1 +
 drivers/net/mlx5/mlx5_flow.c            | 292 ++++++++++++++++++++++--
 drivers/net/mlx5/mlx5_flow.h            |   6 +
 drivers/net/mlx5/mlx5_flow_dv.c         | 103 +++++++++
 drivers/net/mlx5/mlx5_flow_verbs.c      |  77 +------
 drivers/net/mlx5/mlx5_rx.h              |   4 +
 drivers/net/mlx5/mlx5_rxq.c             |  99 +++++++-
 drivers/net/mlx5/mlx5_trigger.c         |  10 +
 drivers/net/mvpp2/mrvl_ethdev.c         |   2 +
 drivers/net/octeontx2/otx2_ethdev_ops.c |   1 +
 drivers/net/qede/qede_ethdev.c          |   1 +
 drivers/net/sfc/sfc_ethdev.c            |   1 +
 drivers/net/softnic/rte_eth_softnic.c   |   1 +
 drivers/net/tap/rte_eth_tap.c           |   1 +
 drivers/net/txgbe/txgbe_ethdev.c        |   1 +
 drivers/net/txgbe/txgbe_ethdev_vf.c     |   1 +
 lib/ethdev/rte_ethdev.h                 |  10 +
 lib/ethdev/rte_flow.h                   |   1 +
 39 files changed, 770 insertions(+), 137 deletions(-)
  

On 10/21/2021 7:34 AM, Dmitry Kozlyuk wrote:
> It is unspecified whether flow rules and indirect actions are kept
> when a port is stopped, possibly reconfigured, and started again.
> Vendors approach the topic differently, e.g. mlx5 and i40e PMD
> disagree in whether flow rules can be kept, and mlx5 PMD would keep
> indirect actions. In the end, applications are greatly affected
> by whatever contract there is and need to know it.
> 
> Applications may wish to restart the port to reconfigure it,
> e.g. switch offloads or even modify queues.
> Keeping rte_flow entities enables application improvements:
> 1. Since keeping the rules across restart comes with the ability
>     to create rules before the device is started. This allows
>     to have all the rules created at the moment of start,
>     so that there is no time frame when traffic is coming already,
>     but the rules are not yet created (restored).
> 2. When a rule or an indirect action has some associated state,
>     such as a counter, application saves the need to keep
>     additional state in order to cope with information loss
>     if such an entity would be destroyed.
> 
> It is proposed to advertise capabilities of keeping flow rules
> and indirect actions (as a special case of shared object)
> using a combination of ethdev info and rte_flow calls.
> Then a bug is fixed in mlx5 PMD that prevented indirect RSS action
> from being kept, and the driver starts advertising the new capability.
> 
> Prior discussions:
> 1) http://inbox.dpdk.org/dev/20210727073121.895620-1-dkozlyuk@nvidia.com/
> 2) http://inbox.dpdk.org/dev/20210901085516.3647814-1-dkozlyuk@nvidia.com/
> 
> v4:  1. Fix rebase conflicts (CI).
>       2. State rule behavior when a port is not started or stopped (Ori).
>       3. Improve wording on rule features, add examples (Andrew).
>       4. State that rules/actions that cannot be kept while other can be
>          must be destroyed by the application (Andrew/Ori).
>       5. Add rationale to the cover letter (Andrew).
> 
> Dmitry Kozlyuk (6):
>    ethdev: add capability to keep flow rules on restart
>    ethdev: add capability to keep shared objects on restart
>    net: advertise no support for keeping flow rules
>    net/mlx5: discover max flow priority using DevX
>    net/mlx5: create drop queue using DevX
>    net/mlx5: preserve indirect actions on restart
> 

Requesting review from PMD maintainers.

Since this patch tries to define behavior on keeping/flushing flow rules
after port stop/start/configure, better to get more feedback from various
vendors, please review/comment on patch so that we can get it for -rc2.

If there is no comment the patch can go in as it is for -rc2.

Thanks,
ferruh
  

On 10/26/2021 12:46 PM, Ferruh Yigit wrote:
> On 10/21/2021 7:34 AM, Dmitry Kozlyuk wrote:
>> It is unspecified whether flow rules and indirect actions are kept
>> when a port is stopped, possibly reconfigured, and started again.
>> Vendors approach the topic differently, e.g. mlx5 and i40e PMD
>> disagree in whether flow rules can be kept, and mlx5 PMD would keep
>> indirect actions. In the end, applications are greatly affected
>> by whatever contract there is and need to know it.
>>
>> Applications may wish to restart the port to reconfigure it,
>> e.g. switch offloads or even modify queues.
>> Keeping rte_flow entities enables application improvements:
>> 1. Since keeping the rules across restart comes with the ability
>>     to create rules before the device is started. This allows
>>     to have all the rules created at the moment of start,
>>     so that there is no time frame when traffic is coming already,
>>     but the rules are not yet created (restored).
>> 2. When a rule or an indirect action has some associated state,
>>     such as a counter, application saves the need to keep
>>     additional state in order to cope with information loss
>>     if such an entity would be destroyed.
>>
>> It is proposed to advertise capabilities of keeping flow rules
>> and indirect actions (as a special case of shared object)
>> using a combination of ethdev info and rte_flow calls.
>> Then a bug is fixed in mlx5 PMD that prevented indirect RSS action
>> from being kept, and the driver starts advertising the new capability.
>>
>> Prior discussions:
>> 1) http://inbox.dpdk.org/dev/20210727073121.895620-1-dkozlyuk@nvidia.com/
>> 2) http://inbox.dpdk.org/dev/20210901085516.3647814-1-dkozlyuk@nvidia.com/
>>
>> v4:  1. Fix rebase conflicts (CI).
>>       2. State rule behavior when a port is not started or stopped (Ori).
>>       3. Improve wording on rule features, add examples (Andrew).
>>       4. State that rules/actions that cannot be kept while other can be
>>          must be destroyed by the application (Andrew/Ori).
>>       5. Add rationale to the cover letter (Andrew).
>>
>> Dmitry Kozlyuk (6):
>>    ethdev: add capability to keep flow rules on restart
>>    ethdev: add capability to keep shared objects on restart
>>    net: advertise no support for keeping flow rules
>>    net/mlx5: discover max flow priority using DevX
>>    net/mlx5: create drop queue using DevX
>>    net/mlx5: preserve indirect actions on restart
>>
> 
> Requesting review from PMD maintainers.
> 
> Since this patch tries to define behavior on keeping/flushing flow rules
> after port stop/start/configure, better to get more feedback from various
> vendors, please review/comment on patch so that we can get it for -rc2.
> 
> If there is no comment the patch can go in as it is for -rc2.
> 

Last review call before merge.
  

On 10/21/2021 7:34 AM, Dmitry Kozlyuk wrote:
> It is unspecified whether flow rules and indirect actions are kept
> when a port is stopped, possibly reconfigured, and started again.
> Vendors approach the topic differently, e.g. mlx5 and i40e PMD
> disagree in whether flow rules can be kept, and mlx5 PMD would keep
> indirect actions. In the end, applications are greatly affected
> by whatever contract there is and need to know it.
> 
> Applications may wish to restart the port to reconfigure it,
> e.g. switch offloads or even modify queues.
> Keeping rte_flow entities enables application improvements:
> 1. Since keeping the rules across restart comes with the ability
>     to create rules before the device is started. This allows
>     to have all the rules created at the moment of start,
>     so that there is no time frame when traffic is coming already,
>     but the rules are not yet created (restored).
> 2. When a rule or an indirect action has some associated state,
>     such as a counter, application saves the need to keep
>     additional state in order to cope with information loss
>     if such an entity would be destroyed.
> 
> It is proposed to advertise capabilities of keeping flow rules
> and indirect actions (as a special case of shared object)
> using a combination of ethdev info and rte_flow calls.
> Then a bug is fixed in mlx5 PMD that prevented indirect RSS action
> from being kept, and the driver starts advertising the new capability.
> 
> Prior discussions:
> 1) http://inbox.dpdk.org/dev/20210727073121.895620-1-dkozlyuk@nvidia.com/
> 2) http://inbox.dpdk.org/dev/20210901085516.3647814-1-dkozlyuk@nvidia.com/
> 
> v4:  1. Fix rebase conflicts (CI).
>       2. State rule behavior when a port is not started or stopped (Ori).
>       3. Improve wording on rule features, add examples (Andrew).
>       4. State that rules/actions that cannot be kept while other can be
>          must be destroyed by the application (Andrew/Ori).
>       5. Add rationale to the cover letter (Andrew).
> 
> Dmitry Kozlyuk (6):
>    ethdev: add capability to keep flow rules on restart
>    ethdev: add capability to keep shared objects on restart
>    net: advertise no support for keeping flow rules
>    net/mlx5: discover max flow priority using DevX
>    net/mlx5: create drop queue using DevX
>    net/mlx5: preserve indirect actions on restart
> 

Hi Dmitry,

Can you please rebase this set on latest next-net?
There are some changes both in ethdev and mlx5.