[RFC,0/1] app/testpmd: add l3fwd mode to testpmd

  Performance of the LPM mode in L3fwd example application is used as an industry
standard to compare between various platforms.

Unfortunately, L3fwd example application lacks debugging capabilities to
understand the performance bottlenecks and fix them.

While debugging performance issues we need all the flexibility possible.
Some of the capabilities we have used are:
1) ability to print hardware and software statistics - xstats, stats at
   port/queue level, burst stats to identify any headroom available,
   CPU cycles/packet etc
2) ability to modify all possible configurable parameters for the PMD
   as well as the application at run time without recompiling the code.
   Some of the parameters we have used are RX/TX queue depths, burst size,
   number of receive queues, PMD specific parameters etc. This
   configurability at runtime helps to understand and debug L3fwd
   performance issues quickly and effectively.

It is possible to add all these capabilities to L3fwd example application.
However, doing that we will result in L3fwd example application losing
its purpose (of being a sample application). At the same time, testpmd
application has all these capabilities to debug an application. In my opinion
it makes sense to add L3fwd mode to testpmd.

This patch adds l3fwd mode into testpmd to take advantage of the
existing infrastructure in testpmd.

I'd like to hear from the community if the structure of this change makes sense,
namely, adding l3fwd as a separate fwd_engine into testpmd.

This feature is not yet implemeted for SSE or AltiVec.

Kathleen Capella (1):
  app/testpmd: add l3fwd mode to testpmd

 app/test-pmd/config.c         |  66 +++++++
 app/test-pmd/l3fwd.c          | 356 ++++++++++++++++++++++++++++++++++
 app/test-pmd/l3fwd.h          | 143 ++++++++++++++
 app/test-pmd/l3fwd_common.h   | 268 +++++++++++++++++++++++++
 app/test-pmd/l3fwd_lpm.h      | 107 ++++++++++
 app/test-pmd/l3fwd_lpm_neon.h | 169 ++++++++++++++++
 app/test-pmd/l3fwd_neon.h     | 234 ++++++++++++++++++++++
 app/test-pmd/meson.build      |   3 +-
 app/test-pmd/testpmd.c        |   4 +-
 app/test-pmd/testpmd.h        |  20 ++
 10 files changed, 1368 insertions(+), 2 deletions(-)
 create mode 100644 app/test-pmd/l3fwd.c
 create mode 100644 app/test-pmd/l3fwd.h
 create mode 100644 app/test-pmd/l3fwd_common.h
 create mode 100644 app/test-pmd/l3fwd_lpm.h
 create mode 100644 app/test-pmd/l3fwd_lpm_neon.h
 create mode 100644 app/test-pmd/l3fwd_neon.h
  

@Xiaoyun could you share your thoughts on it?

As far as I remember there is no agreement on the topic. Adding
more people in Cc.

On 5/1/21 12:37 AM, Kathleen Capella wrote:
> Performance of the LPM mode in L3fwd example application is used as an industry
> standard to compare between various platforms.
> 
> Unfortunately, L3fwd example application lacks debugging capabilities to
> understand the performance bottlenecks and fix them.
> 
> While debugging performance issues we need all the flexibility possible.
> Some of the capabilities we have used are:
> 1) ability to print hardware and software statistics - xstats, stats at
>    port/queue level, burst stats to identify any headroom available,
>    CPU cycles/packet etc
> 2) ability to modify all possible configurable parameters for the PMD
>    as well as the application at run time without recompiling the code.
>    Some of the parameters we have used are RX/TX queue depths, burst size,
>    number of receive queues, PMD specific parameters etc. This
>    configurability at runtime helps to understand and debug L3fwd
>    performance issues quickly and effectively.
> 
> It is possible to add all these capabilities to L3fwd example application.
> However, doing that we will result in L3fwd example application losing
> its purpose (of being a sample application). At the same time, testpmd
> application has all these capabilities to debug an application. In my opinion
> it makes sense to add L3fwd mode to testpmd.
> 
> This patch adds l3fwd mode into testpmd to take advantage of the
> existing infrastructure in testpmd.
> 
> I'd like to hear from the community if the structure of this change makes sense,
> namely, adding l3fwd as a separate fwd_engine into testpmd.
> 
> This feature is not yet implemeted for SSE or AltiVec.
> 
> Kathleen Capella (1):
>   app/testpmd: add l3fwd mode to testpmd
> 
>  app/test-pmd/config.c         |  66 +++++++
>  app/test-pmd/l3fwd.c          | 356 ++++++++++++++++++++++++++++++++++
>  app/test-pmd/l3fwd.h          | 143 ++++++++++++++
>  app/test-pmd/l3fwd_common.h   | 268 +++++++++++++++++++++++++
>  app/test-pmd/l3fwd_lpm.h      | 107 ++++++++++
>  app/test-pmd/l3fwd_lpm_neon.h | 169 ++++++++++++++++
>  app/test-pmd/l3fwd_neon.h     | 234 ++++++++++++++++++++++
>  app/test-pmd/meson.build      |   3 +-
>  app/test-pmd/testpmd.c        |   4 +-
>  app/test-pmd/testpmd.h        |  20 ++
>  10 files changed, 1368 insertions(+), 2 deletions(-)
>  create mode 100644 app/test-pmd/l3fwd.c
>  create mode 100644 app/test-pmd/l3fwd.h
>  create mode 100644 app/test-pmd/l3fwd_common.h
>  create mode 100644 app/test-pmd/l3fwd_lpm.h
>  create mode 100644 app/test-pmd/l3fwd_lpm_neon.h
>  create mode 100644 app/test-pmd/l3fwd_neon.h
>
  

On 7/2/2021 11:15 AM, Andrew Rybchenko wrote:
> @Xiaoyun could you share your thoughts on it?
> 
> As far as I remember there is no agreement on the topic. Adding
> more people in Cc.
> 

I was OK for adding simple l3fwd forwarding engine to testpmd, to benefit from
configuration/debugging/measurement benefits testpmd brings as patch mentions.

But adding neon will likely bring other architecture specific implementations,
and there will be more code duplicates, which is not good.
Also it is possible that people may want to add more lookup methods (em, fib..)
making things worse.

If we get the feature, what about limiting it to scalar implementation and LPM?
Still it is questionable to have the feature in the testpmd, but at least we
limit the scope.
For performance measurements can use the l3fwd sample application.

Another point is testing, this feature should come with dts updates to test
testpmd l3fwd, otherwise it may not be tested and turn into dead code easily.


> On 5/1/21 12:37 AM, Kathleen Capella wrote:
>> Performance of the LPM mode in L3fwd example application is used as an industry
>> standard to compare between various platforms.
>>
>> Unfortunately, L3fwd example application lacks debugging capabilities to
>> understand the performance bottlenecks and fix them.
>>
>> While debugging performance issues we need all the flexibility possible.
>> Some of the capabilities we have used are:
>> 1) ability to print hardware and software statistics - xstats, stats at
>>    port/queue level, burst stats to identify any headroom available,
>>    CPU cycles/packet etc
>> 2) ability to modify all possible configurable parameters for the PMD
>>    as well as the application at run time without recompiling the code.
>>    Some of the parameters we have used are RX/TX queue depths, burst size,
>>    number of receive queues, PMD specific parameters etc. This
>>    configurability at runtime helps to understand and debug L3fwd
>>    performance issues quickly and effectively.
>>
>> It is possible to add all these capabilities to L3fwd example application.
>> However, doing that we will result in L3fwd example application losing
>> its purpose (of being a sample application). At the same time, testpmd
>> application has all these capabilities to debug an application. In my opinion
>> it makes sense to add L3fwd mode to testpmd.
>>
>> This patch adds l3fwd mode into testpmd to take advantage of the
>> existing infrastructure in testpmd.
>>
>> I'd like to hear from the community if the structure of this change makes sense,
>> namely, adding l3fwd as a separate fwd_engine into testpmd.
>>
>> This feature is not yet implemeted for SSE or AltiVec.
>>
>> Kathleen Capella (1):
>>   app/testpmd: add l3fwd mode to testpmd
>>
>>  app/test-pmd/config.c         |  66 +++++++
>>  app/test-pmd/l3fwd.c          | 356 ++++++++++++++++++++++++++++++++++
>>  app/test-pmd/l3fwd.h          | 143 ++++++++++++++
>>  app/test-pmd/l3fwd_common.h   | 268 +++++++++++++++++++++++++
>>  app/test-pmd/l3fwd_lpm.h      | 107 ++++++++++
>>  app/test-pmd/l3fwd_lpm_neon.h | 169 ++++++++++++++++
>>  app/test-pmd/l3fwd_neon.h     | 234 ++++++++++++++++++++++
>>  app/test-pmd/meson.build      |   3 +-
>>  app/test-pmd/testpmd.c        |   4 +-
>>  app/test-pmd/testpmd.h        |  20 ++
>>  10 files changed, 1368 insertions(+), 2 deletions(-)
>>  create mode 100644 app/test-pmd/l3fwd.c
>>  create mode 100644 app/test-pmd/l3fwd.h
>>  create mode 100644 app/test-pmd/l3fwd_common.h
>>  create mode 100644 app/test-pmd/l3fwd_lpm.h
>>  create mode 100644 app/test-pmd/l3fwd_lpm_neon.h
>>  create mode 100644 app/test-pmd/l3fwd_neon.h
>>
>
  

<snip>

> 
> On 7/2/2021 11:15 AM, Andrew Rybchenko wrote:
> > @Xiaoyun could you share your thoughts on it?
> >
> > As far as I remember there is no agreement on the topic. Adding more
> > people in Cc.
> >
> 
> I was OK for adding simple l3fwd forwarding engine to testpmd, to benefit from
> configuration/debugging/measurement benefits testpmd brings as patch
> mentions.
> 
> But adding neon will likely bring other architecture specific implementations,
> and there will be more code duplicates, which is not good.
> Also it is possible that people may want to add more lookup methods (em, fib..)
> making things worse.
The main goal we are trying to address is the ability to debugging the performance issues of the L3fwd application. As far as I know, the marketing folks care about LPM (may be replace LPM with fib). We could definitely avoid adding exact match.
Since the L3fwd application is about showcasing the best possible performance, it is better to keep vector implementation and skip scalar code. This will help debug the correct code path.

> 
> If we get the feature, what about limiting it to scalar implementation and LPM?
I agree with LPM, scalar only might not be very useful.

> Still it is questionable to have the feature in the testpmd, but at least we limit
> the scope.
> For performance measurements can use the l3fwd sample application.
> 
> Another point is testing, this feature should come with dts updates to test
> testpmd l3fwd, otherwise it may not be tested and turn into dead code easily.
Agree

> 
> 
> > On 5/1/21 12:37 AM, Kathleen Capella wrote:
> >> Performance of the LPM mode in L3fwd example application is used as
> >> an industry standard to compare between various platforms.
> >>
> >> Unfortunately, L3fwd example application lacks debugging capabilities
> >> to understand the performance bottlenecks and fix them.
> >>
> >> While debugging performance issues we need all the flexibility possible.
> >> Some of the capabilities we have used are:
> >> 1) ability to print hardware and software statistics - xstats, stats at
> >>    port/queue level, burst stats to identify any headroom available,
> >>    CPU cycles/packet etc
> >> 2) ability to modify all possible configurable parameters for the PMD
> >>    as well as the application at run time without recompiling the code.
> >>    Some of the parameters we have used are RX/TX queue depths, burst size,
> >>    number of receive queues, PMD specific parameters etc. This
> >>    configurability at runtime helps to understand and debug L3fwd
> >>    performance issues quickly and effectively.
> >>
> >> It is possible to add all these capabilities to L3fwd example application.
> >> However, doing that we will result in L3fwd example application
> >> losing its purpose (of being a sample application). At the same time,
> >> testpmd application has all these capabilities to debug an
> >> application. In my opinion it makes sense to add L3fwd mode to testpmd.
> >>
> >> This patch adds l3fwd mode into testpmd to take advantage of the
> >> existing infrastructure in testpmd.
> >>
> >> I'd like to hear from the community if the structure of this change
> >> makes sense, namely, adding l3fwd as a separate fwd_engine into testpmd.
> >>
> >> This feature is not yet implemeted for SSE or AltiVec.
> >>
> >> Kathleen Capella (1):
> >>   app/testpmd: add l3fwd mode to testpmd
> >>
> >>  app/test-pmd/config.c         |  66 +++++++
> >>  app/test-pmd/l3fwd.c          | 356 ++++++++++++++++++++++++++++++++++
> >>  app/test-pmd/l3fwd.h          | 143 ++++++++++++++
> >>  app/test-pmd/l3fwd_common.h   | 268 +++++++++++++++++++++++++
> >>  app/test-pmd/l3fwd_lpm.h      | 107 ++++++++++
> >>  app/test-pmd/l3fwd_lpm_neon.h | 169 ++++++++++++++++
> >>  app/test-pmd/l3fwd_neon.h     | 234 ++++++++++++++++++++++
> >>  app/test-pmd/meson.build      |   3 +-
> >>  app/test-pmd/testpmd.c        |   4 +-
> >>  app/test-pmd/testpmd.h        |  20 ++
> >>  10 files changed, 1368 insertions(+), 2 deletions(-)  create mode
> >> 100644 app/test-pmd/l3fwd.c  create mode 100644 app/test-pmd/l3fwd.h
> >> create mode 100644 app/test-pmd/l3fwd_common.h  create mode 100644
> >> app/test-pmd/l3fwd_lpm.h  create mode 100644
> >> app/test-pmd/l3fwd_lpm_neon.h  create mode 100644
> >> app/test-pmd/l3fwd_neon.h
> >>
> >