[v6,00/10] example/l2fwd-event: introduce l2fwd-event example

  From: Pavan Nikhilesh <pbhagavatula@marvell.com>

This patchset adds a new application to demonstrate the usage of event
mode. The poll mode is also available to help with the transition.

The following new command line parameters are added:
 --mode: Dictates the mode of operation either poll or event.
 --eventq_sched: Dictates event scheduling mode ordered, atomic or
		 parallel.

Based on event device capability the configuration is done as follows:
    - A single event device is enabled.
    - The number of event ports is equal to the number of worker
      cores enabled in the core mask. Additional event ports might
      be configured based on Rx/Tx adapter capability.
    - The number of event queues is equal to the number of ethernet
      ports. If Tx adapter doesn't have internal port capability then
      an additional single link event queue is used to enqueue events
      to Tx adapter.
    - Each event port is linked to all existing event queues.
    - Dedicated Rx/Tx adapters for each Ethernet port.

v6 Changes:
- Shorten the structure name `s/event_rsrc/evt_rsrc` `s/l2fwd_rsrc/rsrc`.
- Use rte_panic instead of rte_exit.
- Rebase on top of Tx adapter change http://patches.dpdk.org/patch/60971.
- Update documentation and fix error, spellcheck.
- Fix meson build.
- Split functions into smaller functions for redability.
- Add parallel mode support.

v5 Changes:
- Redo poll mode datapath by removing all the static globals.
- Fix event queue configuration when required queues are not available.
- Fix Rx/Tx adapter creation based on portmask.
- Update release notes.
- Unroll macro used to generate event mode functions.

v4 Changes:
- Fix missing eventdev args parsing.

v3 Changes:
- Remove unwanted change to example/l2fwd.
- Fix checkpatch issue
        http://mails.dpdk.org/archives/test-report/2019-September/098053.html

v2 Changes:
- Remove global variables.
- Split patches to make reviews friendlier.
- Split datapath based on eventdev capability.

Pavan Nikhilesh (5):
  examples/l2fwd-event: add default poll mode routines
  examples/l2fwd-event: add infra for eventdev
  examples/l2fwd-event: add service core setup
  examples/l2fwd-event: add eventdev main loop
  examples/l2fwd-event: add graceful teardown

Sunil Kumar Kori (5):
  examples/l2fwd-event: add infra to split eventdev framework
  examples/l2fwd-event: add event device setup
  examples/l2fwd-event: add eventdev queue and port setup
  examples/l2fwd-event: add event Rx/Tx adapter setup
  doc: add application usage guide for l2fwd-event

 MAINTAINERS                                   |   6 +
 doc/guides/rel_notes/release_19_11.rst        |   6 +
 doc/guides/sample_app_ug/index.rst            |   1 +
 doc/guides/sample_app_ug/intro.rst            |   5 +
 doc/guides/sample_app_ug/l2_forward_event.rst | 711 ++++++++++++++++++
 examples/Makefile                             |   1 +
 examples/l2fwd-event/Makefile                 |  62 ++
 examples/l2fwd-event/l2fwd_common.c           | 144 ++++
 examples/l2fwd-event/l2fwd_common.h           | 133 ++++
 examples/l2fwd-event/l2fwd_event.c            | 431 +++++++++++
 examples/l2fwd-event/l2fwd_event.h            |  73 ++
 examples/l2fwd-event/l2fwd_event_generic.c    | 315 ++++++++
 .../l2fwd-event/l2fwd_event_internal_port.c   | 294 ++++++++
 examples/l2fwd-event/l2fwd_poll.c             | 193 +++++
 examples/l2fwd-event/l2fwd_poll.h             |  25 +
 examples/l2fwd-event/main.c                   | 456 +++++++++++
 examples/l2fwd-event/meson.build              |  18 +
 examples/meson.build                          |   2 +-
 18 files changed, 2875 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/l2_forward_event.rst
 create mode 100644 examples/l2fwd-event/Makefile
 create mode 100644 examples/l2fwd-event/l2fwd_common.c
 create mode 100644 examples/l2fwd-event/l2fwd_common.h
 create mode 100644 examples/l2fwd-event/l2fwd_event.c
 create mode 100644 examples/l2fwd-event/l2fwd_event.h
 create mode 100644 examples/l2fwd-event/l2fwd_event_generic.c
 create mode 100644 examples/l2fwd-event/l2fwd_event_internal_port.c
 create mode 100644 examples/l2fwd-event/l2fwd_poll.c
 create mode 100644 examples/l2fwd-event/l2fwd_poll.h
 create mode 100644 examples/l2fwd-event/main.c
 create mode 100644 examples/l2fwd-event/meson.build

--
2.17.1
  

On Mon, Oct 14, 2019 at 11:52 PM <pbhagavatula@marvell.com> wrote:
>
> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>
> This patchset adds a new application to demonstrate the usage of event
> mode. The poll mode is also available to help with the transition.
>
> The following new command line parameters are added:
>  --mode: Dictates the mode of operation either poll or event.
>  --eventq_sched: Dictates event scheduling mode ordered, atomic or
>                  parallel.
>
> Based on event device capability the configuration is done as follows:
>     - A single event device is enabled.
>     - The number of event ports is equal to the number of worker
>       cores enabled in the core mask. Additional event ports might
>       be configured based on Rx/Tx adapter capability.
>     - The number of event queues is equal to the number of ethernet
>       ports. If Tx adapter doesn't have internal port capability then
>       an additional single link event queue is used to enqueue events
>       to Tx adapter.
>     - Each event port is linked to all existing event queues.
>     - Dedicated Rx/Tx adapters for each Ethernet port.


+ Adding all eventdev PMD maintainers.

# Got the ACK from NXP after testing with their HW.
# I will merge this patch after the RC1. i.e End of next week if there
are no more review comments.
# Thanks Anoob and Lukas for the initial versions of this l2fwd-event support.

>
> v6 Changes:
> - Shorten the structure name `s/event_rsrc/evt_rsrc` `s/l2fwd_rsrc/rsrc`.
> - Use rte_panic instead of rte_exit.
> - Rebase on top of Tx adapter change http://patches.dpdk.org/patch/60971.
> - Update documentation and fix error, spellcheck.
> - Fix meson build.
> - Split functions into smaller functions for redability.
> - Add parallel mode support.
>
> v5 Changes:
> - Redo poll mode datapath by removing all the static globals.
> - Fix event queue configuration when required queues are not available.
> - Fix Rx/Tx adapter creation based on portmask.
> - Update release notes.
> - Unroll macro used to generate event mode functions.
>
> v4 Changes:
> - Fix missing eventdev args parsing.
>
> v3 Changes:
> - Remove unwanted change to example/l2fwd.
> - Fix checkpatch issue
>         http://mails.dpdk.org/archives/test-report/2019-September/098053.html
>
> v2 Changes:
> - Remove global variables.
> - Split patches to make reviews friendlier.
> - Split datapath based on eventdev capability.
>
> Pavan Nikhilesh (5):
>   examples/l2fwd-event: add default poll mode routines
>   examples/l2fwd-event: add infra for eventdev
>   examples/l2fwd-event: add service core setup
>   examples/l2fwd-event: add eventdev main loop
>   examples/l2fwd-event: add graceful teardown
>
> Sunil Kumar Kori (5):
>   examples/l2fwd-event: add infra to split eventdev framework
>   examples/l2fwd-event: add event device setup
>   examples/l2fwd-event: add eventdev queue and port setup
>   examples/l2fwd-event: add event Rx/Tx adapter setup
>   doc: add application usage guide for l2fwd-event
>
>  MAINTAINERS                                   |   6 +
>  doc/guides/rel_notes/release_19_11.rst        |   6 +
>  doc/guides/sample_app_ug/index.rst            |   1 +
>  doc/guides/sample_app_ug/intro.rst            |   5 +
>  doc/guides/sample_app_ug/l2_forward_event.rst | 711 ++++++++++++++++++
>  examples/Makefile                             |   1 +
>  examples/l2fwd-event/Makefile                 |  62 ++
>  examples/l2fwd-event/l2fwd_common.c           | 144 ++++
>  examples/l2fwd-event/l2fwd_common.h           | 133 ++++
>  examples/l2fwd-event/l2fwd_event.c            | 431 +++++++++++
>  examples/l2fwd-event/l2fwd_event.h            |  73 ++
>  examples/l2fwd-event/l2fwd_event_generic.c    | 315 ++++++++
>  .../l2fwd-event/l2fwd_event_internal_port.c   | 294 ++++++++
>  examples/l2fwd-event/l2fwd_poll.c             | 193 +++++
>  examples/l2fwd-event/l2fwd_poll.h             |  25 +
>  examples/l2fwd-event/main.c                   | 456 +++++++++++
>  examples/l2fwd-event/meson.build              |  18 +
>  examples/meson.build                          |   2 +-
>  18 files changed, 2875 insertions(+), 1 deletion(-)
>  create mode 100644 doc/guides/sample_app_ug/l2_forward_event.rst
>  create mode 100644 examples/l2fwd-event/Makefile
>  create mode 100644 examples/l2fwd-event/l2fwd_common.c
>  create mode 100644 examples/l2fwd-event/l2fwd_common.h
>  create mode 100644 examples/l2fwd-event/l2fwd_event.c
>  create mode 100644 examples/l2fwd-event/l2fwd_event.h
>  create mode 100644 examples/l2fwd-event/l2fwd_event_generic.c
>  create mode 100644 examples/l2fwd-event/l2fwd_event_internal_port.c
>  create mode 100644 examples/l2fwd-event/l2fwd_poll.c
>  create mode 100644 examples/l2fwd-event/l2fwd_poll.h
>  create mode 100644 examples/l2fwd-event/main.c
>  create mode 100644 examples/l2fwd-event/meson.build
>
> --
> 2.17.1
>
  

Hi Pavan,

Thanks for sharing the write up, following are some of my thoughts.

Snipped
> 
> Based on event device capability the configuration is done as follows:
>     - A single event device is enabled.
>     - The number of event ports is equal to the number of worker
>       cores enabled in the core mask. Additional event ports might
>       be configured based on Rx/Tx adapter capability.
>     - The number of event queues is equal to the number of ethernet
>       ports. If Tx adapter doesn't have internal port capability then
>       an additional single link event queue is used to enqueue events
>       to Tx adapter.
1. Are we support per port function as 'mac_updating' and 'non_macupdating'? 
2. If no for above, why is that each port should have ` number of event queues is equal to the number of ethernet`? 
3. With this work for vdev ports, if no are we adding check for the same in `main` function?

snipped
  

Hi Vipin,

>Hi Pavan,
>
>Thanks for sharing the write up, following are some of my thoughts.
>
>Snipped
>>
>> Based on event device capability the configuration is done as follows:
>>     - A single event device is enabled.
>>     - The number of event ports is equal to the number of worker
>>       cores enabled in the core mask. Additional event ports might
>>       be configured based on Rx/Tx adapter capability.
>>     - The number of event queues is equal to the number of ethernet
>>       ports. If Tx adapter doesn't have internal port capability then
>>       an additional single link event queue is used to enqueue events
>>       to Tx adapter.
>1. Are we support per port function as 'mac_updating' and
>'non_macupdating'?
No, its global.

>2. If no for above, why is that each port should have ` number of event
>queues is equal to the number of ethernet`?

It's not a quantifier of each port, It across the eventdevice that the number 
of event queues is equal to number of ethdevices used.

This is to prevent event queues being overcrowded i.e. in case there is only 
one event queue and multiple Eth devices then SW/HW will have a bottleneck
in enqueueing all the mbufs to a single event queue. 

>3. With this work for vdev ports, if no are we adding check for the same
>in `main` function?

I have verified the functionality for --vdev=event_sw0 and it seems to work fine.

>
>snipped
  

Hi Pavan,

snipped
> >>
> >> Based on event device capability the configuration is done as follows:
> >>     - A single event device is enabled.
> >>     - The number of event ports is equal to the number of worker
> >>       cores enabled in the core mask. Additional event ports might
> >>       be configured based on Rx/Tx adapter capability.
> >>     - The number of event queues is equal to the number of ethernet
> >>       ports. If Tx adapter doesn't have internal port capability then
> >>       an additional single link event queue is used to enqueue events
> >>       to Tx adapter.
> >1. Are we support per port function as 'mac_updating' and
> >'non_macupdating'?
> No, its global.
Thanks for confirming, so all ports behaviour are affected by this flag.

> 
> >2. If no for above, why is that each port should have ` number of event
> >queues is equal to the number of ethernet`?
> 
> It's not a quantifier of each port, It across the eventdevice that the number of
> event queues is equal to number of ethdevices used.
> 
> This is to prevent event queues being overcrowded i.e. in case there is only one
> event queue and multiple Eth devices then SW/HW will have a bottleneck in
> enqueueing all the mbufs to a single event queue.
Yes you are correct, but here is my confusion 

1. let us assume there are 2 ports, ie: port 0 <==> port 1.
2. We have 4 workers and 2 event queue (2 ports, so Q-0 for port-0 and Q-11 for port-1)
3. The event mode (SW or HW) is parallel.
4. In this case we need to rely of Q-2 which can absorb the events for single-event TX adapter for L2FWD_EVENT_TX_ENQ.
5. But for L2FWD_EVENT_TX_DIRECT (which is to send packets out directly) this is not right form as multiple parallel workers may concurrently send out same destination port queue (as there is 1 TX configured).

Can you help me understand, how this is worked around?

> 
> >3. With this work for vdev ports, if no are we adding check for the
> >same in `main` function?
> 
> I have verified the functionality for --vdev=event_sw0 and it seems to work fine.
Thanks, so whether it is physical or virtual ethernet device all packets has to come to worker cores for 'no-mac-updating'.

> 
> >
> >snipped
  

<snip>
>>
>> It's not a quantifier of each port, It across the eventdevice that the
>number of
>> event queues is equal to number of ethdevices used.
>>
>> This is to prevent event queues being overcrowded i.e. in case there
>is only one
>> event queue and multiple Eth devices then SW/HW will have a
>bottleneck in
>> enqueueing all the mbufs to a single event queue.
>Yes you are correct, but here is my confusion
>
>1. let us assume there are 2 ports, ie: port 0 <==> port 1.
>2. We have 4 workers and 2 event queue (2 ports, so Q-0 for port-0 and
>Q-11 for port-1)
>3. The event mode (SW or HW) is parallel.
>4. In this case we need to rely of Q-2 which can absorb the events for
>single-event TX adapter for L2FWD_EVENT_TX_ENQ.
>5. But for L2FWD_EVENT_TX_DIRECT (which is to send packets out
>directly) this is not right form as multiple parallel workers may
>concurrently send out same destination port queue (as there is 1 TX
>configured).
>
>Can you help me understand, how this is worked around?

We only select TX_DIRECT when the eventdev coupled with ethdev has 
RTE_EVENT_ETH_TX_ADAPTER_CAP_INTERNAL_PORT capability.
It is the PMDs responsibility to synchronize concurrent Tx queues access
across queues if it exposes the above capability.

In case of octeontx2 we expose RTE_EVENT_ETH_TX_ADAPTER_CAP_INTERNAL_PORT
only when both eventdev and ethdev are octeontx2 pmds. 
Since octeontx2 has DEV_TX_OFFLOAD_MT_LOCKFREE multicore Tx queue atomicity is
taken care by HW.

>
>>
>> >3. With this work for vdev ports, if no are we adding check for the
>> >same in `main` function?
>>
>> I have verified the functionality for --vdev=event_sw0 and it seems to
>work fine.
>Thanks, so whether it is physical or virtual ethernet device all packets
>has to come to worker cores for 'no-mac-updating'.
>
>>
>> >
>> >snipped