[v3,0/9] GPU library

  From: eagostini <eagostini@nvidia.com>

In heterogeneous computing system, processing is not only in the CPU.
Some tasks can be delegated to devices working in parallel.

The goal of this new library is to enhance the collaboration between
DPDK, that's primarily a CPU framework, and GPU devices.

When mixing network activity with task processing on a non-CPU device,
there may be the need to put in communication the CPU with the device
in order to manage the memory, synchronize operations, exchange info, etc..

This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPU

The infrastructure is prepared to welcome drivers in drivers/gpu/
as the CUDA one, sent as draft:
https://patches.dpdk.org/project/dpdk/patch/20211005224905.13505-1-eagostini@nvidia.com/


Elena Agostini (6):
  gpudev: introduce GPU device class library
  gpudev: add memory API
  gpudev: add memory barrier
  gpudev: add communication flag
  gpudev: add communication list
  doc: add CUDA example in GPU guide

Thomas Monjalon (3):
  gpudev: add event notification
  gpudev: add child device representing a device context
  gpudev: support multi-process

 .gitignore                             |   1 +
 MAINTAINERS                            |   6 +
 app/meson.build                        |   1 +
 app/test-gpudev/main.c                 | 394 +++++++++++
 app/test-gpudev/meson.build            |   5 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 doc/guides/conf.py                     |   8 +
 doc/guides/gpus/features/default.ini   |  13 +
 doc/guides/gpus/index.rst              |  11 +
 doc/guides/gpus/overview.rst           |  10 +
 doc/guides/index.rst                   |   1 +
 doc/guides/prog_guide/gpudev.rst       | 226 +++++++
 doc/guides/prog_guide/index.rst        |   1 +
 doc/guides/rel_notes/release_21_11.rst |   6 +
 drivers/gpu/meson.build                |   4 +
 drivers/meson.build                    |   1 +
 lib/gpudev/gpudev.c                    | 904 +++++++++++++++++++++++++
 lib/gpudev/gpudev_driver.h             | 102 +++
 lib/gpudev/meson.build                 |  12 +
 lib/gpudev/rte_gpudev.h                | 649 ++++++++++++++++++
 lib/gpudev/version.map                 |  38 ++
 lib/meson.build                        |   1 +
 23 files changed, 2396 insertions(+)
 create mode 100644 app/test-gpudev/main.c
 create mode 100644 app/test-gpudev/meson.build
 create mode 100644 doc/guides/gpus/features/default.ini
 create mode 100644 doc/guides/gpus/index.rst
 create mode 100644 doc/guides/gpus/overview.rst
 create mode 100644 doc/guides/prog_guide/gpudev.rst
 create mode 100644 drivers/gpu/meson.build
 create mode 100644 lib/gpudev/gpudev.c
 create mode 100644 lib/gpudev/gpudev_driver.h
 create mode 100644 lib/gpudev/meson.build
 create mode 100644 lib/gpudev/rte_gpudev.h
 create mode 100644 lib/gpudev/version.map
  

On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
>
> From: eagostini <eagostini@nvidia.com>
>
> In heterogeneous computing system, processing is not only in the CPU.
> Some tasks can be delegated to devices working in parallel.
>
> The goal of this new library is to enhance the collaboration between
> DPDK, that's primarily a CPU framework, and GPU devices.
>
> When mixing network activity with task processing on a non-CPU device,
> there may be the need to put in communication the CPU with the device
> in order to manage the memory, synchronize operations, exchange info, etc..
>
> This library provides a number of new features:
> - Interoperability with GPU-specific library with generic handlers
> - Possibility to allocate and free memory on the GPU
> - Possibility to allocate and free memory on the CPU but visible from the GPU
> - Communication functions to enhance the dialog between the CPU and the GPU

In the RFC thread, There was one outstanding non technical issues on this,

i.e
The above features are driver specific details. Does the DPDK
_application_ need to be aware of this?
aka DPDK device class has a fixed personality and it has API to deal
with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective
of underlying bus/device properties.

Even similar semantics are required for DPU(Smart NIC)
communitication. I am planning to
send RFC in coming days to address the issue without the application
knowing the Bus/HW/Driver details.

Irrespective of the RFC I am planning to send and since the new
library needs techboard approval, You may
request that the techboard decide approval for this library. Also, As
far as I remember a minimum a SW driver in
additional to HW driver to accept a new driver class.

Just my 2c to save your cycles.
  

10/10/2021 12:16, Jerin Jacob:
> On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> >
> > From: eagostini <eagostini@nvidia.com>
> >
> > In heterogeneous computing system, processing is not only in the CPU.
> > Some tasks can be delegated to devices working in parallel.
> >
> > The goal of this new library is to enhance the collaboration between
> > DPDK, that's primarily a CPU framework, and GPU devices.
> >
> > When mixing network activity with task processing on a non-CPU device,
> > there may be the need to put in communication the CPU with the device
> > in order to manage the memory, synchronize operations, exchange info, etc..
> >
> > This library provides a number of new features:
> > - Interoperability with GPU-specific library with generic handlers
> > - Possibility to allocate and free memory on the GPU
> > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > - Communication functions to enhance the dialog between the CPU and the GPU
> 
> In the RFC thread, There was one outstanding non technical issues on this,
> 
> i.e
> The above features are driver specific details. Does the DPDK
> _application_ need to be aware of this?

I don't see these features as driver-specific.

> aka DPDK device class has a fixed personality and it has API to deal
> with abstracting specific application specific
> end user functionality like ethdev, cryptodev, eventdev irrespective
> of underlying bus/device properties.

The goal of the lib is to allow anyone to invent any feature
which is not already available in DPDK.

> Even similar semantics are required for DPU(Smart NIC)
> communitication. I am planning to
> send RFC in coming days to address the issue without the application
> knowing the Bus/HW/Driver details.

gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.

> Irrespective of the RFC I am planning to send and since the new
> library needs techboard approval, You may
> request that the techboard decide approval for this library. Also, As
> far as I remember a minimum a SW driver in
> additional to HW driver to accept a new driver class.

No, only one driver is required:
"When introducing a new device API, at least one driver should implement it."
Anyway, a SW driver doesn't make sense for gpudev.
  

On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 10/10/2021 12:16, Jerin Jacob:
> > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > >
> > > From: eagostini <eagostini@nvidia.com>
> > >
> > > In heterogeneous computing system, processing is not only in the CPU.
> > > Some tasks can be delegated to devices working in parallel.
> > >
> > > The goal of this new library is to enhance the collaboration between
> > > DPDK, that's primarily a CPU framework, and GPU devices.
> > >
> > > When mixing network activity with task processing on a non-CPU device,
> > > there may be the need to put in communication the CPU with the device
> > > in order to manage the memory, synchronize operations, exchange info, etc..
> > >
> > > This library provides a number of new features:
> > > - Interoperability with GPU-specific library with generic handlers
> > > - Possibility to allocate and free memory on the GPU
> > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > - Communication functions to enhance the dialog between the CPU and the GPU
> >
> > In the RFC thread, There was one outstanding non technical issues on this,
> >
> > i.e
> > The above features are driver specific details. Does the DPDK
> > _application_ need to be aware of this?
>
> I don't see these features as driver-specific.

That is the disconnect. I see this as more driver-specific details
which are not required
to implement an "application" facing API.

For example, If we need to implement application facing" subsystems like bbdev,
If we make all this driver interface, you can still implement the
bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how
memory is allocated etc
 to "application".



>
> > aka DPDK device class has a fixed personality and it has API to deal
> > with abstracting specific application specific
> > end user functionality like ethdev, cryptodev, eventdev irrespective
> > of underlying bus/device properties.
>
> The goal of the lib is to allow anyone to invent any feature
> which is not already available in DPDK.
>
> > Even similar semantics are required for DPU(Smart NIC)
> > communitication. I am planning to
> > send RFC in coming days to address the issue without the application
> > knowing the Bus/HW/Driver details.
>
> gpudev is not exposing bus/hw/driver details.
> I don't understand what you mean.


See above.

>
> > Irrespective of the RFC I am planning to send and since the new
> > library needs techboard approval, You may
> > request that the techboard decide approval for this library. Also, As
> > far as I remember a minimum a SW driver in
> > additional to HW driver to accept a new driver class.
>
> No, only one driver is required:

That is good news.

> "When introducing a new device API, at least one driver should implement it."
> Anyway, a SW driver doesn't make sense for gpudev.

OK

>
>
>
  

11/10/2021 10:43, Jerin Jacob:
> On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > 10/10/2021 12:16, Jerin Jacob:
> > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > >
> > > > From: eagostini <eagostini@nvidia.com>
> > > >
> > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > Some tasks can be delegated to devices working in parallel.
> > > >
> > > > The goal of this new library is to enhance the collaboration between
> > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > >
> > > > When mixing network activity with task processing on a non-CPU device,
> > > > there may be the need to put in communication the CPU with the device
> > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > >
> > > > This library provides a number of new features:
> > > > - Interoperability with GPU-specific library with generic handlers
> > > > - Possibility to allocate and free memory on the GPU
> > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > >
> > > In the RFC thread, There was one outstanding non technical issues on this,
> > >
> > > i.e
> > > The above features are driver specific details. Does the DPDK
> > > _application_ need to be aware of this?
> >
> > I don't see these features as driver-specific.
> 
> That is the disconnect. I see this as more driver-specific details
> which are not required to implement an "application" facing API.

Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.

First, this is not driver-specific. It is a low-level API.

> For example, If we need to implement application facing" subsystems like bbdev,
> If we make all this driver interface, you can still implement the
> bbdev API as a driver without
> exposing HW specific details like how devices communicate to CPU, how
> memory is allocated etc
>  to "application".

There are 2 things to understand here.

First we want to allow the application using the GPU for needs which are
not exposed by any other DPDK API.

Second, if we want to implement another DPDK API like bbdev,
then the GPU implementation would be exposed as a vdev in bbdev,
using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?

> > > aka DPDK device class has a fixed personality and it has API to deal
> > > with abstracting specific application specific
> > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > of underlying bus/device properties.
> >
> > The goal of the lib is to allow anyone to invent any feature
> > which is not already available in DPDK.
> >
> > > Even similar semantics are required for DPU(Smart NIC)
> > > communitication. I am planning to
> > > send RFC in coming days to address the issue without the application
> > > knowing the Bus/HW/Driver details.
> >
> > gpudev is not exposing bus/hw/driver details.
> > I don't understand what you mean.
> 
> See above.
  

On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 11/10/2021 10:43, Jerin Jacob:
> > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 10/10/2021 12:16, Jerin Jacob:
> > > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > > >
> > > > > From: eagostini <eagostini@nvidia.com>
> > > > >
> > > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > > Some tasks can be delegated to devices working in parallel.
> > > > >
> > > > > The goal of this new library is to enhance the collaboration between
> > > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > > >
> > > > > When mixing network activity with task processing on a non-CPU device,
> > > > > there may be the need to put in communication the CPU with the device
> > > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > > >
> > > > > This library provides a number of new features:
> > > > > - Interoperability with GPU-specific library with generic handlers
> > > > > - Possibility to allocate and free memory on the GPU
> > > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > > >
> > > > In the RFC thread, There was one outstanding non technical issues on this,
> > > >
> > > > i.e
> > > > The above features are driver specific details. Does the DPDK
> > > > _application_ need to be aware of this?
> > >
> > > I don't see these features as driver-specific.
> >
> > That is the disconnect. I see this as more driver-specific details
> > which are not required to implement an "application" facing API.
>
> Indeed this is the disconnect.
> I already answered but it seems you don't accept the answer.

Same with you. That is why I requested, we need to get opinions from others.
Some of them already provided opinions in RFC.

>
> First, this is not driver-specific. It is a low-level API.

What is the difference between low-level API and driver-level API.


>
> > For example, If we need to implement application facing" subsystems like bbdev,
> > If we make all this driver interface, you can still implement the
> > bbdev API as a driver without
> > exposing HW specific details like how devices communicate to CPU, how
> > memory is allocated etc
> >  to "application".
>
> There are 2 things to understand here.
>
> First we want to allow the application using the GPU for needs which are
> not exposed by any other DPDK API.
>
> Second, if we want to implement another DPDK API like bbdev,
> then the GPU implementation would be exposed as a vdev in bbdev,
> using the HW GPU device being a PCI in gpudev.
> They are two different levels, got it?

Exactly. So what is the point of exposing low-level driver API to
"application",
why not it is part of the internal driver API. My point is, why the
application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory
visibility etc.

>
> > > > aka DPDK device class has a fixed personality and it has API to deal
> > > > with abstracting specific application specific
> > > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > > of underlying bus/device properties.
> > >
> > > The goal of the lib is to allow anyone to invent any feature
> > > which is not already available in DPDK.
> > >
> > > > Even similar semantics are required for DPU(Smart NIC)
> > > > communitication. I am planning to
> > > > send RFC in coming days to address the issue without the application
> > > > knowing the Bus/HW/Driver details.
> > >
> > > gpudev is not exposing bus/hw/driver details.
> > > I don't understand what you mean.
> >
> > See above.
>
>
>
  

11/10/2021 11:29, Jerin Jacob:
> On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > 11/10/2021 10:43, Jerin Jacob:
> > > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > 10/10/2021 12:16, Jerin Jacob:
> > > > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > > > >
> > > > > > From: eagostini <eagostini@nvidia.com>
> > > > > >
> > > > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > > > Some tasks can be delegated to devices working in parallel.
> > > > > >
> > > > > > The goal of this new library is to enhance the collaboration between
> > > > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > > > >
> > > > > > When mixing network activity with task processing on a non-CPU device,
> > > > > > there may be the need to put in communication the CPU with the device
> > > > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > > > >
> > > > > > This library provides a number of new features:
> > > > > > - Interoperability with GPU-specific library with generic handlers
> > > > > > - Possibility to allocate and free memory on the GPU
> > > > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > > > >
> > > > > In the RFC thread, There was one outstanding non technical issues on this,
> > > > >
> > > > > i.e
> > > > > The above features are driver specific details. Does the DPDK
> > > > > _application_ need to be aware of this?
> > > >
> > > > I don't see these features as driver-specific.
> > >
> > > That is the disconnect. I see this as more driver-specific details
> > > which are not required to implement an "application" facing API.
> >
> > Indeed this is the disconnect.
> > I already answered but it seems you don't accept the answer.
> 
> Same with you. That is why I requested, we need to get opinions from others.
> Some of them already provided opinions in RFC.

This is why I Cc'ed techboard.

> > First, this is not driver-specific. It is a low-level API.
> 
> What is the difference between low-level API and driver-level API.

The low-level API provides tools to build a feature,
but no specific feature.

> > > For example, If we need to implement application facing" subsystems like bbdev,
> > > If we make all this driver interface, you can still implement the
> > > bbdev API as a driver without
> > > exposing HW specific details like how devices communicate to CPU, how
> > > memory is allocated etc
> > >  to "application".
> >
> > There are 2 things to understand here.
> >
> > First we want to allow the application using the GPU for needs which are
> > not exposed by any other DPDK API.
> >
> > Second, if we want to implement another DPDK API like bbdev,
> > then the GPU implementation would be exposed as a vdev in bbdev,
> > using the HW GPU device being a PCI in gpudev.
> > They are two different levels, got it?
> 
> Exactly. So what is the point of exposing low-level driver API to
> "application",
> why not it is part of the internal driver API. My point is, why the
> application needs to worry
> about, How the CPU <-> Device communicated? CPU < -> Device memory
> visibility etc.

There are two reasons.

1/ The application may want to use the GPU for some application-specific
needs which are not abstracted in DPDK API.

2/ This API may also be used by some feature implementation internally
in some DPDK libs or drivers.
We cannot skip the gpudev layer because this is what allows generic probing
of the HW, and gpudev allows to share the GPU with multiple features
implemented in different libs or drivers, thanks to the "child" concept.


> > > > > aka DPDK device class has a fixed personality and it has API to deal
> > > > > with abstracting specific application specific
> > > > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > > > of underlying bus/device properties.
> > > >
> > > > The goal of the lib is to allow anyone to invent any feature
> > > > which is not already available in DPDK.
> > > >
> > > > > Even similar semantics are required for DPU(Smart NIC)
> > > > > communitication. I am planning to
> > > > > send RFC in coming days to address the issue without the application
> > > > > knowing the Bus/HW/Driver details.
> > > >
> > > > gpudev is not exposing bus/hw/driver details.
> > > > I don't understand what you mean.
> > >
> > > See above.
  

On Mon, Oct 11, 2021 at 3:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 11/10/2021 11:29, Jerin Jacob:
> > On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 11/10/2021 10:43, Jerin Jacob:
> > > > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > 10/10/2021 12:16, Jerin Jacob:
> > > > > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > > > > >
> > > > > > > From: eagostini <eagostini@nvidia.com>
> > > > > > >
> > > > > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > > > > Some tasks can be delegated to devices working in parallel.
> > > > > > >
> > > > > > > The goal of this new library is to enhance the collaboration between
> > > > > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > > > > >
> > > > > > > When mixing network activity with task processing on a non-CPU device,
> > > > > > > there may be the need to put in communication the CPU with the device
> > > > > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > > > > >
> > > > > > > This library provides a number of new features:
> > > > > > > - Interoperability with GPU-specific library with generic handlers
> > > > > > > - Possibility to allocate and free memory on the GPU
> > > > > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > > > > >
> > > > > > In the RFC thread, There was one outstanding non technical issues on this,
> > > > > >
> > > > > > i.e
> > > > > > The above features are driver specific details. Does the DPDK
> > > > > > _application_ need to be aware of this?
> > > > >
> > > > > I don't see these features as driver-specific.
> > > >
> > > > That is the disconnect. I see this as more driver-specific details
> > > > which are not required to implement an "application" facing API.
> > >
> > > Indeed this is the disconnect.
> > > I already answered but it seems you don't accept the answer.
> >
> > Same with you. That is why I requested, we need to get opinions from others.
> > Some of them already provided opinions in RFC.
>
> This is why I Cc'ed techboard.

Yes. Indeed.

>
> > > First, this is not driver-specific. It is a low-level API.
> >
> > What is the difference between low-level API and driver-level API.
>
> The low-level API provides tools to build a feature,
> but no specific feature.
>
> > > > For example, If we need to implement application facing" subsystems like bbdev,
> > > > If we make all this driver interface, you can still implement the
> > > > bbdev API as a driver without
> > > > exposing HW specific details like how devices communicate to CPU, how
> > > > memory is allocated etc
> > > >  to "application".
> > >
> > > There are 2 things to understand here.
> > >
> > > First we want to allow the application using the GPU for needs which are
> > > not exposed by any other DPDK API.
> > >
> > > Second, if we want to implement another DPDK API like bbdev,
> > > then the GPU implementation would be exposed as a vdev in bbdev,
> > > using the HW GPU device being a PCI in gpudev.
> > > They are two different levels, got it?
> >
> > Exactly. So what is the point of exposing low-level driver API to
> > "application",
> > why not it is part of the internal driver API. My point is, why the
> > application needs to worry
> > about, How the CPU <-> Device communicated? CPU < -> Device memory
> > visibility etc.
>
> There are two reasons.
>
> 1/ The application may want to use the GPU for some application-specific
> needs which are not abstracted in DPDK API.

Yes. Exactly, That's where my concern, If we take this path, What is
the motivation to contribute to DPDK abstracted subsystem APIs which
make sense for multiple vendors and every
Similar stuff applicable for DPU,

Otherway to put, if GPU is doing some ethdev offload, why not making
as ethdev offload in ethdev spec so that
another type of device can be used and make sense for application writters.

For example, In the future, If someone needs to add ML(Machine
learning) subsystem and enable a proper subsystem
interface that is good for DPDK. If this path is open, there is no
motivation for contribution and the application
will not have a standard interface doing the ML job across multiple vendors.

That's is the only reason why saying it should not APPLICATION
interface it can be DRIVER interface.

>
> 2/ This API may also be used by some feature implementation internally
> in some DPDK libs or drivers.
> We cannot skip the gpudev layer because this is what allows generic probing
> of the HW, and gpudev allows to share the GPU with multiple features
> implemented in different libs or drivers, thanks to the "child" concept.

Again, why do applications need to know it? It is similar to `bus`
kind of this where it
sharing the physical resouces.


>
>
> > > > > > aka DPDK device class has a fixed personality and it has API to deal
> > > > > > with abstracting specific application specific
> > > > > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > > > > of underlying bus/device properties.
> > > > >
> > > > > The goal of the lib is to allow anyone to invent any feature
> > > > > which is not already available in DPDK.
> > > > >
> > > > > > Even similar semantics are required for DPU(Smart NIC)
> > > > > > communitication. I am planning to
> > > > > > send RFC in coming days to address the issue without the application
> > > > > > knowing the Bus/HW/Driver details.
> > > > >
> > > > > gpudev is not exposing bus/hw/driver details.
> > > > > I don't understand what you mean.
> > > >
> > > > See above.
>
>
>
  

11/10/2021 13:41, Jerin Jacob:
> On Mon, Oct 11, 2021 at 3:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > 11/10/2021 11:29, Jerin Jacob:
> > > On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > 11/10/2021 10:43, Jerin Jacob:
> > > > > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > > 10/10/2021 12:16, Jerin Jacob:
> > > > > > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > > > > > >
> > > > > > > > From: eagostini <eagostini@nvidia.com>
> > > > > > > >
> > > > > > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > > > > > Some tasks can be delegated to devices working in parallel.
> > > > > > > >
> > > > > > > > The goal of this new library is to enhance the collaboration between
> > > > > > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > > > > > >
> > > > > > > > When mixing network activity with task processing on a non-CPU device,
> > > > > > > > there may be the need to put in communication the CPU with the device
> > > > > > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > > > > > >
> > > > > > > > This library provides a number of new features:
> > > > > > > > - Interoperability with GPU-specific library with generic handlers
> > > > > > > > - Possibility to allocate and free memory on the GPU
> > > > > > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > > > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > > > > > >
> > > > > > > In the RFC thread, There was one outstanding non technical issues on this,
> > > > > > >
> > > > > > > i.e
> > > > > > > The above features are driver specific details. Does the DPDK
> > > > > > > _application_ need to be aware of this?
> > > > > >
> > > > > > I don't see these features as driver-specific.
> > > > >
> > > > > That is the disconnect. I see this as more driver-specific details
> > > > > which are not required to implement an "application" facing API.
> > > >
> > > > Indeed this is the disconnect.
> > > > I already answered but it seems you don't accept the answer.
> > >
> > > Same with you. That is why I requested, we need to get opinions from others.
> > > Some of them already provided opinions in RFC.
> >
> > This is why I Cc'ed techboard.
> 
> Yes. Indeed.
> 
> >
> > > > First, this is not driver-specific. It is a low-level API.
> > >
> > > What is the difference between low-level API and driver-level API.
> >
> > The low-level API provides tools to build a feature,
> > but no specific feature.
> >
> > > > > For example, If we need to implement application facing" subsystems like bbdev,
> > > > > If we make all this driver interface, you can still implement the
> > > > > bbdev API as a driver without
> > > > > exposing HW specific details like how devices communicate to CPU, how
> > > > > memory is allocated etc
> > > > >  to "application".
> > > >
> > > > There are 2 things to understand here.
> > > >
> > > > First we want to allow the application using the GPU for needs which are
> > > > not exposed by any other DPDK API.
> > > >
> > > > Second, if we want to implement another DPDK API like bbdev,
> > > > then the GPU implementation would be exposed as a vdev in bbdev,
> > > > using the HW GPU device being a PCI in gpudev.
> > > > They are two different levels, got it?
> > >
> > > Exactly. So what is the point of exposing low-level driver API to
> > > "application",
> > > why not it is part of the internal driver API. My point is, why the
> > > application needs to worry
> > > about, How the CPU <-> Device communicated? CPU < -> Device memory
> > > visibility etc.
> >
> > There are two reasons.
> >
> > 1/ The application may want to use the GPU for some application-specific
> > needs which are not abstracted in DPDK API.
> 
> Yes. Exactly, That's where my concern, If we take this path, What is
> the motivation to contribute to DPDK abstracted subsystem APIs which
> make sense for multiple vendors and every
> Similar stuff applicable for DPU,

A feature-specific API is better of course, there is no lose of motivation.
But you cannot forbid applications to have their own features on GPU.

> Otherway to put, if GPU is doing some ethdev offload, why not making
> as ethdev offload in ethdev spec so that
> another type of device can be used and make sense for application writters.

If we do ethdev offload, yes we'll implement it.
And we'll do it on top of gpudev, which is the only way to share the CPU.

> For example, In the future, If someone needs to add ML(Machine
> learning) subsystem and enable a proper subsystem
> interface that is good for DPDK. If this path is open, there is no
> motivation for contribution and the application
> will not have a standard interface doing the ML job across multiple vendors.

Wrong. It does remove the motivation, it is a first step to build on top of it.

> That's is the only reason why saying it should not APPLICATION
> interface it can be DRIVER interface.
> 
> >
> > 2/ This API may also be used by some feature implementation internally
> > in some DPDK libs or drivers.
> > We cannot skip the gpudev layer because this is what allows generic probing
> > of the HW, and gpudev allows to share the GPU with multiple features
> > implemented in different libs or drivers, thanks to the "child" concept.
> 
> Again, why do applications need to know it? It is similar to `bus`
> kind of this where it sharing the physical resouces.

No it's not a bus, it is a device that we need to share.

> > > > > > > aka DPDK device class has a fixed personality and it has API to deal
> > > > > > > with abstracting specific application specific
> > > > > > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > > > > > of underlying bus/device properties.
> > > > > >
> > > > > > The goal of the lib is to allow anyone to invent any feature
> > > > > > which is not already available in DPDK.
> > > > > >
> > > > > > > Even similar semantics are required for DPU(Smart NIC)
> > > > > > > communitication. I am planning to
> > > > > > > send RFC in coming days to address the issue without the application
> > > > > > > knowing the Bus/HW/Driver details.
> > > > > >
> > > > > > gpudev is not exposing bus/hw/driver details.
> > > > > > I don't understand what you mean.
> > > > >
> > > > > See above.

We are going into circles.
In short, Jerin wants to forbid the generic use of GPU in DPDK.
He wants only feature-specific API.
It is like restricting the functions we can run on a CPU.

And anyway we need this layer to share the GPU between multiple features.

Techboard please vote.
  

On Mon, Oct 11, 2021 at 6:14 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 11/10/2021 13:41, Jerin Jacob:
> > On Mon, Oct 11, 2021 at 3:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > 11/10/2021 11:29, Jerin Jacob:
> > > > On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > 11/10/2021 10:43, Jerin Jacob:
> > > > > > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > > > 10/10/2021 12:16, Jerin Jacob:
> > > > > > > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > > > > > > >
> > > > > > > > > From: eagostini <eagostini@nvidia.com>
> > > > > > > > >
> > > > > > > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > > > > > > Some tasks can be delegated to devices working in parallel.
> > > > > > > > >
> > > > > > > > > The goal of this new library is to enhance the collaboration between
> > > > > > > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > > > > > > >
> > > > > > > > > When mixing network activity with task processing on a non-CPU device,
> > > > > > > > > there may be the need to put in communication the CPU with the device
> > > > > > > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > > > > > > >
> > > > > > > > > This library provides a number of new features:
> > > > > > > > > - Interoperability with GPU-specific library with generic handlers
> > > > > > > > > - Possibility to allocate and free memory on the GPU
> > > > > > > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > > > > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > > > > > > >
> > > > > > > > In the RFC thread, There was one outstanding non technical issues on this,
> > > > > > > >
> > > > > > > > i.e
> > > > > > > > The above features are driver specific details. Does the DPDK
> > > > > > > > _application_ need to be aware of this?
> > > > > > >
> > > > > > > I don't see these features as driver-specific.
> > > > > >
> > > > > > That is the disconnect. I see this as more driver-specific details
> > > > > > which are not required to implement an "application" facing API.
> > > > >
> > > > > Indeed this is the disconnect.
> > > > > I already answered but it seems you don't accept the answer.
> > > >
> > > > Same with you. That is why I requested, we need to get opinions from others.
> > > > Some of them already provided opinions in RFC.
> > >
> > > This is why I Cc'ed techboard.
> >
> > Yes. Indeed.
> >
> > >
> > > > > First, this is not driver-specific. It is a low-level API.
> > > >
> > > > What is the difference between low-level API and driver-level API.
> > >
> > > The low-level API provides tools to build a feature,
> > > but no specific feature.
> > >
> > > > > > For example, If we need to implement application facing" subsystems like bbdev,
> > > > > > If we make all this driver interface, you can still implement the
> > > > > > bbdev API as a driver without
> > > > > > exposing HW specific details like how devices communicate to CPU, how
> > > > > > memory is allocated etc
> > > > > >  to "application".
> > > > >
> > > > > There are 2 things to understand here.
> > > > >
> > > > > First we want to allow the application using the GPU for needs which are
> > > > > not exposed by any other DPDK API.
> > > > >
> > > > > Second, if we want to implement another DPDK API like bbdev,
> > > > > then the GPU implementation would be exposed as a vdev in bbdev,
> > > > > using the HW GPU device being a PCI in gpudev.
> > > > > They are two different levels, got it?
> > > >
> > > > Exactly. So what is the point of exposing low-level driver API to
> > > > "application",
> > > > why not it is part of the internal driver API. My point is, why the
> > > > application needs to worry
> > > > about, How the CPU <-> Device communicated? CPU < -> Device memory
> > > > visibility etc.
> > >
> > > There are two reasons.
> > >
> > > 1/ The application may want to use the GPU for some application-specific
> > > needs which are not abstracted in DPDK API.
> >
> > Yes. Exactly, That's where my concern, If we take this path, What is
> > the motivation to contribute to DPDK abstracted subsystem APIs which
> > make sense for multiple vendors and every
> > Similar stuff applicable for DPU,
>
> A feature-specific API is better of course, there is no lose of motivation.
> But you cannot forbid applications to have their own features on GPU.

it still can use it. We don't need DPDK APIs for that.

>
> > Otherway to put, if GPU is doing some ethdev offload, why not making
> > as ethdev offload in ethdev spec so that
> > another type of device can be used and make sense for application writters.
>
> If we do ethdev offload, yes we'll implement it.
> And we'll do it on top of gpudev, which is the only way to share the CPU.
>
> > For example, In the future, If someone needs to add ML(Machine
> > learning) subsystem and enable a proper subsystem
> > interface that is good for DPDK. If this path is open, there is no
> > motivation for contribution and the application
> > will not have a standard interface doing the ML job across multiple vendors.
>
> Wrong. It does remove the motivation, it is a first step to build on top of it.

IMO, No need to make driver API to the public to feature API.

>
> > That's is the only reason why saying it should not APPLICATION
> > interface it can be DRIVER interface.
> >
> > >
> > > 2/ This API may also be used by some feature implementation internally
> > > in some DPDK libs or drivers.
> > > We cannot skip the gpudev layer because this is what allows generic probing
> > > of the HW, and gpudev allows to share the GPU with multiple features
> > > implemented in different libs or drivers, thanks to the "child" concept.
> >
> > Again, why do applications need to know it? It is similar to `bus`
> > kind of this where it sharing the physical resouces.
>
> No it's not a bus, it is a device that we need to share.
>
> > > > > > > > aka DPDK device class has a fixed personality and it has API to deal
> > > > > > > > with abstracting specific application specific
> > > > > > > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > > > > > > of underlying bus/device properties.
> > > > > > >
> > > > > > > The goal of the lib is to allow anyone to invent any feature
> > > > > > > which is not already available in DPDK.
> > > > > > >
> > > > > > > > Even similar semantics are required for DPU(Smart NIC)
> > > > > > > > communitication. I am planning to
> > > > > > > > send RFC in coming days to address the issue without the application
> > > > > > > > knowing the Bus/HW/Driver details.
> > > > > > >
> > > > > > > gpudev is not exposing bus/hw/driver details.
> > > > > > > I don't understand what you mean.
> > > > > >
 > > > > > > See above.
>
> We are going into circles.

Yes.

> In short, Jerin wants to forbid the generic use of GPU in DPDK.

See below.

> He wants only feature-specific API.

To re-reiterate, feature-specific "application" API. A device-specific
bit can be
driver API and accessible to the out-of-tree driver if needed.

IMO, if we take this path, DPU, XPU, GPU, etc we need N different libraries to
get the job done for a specific feature for the dataplane.
Instead, Enabling public feature APIs will make the application
portable and does not
need to worry about which type of *PU it runs.


> It is like restricting the functions we can run on a CPU.
>
> And anyway we need this layer to share the GPU between multiple features.

No disagreement there. Is that layer public application API or not is
the question.
it is like PCI device API calls over of the application and makes the
application device specific.

>
> Techboard please vote.

Yes.

>
>
  

From: Jerin Jacob <jerinjacobk@gmail.com>
Date: Monday, 11 October 2021 at 15:30
To: NBU-Contact-Thomas Monjalon <thomas@monjalon.net>
Cc: techboard@dpdk.org <techboard@dpdk.org>, Elena Agostini <eagostini@nvidia.com>, dpdk-dev <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v3 0/9] GPU library
> On Mon, Oct 11, 2021 at 6:14 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 11/10/2021 13:41, Jerin Jacob:
> > > On Mon, Oct 11, 2021 at 3:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > 11/10/2021 11:29, Jerin Jacob:
> > > > > On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > > 11/10/2021 10:43, Jerin Jacob:
> > > > > > > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > > > > 10/10/2021 12:16, Jerin Jacob:
> > > > > > > > > On Fri, Oct 8, 2021 at 11:13 PM <eagostini@nvidia.com> wrote:
> > > > > > > > > >
> > > > > > > > > > From: eagostini <eagostini@nvidia.com>
> > > > > > > > > >
> > > > > > > > > > In heterogeneous computing system, processing is not only in the CPU.
> > > > > > > > > > Some tasks can be delegated to devices working in parallel.
> > > > > > > > > >
> > > > > > > > > > The goal of this new library is to enhance the collaboration between
> > > > > > > > > > DPDK, that's primarily a CPU framework, and GPU devices.
> > > > > > > > > >
> > > > > > > > > > When mixing network activity with task processing on a non-CPU device,
> > > > > > > > > > there may be the need to put in communication the CPU with the device
> > > > > > > > > > in order to manage the memory, synchronize operations, exchange info, etc..
> > > > > > > > > >
> > > > > > > > > > This library provides a number of new features:
> > > > > > > > > > - Interoperability with GPU-specific library with generic handlers
> > > > > > > > > > - Possibility to allocate and free memory on the GPU
> > > > > > > > > > - Possibility to allocate and free memory on the CPU but visible from the GPU
> > > > > > > > > > - Communication functions to enhance the dialog between the CPU and the GPU
> > > > > > > > >
> > > > > > > > > In the RFC thread, There was one outstanding non technical issues on this,
> > > > > > > > >
> > > > > > > > > i.e
> > > > > > > > > The above features are driver specific details. Does the DPDK
> > > > > > > > > _application_ need to be aware of this?
> > > > > > > >
> > > > > > > > I don't see these features as driver-specific.
> > > > > > >
> > > > > > > That is the disconnect. I see this as more driver-specific details
> > > > > > > which are not required to implement an "application" facing API.
> > > > > >
> > > > > > Indeed this is the disconnect.
> > > > > > I already answered but it seems you don't accept the answer.
> > > > >
> > > > > Same with you. That is why I requested, we need to get opinions from others.
> > > > > Some of them already provided opinions in RFC.
> > > >
> > > > This is why I Cc'ed techboard.
> > >
> > > Yes. Indeed.
> > >
> > > >
> > > > > > First, this is not driver-specific. It is a low-level API.
> > > > >
> > > > > What is the difference between low-level API and driver-level API.
> > > >
> > > > The low-level API provides tools to build a feature,
> > > > but no specific feature.
> > > >
> > > > > > > For example, If we need to implement application facing" subsystems like bbdev,
> > > > > > > If we make all this driver interface, you can still implement the
> > > > > > > bbdev API as a driver without
> > > > > > > exposing HW specific details like how devices communicate to CPU, how
> > > > > > > memory is allocated etc
> > > > > > >  to "application".
> > > > > >
> > > > > > There are 2 things to understand here.
> > > > > >
> > > > > > First we want to allow the application using the GPU for needs which are
> > > > > > not exposed by any other DPDK API.
> > > > > >
> > > > > > Second, if we want to implement another DPDK API like bbdev,
> > > > > > then the GPU implementation would be exposed as a vdev in bbdev,
> > > > > > using the HW GPU device being a PCI in gpudev.
> > > > > > They are two different levels, got it?
> > > > >
> > > > > Exactly. So what is the point of exposing low-level driver API to
> > > > > "application",
> > > > > why not it is part of the internal driver API. My point is, why the
> > > > > application needs to worry
> > > > > about, How the CPU <-> Device communicated? CPU < -> Device memory
> > > > > visibility etc.
> > > >
> > > > There are two reasons.
> > > >
> > > > 1/ The application may want to use the GPU for some application-specific
> > > > needs which are not abstracted in DPDK API.
> > >
> > > Yes. Exactly, That's where my concern, If we take this path, What is
> > > the motivation to contribute to DPDK abstracted subsystem APIs which
> > > make sense for multiple vendors and every
> > > Similar stuff applicable for DPU,
> >
> > A feature-specific API is better of course, there is no lose of motivation.
> > But you cannot forbid applications to have their own features on GPU.
>
> it still can use it. We don't need DPDK APIs for that.
>
> >
> > > Otherway to put, if GPU is doing some ethdev offload, why not making
> > > as ethdev offload in ethdev spec so that
> > > another type of device can be used and make sense for application writters.
> >
> > If we do ethdev offload, yes we'll implement it.
> > And we'll do it on top of gpudev, which is the only way to share the CPU.
> >
> > > For example, In the future, If someone needs to add ML(Machine
> > > learning) subsystem and enable a proper subsystem
> > > interface that is good for DPDK. If this path is open, there is no
> > > motivation for contribution and the application
> > > will not have a standard interface doing the ML job across multiple vendors.
> >
> > Wrong. It does remove the motivation, it is a first step to build on top of it.
>
> IMO, No need to make driver API to the public to feature API.
>
> >
> > > That's is the only reason why saying it should not APPLICATION
> > > interface it can be DRIVER interface.
> > >
> > > >
> > > > 2/ This API may also be used by some feature implementation internally
> > > > in some DPDK libs or drivers.
> > > > We cannot skip the gpudev layer because this is what allows generic probing
> > > > of the HW, and gpudev allows to share the GPU with multiple features
> > > > implemented in different libs or drivers, thanks to the "child" concept.
> > >
> > > Again, why do applications need to know it? It is similar to `bus`
> > > kind of this where it sharing the physical resouces.
> >
> > No it's not a bus, it is a device that we need to share.
> >
> > > > > > > > > aka DPDK device class has a fixed personality and it has API to deal
> > > > > > > > > with abstracting specific application specific
> > > > > > > > > end user functionality like ethdev, cryptodev, eventdev irrespective
> > > > > > > > > of underlying bus/device properties.
> > > > > > > >
> > > > > > > > The goal of the lib is to allow anyone to invent any feature
> > > > > > > > which is not already available in DPDK.
> > > > > > > >
> > > > > > > > > Even similar semantics are required for DPU(Smart NIC)
> > > > > > > > > communitication. I am planning to
> > > > > > > > > send RFC in coming days to address the issue without the application
> > > > > > > > > knowing the Bus/HW/Driver details.
> > > > > > > >
> > > > > > > > gpudev is not exposing bus/hw/driver details.
> > > > > > > > I don't understand what you mean.
> > > > > > >
> > > > > > > See above.
> >
> > We are going into circles.
>
> Yes.
>
> > In short, Jerin wants to forbid the generic use of GPU in DPDK.
>
> See below.

Honestly I don’t see a real problem releasing the library at application level.
It doesn’t prevent to use it internally by other DPDK libraries/drivers if needed.

Applications can benefit of this library for a number of reasons:

  *   Enhance the interaction between GPU specific library and DPDK
  *   Hide GPU specific implementation details to the “final” user that wants to build a GPU + DPDK application
  *   Measure network throughput with common tools like testpmd using GPU memory

Please be aware that this is just a starting point.
I’m planning to expose a number of features (at memory and processing levels) that can be useful
to enhance the communication among GPU, CPU and NIC hiding the implementation
details within the library/driver.

>
> > He wants only feature-specific API.
>
> To re-reiterate, feature-specific "application" API. A device-specific
> bit can be
> driver API and accessible to the out-of-tree driver if needed.
>
> IMO, if we take this path, DPU, XPU, GPU, etc we need N different libraries to
> get the job done for a specific feature for the dataplane.
> Instead, Enabling public feature APIs will make the application
> portable and does not
> need to worry about which type of *PU it runs.
>

As I stated multiple times, let’s start with something simple that works and
then think about how to enhance the library/driver.
IMHO it doesn’t make sense to address all the use-cases now.
This is a completely new scenario we’re opening in the DPDK context, let’s start
from the basis.

>
> > It is like restricting the functions we can run on a CPU.
> >
> > And anyway we need this layer to share the GPU between multiple features.
>
> No disagreement there. Is that layer public application API or not is
> the question.
> it is like PCI device API calls over of the application and makes the
> application device specific.
>
> >
> > Techboard please vote.
>
> Yes.
>
> >
> >
  

>
> >
>
> > > He wants only feature-specific API.
>
> >
>
> > To re-reiterate, feature-specific "application" API. A device-specific
>
> > bit can be
>
> > driver API and accessible to the out-of-tree driver if needed.
>
> >
>
> > IMO, if we take this path, DPU, XPU, GPU, etc we need N different libraries to
>
> > get the job done for a specific feature for the dataplane.
>
> > Instead, Enabling public feature APIs will make the application
>
> > portable and does not
>
> > need to worry about which type of *PU it runs.
>
> >
>
>
>
> As I stated multiple times, let’s start with something simple that works and
>
> then think about how to enhance the library/driver.

I have submitted RFC[1] to abstract in a generic way so that
workload-specific _driver_ aspects are NOT EXPOSED to the application.

The RFC explains with application code, how any workload can be used
by the application without exposing the driver details.
Also, shows how to enable a framework to abstract a different form of
workload acceletor(DPU, GPU, XPU, IPU....)

In order to map to this discussion with RFC:
You may need to add a new host port for GPU which as
lib/dwa/rte_dwa_port_host_xxxxx.h and adding new workload as
lib/dwa/rte_dwa_profile_xxxx.h(Which can be reused by
all dataplane workload accelerator)



[1]
http://mails.dpdk.org/archives/dev/2021-October/226070.html


>
> IMHO it doesn’t make sense to address all the use-cases now.
>
> This is a completely new scenario we’re opening in the DPDK context, let’s start
>
> from the basis.
  

19/10/2021 20:47, Jerin Jacob:
> > As I stated multiple times, let’s start with something simple that works and
> > then think about how to enhance the library/driver.
> 
> I have submitted RFC[1] to abstract in a generic way so that
> workload-specific _driver_ aspects are NOT EXPOSED to the application.
> 
> The RFC explains with application code, how any workload can be used
> by the application without exposing the driver details.
> Also, shows how to enable a framework to abstract a different form of
> workload acceletor(DPU, GPU, XPU, IPU....)
> 
> In order to map to this discussion with RFC:
> You may need to add a new host port for GPU which as
> lib/dwa/rte_dwa_port_host_xxxxx.h and adding new workload as
> lib/dwa/rte_dwa_profile_xxxx.h(Which can be reused by
> all dataplane workload accelerator)
> 
> [1] http://mails.dpdk.org/archives/dev/2021-October/226070.html

That's really 2 different points of view:
	- expose only some predefined dataplane workloads
	- or allow to build any custom worload
  

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> Sent: Wednesday, October 20, 2021 12:42 AM
> To: Jerin Jacob <jerinjacobk@gmail.com>
> Cc: Elena Agostini <eagostini@nvidia.com>; techboard@dpdk.org; dpdk-dev
> <dev@dpdk.org>
> Subject: [EXT] Re: [dpdk-dev] [PATCH v3 0/9] GPU library
> 
> External Email
> 
> ----------------------------------------------------------------------
> 19/10/2021 20:47, Jerin Jacob:
> > > As I stated multiple times, let’s start with something simple that
> > > works and then think about how to enhance the library/driver.
> >
> > I have submitted RFC[1] to abstract in a generic way so that
> > workload-specific _driver_ aspects are NOT EXPOSED to the application.
> >
> > The RFC explains with application code, how any workload can be used
> > by the application without exposing the driver details.
> > Also, shows how to enable a framework to abstract a different form of
> > workload acceletor(DPU, GPU, XPU, IPU....)
> >
> > In order to map to this discussion with RFC:
> > You may need to add a new host port for GPU which as
> > lib/dwa/rte_dwa_port_host_xxxxx.h and adding new workload as
> > lib/dwa/rte_dwa_profile_xxxx.h(Which can be reused by all dataplane
> > workload accelerator)
> >
> > [1]
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_arc
> > hives_dev_2021-
> 2DOctober_226070.html&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ
> >
> &r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=8RtZAL1mpqAKtA
> i3Fpryk
> >
> gq7CJ_9Xlb_cnR8gVPZPAc&s=aWBFPWzbVCCBZ9HSTIwBD5Mw9wdBYje1ad8zY
> j7oeiI&e
> > =
> 
> That's really 2 different points of view:
> 	- expose only some predefined dataplane workloads
> 	- or allow to build any custom workload

That’s one way to look at. Other way to look at, If you split the workload as quantifiable piece.
We could chain the workload to create any custom workload.
Another aspect is how can application writer  custom Dataplane workload which is not
specific to acceletor HW rather it express what need to do for the application .


> 
>