[24.03,v2] build: track mandatory rather than optional libs
Checks
Commit Message
DPDK now has more optional libraries than mandatory ones, so invert the
list stored in the meson.build file from the optional ones to the
"always_enable" ones. As well as being a shorter list:
* we can remove the loop building up the "always_enable" list
dynamically from the optional list
* it better aligns with the drivers/meson.build file which maintains an
always_enable list.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
V2: List of mandatory libraries is now sorted alphabetically
---
lib/meson.build | 70 +++++++++++++++++--------------------------------
1 file changed, 24 insertions(+), 46 deletions(-)
Comments
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 3 November 2023 17.52
>
> DPDK now has more optional libraries than mandatory ones, so invert the
> list stored in the meson.build file from the optional ones to the
> "always_enable" ones. As well as being a shorter list:
>
> * we can remove the loop building up the "always_enable" list
> dynamically from the optional list
> * it better aligns with the drivers/meson.build file which maintains an
> always_enable list.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Excellent!
It really shows how bloated DPDK CORE still is. I would like to see these go optional:
acl - OPTIONAL
cmdline - OPTIONAL (if not used by EAL)
eal - CORE
ethdev - CORE, or OPTIONAL for secondary process
fib - OPTIONAL
hash - OPTIONAL (if not used by CORE libs)
kvargs - OPTIONAL (if not used by EAL)
log - CORE
lpm - OPTIONAL
mbuf - CORE
mempool - CORE
meter - OPTIONAL
net - CORE
pci - CORE, or OPTIONAL for secondary process
rcu - OPTIONAL (if not used by CORE libs)
rib - OPTIONAL
ring - CORE
stack - OPTIONAL (if not used by CORE libs)
telemetry - OPTIONAL
timer - OPTIONAL (if not used by CORE libs)
Anyway,
Acked-by: Morten Brørup <mb@smartsharesystems.com>
On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Friday, 3 November 2023 17.52
> >
> > DPDK now has more optional libraries than mandatory ones, so invert the
> > list stored in the meson.build file from the optional ones to the
> > "always_enable" ones. As well as being a shorter list:
> >
> > * we can remove the loop building up the "always_enable" list
> > dynamically from the optional list
> > * it better aligns with the drivers/meson.build file which maintains an
> > always_enable list.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>
> Excellent!
>
> It really shows how bloated DPDK CORE still is. I would like to see these go optional:
>
For some I agree, but we need to decide what optional really means. :-)
For my mind, there are 3 (maybe 4) key components that need to be built for
me to consider a build to be a valid DPDK one:
* EAL obviously,
* testpmd, because everyone seems to use it
* l3fwd, becaues it's the most commonly referenced example and used for
benchmarking, and build testing in test-meson-builds. (There are others,
but they are pretty likely to build if l3fwd does!)
* dpdk-test - I feel this should always be buildable, but for me it's the
optional 4th component.
Now, the obviously one to relax here is l3fwd, since it is just an example,
but I wonder if that may cause some heartache.
Anyway some notes:
> acl - OPTIONAL
Used by l3fwd
> cmdline - OPTIONAL (if not used by EAL)
I'd consider this core - used by testpmd and the auto-tests. I'd push this
down the list to consider making optional.
> eal - CORE
> ethdev - CORE, or OPTIONAL for secondary process
Is there really much use of DPDK without ethdev? Let's just keep core for
now.
> fib - OPTIONAL
Used by l3fwd
> hash - OPTIONAL (if not used by CORE libs)
Needed by l3fwd. Also listed as a dependency by a number of drivers, e.g.
i40e, mlx5, nfp, tap. I think they use it for holding filters and things.
This I'd be wary about allowing disabling without some work, as it could
cause users surprise when some drivers unexpectedly stop building.
> kvargs - OPTIONAL (if not used by EAL)
Used by EAL
> log - CORE
> lpm - OPTIONAL
Used by l3fwd
> mbuf - CORE
> mempool - CORE
> meter - OPTIONAL
Used by ethdev.
> net - CORE
> pci - CORE, or OPTIONAL for secondary process
> rcu - OPTIONAL (if not used by CORE libs)
Used by lpm and hash, which are used by l3fwd.
> rib - OPTIONAL
Used by fib, used by l3fwd.
> ring - CORE
Actually, this is one I'd look to maybe say optional. :-)
I think we should switch our default mempool backend from ring to stack,
and make that mandatory. The reason being that run-to-completion apps
should be running entirely out of mempool cache and not care about the
underlying driver, while pipelined apps pushing buffers across cores would
be better with a LIFO-based rather than FIFO-based mempool driver.
Therefore, I think overall switching to stack from ring would be an overall
win for performance.
> stack - OPTIONAL (if not used by CORE libs)
For reasons above, I'd make this core.
> telemetry - OPTIONAL
Used by EAL
> timer - OPTIONAL (if not used by CORE libs)
Only lib that seems to depend on this is eventdev, so maybe this can be an
easy removal. Though I do see it listed as a dependency of ENA driver too.
>
> Anyway,
>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>
Overall, if we want to make more libs optional, I would start looking at
l3fwd and making it a bit more modular. I previously made its support for
eventdev optional, we should do the same for lpm and fib. Beyond that, we
need to decide what core really means.
/Bruce
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Friday, 3 November 2023 19.09
>
> On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Friday, 3 November 2023 17.52
> > >
> > > DPDK now has more optional libraries than mandatory ones, so invert
> the
> > > list stored in the meson.build file from the optional ones to the
> > > "always_enable" ones. As well as being a shorter list:
> > >
> > > * we can remove the loop building up the "always_enable" list
> > > dynamically from the optional list
> > > * it better aligns with the drivers/meson.build file which
> maintains an
> > > always_enable list.
> > >
> > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> > Excellent!
> >
> > It really shows how bloated DPDK CORE still is. I would like to see
> these go optional:
> >
>
> For some I agree, but we need to decide what optional really means. :-)
>
> For my mind, there are 3 (maybe 4) key components that need to be built
> for
> me to consider a build to be a valid DPDK one:
> * EAL obviously,
> * testpmd, because everyone seems to use it
> * l3fwd, becaues it's the most commonly referenced example and used for
> benchmarking, and build testing in test-meson-builds. (There are
> others,
> but they are pretty likely to build if l3fwd does!)
> * dpdk-test - I feel this should always be buildable, but for me it's
> the
> optional 4th component.
>
> Now, the obviously one to relax here is l3fwd, since it is just an
> example,
> but I wonder if that may cause some heartache.
I don't consider any DPDK lib CORE just because the lib is used by testpmd and/or l3fwd. I agree that all libs should be included by default, so you can run testpmd, l3fwd, and other apps and examples.
However, many libs are not needed for *all* DPDK applications, so I would like other apps to be able to build DPDK without superfluous libs.
E.g. our StraightShaper CSP appliance is deployed at Layer 2, and doesn't use any of DPDK's L3 libs, so why should the DPDK L3 libs be considered CORE and thus included in our application? I suppose other companies are also using DPDK for other purposes than L3 routing, and don't need the DPDK L3 libs.
Furthermore, I suppose that some Layer 3 applications use their own RIB/FIB/LPM libraries. Does OVS use DPDK's rib/fib/lpm libraries?
>
> Anyway some notes:
>
> > acl - OPTIONAL
> Used by l3fwd
But not *all* apps.
>
> > cmdline - OPTIONAL (if not used by EAL)
> I'd consider this core - used by testpmd and the auto-tests. I'd push
> this
> down the list to consider making optional.
>
> > eal - CORE
> > ethdev - CORE, or OPTIONAL for secondary process
> Is there really much use of DPDK without ethdev? Let's just keep core
> for
> now.
Here, I am imagining a secondary process attaching to DPDK shared memory, but not doing any device I/O.
For a primary process, I agree 100 % that it is not much of a DPDK application without ethdev.
>
> > fib - OPTIONAL
> Used by l3fwd
>
> > hash - OPTIONAL (if not used by CORE libs)
> Needed by l3fwd. Also listed as a dependency by a number of drivers,
> e.g.
> i40e, mlx5, nfp, tap. I think they use it for holding filters and
> things.
> This I'd be wary about allowing disabling without some work, as it
> could
> cause users surprise when some drivers unexpectedly stop building.
It is fair enough if some drivers depend on the hash lib; then they should register that dependency in the build system.
In principle, if someone builds DPDK without any of those drivers, so this lib is unused, why should it be included?
This lib has a small footprint, so I don't feel strongly about it.
>
> > kvargs - OPTIONAL (if not used by EAL)
> Used by EAL
>
> > log - CORE
> > lpm - OPTIONAL
> Used by l3fwd
>
> > mbuf - CORE
> > mempool - CORE
> > meter - OPTIONAL
> Used by ethdev.
I wonder how this became a mandatory part of ethdev... It should be optional if not used by the application.
This seems to be EAL all over again... bloat is integrated deeply into EAL, and cannot be removed/disabled. Now, this other bloat is deeply integrated into ethdev, and cannot be removed/disabled.
>
> > net - CORE
> > pci - CORE, or OPTIONAL for secondary process
> > rcu - OPTIONAL (if not used by CORE libs)
> Used by lpm and hash, which are used by l3fwd.
>
> > rib - OPTIONAL
> Used by fib, used by l3fwd.
>
> > ring - CORE
> Actually, this is one I'd look to maybe say optional. :-)
The ring is also used for pipelined applications. (Which in itself doesn't make it mandatory; run-to-completion applications might not need it.)
>
> I think we should switch our default mempool backend from ring to
> stack,
> and make that mandatory. The reason being that run-to-completion apps
> should be running entirely out of mempool cache and not care about the
> underlying driver, while pipelined apps pushing buffers across cores
> would
> be better with a LIFO-based rather than FIFO-based mempool driver.
> Therefore, I think overall switching to stack from ring would be an
> overall
> win for performance.
As a mempool backend, you are probably right that a stack would provide better performance for pipelined applications (due to improved cache utilization).
But isn't the stack backend using locks, so it would run into problems if used by an unregistered non-EAL thread?
Although I haven't looked into the details, I am under the impression that some of the newer ring backends (Relaxed Tail Sync or Head/Tail Sync) don't have this problem.
>
> > stack - OPTIONAL (if not used by CORE libs)
> For reasons above, I'd make this core.
We need at least one mempool backend, so either ring or stack must be mandatory.
>
> > telemetry - OPTIONAL
> Used by EAL
This is mega-bloat, and I am surprised that it is not optional!
Not all apps use telemetry, and previous DPDK releases were able to provide an *environment abstraction layer* without considering telemetry a CORE component of making applications run on some underlying hardware/environment.
An excellent example of non-mandatory stuff added to EAL, instead of keeping EAL lean and mean. This also raises the barrier to implementing an EAL for some other hardware/environment.
Long term, I am hoping for a barebone DPDK be able to run on low-end CPE hardware. I we keep moving down the path of adding bloat to DPDK EAL and CORE, it will eventually require 32 GB RAM and 4 CPU cores to fire up DPDK.
>
> > timer - OPTIONAL (if not used by CORE libs)
> Only lib that seems to depend on this is eventdev, so maybe this can be
> an
> easy removal. Though I do see it listed as a dependency of ENA driver
> too.
Excellent. If dependencies are set up correctly, it can be omitted from CORE.
>
> >
> > Anyway,
> >
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> >
>
> Overall, if we want to make more libs optional, I would start looking
> at
> l3fwd and making it a bit more modular. I previously made its support
> for
> eventdev optional, we should do the same for lpm and fib. Beyond that,
> we
> need to decide what core really means.
Yes - defining CORE is the key to setting the goal here.
In my mind, CORE is the minimum requirement to running an absolutely minimal DPDK application.
A primary DPDK application would probably need to do some packet I/O; but it might be a simple layer two bridge, not using any of the L3 libs.
And a secondary DPDK application might attach to a primary DPDK application only to work on its data structures, e.g. to collect statistics, but not do any packet processing, so that application doesn't need any of those libs (not even the ethdev lib).
In reality, DPDK applications would probably need to build more libs than just CORE. But some application might need CORE + lib A, and some other application might need CORE + lib B. In essence, I don't want application A to drag around some unused lib B, and application B to drag around some unused lib A.
It's an optimization only available a build time. Distros should continue providing all DPDK libs.
There's also system testing and system attack surface to consider... all that bloat makes production systems more fragile and vulnerable.
-Morten
On Fri, Nov 03, 2023 at 09:19:53PM +0100, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Friday, 3 November 2023 19.09
> >
> > On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote:
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent: Friday, 3 November 2023 17.52
> > > >
> > > > DPDK now has more optional libraries than mandatory ones, so invert
> > the
> > > > list stored in the meson.build file from the optional ones to the
> > > > "always_enable" ones. As well as being a shorter list:
> > > >
> > > > * we can remove the loop building up the "always_enable" list
> > > > dynamically from the optional list
> > > > * it better aligns with the drivers/meson.build file which
> > maintains an
> > > > always_enable list.
> > > >
> > > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > >
> > > Excellent!
> > >
> > > It really shows how bloated DPDK CORE still is. I would like to see
> > these go optional:
> > >
> >
> > For some I agree, but we need to decide what optional really means. :-)
> >
> > For my mind, there are 3 (maybe 4) key components that need to be built
> > for
> > me to consider a build to be a valid DPDK one:
> > * EAL obviously,
> > * testpmd, because everyone seems to use it
> > * l3fwd, becaues it's the most commonly referenced example and used for
> > benchmarking, and build testing in test-meson-builds. (There are
> > others,
> > but they are pretty likely to build if l3fwd does!)
> > * dpdk-test - I feel this should always be buildable, but for me it's
> > the
> > optional 4th component.
> >
> > Now, the obviously one to relax here is l3fwd, since it is just an
> > example,
> > but I wonder if that may cause some heartache.
>
> I don't consider any DPDK lib CORE just because the lib is used by testpmd and/or l3fwd. I agree that all libs should be included by default, so you can run testpmd, l3fwd, and other apps and examples.
>
> However, many libs are not needed for *all* DPDK applications, so I would like other apps to be able to build DPDK without superfluous libs.
>
> E.g. our StraightShaper CSP appliance is deployed at Layer 2, and doesn't use any of DPDK's L3 libs, so why should the DPDK L3 libs be considered CORE and thus included in our application? I suppose other companies are also using DPDK for other purposes than L3 routing, and don't need the DPDK L3 libs.
>
> Furthermore, I suppose that some Layer 3 applications use their own RIB/FIB/LPM libraries. Does OVS use DPDK's rib/fib/lpm libraries?
>
<snip for brevity>
> > Overall, if we want to make more libs optional, I would start looking
> > at
> > l3fwd and making it a bit more modular. I previously made its support
> > for
> > eventdev optional, we should do the same for lpm and fib. Beyond that,
> > we
> > need to decide what core really means.
>
> Yes - defining CORE is the key to setting the goal here.
>
> In my mind, CORE is the minimum requirement to running an absolutely minimal DPDK application.
>
> A primary DPDK application would probably need to do some packet I/O; but it might be a simple layer two bridge, not using any of the L3 libs.
>
> And a secondary DPDK application might attach to a primary DPDK application only to work on its data structures, e.g. to collect statistics, but not do any packet processing, so that application doesn't need any of those libs (not even the ethdev lib).
>
> In reality, DPDK applications would probably need to build more libs than just CORE. But some application might need CORE + lib A, and some other application might need CORE + lib B. In essence, I don't want application A to drag around some unused lib B, and application B to drag around some unused lib A.
>
> It's an optimization only available a build time. Distros should continue providing all DPDK libs.
>
> There's also system testing and system attack surface to consider... all that bloat makes production systems more fragile and vulnerable.
>
I largely agree, though I do think that trying to split primary-secondary
as having different builds could lead to some headaches, so I'd push any
work around that further down the road.
Some thoughts on next steps:
* From looks of my original list above, it appears the low-hanging fruit is
largely gone, in terms of being able to turn off libs that have few
dependencies, timer being one possible exception
* I think it's worth looking into making l3fwd more modular so it can be
build only with backend x or y or z in it. However, if agreeable, we can
just start marking lpm and rib/fib libs as optional directly and have
l3fwd not buildable in those cases.
* For libs that depend on other libs for bits of functionality, we are
getting into the realm of using ifdefs to start selectively removing
bits. This is the not-so-nice bit as:
- it makes it a lot harder to do proper build testing, as we now have to
test with individual bits on and off. So long as we were just enabling/
disabling whole components, the build-minimal target was good enough to
test we had a working build. With some libs partially depending on
others - both of which may be disablable independently - our build test
matrix just explodes.
- #ifdefs are just really, really ugly in the code, and make it far
harder to maintain and manage.
Therefore, I'm wondering if we can come up with some sort of neater
solution here.
For example, can we add support to the build system that allows some form
of stubbing out of libraries when they are disabled? That would save the
putting of ifdefs throughout other parts of DPDK and keep the management of
the disabling of the library someway inside the library itself. One way to
do this might be to have a "stub" folder inside a library folder, where
that contains a minimal header file to be used to provide empty functions
in case where the lib itself is disabled. Other, more interesting schemes,
might involve the automatic creation - from the version.map file - of dummy
functions for dependent libs to link against on build.
/Bruce
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 6 November 2023 11.29
>
> On Fri, Nov 03, 2023 at 09:19:53PM +0100, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Friday, 3 November 2023 19.09
> > >
> > > On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote:
> > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > Sent: Friday, 3 November 2023 17.52
> > > > >
> > > > > DPDK now has more optional libraries than mandatory ones, so
> invert
> > > the
> > > > > list stored in the meson.build file from the optional ones to
> the
> > > > > "always_enable" ones. As well as being a shorter list:
> > > > >
> > > > > * we can remove the loop building up the "always_enable" list
> > > > > dynamically from the optional list
> > > > > * it better aligns with the drivers/meson.build file which
> > > maintains an
> > > > > always_enable list.
> > > > >
> > > > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > > >
> > > > Excellent!
> > > >
> > > > It really shows how bloated DPDK CORE still is. I would like to
> see
> > > these go optional:
> > > >
> > >
> > > For some I agree, but we need to decide what optional really means.
> :-)
> > >
> > > For my mind, there are 3 (maybe 4) key components that need to be
> built
> > > for
> > > me to consider a build to be a valid DPDK one:
> > > * EAL obviously,
> > > * testpmd, because everyone seems to use it
> > > * l3fwd, becaues it's the most commonly referenced example and used
> for
> > > benchmarking, and build testing in test-meson-builds. (There are
> > > others,
> > > but they are pretty likely to build if l3fwd does!)
> > > * dpdk-test - I feel this should always be buildable, but for me
> it's
> > > the
> > > optional 4th component.
> > >
> > > Now, the obviously one to relax here is l3fwd, since it is just an
> > > example,
> > > but I wonder if that may cause some heartache.
> >
> > I don't consider any DPDK lib CORE just because the lib is used by
> testpmd and/or l3fwd. I agree that all libs should be included by
> default, so you can run testpmd, l3fwd, and other apps and examples.
> >
> > However, many libs are not needed for *all* DPDK applications, so I
> would like other apps to be able to build DPDK without superfluous
> libs.
> >
> > E.g. our StraightShaper CSP appliance is deployed at Layer 2, and
> doesn't use any of DPDK's L3 libs, so why should the DPDK L3 libs be
> considered CORE and thus included in our application? I suppose other
> companies are also using DPDK for other purposes than L3 routing, and
> don't need the DPDK L3 libs.
> >
> > Furthermore, I suppose that some Layer 3 applications use their own
> RIB/FIB/LPM libraries. Does OVS use DPDK's rib/fib/lpm libraries?
> >
>
> <snip for brevity>
>
> > > Overall, if we want to make more libs optional, I would start
> looking
> > > at
> > > l3fwd and making it a bit more modular. I previously made its
> support
> > > for
> > > eventdev optional, we should do the same for lpm and fib. Beyond
> that,
> > > we
> > > need to decide what core really means.
> >
> > Yes - defining CORE is the key to setting the goal here.
> >
> > In my mind, CORE is the minimum requirement to running an absolutely
> minimal DPDK application.
> >
> > A primary DPDK application would probably need to do some packet I/O;
> but it might be a simple layer two bridge, not using any of the L3
> libs.
> >
> > And a secondary DPDK application might attach to a primary DPDK
> application only to work on its data structures, e.g. to collect
> statistics, but not do any packet processing, so that application
> doesn't need any of those libs (not even the ethdev lib).
> >
> > In reality, DPDK applications would probably need to build more libs
> than just CORE. But some application might need CORE + lib A, and some
> other application might need CORE + lib B. In essence, I don't want
> application A to drag around some unused lib B, and application B to
> drag around some unused lib A.
> >
> > It's an optimization only available a build time. Distros should
> continue providing all DPDK libs.
> >
> > There's also system testing and system attack surface to consider...
> all that bloat makes production systems more fragile and vulnerable.
> >
>
> I largely agree, though I do think that trying to split primary-
> secondary
> as having different builds could lead to some headaches, so I'd push
> any
> work around that further down the road.
You are probably right that running a secondary process built differently than the primary process will cause an avalanche of new challenges, so I strongly agree to pushing it further down the road. I don't even know if there is any demand for such a secondary process. (We considered something like this for our application, but did something else instead.) Starting the secondary process with some additional run-time parameters will have to suffice.
>
> Some thoughts on next steps:
> * From looks of my original list above, it appears the low-hanging
> fruit is
> largely gone, in terms of being able to turn off libs that have few
> dependencies, timer being one possible exception
> * I think it's worth looking into making l3fwd more modular so it can
> be
> build only with backend x or y or z in it. However, if agreeable, we
> can
> just start marking lpm and rib/fib libs as optional directly and have
> l3fwd not buildable in those cases.
I agree with that. (It would also affect the variants of l3fwd.)
> * For libs that depend on other libs for bits of functionality, we are
> getting into the realm of using ifdefs to start selectively removing
> bits. This is the not-so-nice bit as:
>
> - it makes it a lot harder to do proper build testing, as we now have
> to
> test with individual bits on and off. So long as we were just
> enabling/
> disabling whole components, the build-minimal target was good
> enough to
> test we had a working build. With some libs partially depending on
> others - both of which may be disablable independently - our build
> test
> matrix just explodes.
We could start without the matrix, and have the CI build just two or three variants:
1. Everything (like now),
2. CORE only, and
3. CORE + all drivers with their dependencies.
> - #ifdefs are just really, really ugly in the code, and make it far
> harder to maintain and manage.
>
> Therefore, I'm wondering if we can come up with some sort of neater
> solution here.
>
> For example, can we add support to the build system that allows some
> form
> of stubbing out of libraries when they are disabled? That would save
> the
> putting of ifdefs throughout other parts of DPDK and keep the
> management of
> the disabling of the library someway inside the library itself. One way
> to
> do this might be to have a "stub" folder inside a library folder, where
> that contains a minimal header file to be used to provide empty
> functions
> in case where the lib itself is disabled. Other, more interesting
> schemes,
> might involve the automatic creation - from the version.map file - of
> dummy
> functions for dependent libs to link against on build.
If we stub out a library, we have to somehow ensure that no application/driver/library calls that library, expecting it to work, if the library disabled. Preferably, this should fail at build time.
Although stubbing out libraries might not be a generic solution, it would probably work for some libraries, e.g. trace points, where empty functions do no harm. But they have to be empty in the header file, so the calling code (and associated data) will be optimized away by the compiler. I can come up with one example where we already do this: __rte_mbuf_sanity_check() [1].
[1]: https://elixir.bootlin.com/dpdk/v23.11-rc1/source/lib/mbuf/rte_mbuf.h#L340
PS: I'm not scared of #ifdefs. Many years ago I was involved in a project where one code base was used for generating a variety of firmware for both unmanaged and managed switches (using related families of silicon), controlled at build time by #ifdefs. Consolidating the source code like that worked very well there.
On Mon, Nov 06, 2023 at 12:22:57PM +0100, Morten Brørup wrote:
> > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > Sent: Monday, 6 November 2023 11.29
> >
> > On Fri, Nov 03, 2023 at 09:19:53PM +0100, Morten Brørup wrote:
> > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > Sent: Friday, 3 November 2023 19.09
> > > >
> > > > On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote:
> > > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > > Sent: Friday, 3 November 2023 17.52
> > > > > >
> > > > > > DPDK now has more optional libraries than mandatory ones, so
> > invert
> > > > the
> > > > > > list stored in the meson.build file from the optional ones to
> > the
> > > > > > "always_enable" ones. As well as being a shorter list:
> > > > > >
> > > > > > * we can remove the loop building up the "always_enable" list
> > > > > > dynamically from the optional list
> > > > > > * it better aligns with the drivers/meson.build file which
> > > > maintains an
> > > > > > always_enable list.
> > > > > >
> > > > > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > > > >
> > > > > Excellent!
> > > > >
> > > > > It really shows how bloated DPDK CORE still is. I would like to
> > see
> > > > these go optional:
> > > > >
> > > >
> > > > For some I agree, but we need to decide what optional really means.
> > :-)
> > > >
> > > > For my mind, there are 3 (maybe 4) key components that need to be
> > built
> > > > for
> > > > me to consider a build to be a valid DPDK one:
> > > > * EAL obviously,
> > > > * testpmd, because everyone seems to use it
> > > > * l3fwd, becaues it's the most commonly referenced example and used
> > for
> > > > benchmarking, and build testing in test-meson-builds. (There are
> > > > others,
> > > > but they are pretty likely to build if l3fwd does!)
> > > > * dpdk-test - I feel this should always be buildable, but for me
> > it's
> > > > the
> > > > optional 4th component.
> > > >
> > > > Now, the obviously one to relax here is l3fwd, since it is just an
> > > > example,
> > > > but I wonder if that may cause some heartache.
> > >
> > > I don't consider any DPDK lib CORE just because the lib is used by
> > testpmd and/or l3fwd. I agree that all libs should be included by
> > default, so you can run testpmd, l3fwd, and other apps and examples.
> > >
> > > However, many libs are not needed for *all* DPDK applications, so I
> > would like other apps to be able to build DPDK without superfluous
> > libs.
> > >
> > > E.g. our StraightShaper CSP appliance is deployed at Layer 2, and
> > doesn't use any of DPDK's L3 libs, so why should the DPDK L3 libs be
> > considered CORE and thus included in our application? I suppose other
> > companies are also using DPDK for other purposes than L3 routing, and
> > don't need the DPDK L3 libs.
> > >
> > > Furthermore, I suppose that some Layer 3 applications use their own
> > RIB/FIB/LPM libraries. Does OVS use DPDK's rib/fib/lpm libraries?
> > >
> >
> > <snip for brevity>
> >
> > > > Overall, if we want to make more libs optional, I would start
> > looking
> > > > at
> > > > l3fwd and making it a bit more modular. I previously made its
> > support
> > > > for
> > > > eventdev optional, we should do the same for lpm and fib. Beyond
> > that,
> > > > we
> > > > need to decide what core really means.
> > >
> > > Yes - defining CORE is the key to setting the goal here.
> > >
> > > In my mind, CORE is the minimum requirement to running an absolutely
> > minimal DPDK application.
> > >
> > > A primary DPDK application would probably need to do some packet I/O;
> > but it might be a simple layer two bridge, not using any of the L3
> > libs.
> > >
> > > And a secondary DPDK application might attach to a primary DPDK
> > application only to work on its data structures, e.g. to collect
> > statistics, but not do any packet processing, so that application
> > doesn't need any of those libs (not even the ethdev lib).
> > >
> > > In reality, DPDK applications would probably need to build more libs
> > than just CORE. But some application might need CORE + lib A, and some
> > other application might need CORE + lib B. In essence, I don't want
> > application A to drag around some unused lib B, and application B to
> > drag around some unused lib A.
> > >
> > > It's an optimization only available a build time. Distros should
> > continue providing all DPDK libs.
> > >
> > > There's also system testing and system attack surface to consider...
> > all that bloat makes production systems more fragile and vulnerable.
> > >
> >
> > I largely agree, though I do think that trying to split primary-
> > secondary
> > as having different builds could lead to some headaches, so I'd push
> > any
> > work around that further down the road.
>
> You are probably right that running a secondary process built differently than the primary process will cause an avalanche of new challenges, so I strongly agree to pushing it further down the road. I don't even know if there is any demand for such a secondary process. (We considered something like this for our application, but did something else instead.) Starting the secondary process with some additional run-time parameters will have to suffice.
>
> >
> > Some thoughts on next steps:
> > * From looks of my original list above, it appears the low-hanging
> > fruit is
> > largely gone, in terms of being able to turn off libs that have few
> > dependencies, timer being one possible exception
> > * I think it's worth looking into making l3fwd more modular so it can
> > be
> > build only with backend x or y or z in it. However, if agreeable, we
> > can
> > just start marking lpm and rib/fib libs as optional directly and have
> > l3fwd not buildable in those cases.
>
> I agree with that. (It would also affect the variants of l3fwd.)
>
> > * For libs that depend on other libs for bits of functionality, we are
> > getting into the realm of using ifdefs to start selectively removing
> > bits. This is the not-so-nice bit as:
> >
> > - it makes it a lot harder to do proper build testing, as we now have
> > to
> > test with individual bits on and off. So long as we were just
> > enabling/
> > disabling whole components, the build-minimal target was good
> > enough to
> > test we had a working build. With some libs partially depending on
> > others - both of which may be disablable independently - our build
> > test
> > matrix just explodes.
>
> We could start without the matrix, and have the CI build just two or three variants:
> 1. Everything (like now),
> 2. CORE only, and
> 3. CORE + all drivers with their dependencies.
>
> > - #ifdefs are just really, really ugly in the code, and make it far
> > harder to maintain and manage.
> >
> > Therefore, I'm wondering if we can come up with some sort of neater
> > solution here.
> >
> > For example, can we add support to the build system that allows some
> > form
> > of stubbing out of libraries when they are disabled? That would save
> > the
> > putting of ifdefs throughout other parts of DPDK and keep the
> > management of
> > the disabling of the library someway inside the library itself. One way
> > to
> > do this might be to have a "stub" folder inside a library folder, where
> > that contains a minimal header file to be used to provide empty
> > functions
> > in case where the lib itself is disabled. Other, more interesting
> > schemes,
> > might involve the automatic creation - from the version.map file - of
> > dummy
> > functions for dependent libs to link against on build.
>
> If we stub out a library, we have to somehow ensure that no application/driver/library calls that library, expecting it to work, if the library disabled. Preferably, this should fail at build time.
>
My thinking was that any stubs would only be available internally at build
time. For example, we could have libname.h and stubs/libname.h, where
stubs/libname.h is never installed or exported for application use. We
definitely cannot have stubs generally available to apps.
/Bruce
> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 6 November 2023 12.27
>
> On Mon, Nov 06, 2023 at 12:22:57PM +0100, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Monday, 6 November 2023 11.29
> > >
> > > On Fri, Nov 03, 2023 at 09:19:53PM +0100, Morten Brørup wrote:
> > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > Sent: Friday, 3 November 2023 19.09
> > > > >
> > > > > On Fri, Nov 03, 2023 at 06:31:30PM +0100, Morten Brørup wrote:
> > > > > > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > > > > > Sent: Friday, 3 November 2023 17.52
> > > > > > >
> > > > > > > DPDK now has more optional libraries than mandatory ones,
> so
> > > invert
> > > > > the
> > > > > > > list stored in the meson.build file from the optional ones
> to
> > > the
> > > > > > > "always_enable" ones. As well as being a shorter list:
> > > > > > >
> > > > > > > * we can remove the loop building up the "always_enable"
> list
> > > > > > > dynamically from the optional list
> > > > > > > * it better aligns with the drivers/meson.build file which
> > > > > maintains an
> > > > > > > always_enable list.
> > > > > > >
> > > > > > > Signed-off-by: Bruce Richardson
> <bruce.richardson@intel.com>
> > > > > >
> > > > > > Excellent!
> > > > > >
> > > > > > It really shows how bloated DPDK CORE still is. I would like
> to
> > > see
> > > > > these go optional:
> > > > > >
> > > > >
> > > > > For some I agree, but we need to decide what optional really
> means.
> > > :-)
> > > > >
> > > > > For my mind, there are 3 (maybe 4) key components that need to
> be
> > > built
> > > > > for
> > > > > me to consider a build to be a valid DPDK one:
> > > > > * EAL obviously,
> > > > > * testpmd, because everyone seems to use it
> > > > > * l3fwd, becaues it's the most commonly referenced example and
> used
> > > for
> > > > > benchmarking, and build testing in test-meson-builds. (There
> are
> > > > > others,
> > > > > but they are pretty likely to build if l3fwd does!)
> > > > > * dpdk-test - I feel this should always be buildable, but for
> me
> > > it's
> > > > > the
> > > > > optional 4th component.
> > > > >
> > > > > Now, the obviously one to relax here is l3fwd, since it is just
> an
> > > > > example,
> > > > > but I wonder if that may cause some heartache.
> > > >
> > > > I don't consider any DPDK lib CORE just because the lib is used
> by
> > > testpmd and/or l3fwd. I agree that all libs should be included by
> > > default, so you can run testpmd, l3fwd, and other apps and
> examples.
> > > >
> > > > However, many libs are not needed for *all* DPDK applications, so
> I
> > > would like other apps to be able to build DPDK without superfluous
> > > libs.
> > > >
> > > > E.g. our StraightShaper CSP appliance is deployed at Layer 2, and
> > > doesn't use any of DPDK's L3 libs, so why should the DPDK L3 libs
> be
> > > considered CORE and thus included in our application? I suppose
> other
> > > companies are also using DPDK for other purposes than L3 routing,
> and
> > > don't need the DPDK L3 libs.
> > > >
> > > > Furthermore, I suppose that some Layer 3 applications use their
> own
> > > RIB/FIB/LPM libraries. Does OVS use DPDK's rib/fib/lpm libraries?
> > > >
> > >
> > > <snip for brevity>
> > >
> > > > > Overall, if we want to make more libs optional, I would start
> > > looking
> > > > > at
> > > > > l3fwd and making it a bit more modular. I previously made its
> > > support
> > > > > for
> > > > > eventdev optional, we should do the same for lpm and fib.
> Beyond
> > > that,
> > > > > we
> > > > > need to decide what core really means.
> > > >
> > > > Yes - defining CORE is the key to setting the goal here.
> > > >
> > > > In my mind, CORE is the minimum requirement to running an
> absolutely
> > > minimal DPDK application.
> > > >
> > > > A primary DPDK application would probably need to do some packet
> I/O;
> > > but it might be a simple layer two bridge, not using any of the L3
> > > libs.
> > > >
> > > > And a secondary DPDK application might attach to a primary DPDK
> > > application only to work on its data structures, e.g. to collect
> > > statistics, but not do any packet processing, so that application
> > > doesn't need any of those libs (not even the ethdev lib).
> > > >
> > > > In reality, DPDK applications would probably need to build more
> libs
> > > than just CORE. But some application might need CORE + lib A, and
> some
> > > other application might need CORE + lib B. In essence, I don't want
> > > application A to drag around some unused lib B, and application B
> to
> > > drag around some unused lib A.
> > > >
> > > > It's an optimization only available a build time. Distros should
> > > continue providing all DPDK libs.
> > > >
> > > > There's also system testing and system attack surface to
> consider...
> > > all that bloat makes production systems more fragile and
> vulnerable.
> > > >
> > >
> > > I largely agree, though I do think that trying to split primary-
> > > secondary
> > > as having different builds could lead to some headaches, so I'd
> push
> > > any
> > > work around that further down the road.
> >
> > You are probably right that running a secondary process built
> differently than the primary process will cause an avalanche of new
> challenges, so I strongly agree to pushing it further down the road. I
> don't even know if there is any demand for such a secondary process.
> (We considered something like this for our application, but did
> something else instead.) Starting the secondary process with some
> additional run-time parameters will have to suffice.
> >
> > >
> > > Some thoughts on next steps:
> > > * From looks of my original list above, it appears the low-hanging
> > > fruit is
> > > largely gone, in terms of being able to turn off libs that have
> few
> > > dependencies, timer being one possible exception
> > > * I think it's worth looking into making l3fwd more modular so it
> can
> > > be
> > > build only with backend x or y or z in it. However, if agreeable,
> we
> > > can
> > > just start marking lpm and rib/fib libs as optional directly and
> have
> > > l3fwd not buildable in those cases.
> >
> > I agree with that. (It would also affect the variants of l3fwd.)
> >
> > > * For libs that depend on other libs for bits of functionality, we
> are
> > > getting into the realm of using ifdefs to start selectively
> removing
> > > bits. This is the not-so-nice bit as:
> > >
> > > - it makes it a lot harder to do proper build testing, as we now
> have
> > > to
> > > test with individual bits on and off. So long as we were just
> > > enabling/
> > > disabling whole components, the build-minimal target was good
> > > enough to
> > > test we had a working build. With some libs partially depending
> on
> > > others - both of which may be disablable independently - our
> build
> > > test
> > > matrix just explodes.
> >
> > We could start without the matrix, and have the CI build just two or
> three variants:
> > 1. Everything (like now),
> > 2. CORE only, and
> > 3. CORE + all drivers with their dependencies.
> >
> > > - #ifdefs are just really, really ugly in the code, and make it
> far
> > > harder to maintain and manage.
> > >
> > > Therefore, I'm wondering if we can come up with some sort of neater
> > > solution here.
> > >
> > > For example, can we add support to the build system that allows
> some
> > > form
> > > of stubbing out of libraries when they are disabled? That would
> save
> > > the
> > > putting of ifdefs throughout other parts of DPDK and keep the
> > > management of
> > > the disabling of the library someway inside the library itself. One
> way
> > > to
> > > do this might be to have a "stub" folder inside a library folder,
> where
> > > that contains a minimal header file to be used to provide empty
> > > functions
> > > in case where the lib itself is disabled. Other, more interesting
> > > schemes,
> > > might involve the automatic creation - from the version.map file -
> of
> > > dummy
> > > functions for dependent libs to link against on build.
> >
> > If we stub out a library, we have to somehow ensure that no
> application/driver/library calls that library, expecting it to work, if
> the library disabled. Preferably, this should fail at build time.
> >
>
> My thinking was that any stubs would only be available internally at
> build
> time. For example, we could have libname.h and stubs/libname.h, where
> stubs/libname.h is never installed or exported for application use. We
> definitely cannot have stubs generally available to apps.
That would eliminate the risk for applications, yes.
The risk of using them (and expecting them to work) would still be there for DPDK libs and drivers.
Nonetheless, it might the preferred alternative to #ifdefs is some cases.
And in some cases it might be an easier-to-reach first step for making a lib optional.
@@ -75,52 +75,29 @@ if is_ms_compiler
]
endif
-optional_libs = [
- 'bbdev',
- 'bitratestats',
- 'bpf',
- 'cfgfile',
- 'compressdev',
- 'cryptodev',
- 'dispatcher',
- 'distributor',
- 'dmadev',
- 'efd',
- 'eventdev',
- 'gpudev',
- 'graph',
- 'gro',
- 'gso',
- 'ip_frag',
- 'ipsec',
- 'jobstats',
- 'latencystats',
- 'member',
- 'metrics',
- 'mldev',
- 'node',
- 'pcapng',
- 'pdcp',
- 'pdump',
- 'pipeline',
- 'port',
- 'power',
- 'rawdev',
- 'regexdev',
- 'reorder',
- 'sched',
- 'security',
- 'table',
- 'vhost',
+always_enable = [
+ 'acl',
+ 'cmdline',
+ 'eal',
+ 'ethdev',
+ 'fib',
+ 'hash',
+ 'kvargs',
+ 'log',
+ 'lpm',
+ 'mbuf',
+ 'mempool',
+ 'meter',
+ 'net',
+ 'pci',
+ 'rcu',
+ 'rib',
+ 'ring',
+ 'stack',
+ 'telemetry',
+ 'timer',
]
-always_enable = []
-foreach l:libraries
- if not optional_libs.contains(l)
- always_enable += l
- endif
-endforeach
-
enable_deprecated_libs = []
foreach l:run_command(list_dir_globs, get_option('enable_deprecated_libs'),
check: true).stdout().split()
@@ -136,9 +113,10 @@ enable_libs = run_command(list_dir_globs, get_option('enable_libs'), check: true
require_libs = true
if enable_libs.length() == 0
require_libs = false
- enable_libs += optional_libs
+ enable_libs = libraries
+else
+ enable_libs += always_enable
endif
-enable_libs += always_enable
default_cflags = machine_args
default_cflags += ['-DALLOW_EXPERIMENTAL_API']