[v2,0/10] dpdk: introduce __rte_internal tag

  Hey-
        Based on our recent conversations regarding the use of symbols only
meant for internal dpdk consumption (between dpdk libraries), this is an idea
that I've come up with that I'd like to get some feedback on

Summary:
1) We have symbols in the DPDK that are meant to be used between DPDK libraries,
but not by applications linking to them
2) We would like to document those symbols in the code, so as to note them
clearly as for being meant for internal use only
3) Linker symbol visibility is a very coarse grained tool, and so there is no
good way in a single library to mark items as being meant for use only by other
DPDK libraries, at least not without some extensive runtime checking


Proposal:
I'm proposing that we introduce the __rte_internal tag.  From a coding
standpoint it works a great deal like the __rte_experimental tag in that it
expempts the tagged symbol from ABI constraints (as the only users should be
represented in the DPDK build environment).  Additionally, the __rte_internal
macro resolves differently based on the definition of the BUILDING_RTE_SDK flag
(working under the assumption that said flag should only ever be set if we are
actually building DPDK libraries which will make use of internal calls).  If the
BUILDING_RTE_SDK flag is set __rte_internal resolves to __attribute__((section
"text.internal)), placing it in a special text section which is then used to
validate that the the symbol appears in the INTERNAL section of the
corresponding library version map).  If BUILDING_RTE_SDK is not set, then
__rte_internal resolves to __attribute__((error("..."))), which causes any
caller of the tagged function to throw an error at compile time, indicating that
the symbol is not available for external use.

This isn't a perfect solution, as applications can still hack around it of
course, but I think it hits some of the high points, restricting symbol access
for any library that prototypes its public and private symbols in the same
header file, excluding the internal symbols from ABI constraints, and clearly
documenting those symbols which we wish to limit to internal usage.

Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
CC: Bruce Richardson <bruce.richardson@intel.com>
CC: Thomas Monjalon <thomas@monjalon.net>

---
Change notes

v1->v2
Moved check-experimental-syms.sh rename to earlier in the series
Updated meson build environment to support BUILDING_RTE_SDK
  

I think it would be good to rebase and send at the beginning of the 19.11 cycle.
Thank you

13/06/2019 16:23, Neil Horman:
> Hey-
>         Based on our recent conversations regarding the use of symbols only
> meant for internal dpdk consumption (between dpdk libraries), this is an idea
> that I've come up with that I'd like to get some feedback on
> 
> Summary:
> 1) We have symbols in the DPDK that are meant to be used between DPDK libraries,
> but not by applications linking to them
> 2) We would like to document those symbols in the code, so as to note them
> clearly as for being meant for internal use only
> 3) Linker symbol visibility is a very coarse grained tool, and so there is no
> good way in a single library to mark items as being meant for use only by other
> DPDK libraries, at least not without some extensive runtime checking
> 
> 
> Proposal:
> I'm proposing that we introduce the __rte_internal tag.  From a coding
> standpoint it works a great deal like the __rte_experimental tag in that it
> expempts the tagged symbol from ABI constraints (as the only users should be
> represented in the DPDK build environment).  Additionally, the __rte_internal
> macro resolves differently based on the definition of the BUILDING_RTE_SDK flag
> (working under the assumption that said flag should only ever be set if we are
> actually building DPDK libraries which will make use of internal calls).  If the
> BUILDING_RTE_SDK flag is set __rte_internal resolves to __attribute__((section
> "text.internal)), placing it in a special text section which is then used to
> validate that the the symbol appears in the INTERNAL section of the
> corresponding library version map).  If BUILDING_RTE_SDK is not set, then
> __rte_internal resolves to __attribute__((error("..."))), which causes any
> caller of the tagged function to throw an error at compile time, indicating that
> the symbol is not available for external use.
> 
> This isn't a perfect solution, as applications can still hack around it of
> course, but I think it hits some of the high points, restricting symbol access
> for any library that prototypes its public and private symbols in the same
> header file, excluding the internal symbols from ABI constraints, and clearly
> documenting those symbols which we wish to limit to internal usage.
  

On Tue, Aug 06, 2019 at 12:03:38PM +0200, Thomas Monjalon wrote:
> I think it would be good to rebase and send at the beginning of the 19.11 cycle.
> Thank you
> 
I'm on PTO for the next 10 days or so, but yes, I'll take care of that asap.  

Thanks
Neil

> 13/06/2019 16:23, Neil Horman:
> > Hey-
> >         Based on our recent conversations regarding the use of symbols only
> > meant for internal dpdk consumption (between dpdk libraries), this is an idea
> > that I've come up with that I'd like to get some feedback on
> > 
> > Summary:
> > 1) We have symbols in the DPDK that are meant to be used between DPDK libraries,
> > but not by applications linking to them
> > 2) We would like to document those symbols in the code, so as to note them
> > clearly as for being meant for internal use only
> > 3) Linker symbol visibility is a very coarse grained tool, and so there is no
> > good way in a single library to mark items as being meant for use only by other
> > DPDK libraries, at least not without some extensive runtime checking
> > 
> > 
> > Proposal:
> > I'm proposing that we introduce the __rte_internal tag.  From a coding
> > standpoint it works a great deal like the __rte_experimental tag in that it
> > expempts the tagged symbol from ABI constraints (as the only users should be
> > represented in the DPDK build environment).  Additionally, the __rte_internal
> > macro resolves differently based on the definition of the BUILDING_RTE_SDK flag
> > (working under the assumption that said flag should only ever be set if we are
> > actually building DPDK libraries which will make use of internal calls).  If the
> > BUILDING_RTE_SDK flag is set __rte_internal resolves to __attribute__((section
> > "text.internal)), placing it in a special text section which is then used to
> > validate that the the symbol appears in the INTERNAL section of the
> > corresponding library version map).  If BUILDING_RTE_SDK is not set, then
> > __rte_internal resolves to __attribute__((error("..."))), which causes any
> > caller of the tagged function to throw an error at compile time, indicating that
> > the symbol is not available for external use.
> > 
> > This isn't a perfect solution, as applications can still hack around it of
> > course, but I think it hits some of the high points, restricting symbol access
> > for any library that prototypes its public and private symbols in the same
> > header file, excluding the internal symbols from ABI constraints, and clearly
> > documenting those symbols which we wish to limit to internal usage.
> 
> 
> 
> 
>
  

Hello Neil,

On Tue, Aug 6, 2019 at 2:22 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> On Tue, Aug 06, 2019 at 12:03:38PM +0200, Thomas Monjalon wrote:
> > I think it would be good to rebase and send at the beginning of the 19.11 cycle.
> > Thank you
> >
> I'm on PTO for the next 10 days or so, but yes, I'll take care of that asap.

Looks like your PTO were a bit longer than 10 days ;-).
Do you have some cycles to resurrect this series?


--
David Marchand
  

On Thu, Jan 09, 2020 at 10:55:04AM +0100, David Marchand wrote:
> Hello Neil,
> 
> On Tue, Aug 6, 2019 at 2:22 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> >
> > On Tue, Aug 06, 2019 at 12:03:38PM +0200, Thomas Monjalon wrote:
> > > I think it would be good to rebase and send at the beginning of the 19.11 cycle.
> > > Thank you
> > >
> > I'm on PTO for the next 10 days or so, but yes, I'll take care of that asap.
> 
> Looks like your PTO were a bit longer than 10 days ;-).
> Do you have some cycles to resurrect this series?
> 
Shoot, apologies, it completely slipped my mind.  Yes, I'll get back to this
today
Neil

> 
> --
> David Marchand
> 
>
  

On Thu, Jan 9, 2020 at 12:46 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> On Thu, Jan 09, 2020 at 10:55:04AM +0100, David Marchand wrote:
> > Hello Neil,
> >
> > On Tue, Aug 6, 2019 at 2:22 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> > >
> > > On Tue, Aug 06, 2019 at 12:03:38PM +0200, Thomas Monjalon wrote:
> > > > I think it would be good to rebase and send at the beginning of the 19.11 cycle.
> > > > Thank you
> > > >
> > > I'm on PTO for the next 10 days or so, but yes, I'll take care of that asap.
> >
> > Looks like your PTO were a bit longer than 10 days ;-).
> > Do you have some cycles to resurrect this series?
> >
> Shoot, apologies, it completely slipped my mind.  Yes, I'll get back to this
> today

Cool!
Thanks Neil.