[dpdk-dev,v6,06/19] eal: new TLS definition and API declaration

1. add two TLS *_socket_id* and *_cpuset*
2. add two external API rte_thread_set/get_affinity
3. add one internal API eal_thread_dump_affinity

Signed-off-by: Cunming Liang <cunming.liang@intel.com>
---
 v5 changes:
   add comments for RTE_CPU_AFFINITY_STR_LEN
   update comments for eal_thread_dump_affinity()
   return void for rte_thread_get_affinity()
   move rte_socket_id() change to another patch

 lib/librte_eal/bsdapp/eal/eal_thread.c    |  2 ++
 lib/librte_eal/common/eal_thread.h        | 36 +++++++++++++++++++++++++++++++
 lib/librte_eal/common/include/rte_lcore.h | 26 +++++++++++++++++++++-
 lib/librte_eal/linuxapp/eal/eal_thread.c  |  2 ++
 4 files changed, 65 insertions(+), 1 deletion(-)
  

On Fri, Feb 13, 2015 at 09:38:08AM +0800, Cunming Liang wrote:
> 1. add two TLS *_socket_id* and *_cpuset*
> 2. add two external API rte_thread_set/get_affinity
> 3. add one internal API eal_thread_dump_affinity
> 
> Signed-off-by: Cunming Liang <cunming.liang@intel.com>
> ---
>  v5 changes:
>    add comments for RTE_CPU_AFFINITY_STR_LEN
>    update comments for eal_thread_dump_affinity()
>    return void for rte_thread_get_affinity()
>    move rte_socket_id() change to another patch
> 
>  lib/librte_eal/bsdapp/eal/eal_thread.c    |  2 ++
>  lib/librte_eal/common/eal_thread.h        | 36 +++++++++++++++++++++++++++++++
>  lib/librte_eal/common/include/rte_lcore.h | 26 +++++++++++++++++++++-
>  lib/librte_eal/linuxapp/eal/eal_thread.c  |  2 ++
>  4 files changed, 65 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/bsdapp/eal/eal_thread.c b/lib/librte_eal/bsdapp/eal/eal_thread.c
> index ab05368..10220c7 100644
> --- a/lib/librte_eal/bsdapp/eal/eal_thread.c
> +++ b/lib/librte_eal/bsdapp/eal/eal_thread.c
> @@ -56,6 +56,8 @@
>  #include "eal_thread.h"
>  
>  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
>  
>  /*
>   * Send a message to a slave lcore identified by slave_id to call a
> diff --git a/lib/librte_eal/common/eal_thread.h b/lib/librte_eal/common/eal_thread.h
> index f1ce0bd..e4e76b9 100644
> --- a/lib/librte_eal/common/eal_thread.h
> +++ b/lib/librte_eal/common/eal_thread.h
> @@ -34,6 +34,8 @@
>  #ifndef EAL_THREAD_H
>  #define EAL_THREAD_H
>  
> +#include <rte_lcore.h>
> +
>  /**
>   * basic loop of thread, called for each thread by eal_init().
>   *
> @@ -61,4 +63,38 @@ void eal_thread_init_master(unsigned lcore_id);
>   */
>  unsigned eal_cpu_socket_id(unsigned cpu_id);
>  
> +/**
> + * Get the NUMA socket id from cpuset.
> + * This function is private to EAL.
> + *
> + * @param cpusetp
> + *   The point to a valid cpu set.
> + * @return
> + *   socket_id or SOCKET_ID_ANY
> + */
> +int eal_cpuset_socket_id(rte_cpuset_t *cpusetp);
> +
> +/**
> + * Default buffer size to use with eal_thread_dump_affinity()
> + */
> +#define RTE_CPU_AFFINITY_STR_LEN            256
> +
> +/**
> + * Dump the current pthread cpuset.
> + * This function is private to EAL.
> + *
> + * Note:
> + *   If the dump size is greater than the size of given buffer,
> + *   the string will be truncated and with '\0' at the end.
> + *
> + * @param str
> + *   The string buffer the cpuset will dump to.
> + * @param size
> + *   The string buffer size.
> + * @return
> + *   0 for success, -1 if truncation happens.
> + */
> +int
> +eal_thread_dump_affinity(char *str, unsigned size);
> +
>  #endif /* EAL_THREAD_H */
> diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h
> index 4c7d6bb..33f558e 100644
> --- a/lib/librte_eal/common/include/rte_lcore.h
> +++ b/lib/librte_eal/common/include/rte_lcore.h
> @@ -80,7 +80,9 @@ struct lcore_config {
>   */
>  extern struct lcore_config lcore_config[RTE_MAX_LCORE];
>  
> -RTE_DECLARE_PER_LCORE(unsigned, _lcore_id); /**< Per core "core id". */
> +RTE_DECLARE_PER_LCORE(unsigned, _lcore_id);  /**< Per thread "lcore id". */
> +RTE_DECLARE_PER_LCORE(unsigned, _socket_id); /**< Per thread "socket id". */
> +RTE_DECLARE_PER_LCORE(rte_cpuset_t, _cpuset); /**< Per thread "cpuset". */
>  
>  /**
>   * Return the ID of the execution unit we are running on.
> @@ -229,6 +231,28 @@ rte_get_next_lcore(unsigned i, int skip_master, int wrap)
>  	     i<RTE_MAX_LCORE;						\
>  	     i = rte_get_next_lcore(i, 1, 0))
>  
> +/**
> + * Set core affinity of the current thread.
> + * Support both EAL and none-EAL thread and update TLS.
> + *
> + * @param cpusetp
> + *   Point to cpu_set_t for setting current thread affinity.
> + * @return
> + *   On success, return 0; otherwise return -1;
> + */
> +int rte_thread_set_affinity(rte_cpuset_t *cpusetp);
> +
> +/**
> + * Get core affinity of the current thread.
> + *
> + * @param cpusetp
> + *   Point to cpu_set_t for getting current thread cpu affinity.
> + *   It presumes input is not NULL, otherwise it causes panic.
> + *
> + */
> +void rte_thread_get_affinity(rte_cpuset_t *cpusetp);
> +
> +
>  #ifdef __cplusplus
>  }
>  #endif
> diff --git a/lib/librte_eal/linuxapp/eal/eal_thread.c b/lib/librte_eal/linuxapp/eal/eal_thread.c
> index 80a985f..748a83a 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_thread.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_thread.c
> @@ -56,6 +56,8 @@
>  #include "eal_thread.h"
>  
>  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
>  
>  /*
>   * Send a message to a slave lcore identified by slave_id to call a
> -- 
> 1.8.1.4
> 
> 
All of these exported functions need to be exported in the version map.  Also, I
don't think its a good idea to simply expose the per lcore cpuset variables.  It
would be far better to create an api around them

Neil
  

Hi,

> -----Original Message-----
> From: Neil Horman [mailto:nhorman@tuxdriver.com]
> Sent: Friday, February 13, 2015 9:58 PM
> To: Liang, Cunming
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v6 06/19] eal: new TLS definition and API
> declaration
> 
> On Fri, Feb 13, 2015 at 09:38:08AM +0800, Cunming Liang wrote:
> > 1. add two TLS *_socket_id* and *_cpuset*
> > 2. add two external API rte_thread_set/get_affinity
> > 3. add one internal API eal_thread_dump_affinity
> >
> > Signed-off-by: Cunming Liang <cunming.liang@intel.com>
> > ---
> >  v5 changes:
> >    add comments for RTE_CPU_AFFINITY_STR_LEN
> >    update comments for eal_thread_dump_affinity()
> >    return void for rte_thread_get_affinity()
> >    move rte_socket_id() change to another patch
> >
> >  lib/librte_eal/bsdapp/eal/eal_thread.c    |  2 ++
> >  lib/librte_eal/common/eal_thread.h        | 36
> +++++++++++++++++++++++++++++++
> >  lib/librte_eal/common/include/rte_lcore.h | 26 +++++++++++++++++++++-
> >  lib/librte_eal/linuxapp/eal/eal_thread.c  |  2 ++
> >  4 files changed, 65 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/librte_eal/bsdapp/eal/eal_thread.c
> b/lib/librte_eal/bsdapp/eal/eal_thread.c
> > index ab05368..10220c7 100644
> > --- a/lib/librte_eal/bsdapp/eal/eal_thread.c
> > +++ b/lib/librte_eal/bsdapp/eal/eal_thread.c
> > @@ -56,6 +56,8 @@
> >  #include "eal_thread.h"
> >
> >  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> > +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> > +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
> >
> >  /*
> >   * Send a message to a slave lcore identified by slave_id to call a
> > diff --git a/lib/librte_eal/common/eal_thread.h
> b/lib/librte_eal/common/eal_thread.h
> > index f1ce0bd..e4e76b9 100644
> > --- a/lib/librte_eal/common/eal_thread.h
> > +++ b/lib/librte_eal/common/eal_thread.h
> > @@ -34,6 +34,8 @@
> >  #ifndef EAL_THREAD_H
> >  #define EAL_THREAD_H
> >
> > +#include <rte_lcore.h>
> > +
> >  /**
> >   * basic loop of thread, called for each thread by eal_init().
> >   *
> > @@ -61,4 +63,38 @@ void eal_thread_init_master(unsigned lcore_id);
> >   */
> >  unsigned eal_cpu_socket_id(unsigned cpu_id);
> >
> > +/**
> > + * Get the NUMA socket id from cpuset.
> > + * This function is private to EAL.
> > + *
> > + * @param cpusetp
> > + *   The point to a valid cpu set.
> > + * @return
> > + *   socket_id or SOCKET_ID_ANY
> > + */
> > +int eal_cpuset_socket_id(rte_cpuset_t *cpusetp);
> > +
> > +/**
> > + * Default buffer size to use with eal_thread_dump_affinity()
> > + */
> > +#define RTE_CPU_AFFINITY_STR_LEN            256
> > +
> > +/**
> > + * Dump the current pthread cpuset.
> > + * This function is private to EAL.
> > + *
> > + * Note:
> > + *   If the dump size is greater than the size of given buffer,
> > + *   the string will be truncated and with '\0' at the end.
> > + *
> > + * @param str
> > + *   The string buffer the cpuset will dump to.
> > + * @param size
> > + *   The string buffer size.
> > + * @return
> > + *   0 for success, -1 if truncation happens.
> > + */
> > +int
> > +eal_thread_dump_affinity(char *str, unsigned size);
> > +
> >  #endif /* EAL_THREAD_H */
> > diff --git a/lib/librte_eal/common/include/rte_lcore.h
> b/lib/librte_eal/common/include/rte_lcore.h
> > index 4c7d6bb..33f558e 100644
> > --- a/lib/librte_eal/common/include/rte_lcore.h
> > +++ b/lib/librte_eal/common/include/rte_lcore.h
> > @@ -80,7 +80,9 @@ struct lcore_config {
> >   */
> >  extern struct lcore_config lcore_config[RTE_MAX_LCORE];
> >
> > -RTE_DECLARE_PER_LCORE(unsigned, _lcore_id); /**< Per core "core id". */
> > +RTE_DECLARE_PER_LCORE(unsigned, _lcore_id);  /**< Per thread "lcore id".
> */
> > +RTE_DECLARE_PER_LCORE(unsigned, _socket_id); /**< Per thread "socket id".
> */
> > +RTE_DECLARE_PER_LCORE(rte_cpuset_t, _cpuset); /**< Per thread "cpuset".
> */
> >
> >  /**
> >   * Return the ID of the execution unit we are running on.
> > @@ -229,6 +231,28 @@ rte_get_next_lcore(unsigned i, int skip_master, int
> wrap)
> >  	     i<RTE_MAX_LCORE;						\
> >  	     i = rte_get_next_lcore(i, 1, 0))
> >
> > +/**
> > + * Set core affinity of the current thread.
> > + * Support both EAL and none-EAL thread and update TLS.
> > + *
> > + * @param cpusetp
> > + *   Point to cpu_set_t for setting current thread affinity.
> > + * @return
> > + *   On success, return 0; otherwise return -1;
> > + */
> > +int rte_thread_set_affinity(rte_cpuset_t *cpusetp);
> > +
> > +/**
> > + * Get core affinity of the current thread.
> > + *
> > + * @param cpusetp
> > + *   Point to cpu_set_t for getting current thread cpu affinity.
> > + *   It presumes input is not NULL, otherwise it causes panic.
> > + *
> > + */
> > +void rte_thread_get_affinity(rte_cpuset_t *cpusetp);
> > +
> > +
> >  #ifdef __cplusplus
> >  }
> >  #endif
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_thread.c
> b/lib/librte_eal/linuxapp/eal/eal_thread.c
> > index 80a985f..748a83a 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_thread.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_thread.c
> > @@ -56,6 +56,8 @@
> >  #include "eal_thread.h"
> >
> >  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> > +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> > +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
> >
> >  /*
> >   * Send a message to a slave lcore identified by slave_id to call a
> > --
> > 1.8.1.4
> >
> >
> All of these exported functions need to be exported in the version map.  Also, I
> don't think its a good idea to simply expose the per lcore cpuset variables.  It
> would be far better to create an api around them
[LCM] Thanks for the remind, I haven't taken care of the version map.
The rte_thread_set/get_affinity() are the api around _cpuset, so do you suggest we don't put 'per_lcore__cpuset' into rte_eal_version.map ?
On this point, I agree with you and think we'd better not expose 'per_lcore__socket_id' as well, what do you think ?
> 
> Neil
  

On Sun, Feb 15, 2015 at 01:13:07AM +0000, Liang, Cunming wrote:
> Hi,
> 
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Friday, February 13, 2015 9:58 PM
> > To: Liang, Cunming
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v6 06/19] eal: new TLS definition and API
> > declaration
> > 
> > On Fri, Feb 13, 2015 at 09:38:08AM +0800, Cunming Liang wrote:
> > > 1. add two TLS *_socket_id* and *_cpuset*
> > > 2. add two external API rte_thread_set/get_affinity
> > > 3. add one internal API eal_thread_dump_affinity
> > >
> > > Signed-off-by: Cunming Liang <cunming.liang@intel.com>
> > > ---
> > >  v5 changes:
> > >    add comments for RTE_CPU_AFFINITY_STR_LEN
> > >    update comments for eal_thread_dump_affinity()
> > >    return void for rte_thread_get_affinity()
> > >    move rte_socket_id() change to another patch
> > >
> > >  lib/librte_eal/bsdapp/eal/eal_thread.c    |  2 ++
> > >  lib/librte_eal/common/eal_thread.h        | 36
> > +++++++++++++++++++++++++++++++
> > >  lib/librte_eal/common/include/rte_lcore.h | 26 +++++++++++++++++++++-
> > >  lib/librte_eal/linuxapp/eal/eal_thread.c  |  2 ++
> > >  4 files changed, 65 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/lib/librte_eal/bsdapp/eal/eal_thread.c
> > b/lib/librte_eal/bsdapp/eal/eal_thread.c
> > > index ab05368..10220c7 100644
> > > --- a/lib/librte_eal/bsdapp/eal/eal_thread.c
> > > +++ b/lib/librte_eal/bsdapp/eal/eal_thread.c
> > > @@ -56,6 +56,8 @@
> > >  #include "eal_thread.h"
> > >
> > >  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> > > +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> > > +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
> > >
> > >  /*
> > >   * Send a message to a slave lcore identified by slave_id to call a
> > > diff --git a/lib/librte_eal/common/eal_thread.h
> > b/lib/librte_eal/common/eal_thread.h
> > > index f1ce0bd..e4e76b9 100644
> > > --- a/lib/librte_eal/common/eal_thread.h
> > > +++ b/lib/librte_eal/common/eal_thread.h
> > > @@ -34,6 +34,8 @@
> > >  #ifndef EAL_THREAD_H
> > >  #define EAL_THREAD_H
> > >
> > > +#include <rte_lcore.h>
> > > +
> > >  /**
> > >   * basic loop of thread, called for each thread by eal_init().
> > >   *
> > > @@ -61,4 +63,38 @@ void eal_thread_init_master(unsigned lcore_id);
> > >   */
> > >  unsigned eal_cpu_socket_id(unsigned cpu_id);
> > >
> > > +/**
> > > + * Get the NUMA socket id from cpuset.
> > > + * This function is private to EAL.
> > > + *
> > > + * @param cpusetp
> > > + *   The point to a valid cpu set.
> > > + * @return
> > > + *   socket_id or SOCKET_ID_ANY
> > > + */
> > > +int eal_cpuset_socket_id(rte_cpuset_t *cpusetp);
> > > +
> > > +/**
> > > + * Default buffer size to use with eal_thread_dump_affinity()
> > > + */
> > > +#define RTE_CPU_AFFINITY_STR_LEN            256
> > > +
> > > +/**
> > > + * Dump the current pthread cpuset.
> > > + * This function is private to EAL.
> > > + *
> > > + * Note:
> > > + *   If the dump size is greater than the size of given buffer,
> > > + *   the string will be truncated and with '\0' at the end.
> > > + *
> > > + * @param str
> > > + *   The string buffer the cpuset will dump to.
> > > + * @param size
> > > + *   The string buffer size.
> > > + * @return
> > > + *   0 for success, -1 if truncation happens.
> > > + */
> > > +int
> > > +eal_thread_dump_affinity(char *str, unsigned size);
> > > +
> > >  #endif /* EAL_THREAD_H */
> > > diff --git a/lib/librte_eal/common/include/rte_lcore.h
> > b/lib/librte_eal/common/include/rte_lcore.h
> > > index 4c7d6bb..33f558e 100644
> > > --- a/lib/librte_eal/common/include/rte_lcore.h
> > > +++ b/lib/librte_eal/common/include/rte_lcore.h
> > > @@ -80,7 +80,9 @@ struct lcore_config {
> > >   */
> > >  extern struct lcore_config lcore_config[RTE_MAX_LCORE];
> > >
> > > -RTE_DECLARE_PER_LCORE(unsigned, _lcore_id); /**< Per core "core id". */
> > > +RTE_DECLARE_PER_LCORE(unsigned, _lcore_id);  /**< Per thread "lcore id".
> > */
> > > +RTE_DECLARE_PER_LCORE(unsigned, _socket_id); /**< Per thread "socket id".
> > */
> > > +RTE_DECLARE_PER_LCORE(rte_cpuset_t, _cpuset); /**< Per thread "cpuset".
> > */
> > >
> > >  /**
> > >   * Return the ID of the execution unit we are running on.
> > > @@ -229,6 +231,28 @@ rte_get_next_lcore(unsigned i, int skip_master, int
> > wrap)
> > >  	     i<RTE_MAX_LCORE;						\
> > >  	     i = rte_get_next_lcore(i, 1, 0))
> > >
> > > +/**
> > > + * Set core affinity of the current thread.
> > > + * Support both EAL and none-EAL thread and update TLS.
> > > + *
> > > + * @param cpusetp
> > > + *   Point to cpu_set_t for setting current thread affinity.
> > > + * @return
> > > + *   On success, return 0; otherwise return -1;
> > > + */
> > > +int rte_thread_set_affinity(rte_cpuset_t *cpusetp);
> > > +
> > > +/**
> > > + * Get core affinity of the current thread.
> > > + *
> > > + * @param cpusetp
> > > + *   Point to cpu_set_t for getting current thread cpu affinity.
> > > + *   It presumes input is not NULL, otherwise it causes panic.
> > > + *
> > > + */
> > > +void rte_thread_get_affinity(rte_cpuset_t *cpusetp);
> > > +
> > > +
> > >  #ifdef __cplusplus
> > >  }
> > >  #endif
> > > diff --git a/lib/librte_eal/linuxapp/eal/eal_thread.c
> > b/lib/librte_eal/linuxapp/eal/eal_thread.c
> > > index 80a985f..748a83a 100644
> > > --- a/lib/librte_eal/linuxapp/eal/eal_thread.c
> > > +++ b/lib/librte_eal/linuxapp/eal/eal_thread.c
> > > @@ -56,6 +56,8 @@
> > >  #include "eal_thread.h"
> > >
> > >  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> > > +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> > > +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
> > >
> > >  /*
> > >   * Send a message to a slave lcore identified by slave_id to call a
> > > --
> > > 1.8.1.4
> > >
> > >
> > All of these exported functions need to be exported in the version map.  Also, I
> > don't think its a good idea to simply expose the per lcore cpuset variables.  It
> > would be far better to create an api around them
> [LCM] Thanks for the remind, I haven't taken care of the version map.
> The rte_thread_set/get_affinity() are the api around _cpuset, so do you suggest we don't put 'per_lcore__cpuset' into rte_eal_version.map ?
> On this point, I agree with you and think we'd better not expose 'per_lcore__socket_id' as well, what do you think ?
Yes, absolutely, you should wrap some API around them, and make them defined
symbols, only inline them if they're going to be in the hot path.

Thanks
Neil

> > 
> > Neil
> 
>
  

> -----Original Message-----
> From: Neil Horman [mailto:nhorman@tuxdriver.com]
> Sent: Sunday, February 15, 2015 1:17 PM
> To: Liang, Cunming
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v6 06/19] eal: new TLS definition and API
> declaration
>
[...] 
> > > >
> > > >  RTE_DEFINE_PER_LCORE(unsigned, _lcore_id);
> > > > +RTE_DEFINE_PER_LCORE(unsigned, _socket_id);
> > > > +RTE_DEFINE_PER_LCORE(rte_cpuset_t, _cpuset);
> > > >
> > > >  /*
> > > >   * Send a message to a slave lcore identified by slave_id to call a
> > > > --
> > > > 1.8.1.4
> > > >
> > > >
> > > All of these exported functions need to be exported in the version map.  Also,
> I
> > > don't think its a good idea to simply expose the per lcore cpuset variables.  It
> > > would be far better to create an api around them
> > [LCM] Thanks for the remind, I haven't taken care of the version map.
> > The rte_thread_set/get_affinity() are the api around _cpuset, so do you
> suggest we don't put 'per_lcore__cpuset' into rte_eal_version.map ?
> > On this point, I agree with you and think we'd better not expose
> 'per_lcore__socket_id' as well, what do you think ?
> Yes, absolutely, you should wrap some API around them, and make them
> defined
> symbols, only inline them if they're going to be in the hot path.
[LCM] _socket_id is wrapped by rte_socket_id() and rte_thread_set_affinity().
rte_socket_id() is defined as inline in rte_lcore.h. So finally two (rte_thread_set/get_affinity()) are added into version map.
All these are updated in v7.
> 
> Thanks
> Neil
> 
> > >
> > > Neil
> >
> >