[v1,1/1] eal: warn user when lcore cpuset includes multiple sockets

Currently, it is allowed to specify a cpuset for lcores such that it
will include physical cores from different NUMA nodes. This has an
effect of setting `rte_socket_id()` for that lcore to SOCKET_ID_ANY,
so add a warning about it.

Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/eal/common/eal_common_thread.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
  

On Tue, May 23, 2023 at 01:06:16PM +0000, Anatoly Burakov wrote:
> Currently, it is allowed to specify a cpuset for lcores such that it
> will include physical cores from different NUMA nodes. This has an
> effect of setting `rte_socket_id()` for that lcore to SOCKET_ID_ANY,
> so add a warning about it.
> 
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
>  lib/eal/common/eal_common_thread.c | 20 +++++++++++++++++---
>  1 file changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c
> index 079a385630..46fd2aca1e 100644
> --- a/lib/eal/common/eal_common_thread.c
> +++ b/lib/eal/common/eal_common_thread.c
> @@ -65,10 +65,13 @@ static void
>  thread_update_affinity(rte_cpuset_t *cpusetp)
>  {
>  	unsigned int lcore_id = rte_lcore_id();
> +	int socket_id;
> +
> +	/* find socket ID from cpuset */
> +	socket_id = eal_cpuset_socket_id(cpusetp);
>  
>  	/* store socket_id in TLS for quick access */
> -	RTE_PER_LCORE(_socket_id) =
> -		eal_cpuset_socket_id(cpusetp);
> +	RTE_PER_LCORE(_socket_id) = socket_id;
>  
>  	/* store cpuset in TLS for quick access */
>  	memmove(&RTE_PER_LCORE(_cpuset), cpusetp,
> @@ -76,9 +79,20 @@ thread_update_affinity(rte_cpuset_t *cpusetp)
>  
>  	if (lcore_id != (unsigned)LCORE_ID_ANY) {
>  		/* EAL thread will update lcore_config */
> -		lcore_config[lcore_id].socket_id = RTE_PER_LCORE(_socket_id);
> +		lcore_config[lcore_id].socket_id = socket_id;
>  		memmove(&lcore_config[lcore_id].cpuset, cpusetp,
>  			sizeof(rte_cpuset_t));
> +		
> +		/*
> +		 * lcore_id is not LCORE_ID_ANY, meaning this is a DPDK lcore,
> +		 * so having a valid NUMA affinity for this lcore is important.
> +		 * However, if cpuset includes cores from multiple NUMA nodes,
> +		 * the socket ID will be set to SOCKET_ID_ANY. Notify the user
> +		 * about it if that happens.
> +		 */
> +		if (socket_id == SOCKET_ID_ANY)
> +			RTE_LOG(INFO, EAL, "DPDK lcore %u has NUMA affinity set to SOCKET_ID_ANY\n",
> +					lcore_id);
>  	}
>  }
While having a warning comment in the code is good, should we not also have
a user visible warning to the user when the specific a corelist parameter
value which includes the cross-socket scenario?

/Bruce
  

On 5/23/2023 2:55 PM, Bruce Richardson wrote:
> On Tue, May 23, 2023 at 01:06:16PM +0000, Anatoly Burakov wrote:
>> Currently, it is allowed to specify a cpuset for lcores such that it
>> will include physical cores from different NUMA nodes. This has an
>> effect of setting `rte_socket_id()` for that lcore to SOCKET_ID_ANY,
>> so add a warning about it.
>>
>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> ---
>>   lib/eal/common/eal_common_thread.c | 20 +++++++++++++++++---
>>   1 file changed, 17 insertions(+), 3 deletions(-)
>>
>> diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c
>> index 079a385630..46fd2aca1e 100644
>> --- a/lib/eal/common/eal_common_thread.c
>> +++ b/lib/eal/common/eal_common_thread.c
>> @@ -65,10 +65,13 @@ static void
>>   thread_update_affinity(rte_cpuset_t *cpusetp)
>>   {
>>   	unsigned int lcore_id = rte_lcore_id();
>> +	int socket_id;
>> +
>> +	/* find socket ID from cpuset */
>> +	socket_id = eal_cpuset_socket_id(cpusetp);
>>   
>>   	/* store socket_id in TLS for quick access */
>> -	RTE_PER_LCORE(_socket_id) =
>> -		eal_cpuset_socket_id(cpusetp);
>> +	RTE_PER_LCORE(_socket_id) = socket_id;
>>   
>>   	/* store cpuset in TLS for quick access */
>>   	memmove(&RTE_PER_LCORE(_cpuset), cpusetp,
>> @@ -76,9 +79,20 @@ thread_update_affinity(rte_cpuset_t *cpusetp)
>>   
>>   	if (lcore_id != (unsigned)LCORE_ID_ANY) {
>>   		/* EAL thread will update lcore_config */
>> -		lcore_config[lcore_id].socket_id = RTE_PER_LCORE(_socket_id);
>> +		lcore_config[lcore_id].socket_id = socket_id;
>>   		memmove(&lcore_config[lcore_id].cpuset, cpusetp,
>>   			sizeof(rte_cpuset_t));
>> +		
>> +		/*
>> +		 * lcore_id is not LCORE_ID_ANY, meaning this is a DPDK lcore,
>> +		 * so having a valid NUMA affinity for this lcore is important.
>> +		 * However, if cpuset includes cores from multiple NUMA nodes,
>> +		 * the socket ID will be set to SOCKET_ID_ANY. Notify the user
>> +		 * about it if that happens.
>> +		 */
>> +		if (socket_id == SOCKET_ID_ANY)
>> +			RTE_LOG(INFO, EAL, "DPDK lcore %u has NUMA affinity set to SOCKET_ID_ANY\n",
>> +					lcore_id);
>>   	}
>>   }
> While having a warning comment in the code is good, should we not also have
> a user visible warning to the user when the specific a corelist parameter
> value which includes the cross-socket scenario?
> 
> /Bruce

We could, yes. We could walk through the cpuset for a given lcore, and 
list their socket ID's, would that work?