[v2,1/1] fbarray: get fbarrays from containerized secondary
Checks
Commit Message
From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
In secondary_msl_create_walk(), it creates a file for fbarrays with its
PID for reserving unique name among secondary processes. However, it
does not work if secondary is run as app container because each of
containerized secondary has PID 1. To reserve unique name, use hostname
instead of PID if the value is 1.
Cc: stable@dpdk.org
Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
---
lib/librte_eal/linux/eal/eal_memalloc.c | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
Comments
We need a review here.
16/04/2019 05:43, ogawa.yasufumi@lab.ntt.co.jp:
> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>
> In secondary_msl_create_walk(), it creates a file for fbarrays with its
> PID for reserving unique name among secondary processes. However, it
> does not work if secondary is run as app container because each of
> containerized secondary has PID 1. To reserve unique name, use hostname
> instead of PID if the value is 1.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
On 16-Apr-19 4:43 AM, ogawa.yasufumi@lab.ntt.co.jp wrote:
> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>
> In secondary_msl_create_walk(), it creates a file for fbarrays with its
> PID for reserving unique name among secondary processes. However, it
> does not work if secondary is run as app container because each of
> containerized secondary has PID 1. To reserve unique name, use hostname
> instead of PID if the value is 1.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
> ---
I'm not too well versed in containers - is this hostname 1) always set,
and 2) always unique?
<snip>
> + if (getpid() == 1) {
> + FILE *hn_fp;
> + hn_fp = fopen("/etc/hostname", "r");
> + if (hn_fp == NULL) {
> + RTE_LOG(ERR, EAL,
> + "Cannot open '/etc/hostname' for secondary\n");
> + return -1;
> + }
> +
> + /* with docker, /etc/hostname just has one entry of hostname */
> + if (fscanf(hn_fp, "%s", proc_id) == EOF)
> + return -1;
Wouldn't an error in fscanf() leak the file handle? I think you need to
fclose() before checking the result.
> + fclose(hn_fp);
> + } else
> + sprintf(proc_id, "%d", (int)getpid());
> +
> + snprintf(name, RTE_FBARRAY_NAME_LEN, "%s_%s",
> + primary_msl->memseg_arr.name, proc_id);
>
> ret = rte_fbarray_init(&local_msl->memseg_arr, name,
> primary_msl->memseg_arr.len,
>
Hi Anatoly,
On 2019/07/05 17:53, Burakov, Anatoly wrote:
> On 16-Apr-19 4:43 AM, ogawa.yasufumi@lab.ntt.co.jp wrote:
>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>
>> In secondary_msl_create_walk(), it creates a file for fbarrays with its
>> PID for reserving unique name among secondary processes. However, it
>> does not work if secondary is run as app container because each of
>> containerized secondary has PID 1. To reserve unique name, use hostname
>> instead of PID if the value is 1.
>>
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>> ---
>
> I'm not too well versed in containers - is this hostname 1) always set,
> and 2) always unique?
For docker, 1) hostname is always set. 2) The hostname is decided as
short form of container ID, so it might not be unique even though very
low possibility.
I found that we can get whole container ID in `/proc/self/cgroup` as
discussed [1]. I think using hostname is reasonable way without running
many secondary processes. However, it might be better to use 64 digits
full container ID instead of 12 digits short ID if ensure uniqueness
strongly. What do yo think?
[1]
https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237
>
> <snip>
>
>> + if (getpid() == 1) {
>> + FILE *hn_fp;
>> + hn_fp = fopen("/etc/hostname", "r");
>> + if (hn_fp == NULL) {
>> + RTE_LOG(ERR, EAL,
>> + "Cannot open '/etc/hostname' for secondary\n");
>> + return -1;
>> + }
>> +
>> + /* with docker, /etc/hostname just has one entry of hostname */
>> + if (fscanf(hn_fp, "%s", proc_id) == EOF)
>> + return -1;
>
> Wouldn't an error in fscanf() leak the file handle? I think you need to
> fclose() before checking the result.
I would like to fix it.
Regards,
Yasufumi
>
>> + fclose(hn_fp);
>> + } else
>> + sprintf(proc_id, "%d", (int)getpid());
>> +
>> + snprintf(name, RTE_FBARRAY_NAME_LEN, "%s_%s",
>> + primary_msl->memseg_arr.name, proc_id);
>> ret = rte_fbarray_init(&local_msl->memseg_arr, name,
>> primary_msl->memseg_arr.len,
>>
>
>
On 09-Jul-19 11:22 AM, Yasufumi Ogawa wrote:
> Hi Anatoly,
>
> On 2019/07/05 17:53, Burakov, Anatoly wrote:
>> On 16-Apr-19 4:43 AM, ogawa.yasufumi@lab.ntt.co.jp wrote:
>>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>
>>> In secondary_msl_create_walk(), it creates a file for fbarrays with its
>>> PID for reserving unique name among secondary processes. However, it
>>> does not work if secondary is run as app container because each of
>>> containerized secondary has PID 1. To reserve unique name, use hostname
>>> instead of PID if the value is 1.
>>>
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>> ---
>>
>> I'm not too well versed in containers - is this hostname 1) always
>> set, and 2) always unique?
> For docker, 1) hostname is always set. 2) The hostname is decided as
> short form of container ID, so it might not be unique even though very
> low possibility.
>
> I found that we can get whole container ID in `/proc/self/cgroup` as
> discussed [1]. I think using hostname is reasonable way without running
> many secondary processes. However, it might be better to use 64 digits
> full container ID instead of 12 digits short ID if ensure uniqueness
> strongly. What do yo think?
>
> [1]
> https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237
>
I think it's better to err on the side of caution and guarantee better
uniqueness. This code will get into an LTS and will be used for years to
come :)
On 09-Jul-19 11:24 AM, Burakov, Anatoly wrote:
> On 09-Jul-19 11:22 AM, Yasufumi Ogawa wrote:
>> Hi Anatoly,
>>
>> On 2019/07/05 17:53, Burakov, Anatoly wrote:
>>> On 16-Apr-19 4:43 AM, ogawa.yasufumi@lab.ntt.co.jp wrote:
>>>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>>
>>>> In secondary_msl_create_walk(), it creates a file for fbarrays with its
>>>> PID for reserving unique name among secondary processes. However, it
>>>> does not work if secondary is run as app container because each of
>>>> containerized secondary has PID 1. To reserve unique name, use hostname
>>>> instead of PID if the value is 1.
>>>>
>>>> Cc: stable@dpdk.org
>>>>
>>>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>> ---
>>>
>>> I'm not too well versed in containers - is this hostname 1) always
>>> set, and 2) always unique?
>> For docker, 1) hostname is always set. 2) The hostname is decided as
>> short form of container ID, so it might not be unique even though very
>> low possibility.
>>
>> I found that we can get whole container ID in `/proc/self/cgroup` as
>> discussed [1]. I think using hostname is reasonable way without
>> running many secondary processes. However, it might be better to use
>> 64 digits full container ID instead of 12 digits short ID if ensure
>> uniqueness strongly. What do yo think?
>>
>> [1]
>> https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237
>>
>
> I think it's better to err on the side of caution and guarantee better
> uniqueness. This code will get into an LTS and will be used for years to
> come :)
>
...however, i think a full 64-digit ID won't even fit into the fbarray
filename, as i believe it's limited to something like 64 chars. Perhaps
hostname would be enough after all... or we can increase fbarray name
length - that would require ABI breakage but the ABI is already broken
in this release, so it's OK i think.
On 2019/07/09 19:26, Burakov, Anatoly wrote:
> On 09-Jul-19 11:24 AM, Burakov, Anatoly wrote:
>> On 09-Jul-19 11:22 AM, Yasufumi Ogawa wrote:
>>> Hi Anatoly,
>>>
>>> On 2019/07/05 17:53, Burakov, Anatoly wrote:
>>>> On 16-Apr-19 4:43 AM, ogawa.yasufumi@lab.ntt.co.jp wrote:
>>>>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>>>
>>>>> In secondary_msl_create_walk(), it creates a file for fbarrays with
>>>>> its
>>>>> PID for reserving unique name among secondary processes. However, it
>>>>> does not work if secondary is run as app container because each of
>>>>> containerized secondary has PID 1. To reserve unique name, use
>>>>> hostname
>>>>> instead of PID if the value is 1.
>>>>>
>>>>> Cc: stable@dpdk.org
>>>>>
>>>>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>>> ---
>>>>
>>>> I'm not too well versed in containers - is this hostname 1) always
>>>> set, and 2) always unique?
>>> For docker, 1) hostname is always set. 2) The hostname is decided as
>>> short form of container ID, so it might not be unique even though
>>> very low possibility.
>>>
>>> I found that we can get whole container ID in `/proc/self/cgroup` as
>>> discussed [1]. I think using hostname is reasonable way without
>>> running many secondary processes. However, it might be better to use
>>> 64 digits full container ID instead of 12 digits short ID if ensure
>>> uniqueness strongly. What do yo think?
>>>
>>> [1]
>>> https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237
>>>
>>
>> I think it's better to err on the side of caution and guarantee better
>> uniqueness. This code will get into an LTS and will be used for years
>> to come :)
>>
>
> ...however, i think a full 64-digit ID won't even fit into the fbarray
> filename, as i believe it's limited to something like 64 chars. Perhaps
> hostname would be enough after all... or we can increase fbarray name
> length - that would require ABI breakage but the ABI is already broken
> in this release, so it's OK i think.
OK.
>> Wouldn't an error in fscanf() leak the file handle? I think you need
to fclose() before checking the result.
> I would like to fix it.
I would like send v3 patch for fixing for fclose().
Thanks,
Yasufumi
On 11-Jul-19 10:37 AM, Yasufumi Ogawa wrote:
> On 2019/07/09 19:26, Burakov, Anatoly wrote:
>> On 09-Jul-19 11:24 AM, Burakov, Anatoly wrote:
>>> On 09-Jul-19 11:22 AM, Yasufumi Ogawa wrote:
>>>> Hi Anatoly,
>>>>
>>>> On 2019/07/05 17:53, Burakov, Anatoly wrote:
>>>>> On 16-Apr-19 4:43 AM, ogawa.yasufumi@lab.ntt.co.jp wrote:
>>>>>> From: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>>>>
>>>>>> In secondary_msl_create_walk(), it creates a file for fbarrays
>>>>>> with its
>>>>>> PID for reserving unique name among secondary processes. However, it
>>>>>> does not work if secondary is run as app container because each of
>>>>>> containerized secondary has PID 1. To reserve unique name, use
>>>>>> hostname
>>>>>> instead of PID if the value is 1.
>>>>>>
>>>>>> Cc: stable@dpdk.org
>>>>>>
>>>>>> Signed-off-by: Yasufumi Ogawa <ogawa.yasufumi@lab.ntt.co.jp>
>>>>>> ---
>>>>>
>>>>> I'm not too well versed in containers - is this hostname 1) always
>>>>> set, and 2) always unique?
>>>> For docker, 1) hostname is always set. 2) The hostname is decided as
>>>> short form of container ID, so it might not be unique even though
>>>> very low possibility.
>>>>
>>>> I found that we can get whole container ID in `/proc/self/cgroup` as
>>>> discussed [1]. I think using hostname is reasonable way without
>>>> running many secondary processes. However, it might be better to use
>>>> 64 digits full container ID instead of 12 digits short ID if ensure
>>>> uniqueness strongly. What do yo think?
>>>>
>>>> [1]
>>>> https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237
>>>>
>>>
>>> I think it's better to err on the side of caution and guarantee
>>> better uniqueness. This code will get into an LTS and will be used
>>> for years to come :)
>>>
>>
>> ...however, i think a full 64-digit ID won't even fit into the fbarray
>> filename, as i believe it's limited to something like 64 chars.
>> Perhaps hostname would be enough after all... or we can increase
>> fbarray name length - that would require ABI breakage but the ABI is
>> already broken in this release, so it's OK i think.
> OK.
Just a note: you're targetting this fix towards stable too. For stable,
you cannot break ABI, so we would have to do with the shorter hostname.
It's only for 19.08 that you can change fbarray length and use the full
64-char container ID for uniqueness.
>
> >> Wouldn't an error in fscanf() leak the file handle? I think you need
> to fclose() before checking the result.
> > I would like to fix it.
> I would like send v3 patch for fixing for fclose().
Please do :)
>
> Thanks,
> Yasufumi
>
>
@@ -1362,6 +1362,7 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,
struct rte_memseg_list *primary_msl, *local_msl;
char name[PATH_MAX];
int msl_idx, ret;
+ char proc_id[16];
if (msl->external)
return 0;
@@ -1371,8 +1372,28 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,
local_msl = &local_memsegs[msl_idx];
/* create distinct fbarrays for each secondary */
- snprintf(name, RTE_FBARRAY_NAME_LEN, "%s_%i",
- primary_msl->memseg_arr.name, getpid());
+ /* if run secondary in a container, the name of fbarray file cannot
+ * be decided with pid because getpid() always returns 1, so use
+ * hostname as a unique identifier among containers instead.
+ */
+ if (getpid() == 1) {
+ FILE *hn_fp;
+ hn_fp = fopen("/etc/hostname", "r");
+ if (hn_fp == NULL) {
+ RTE_LOG(ERR, EAL,
+ "Cannot open '/etc/hostname' for secondary\n");
+ return -1;
+ }
+
+ /* with docker, /etc/hostname just has one entry of hostname */
+ if (fscanf(hn_fp, "%s", proc_id) == EOF)
+ return -1;
+ fclose(hn_fp);
+ } else
+ sprintf(proc_id, "%d", (int)getpid());
+
+ snprintf(name, RTE_FBARRAY_NAME_LEN, "%s_%s",
+ primary_msl->memseg_arr.name, proc_id);
ret = rte_fbarray_init(&local_msl->memseg_arr, name,
primary_msl->memseg_arr.len,