[v6] eal/unix: fix thread creation
Checks
Commit Message
From: Tyler Retzlaff <roretzla@linux.microsoft.com>
In rte_thread_create setting affinity after pthread_create may fail.
Such a failure should result in the entire rte_thread_create failing
but doesn't.
Additionally if there is a failure to set affinity a race exists where
the creating thread will free ctx and depending on scheduling of the new
thread it may also free ctx (double free).
Resolve the above by setting the affinity from the newly created thread
using a condition variable to signal the completion of the thread
start wrapper having completed.
Since we are now waiting for the thread start wrapper to complete we can
allocate the thread start wrapper context on the stack. While here clean
up the variable naming in the context to better highlight the fields of
the context require synchronization between the creating and created
thread.
Fixes: ce6e911d20f6 ("eal: add thread lifetime API")
Cc: stable@dpdk.org
Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since v5:
- dropped volatile and switched to boolean for wrapper_done,
- reverted to rte_thread_set_affinity_by_id() call,
---
lib/eal/unix/rte_thread.c | 73 ++++++++++++++++++++++++---------------
1 file changed, 45 insertions(+), 28 deletions(-)
Comments
On Fri, Mar 17, 2023 at 07:52:29PM +0100, David Marchand wrote:
> From: Tyler Retzlaff <roretzla@linux.microsoft.com>
>
> In rte_thread_create setting affinity after pthread_create may fail.
> Such a failure should result in the entire rte_thread_create failing
> but doesn't.
>
> Additionally if there is a failure to set affinity a race exists where
> the creating thread will free ctx and depending on scheduling of the new
> thread it may also free ctx (double free).
>
> Resolve the above by setting the affinity from the newly created thread
> using a condition variable to signal the completion of the thread
> start wrapper having completed.
>
> Since we are now waiting for the thread start wrapper to complete we can
> allocate the thread start wrapper context on the stack. While here clean
> up the variable naming in the context to better highlight the fields of
> the context require synchronization between the creating and created
> thread.
>
> Fixes: ce6e911d20f6 ("eal: add thread lifetime API")
> Cc: stable@dpdk.org
>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Looks good to me, not sure if you need a Reviewed-by: from me for the
changes but here is one anyway.
v5
Reviewed-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
On Fri, Mar 17, 2023 at 10:24 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> On Fri, Mar 17, 2023 at 07:52:29PM +0100, David Marchand wrote:
> > From: Tyler Retzlaff <roretzla@linux.microsoft.com>
> >
> > In rte_thread_create setting affinity after pthread_create may fail.
> > Such a failure should result in the entire rte_thread_create failing
> > but doesn't.
> >
> > Additionally if there is a failure to set affinity a race exists where
> > the creating thread will free ctx and depending on scheduling of the new
> > thread it may also free ctx (double free).
> >
> > Resolve the above by setting the affinity from the newly created thread
> > using a condition variable to signal the completion of the thread
> > start wrapper having completed.
> >
> > Since we are now waiting for the thread start wrapper to complete we can
> > allocate the thread start wrapper context on the stack. While here clean
> > up the variable naming in the context to better highlight the fields of
> > the context require synchronization between the creating and created
> > thread.
> >
> > Fixes: ce6e911d20f6 ("eal: add thread lifetime API")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> Looks good to me, not sure if you need a Reviewed-by: from me for the
> changes but here is one anyway.
>
> v5
> Reviewed-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Thanks for the review, I'll keep your SoB only on the patch.
On Fri, Mar 17, 2023 at 7:52 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> From: Tyler Retzlaff <roretzla@linux.microsoft.com>
>
> In rte_thread_create setting affinity after pthread_create may fail.
> Such a failure should result in the entire rte_thread_create failing
> but doesn't.
>
> Additionally if there is a failure to set affinity a race exists where
> the creating thread will free ctx and depending on scheduling of the new
> thread it may also free ctx (double free).
>
> Resolve the above by setting the affinity from the newly created thread
> using a condition variable to signal the completion of the thread
> start wrapper having completed.
>
> Since we are now waiting for the thread start wrapper to complete we can
> allocate the thread start wrapper context on the stack. While here clean
> up the variable naming in the context to better highlight the fields of
> the context require synchronization between the creating and created
> thread.
>
> Fixes: ce6e911d20f6 ("eal: add thread lifetime API")
> Cc: stable@dpdk.org
>
> Signed-off-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Applied thanks.
@@ -5,6 +5,7 @@
#include <errno.h>
#include <pthread.h>
+#include <stdbool.h>
#include <stdlib.h>
#include <string.h>
@@ -16,9 +17,14 @@ struct eal_tls_key {
pthread_key_t thread_index;
};
-struct thread_routine_ctx {
+struct thread_start_context {
rte_thread_func thread_func;
- void *routine_args;
+ void *thread_args;
+ const rte_thread_attr_t *thread_attr;
+ pthread_mutex_t wrapper_mutex;
+ pthread_cond_t wrapper_cond;
+ int wrapper_ret;
+ bool wrapper_done;
};
static int
@@ -81,13 +87,29 @@ thread_map_os_priority_to_eal_priority(int policy, int os_pri,
}
static void *
-thread_func_wrapper(void *arg)
+thread_start_wrapper(void *arg)
{
- struct thread_routine_ctx ctx = *(struct thread_routine_ctx *)arg;
+ struct thread_start_context *ctx = (struct thread_start_context *)arg;
+ rte_thread_func thread_func = ctx->thread_func;
+ void *thread_args = ctx->thread_args;
+ int ret = 0;
+
+ if (ctx->thread_attr != NULL && CPU_COUNT(&ctx->thread_attr->cpuset) > 0) {
+ ret = rte_thread_set_affinity_by_id(rte_thread_self(), &ctx->thread_attr->cpuset);
+ if (ret != 0)
+ RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity_by_id failed\n");
+ }
- free(arg);
+ pthread_mutex_lock(&ctx->wrapper_mutex);
+ ctx->wrapper_ret = ret;
+ ctx->wrapper_done = true;
+ pthread_cond_signal(&ctx->wrapper_cond);
+ pthread_mutex_unlock(&ctx->wrapper_mutex);
- return (void *)(uintptr_t)ctx.thread_func(ctx.routine_args);
+ if (ret != 0)
+ return NULL;
+
+ return (void *)(uintptr_t)thread_func(thread_args);
}
int
@@ -98,20 +120,18 @@ rte_thread_create(rte_thread_t *thread_id,
int ret = 0;
pthread_attr_t attr;
pthread_attr_t *attrp = NULL;
- struct thread_routine_ctx *ctx;
struct sched_param param = {
.sched_priority = 0,
};
int policy = SCHED_OTHER;
-
- ctx = calloc(1, sizeof(*ctx));
- if (ctx == NULL) {
- RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n");
- ret = ENOMEM;
- goto cleanup;
- }
- ctx->routine_args = args;
- ctx->thread_func = thread_func;
+ struct thread_start_context ctx = {
+ .thread_func = thread_func,
+ .thread_args = args,
+ .thread_attr = thread_attr,
+ .wrapper_done = false,
+ .wrapper_mutex = PTHREAD_MUTEX_INITIALIZER,
+ .wrapper_cond = PTHREAD_COND_INITIALIZER,
+ };
if (thread_attr != NULL) {
ret = pthread_attr_init(&attr);
@@ -133,7 +153,6 @@ rte_thread_create(rte_thread_t *thread_id,
goto cleanup;
}
-
if (thread_attr->priority ==
RTE_THREAD_PRIORITY_REALTIME_CRITICAL) {
ret = ENOTSUP;
@@ -158,24 +177,22 @@ rte_thread_create(rte_thread_t *thread_id,
}
ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp,
- thread_func_wrapper, ctx);
+ thread_start_wrapper, &ctx);
if (ret != 0) {
RTE_LOG(DEBUG, EAL, "pthread_create failed\n");
goto cleanup;
}
- if (thread_attr != NULL && CPU_COUNT(&thread_attr->cpuset) > 0) {
- ret = rte_thread_set_affinity_by_id(*thread_id,
- &thread_attr->cpuset);
- if (ret != 0) {
- RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity_by_id failed\n");
- goto cleanup;
- }
- }
+ pthread_mutex_lock(&ctx.wrapper_mutex);
+ while (!ctx.wrapper_done)
+ pthread_cond_wait(&ctx.wrapper_cond, &ctx.wrapper_mutex);
+ ret = ctx.wrapper_ret;
+ pthread_mutex_unlock(&ctx.wrapper_mutex);
+
+ if (ret != 0)
+ pthread_join((pthread_t)thread_id->opaque_id, NULL);
- ctx = NULL;
cleanup:
- free(ctx);
if (attrp != NULL)
pthread_attr_destroy(&attr);