From patchwork Thu Mar 16 00:07:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tyler Retzlaff X-Patchwork-Id: 125173 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5D19F41EA5; Thu, 16 Mar 2023 01:07:12 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C1721427F5; Thu, 16 Mar 2023 01:07:08 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id 6435540EF1 for ; Thu, 16 Mar 2023 01:07:06 +0100 (CET) Received: by linux.microsoft.com (Postfix, from userid 1086) id AF95720C7DF8; Wed, 15 Mar 2023 17:07:05 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com AF95720C7DF8 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1678925225; bh=QdVmYb/o7Aq8A11P9wbdx3I8kxxqizdMblRmgtj6ZXc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Fb2nxXTEG9h00kxtMcmHXYdQ04OIU9pq0LurxCtFWgMla2bPA+TKKkoLw82SLAcOU +lXR567MJKGAKXgSauw0FubVD43VxQ05f5m5iOTn7+asnVPPHntm0RLRZFzpgs8eBi benaYodT0YzM1VzyDrpEwX4TIB5LuvkvoKJTu81I= From: Tyler Retzlaff To: dev@dpdk.org Cc: thomas@monjalon.net, david.marchand@redhat.com, stephen@networkplumber.org, Tyler Retzlaff Subject: [PATCH v5 1/2] eal: make cpusetp to rte thread set affinity const Date: Wed, 15 Mar 2023 17:07:03 -0700 Message-Id: <1678925224-2706-2-git-send-email-roretzla@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1678925224-2706-1-git-send-email-roretzla@linux.microsoft.com> References: <1677782682-27200-1-git-send-email-roretzla@linux.microsoft.com> <1678925224-2706-1-git-send-email-roretzla@linux.microsoft.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org const qualify rte_cpuset_t *cpusetp parameter in the rte_thread_set_affinity API. Signed-off-by: Tyler Retzlaff --- lib/eal/common/eal_common_thread.c | 6 +++--- lib/eal/include/rte_thread.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c index 079a385..c9e5619 100644 --- a/lib/eal/common/eal_common_thread.c +++ b/lib/eal/common/eal_common_thread.c @@ -34,7 +34,7 @@ unsigned rte_socket_id(void) } static int -eal_cpuset_socket_id(rte_cpuset_t *cpusetp) +eal_cpuset_socket_id(const rte_cpuset_t *cpusetp) { unsigned cpu = 0; int socket_id = SOCKET_ID_ANY; @@ -62,7 +62,7 @@ unsigned rte_socket_id(void) } static void -thread_update_affinity(rte_cpuset_t *cpusetp) +thread_update_affinity(const rte_cpuset_t *cpusetp) { unsigned int lcore_id = rte_lcore_id(); @@ -83,7 +83,7 @@ unsigned rte_socket_id(void) } int -rte_thread_set_affinity(rte_cpuset_t *cpusetp) +rte_thread_set_affinity(const rte_cpuset_t *cpusetp) { if (rte_thread_set_affinity_by_id(rte_thread_self(), cpusetp) != 0) { RTE_LOG(ERR, EAL, "rte_thread_set_affinity_by_id failed\n"); diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h index ddd411e..e461354 100644 --- a/lib/eal/include/rte_thread.h +++ b/lib/eal/include/rte_thread.h @@ -348,7 +348,7 @@ int rte_thread_get_affinity_by_id(rte_thread_t thread_id, * @return * On success, return 0; otherwise return -1; */ -int rte_thread_set_affinity(rte_cpuset_t *cpusetp); +int rte_thread_set_affinity(const rte_cpuset_t *cpusetp); /** * Get core affinity of the current thread. From patchwork Thu Mar 16 00:07:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tyler Retzlaff X-Patchwork-Id: 125174 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6625741EA5; Thu, 16 Mar 2023 01:07:17 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB1C142BAC; Thu, 16 Mar 2023 01:07:09 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id 6C4B640FDF; Thu, 16 Mar 2023 01:07:06 +0100 (CET) Received: by linux.microsoft.com (Postfix, from userid 1086) id BBBE420C7DF9; Wed, 15 Mar 2023 17:07:05 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com BBBE420C7DF9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1678925225; bh=uss0tPpe1gjBSFeCYcmOvSPibVM0kMWp3ULqOlQLANo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=abMFr/r+Z0JUEAqtfsLJFx8gwbFyMw3o3mrx3S4+f+Hck7AQyzNyuGwWvY2/lUsg1 u9l1D1ToXrbTqniTdaZsIHnvDIhkmgwifGTZN6IUzUqoiAWG8/cH9h4AlIy+fJjBpR 1D8dcDGtfmlDI98bNdCabPbx4k8nxwVYbvhn1VyM= From: Tyler Retzlaff To: dev@dpdk.org Cc: thomas@monjalon.net, david.marchand@redhat.com, stephen@networkplumber.org, Tyler Retzlaff , stable@dpdk.org Subject: [PATCH v5 2/2] eal: fix failure path race setting new thread affinity Date: Wed, 15 Mar 2023 17:07:04 -0700 Message-Id: <1678925224-2706-3-git-send-email-roretzla@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1678925224-2706-1-git-send-email-roretzla@linux.microsoft.com> References: <1677782682-27200-1-git-send-email-roretzla@linux.microsoft.com> <1678925224-2706-1-git-send-email-roretzla@linux.microsoft.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In rte_thread_create setting affinity after pthread_create may fail. Such a failure should result in the entire rte_thread_create failing but doesn't. Additionally if there is a failure to set affinity a race exists where the creating thread will free ctx and depending on scheduling of the new thread it may also free ctx (double free). Resolve the above by setting the affinity from the newly created thread using a condition variable to signal the completion of the thread start wrapper having completed. Since we are now waiting for the thread start wrapper to complete we can allocate the thread start wrapper context on the stack. While here clean up the variable naming in the context to better highlight the fields of the context require synchronization between the creating and created thread. Fixes: ce6e911d20f6 ("eal: add thread lifetime API") Cc: stable@dpdk.org Signed-off-by: Tyler Retzlaff --- lib/eal/unix/rte_thread.c | 70 +++++++++++++++++++++++++++++------------------ 1 file changed, 43 insertions(+), 27 deletions(-) diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c index 37ebfcf..5992b04 100644 --- a/lib/eal/unix/rte_thread.c +++ b/lib/eal/unix/rte_thread.c @@ -16,9 +16,14 @@ struct eal_tls_key { pthread_key_t thread_index; }; -struct thread_routine_ctx { +struct thread_start_context { rte_thread_func thread_func; - void *routine_args; + void *thread_args; + const rte_thread_attr_t *thread_attr; + pthread_mutex_t wrapper_mutex; + pthread_cond_t wrapper_cond; + int wrapper_ret; + volatile int wrapper_done; }; static int @@ -81,13 +86,29 @@ struct thread_routine_ctx { } static void * -thread_func_wrapper(void *arg) +thread_start_wrapper(void *arg) { - struct thread_routine_ctx ctx = *(struct thread_routine_ctx *)arg; + struct thread_start_context *ctx = (struct thread_start_context *)arg; + rte_thread_func thread_func = ctx->thread_func; + void *thread_args = ctx->thread_args; + int ret = 0; - free(arg); + if (ctx->thread_attr != NULL && CPU_COUNT(&ctx->thread_attr->cpuset) > 0) { + ret = rte_thread_set_affinity(&ctx->thread_attr->cpuset); + if (ret != 0) + RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity failed\n"); + } - return (void *)(uintptr_t)ctx.thread_func(ctx.routine_args); + pthread_mutex_lock(&ctx->wrapper_mutex); + ctx->wrapper_ret = ret; + ctx->wrapper_done = 1; + pthread_cond_signal(&ctx->wrapper_cond); + pthread_mutex_unlock(&ctx->wrapper_mutex); + + if (ret != 0) + return NULL; + + return (void *)(uintptr_t)thread_func(thread_args); } int @@ -98,20 +119,17 @@ struct thread_routine_ctx { int ret = 0; pthread_attr_t attr; pthread_attr_t *attrp = NULL; - struct thread_routine_ctx *ctx; struct sched_param param = { .sched_priority = 0, }; int policy = SCHED_OTHER; - - ctx = calloc(1, sizeof(*ctx)); - if (ctx == NULL) { - RTE_LOG(DEBUG, EAL, "Insufficient memory for thread context allocations\n"); - ret = ENOMEM; - goto cleanup; - } - ctx->routine_args = args; - ctx->thread_func = thread_func; + struct thread_start_context ctx = { + .thread_func = thread_func, + .thread_args = args, + .thread_attr = thread_attr, + .wrapper_mutex = PTHREAD_MUTEX_INITIALIZER, + .wrapper_cond = PTHREAD_COND_INITIALIZER, + }; if (thread_attr != NULL) { ret = pthread_attr_init(&attr); @@ -158,24 +176,22 @@ struct thread_routine_ctx { } ret = pthread_create((pthread_t *)&thread_id->opaque_id, attrp, - thread_func_wrapper, ctx); + thread_start_wrapper, &ctx); if (ret != 0) { RTE_LOG(DEBUG, EAL, "pthread_create failed\n"); goto cleanup; } - if (thread_attr != NULL && CPU_COUNT(&thread_attr->cpuset) > 0) { - ret = rte_thread_set_affinity_by_id(*thread_id, - &thread_attr->cpuset); - if (ret != 0) { - RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity_by_id failed\n"); - goto cleanup; - } - } + pthread_mutex_lock(&ctx.wrapper_mutex); + while (ctx.wrapper_done != 1) + pthread_cond_wait(&ctx.wrapper_cond, &ctx.wrapper_mutex); + ret = ctx.wrapper_ret; + pthread_mutex_unlock(&ctx.wrapper_mutex); + + if (ret != 0) + pthread_join((pthread_t)thread_id->opaque_id, NULL); - ctx = NULL; cleanup: - free(ctx); if (attrp != NULL) pthread_attr_destroy(&attr);