From patchwork Thu Feb 14 13:30:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 50317 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 694C72B93; Thu, 14 Feb 2019 14:31:05 +0100 (CET) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 683C091; Thu, 14 Feb 2019 14:31:03 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6B9682F9457; Thu, 14 Feb 2019 13:31:02 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-16.brq.redhat.com [10.40.204.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id 33DC410021B6; Thu, 14 Feb 2019 13:30:46 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: olivier.matz@6wind.com, anatoly.burakov@intel.com, ktraynor@redhat.com, stable@dpdk.org Date: Thu, 14 Feb 2019 14:30:41 +0100 Message-Id: <1550151042-9764-1-git-send-email-david.marchand@redhat.com> In-Reply-To: <1550074412-31285-1-git-send-email-david.marchand@redhat.com> References: <1550074412-31285-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 14 Feb 2019 13:31:02 +0000 (UTC) Subject: [dpdk-dev] [PATCH v2 1/2] eal: fix potential incorrect pinning for ctrl threads X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" pthread_setaffinity_np returns a >0 value on error. We could end up letting the ctrl threads on the current process cpu affinity. Fixes: d651ee4919cd ("eal: set affinity for control threads") Signed-off-by: David Marchand --- lib/librte_eal/common/eal_common_thread.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c index 48ef4d6..a3985ce 100644 --- a/lib/librte_eal/common/eal_common_thread.c +++ b/lib/librte_eal/common/eal_common_thread.c @@ -209,7 +209,7 @@ static void *rte_thread_init(void *arg) CPU_SET(rte_get_master_lcore(), &cpuset); ret = pthread_setaffinity_np(*thread, sizeof(cpuset), &cpuset); - if (ret < 0) + if (ret) goto fail; ret = pthread_barrier_wait(¶ms->configured); From patchwork Thu Feb 14 13:30:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 50318 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 03BD41B3EA; Thu, 14 Feb 2019 14:31:07 +0100 (CET) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 849EB4CA0; Thu, 14 Feb 2019 14:31:05 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8BADE7C0B5; Thu, 14 Feb 2019 13:31:04 +0000 (UTC) Received: from dmarchan.remote.csb (ovpn-204-16.brq.redhat.com [10.40.204.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9563F108BEFE; Thu, 14 Feb 2019 13:31:02 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: olivier.matz@6wind.com, anatoly.burakov@intel.com, ktraynor@redhat.com, stable@dpdk.org Date: Thu, 14 Feb 2019 14:30:42 +0100 Message-Id: <1550151042-9764-2-git-send-email-david.marchand@redhat.com> In-Reply-To: <1550151042-9764-1-git-send-email-david.marchand@redhat.com> References: <1550074412-31285-1-git-send-email-david.marchand@redhat.com> <1550151042-9764-1-git-send-email-david.marchand@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Thu, 14 Feb 2019 13:31:04 +0000 (UTC) Subject: [dpdk-dev] [PATCH v2 2/2] eal: restrict ctrl threads to startup cpu affinity X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Spawning the ctrl threads on anything that is not part of the eal coremask is not that polite to the rest of the system, especially when you took good care to pin your processes on cpu resources with tools like taskset (linux) / cpuset (freebsd). Rather than introduce yet another eal options to control on which cpu those ctrl threads are created, let's take the startup cpu affinity as a reference and remove the eal coremask from it. If no cpu is left, then we default to the master core. The cpuset is computed once at init before the original cpu affinity is lost. Introduced a RTE_CPU_AND macro to abstract the differences between linux and freebsd respective macros. Examples in a 4 cores FreeBSD vm: $ ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ -- -i --total-num-mbufs=2048 $ procstat -S 1057 PID TID COMM TDNAME CPU CSID CPU MASK 1057 100131 testpmd - 2 1 2 1057 100140 testpmd eal-intr-thread 1 1 0-1 1057 100141 testpmd rte_mp_handle 1 1 0-1 1057 100142 testpmd lcore-slave-3 3 1 3 $ cpuset -l 1,2,3 ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ -- -i --total-num-mbufs=2048 $ procstat -S 1061 PID TID COMM TDNAME CPU CSID CPU MASK 1061 100131 testpmd - 2 2 2 1061 100144 testpmd eal-intr-thread 1 2 1 1061 100145 testpmd rte_mp_handle 1 2 1 1061 100147 testpmd lcore-slave-3 3 2 3 $ cpuset -l 2,3 ./build/app/testpmd -l 2,3 --no-huge --no-pci -m 512 \ -- -i --total-num-mbufs=2048 $ procstat -S 1065 PID TID COMM TDNAME CPU CSID CPU MASK 1065 100131 testpmd - 2 2 2 1065 100148 testpmd eal-intr-thread 2 2 2 1065 100149 testpmd rte_mp_handle 2 2 2 1065 100150 testpmd lcore-slave-3 3 2 3 Fixes: d651ee4919cd ("eal: set affinity for control threads") Signed-off-by: David Marchand --- Changes since v1: - added some description in the prog guide - fixed FreeBSD build --- doc/guides/prog_guide/env_abstraction_layer.rst | 14 +++++++++++++ lib/librte_eal/common/eal_common_options.c | 28 +++++++++++++++++++++++++ lib/librte_eal/common/eal_common_thread.c | 21 ++++--------------- lib/librte_eal/common/eal_internal_cfg.h | 3 +++ lib/librte_eal/common/include/rte_lcore.h | 17 +++++++++++---- 5 files changed, 62 insertions(+), 21 deletions(-) diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst index 929d76d..bfc66d9 100644 --- a/doc/guides/prog_guide/env_abstraction_layer.rst +++ b/doc/guides/prog_guide/env_abstraction_layer.rst @@ -498,6 +498,20 @@ Those TLS include *_cpuset* and *_socket_id*: * *_socket_id* stores the NUMA node of the CPU set. If the CPUs in CPU set belong to different NUMA node, the *_socket_id* will be set to SOCKET_ID_ANY. +Control Thread API +~~~~~~~~~~~~~~~~~~ + +It is possible to create Control Threads using the public API ``rte_ctrl_thread_create()``. +Those threads can be used for management/infrastructure tasks and are used internally by DPDK for multi process support and interrupt handling. + +Those threads will be scheduled on cpus part of the original process cpu affinity from which the dataplane and service lcores are excluded. + +For example, on a 8 cpus system, starting a dpdk application with -l 2,3 (dataplane cores), then depending on the affinity configuration which can be controlled with tools like taskset (Linux) or cpuset (FreeBSD), + +- with no affinity configuration, the Control Threads will end up on 0-1,4-7 cpus. +- with affinity restricted to 2-4, the Control Threads will end up on cpu 4. +- with affinity restricted to 2-3, the Control Threads will end up on cpu 2 (master lcore, which is the default when no cpu is available). + .. _known_issue_label: Known Issues diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c index 1f45f82..fca3f83 100644 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -217,6 +217,7 @@ struct device_option { internal_cfg->create_uio_dev = 0; internal_cfg->iova_mode = RTE_IOVA_DC; internal_cfg->user_mbuf_pool_ops_name = NULL; + CPU_ZERO(&internal_cfg->ctrl_cpuset); internal_cfg->init_complete = 0; } @@ -1359,6 +1360,31 @@ static int xdigit2val(unsigned char c) cfg->lcore_count -= removed; } +static void +compute_ctrl_threads_cpuset(struct internal_config *internal_cfg) +{ + rte_cpuset_t *cpuset = &internal_cfg->ctrl_cpuset; + rte_cpuset_t default_set; + unsigned int lcore_id; + + for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { + if (eal_cpu_detected(lcore_id) && + rte_lcore_has_role(lcore_id, ROLE_OFF)) { + CPU_SET(lcore_id, cpuset); + } + } + + if (pthread_getaffinity_np(pthread_self(), sizeof(rte_cpuset_t), + &default_set) < 0) + CPU_ZERO(&default_set); + + RTE_CPU_AND(cpuset, cpuset, &default_set); + + /* if no detected cpu is off, use master core */ + if (!CPU_COUNT(cpuset)) + CPU_SET(rte_get_master_lcore(), cpuset); +} + int eal_cleanup_config(struct internal_config *internal_cfg) { @@ -1392,6 +1418,8 @@ static int xdigit2val(unsigned char c) lcore_config[cfg->master_lcore].core_role = ROLE_RTE; } + compute_ctrl_threads_cpuset(internal_cfg); + /* if no memory amounts were requested, this will result in 0 and * will be overridden later, right after eal_hugepage_info_init() */ for (i = 0; i < RTE_MAX_NUMA_NODES; i++) diff --git a/lib/librte_eal/common/eal_common_thread.c b/lib/librte_eal/common/eal_common_thread.c index a3985ce..14f206c 100644 --- a/lib/librte_eal/common/eal_common_thread.c +++ b/lib/librte_eal/common/eal_common_thread.c @@ -16,6 +16,7 @@ #include #include +#include "eal_internal_cfg.h" #include "eal_private.h" #include "eal_thread.h" @@ -168,10 +169,9 @@ static void *rte_thread_init(void *arg) const pthread_attr_t *attr, void *(*start_routine)(void *), void *arg) { + rte_cpuset_t *cpuset = &internal_config.ctrl_cpuset; struct rte_thread_ctrl_params *params; - unsigned int lcore_id; - rte_cpuset_t cpuset; - int cpu_found, ret; + int ret; params = malloc(sizeof(*params)); if (!params) @@ -195,20 +195,7 @@ static void *rte_thread_init(void *arg) "Cannot set name for ctrl thread\n"); } - cpu_found = 0; - CPU_ZERO(&cpuset); - for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) { - if (eal_cpu_detected(lcore_id) && - rte_lcore_has_role(lcore_id, ROLE_OFF)) { - CPU_SET(lcore_id, &cpuset); - cpu_found = 1; - } - } - /* if no detected cpu is off, use master core */ - if (!cpu_found) - CPU_SET(rte_get_master_lcore(), &cpuset); - - ret = pthread_setaffinity_np(*thread, sizeof(cpuset), &cpuset); + ret = pthread_setaffinity_np(*thread, sizeof(*cpuset), cpuset); if (ret) goto fail; diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h index 60eaead..edff09d 100644 --- a/lib/librte_eal/common/eal_internal_cfg.h +++ b/lib/librte_eal/common/eal_internal_cfg.h @@ -13,6 +13,8 @@ #include #include +#include "eal_thread.h" + #define MAX_HUGEPAGE_SIZES 3 /**< support up to 3 page sizes */ /* @@ -73,6 +75,7 @@ struct internal_config { unsigned num_hugepage_sizes; /**< how many sizes on this system */ struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES]; enum rte_iova_mode iova_mode ; /**< Set IOVA mode on this system */ + rte_cpuset_t ctrl_cpuset; /**< cpuset for ctrl threads */ volatile unsigned int init_complete; /**< indicates whether EAL has completed initialization */ }; diff --git a/lib/librte_eal/common/include/rte_lcore.h b/lib/librte_eal/common/include/rte_lcore.h index 6e09d91..eb49132 100644 --- a/lib/librte_eal/common/include/rte_lcore.h +++ b/lib/librte_eal/common/include/rte_lcore.h @@ -23,10 +23,18 @@ #define LCORE_ID_ANY UINT32_MAX /**< Any lcore. */ #if defined(__linux__) - typedef cpu_set_t rte_cpuset_t; +typedef cpu_set_t rte_cpuset_t; +#define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2) #elif defined(__FreeBSD__) #include - typedef cpuset_t rte_cpuset_t; +typedef cpuset_t rte_cpuset_t; +#define RTE_CPU_AND(dst, src1, src2) do \ +{ \ + cpuset_t tmp; \ + CPU_COPY(src1, &tmp); \ + CPU_AND(&tmp, src2); \ + CPU_COPY(&tmp, dst); \ +} while (0) #endif /** @@ -280,8 +288,9 @@ struct lcore_config { * Create a control thread. * * Wrapper to pthread_create(), pthread_setname_np() and - * pthread_setaffinity_np(). The dataplane and service lcores are - * excluded from the affinity of the new thread. + * pthread_setaffinity_np(). The affinity of the new thread is based + * on the cpu affinity retrieved at the time rte_eal_init() was called, + * the dataplane and service lcores are then excluded. * * @param thread * Filled with the thread id of the new created thread.