From patchwork Fri Sep 11 15:29:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 77437 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DEF1CA04BB; Fri, 11 Sep 2020 17:31:18 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B3F961C0CF; Fri, 11 Sep 2020 17:31:18 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 727461C0CD for ; Fri, 11 Sep 2020 17:31:17 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F29701063; Fri, 11 Sep 2020 08:31:16 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.10.210]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 55D653F73C; Fri, 11 Sep 2020 08:31:15 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, dharmik.thakkar@arm.com, Steven Lariau Date: Fri, 11 Sep 2020 16:29:34 +0100 Message-Id: <20200911152938.8019-2-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200911152938.8019-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH 1/5] lib/stack: fix inconsistent weak / strong cas X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Fix cmpexchange usage of weak / strong. The generated code is the same on x86 and ARM (there is no weak cmpexchange), but the old usage was inconsistent. For push and pop update size, weak is used because cmpexchange is inside a loop. For pop update root, strong is used even though cmpexchange is inside a loop, because there may be a lot of operations to do in a loop iteration (locate the new head). Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 999359f08..1e0ea0bef 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -96,7 +96,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, /* len is updated on failure */ if (__atomic_compare_exchange_n(&list->len, &len, len - num, - 0, __ATOMIC_ACQUIRE, + 1, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE)) break; } @@ -149,7 +149,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, (rte_int128_t *)&list->head, (rte_int128_t *)&old_head, (rte_int128_t *)&new_head, - 1, __ATOMIC_RELEASE, + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED); } while (success == 0); From patchwork Fri Sep 11 15:29:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 77438 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0FF75A04BB; Fri, 11 Sep 2020 17:31:26 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0D7F21C0D7; Fri, 11 Sep 2020 17:31:24 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 632141C0D5 for ; Fri, 11 Sep 2020 17:31:22 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DDE201063; Fri, 11 Sep 2020 08:31:21 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.10.210]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6097B3F73C; Fri, 11 Sep 2020 08:31:20 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, dharmik.thakkar@arm.com, Steven Lariau Date: Fri, 11 Sep 2020 16:29:35 +0100 Message-Id: <20200911152938.8019-3-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200911152938.8019-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH 2/5] lib/stack: remove push acquire fence X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" An acquire fence is used to make sure loads after the fence can observe all store operations before a specific store-release. But push doesn't read any data, except for the head which is part of a CAS operation (the items on the list are not read). So there is no need for the acquire barrier. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 6 ------ 1 file changed, 6 deletions(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 1e0ea0bef..82b7287f1 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -44,12 +44,6 @@ __rte_stack_lf_push_elems(struct rte_stack_lf_list *list, do { struct rte_stack_lf_head new_head; - /* Use an acquire fence to establish a synchronized-with - * relationship between the list->head load and store-release - * operations (as part of the rte_atomic128_cmp_exchange()). - */ - __atomic_thread_fence(__ATOMIC_ACQUIRE); - /* Swing the top pointer to the first element in the list and * make the last element point to the old top. */ From patchwork Fri Sep 11 15:29:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 77439 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1AE48A04BB; Fri, 11 Sep 2020 17:31:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 45B8E1C10A; Fri, 11 Sep 2020 17:31:26 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 2C6E61C10A for ; Fri, 11 Sep 2020 17:31:25 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B32B81063; Fri, 11 Sep 2020 08:31:24 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.10.210]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 545393F73C; Fri, 11 Sep 2020 08:31:23 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, dharmik.thakkar@arm.com, Steven Lariau Date: Fri, 11 Sep 2020 16:29:36 +0100 Message-Id: <20200911152938.8019-4-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200911152938.8019-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH 3/5] lib/stack: remove redundant orderings for list->len X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The load-acquire of list->len on pop function is redundant. Only the CAS success needs to be load-acquire. It synchronizes with the store release in push, to ensure that the updated head is visible when the new length is visible. Without this, one thread in pop could see the increased length but the old list, which doesn't have enough items yet for pop to succeed. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 82b7287f1..2bc639419 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -80,7 +80,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, int success; /* Reserve num elements, if available */ - len = __atomic_load_n(&list->len, __ATOMIC_ACQUIRE); + len = __atomic_load_n(&list->len, __ATOMIC_RELAXED); while (1) { /* Does the list contain enough elements? */ @@ -91,7 +91,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, if (__atomic_compare_exchange_n(&list->len, &len, len - num, 1, __ATOMIC_ACQUIRE, - __ATOMIC_ACQUIRE)) + __ATOMIC_RELAXED)) break; } From patchwork Fri Sep 11 15:29:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 77440 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8320BA04BB; Fri, 11 Sep 2020 17:31:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EFF1B1C112; Fri, 11 Sep 2020 17:31:31 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 6DA801C113 for ; Fri, 11 Sep 2020 17:31:30 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EC34B147A; Fri, 11 Sep 2020 08:31:29 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.10.210]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6E1333F73C; Fri, 11 Sep 2020 08:31:28 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, dharmik.thakkar@arm.com, Steven Lariau Date: Fri, 11 Sep 2020 16:29:37 +0100 Message-Id: <20200911152938.8019-5-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200911152938.8019-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH 4/5] lib/stack: reload head when pop fails X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" List head must be loaded right before continue (when failed to find the new head). Without this, one thread might keep trying and failing to pop items without ever loading the new correct head. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 2bc639419..adb9f590d 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -133,8 +133,10 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, /* If NULL was encountered, the list was modified while * traversing it. Retry. */ - if (i != num) + if (i != num) { + old_head = list->head; continue; + } new_head.top = tmp; new_head.cnt = old_head.cnt + 1; From patchwork Fri Sep 11 15:29:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 77441 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3449DA04BB; Fri, 11 Sep 2020 17:31:58 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1AD311C11D; Fri, 11 Sep 2020 17:31:37 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 302771C11C for ; Fri, 11 Sep 2020 17:31:35 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A5A131063; Fri, 11 Sep 2020 08:31:34 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.10.210]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2800F3F73C; Fri, 11 Sep 2020 08:31:32 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, dharmik.thakkar@arm.com, Steven Lariau Date: Fri, 11 Sep 2020 16:29:38 +0100 Message-Id: <20200911152938.8019-6-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200911152938.8019-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH 5/5] lib/stack: remove pop cas release ordering X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Replace the store-release by relaxed for the CAS success at the end of pop. Release isn't needed, because there is not write to data that need to be synchronized. The only preceding write is when the length is decreased, but the length CAS loop already ensures the right synchronization. The situation to avoid is when a thread sees the old length but the new list, that doesn't have enough items for pop to success. But the CAS success on length before the pop loop ensures any core reads and updates the latest length, preventing this situation. The store-release is also used to make sure that the items are read before the head is updated, in order to prevent a core in pop to read an incorrect value because another core rewrites it with push. But this isn't needed, because items are read only when removed from the used list. Right after this, they are pushed to the free list, and the store-release in push makes sure the items are read before they are visible in the free list. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang --- lib/librte_stack/rte_stack_lf_c11.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index adb9f590d..8800db42e 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -145,7 +145,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, (rte_int128_t *)&list->head, (rte_int128_t *)&old_head, (rte_int128_t *)&new_head, - 0, __ATOMIC_RELEASE, + 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); } while (success == 0);