From patchwork Sun May 5 08:37:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Mattias_R=C3=B6nnblom?= X-Patchwork-Id: 139870 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 23FCE43FAE; Sun, 5 May 2024 10:48:28 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 537E840395; Sun, 5 May 2024 10:48:06 +0200 (CEST) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2057.outbound.protection.outlook.com [40.107.22.57]) by mails.dpdk.org (Postfix) with ESMTP id 37478402E4 for ; Sun, 5 May 2024 10:48:01 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UO9fVczkjoI3KCeYGta8lT8AKjwJD2FSjmXYTdyj3JOL2KypJHYljccqJHjJb6uqCHXT9eir1VtMBd+VtORkikqQGCwZ2x6JZN96NowbRNFNHWYDOh0PK4m4s5BpLe6OWN7LQ7RP3Rdyl6ddYGt3nYiLZtOIGCxbyv++1bXJtSIshn0pi9p3I5Vr2UwX05ZBmC0CxJncOUyBaYXtA+MY5eE+n4K7FN6ARjlGRrCvjZcvit8XQ64YsnOU7eThoPn7lkagXrt1l/r/kUzT3rgc1Gi1JpFmvvHiovC5OSrIA3MLfdgLXuV1vC641JujpgslsO21JpuSF4PqDCDz1hWCUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pWRlgC0WG3OPI0HrFEOZKsI752eHZBpRWP+cxmiblQc=; b=K7fudXbDOgjvaFRT3p7osy+lwVvLrwbgj03RnlZwsOtLoxEfj5ieP6aaQpb5xUT7DVrZsmx43LieXic5yIzrvXvt2b8QV+VnvwEs7txto5J9gn5FO77bj/UpUrKWD0cvJtkHGoQnEs2R0QfgDnqsMN9NVa93w0yIeFsRI6Hd2j9Bpuhm5Ol9ncUZcVJNlK/w6G3vbpb/azPFXNiMgRK0925RSQ+8NowZPBW1FeoUg3uEakrkrirQ+A9rvo688r268dKAE1tCc5IXqs3BGh1N0Y5rnp7C1U/q3gMR2Nd3paJGL0FvqFDdNpgs4Xc7kLlLZSa/dmABzk4KJT9xoduKDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pWRlgC0WG3OPI0HrFEOZKsI752eHZBpRWP+cxmiblQc=; b=HRZrtDG9NtJ0chBcqtzN26SIse1F1zi8nYxjPtzCxg5jMRIMv6oQXDRL0M/EtYjlEpSTyb2/bbKXcsUDFVZRKIxw2OedhsK1pyUCha8mZ/UZKIL4ndAwkPNFfRK7vys0BPqiwLXvXRe7pfY3TvJtXLbG6dt6iYfHqQiTOjoASquKT9hSiQf0smeP3DlovkOww8yMoLk0UJ7XJAERXIzhOCKkb4CbGrQbVy13Zp5Usy/C7nEEToN2afHttfYv5qhmJuJbzf7/WPG523vEgJFX+kZjse3HmsbZrxXFUGAKiQw/zMLXgLPA5T6G6p2BeHyimXV2dBTQ/26QyZmHdxmKqA== Received: from DU7P195CA0005.EURP195.PROD.OUTLOOK.COM (2603:10a6:10:54d::24) by PAWPR07MB9853.eurprd07.prod.outlook.com (2603:10a6:102:38d::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.39; Sun, 5 May 2024 08:47:59 +0000 Received: from DB3PEPF0000885C.eurprd02.prod.outlook.com (2603:10a6:10:54d:cafe::ac) by DU7P195CA0005.outlook.office365.com (2603:10a6:10:54d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.39 via Frontend Transport; Sun, 5 May 2024 08:47:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DB3PEPF0000885C.mail.protection.outlook.com (10.167.242.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.18 via Frontend Transport; Sun, 5 May 2024 08:47:58 +0000 Received: from seliicinfr00049.seli.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.67) with Microsoft SMTP Server id 15.2.1544.9; Sun, 5 May 2024 10:47:58 +0200 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00049.seli.gic.ericsson.se (Postfix) with ESMTP id 94B9138007A; Sun, 5 May 2024 10:47:58 +0200 (CEST) From: =?utf-8?q?Mattias_R=C3=B6nnblom?= To: CC: , Heng Wang , "Stephen Hemminger" , Tyler Retzlaff , =?utf-8?q?Morten_Br=C3=B8rup?= , =?utf-8?q?Mattia?= =?utf-8?q?s_R=C3=B6nnblom?= Subject: [RFC v7 5/6] eal: add atomic bit operations Date: Sun, 5 May 2024 10:37:36 +0200 Message-ID: <20240505083737.118649-6-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240505083737.118649-1-mattias.ronnblom@ericsson.com> References: <20240502055706.112443-2-mattias.ronnblom@ericsson.com> <20240505083737.118649-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB3PEPF0000885C:EE_|PAWPR07MB9853:EE_ X-MS-Office365-Filtering-Correlation-Id: 779eae21-991b-4122-f90a-08dc6ce010d5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230031|82310400017|36860700004|1800799015|376005; X-Microsoft-Antispam-Message-Info: =?utf-8?q?vNn7XEQhFBqKy40SLNjKDx9HsXWTfRe?= =?utf-8?q?WiGgWdqPKhOyP6O3wPaVYyfC/ZfzH5E4OKX9bIh2P1YCX0DekJZPL9XlM3X0W0tiF?= =?utf-8?q?ep/C79wMmI9Jvhe+vRpNIyu+KXfF3BAqsoa5JAAm+O1GR1ydRwtqm1KHzoFZcrrxJ?= =?utf-8?q?U0ilQ8CwaKgN8sKJecVaOCQ7jpgwcm2inhr8q+B8kWDxChKTub5Lvjm/wABjXZ9uj?= =?utf-8?q?5yiIdRX5TVW0xsH84S/19nLBgxmFf99X5xRH2JZcNzAxLJNKUQaMbI7xN3t6fd9un?= =?utf-8?q?m/wT+YPcB3OmvEGqia9DwABe2eMggGNijkNbzmtFXnFxO9691hglxbDVgm0KsIQiI?= =?utf-8?q?MrQajnhPbjd1FbJZIIzdr4Fx4A0w0xT+HqFW8SylNRJaY6SEGppw5/BCyJXnRyQDT?= =?utf-8?q?4harXRp3yDBowhsHpGzpHXpHtj7nudoa7L9EWmo6bXt9Ue2ds4UHUrh2HLGh2zT7c?= =?utf-8?q?y0QNpY13yS8OKaY1KIq84O0b36MiHJd5zATQ3SNkzuuBkGDGDKdIFG6zsAsBw2YdF?= =?utf-8?q?57MTwFGyID5X0pG2yEiFe9DAThwMMgvsqGRtAcLtmscXBspNh6WH+DfLlmvpZv4t5?= =?utf-8?q?jd1VbKD2iXBLYmMZqCUS84wklVgyTq/HbjwgjuJ7eT90GfsMRS4Gb9C2LA80HPYQw?= =?utf-8?q?vX5Nfx7A9bXlvP8alabi6UvfXvOl98uhWl9up4fwvzMX6iS7DQ/IrNdW9tNfMaKMJ?= =?utf-8?q?+2UpFKz3O2l42UGE8R2f3JtchcMh1d2CxK7jK/IGbpB3tT/zTVcZiBGI3B6ZOfZUF?= =?utf-8?q?Pfxf7OAzHEu5X+ma196cOZeOr84oo3Qk4s+OhVrkoURepO+K5wELIVT2/WywY+h1p?= =?utf-8?q?KWnej8K8Fc25lisH5W3Qv0YOvUj913L1qV/0u7mHbYTo2Vva+4OohdjoFIDndH1Pg?= =?utf-8?q?ZrrhkNu+Ct/tuFVXwphUsVboIWU5AFefAegsffuqn4qCeTGuWA15/ll9DCs8Uq4Aw?= =?utf-8?q?hXtRyEt/xLo6tDt8j5htmp+p3YA1Gd38g0ZaQVDMD2itP/svEVW3cnuxCL/PT1are?= =?utf-8?q?MVBPXmuBD9FE0zW+HPeScyzxKao7+zujAF505Jkas8FYLBTN7dnWvzAlKT5HLKw5y?= =?utf-8?q?Sewp7x9v+t87vyDnQ9wB47RslQW8H/45s3xR1poC9cv8pTVeiGvtMZjyLXAuX2YNN?= =?utf-8?q?tOtCG3IQBXOVYHFX0JmM5uhy5PsxqFALidtCn1K1ax2bQD8Q065uzCYwZaPkfdMu+?= =?utf-8?q?Fxg+h7Gaw1Zcp3E/n93TkRq6GJSQTkWXCf/+6v3xdJf+wPOwpNFxXBXFDrC4gZPQs?= =?utf-8?q?NOxoQRQbtkn47StdzUd7DjIOhGZ0ZzRyKzWjckQUbdEMCWQbCxwX47aXmQmNotiUR?= =?utf-8?q?sPWyu1dFxiOR?= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230031)(82310400017)(36860700004)(1800799015)(376005); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2024 08:47:58.9762 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 779eae21-991b-4122-f90a-08dc6ce010d5 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DB3PEPF0000885C.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR07MB9853 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add atomic bit test/set/clear/assign/flip and test-and-set/clear/assign/flip functions. All atomic bit functions allow (and indeed, require) the caller to specify a memory order. RFC v7: * Replace compare-exchange-based rte_bitset_atomic_test_and_*() and flip() with implementations that use the previous value as returned by the atomic fetch function. * Reword documentation to match the non-atomic macro variants. * Remove pointer to for memory model documentation, since there is no documentation for that API. RFC v6: * Have rte_bit_atomic_test() accept const-marked bitsets. RFC v4: * Add atomic bit flip. * Mark macro-generated private functions experimental. RFC v3: * Work around lack of C++ support for _Generic (Tyler Retzlaff). RFC v2: o Add rte_bit_atomic_test_and_assign() (for consistency). o Fix bugs in rte_bit_atomic_test_and_[set|clear](). o Use to support MSVC. Signed-off-by: Mattias Rönnblom Acked-by: Morten Brørup Acked-by: Tyler Retzlaff --- lib/eal/include/rte_bitops.h | 413 +++++++++++++++++++++++++++++++++++ 1 file changed, 413 insertions(+) diff --git a/lib/eal/include/rte_bitops.h b/lib/eal/include/rte_bitops.h index 3644aa115c..673b888c1a 100644 --- a/lib/eal/include/rte_bitops.h +++ b/lib/eal/include/rte_bitops.h @@ -21,6 +21,7 @@ #include #include +#include #ifdef __cplusplus extern "C" { @@ -403,6 +404,204 @@ extern "C" { uint32_t *: __rte_bit_once_flip32, \ uint64_t *: __rte_bit_once_flip64)(addr, nr) +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Test if a particular bit in a word is set with a particular memory + * order. + * + * Test a bit with the resulting memory load ordered as per the + * specified memory order. + * + * @param addr + * A pointer to the word to query. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit is set, and false otherwise. + */ +#define rte_bit_atomic_test(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test32, \ + const uint32_t *: __rte_bit_atomic_test32, \ + uint64_t *: __rte_bit_atomic_test64, \ + const uint64_t *: __rte_bit_atomic_test64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically set bit in word. + * + * Generic selection macro to atomically set bit specified by @c nr in + * the word pointed to by @c addr to '1', with the memory ordering as + * specified by @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_set(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_set32, \ + uint64_t *: __rte_bit_atomic_set64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically clear bit in word. + * + * Generic selection macro to atomically set bit specified by @c nr in + * the word pointed to by @c addr to '0', with the memory ordering as + * specified by @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_clear(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_clear32, \ + uint64_t *: __rte_bit_atomic_clear64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically assign a value to bit in word. + * + * Generic selection macro to atomically set bit specified by @c nr in the + * word pointed to by @c addr to the value indicated by @c value, with + * the memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_assign(addr, nr, value, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_assign32, \ + uint64_t *: __rte_bit_atomic_assign64)(addr, nr, value, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically flip bit in word. + * + * Generic selection macro to atomically negate the value of the bit + * specified by @c nr in the word pointed to by @c addr to the value + * indicated by @c value, with the memory ordering as specified with + * @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + */ +#define rte_bit_atomic_flip(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_flip32, \ + uint64_t *: __rte_bit_atomic_flip64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and set a bit in word. + * + * Generic selection macro to atomically test and set bit specified by + * @c nr in the word pointed to by @c addr to '1', with the memory + * ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_set(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_set32, \ + uint64_t *: __rte_bit_atomic_test_and_set64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and clear a bit in word. + * + * Generic selection macro to atomically test and clear bit specified + * by @c nr in the word pointed to by @c addr to '0', with the memory + * ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_clear(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_clear32, \ + uint64_t *: __rte_bit_atomic_test_and_clear64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and assign a bit in word. + * + * Generic selection macro to atomically test and assign bit specified + * by @c nr in the word pointed to by @c addr the value specified by + * @c value, with the memory ordering as specified with @c + * memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_assign(addr, nr, value, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_assign32, \ + uint64_t *: __rte_bit_atomic_test_and_assign64)(addr, nr, \ + value, \ + memory_order) + #define __RTE_GEN_BIT_TEST(family, fun, qualifier, size) \ __rte_experimental \ static inline bool \ @@ -487,6 +686,145 @@ __RTE_GEN_BIT_CLEAR(once_, clear, volatile, 64) __RTE_GEN_BIT_ASSIGN(once_, assign, volatile, 64) __RTE_GEN_BIT_FLIP(once_, flip, volatile, 64) +#define __RTE_GEN_BIT_ATOMIC_TEST(size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_test ## size(const uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + const RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (const RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + return rte_atomic_load_explicit(a_addr, memory_order) & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_SET(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_set ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_or_explicit(a_addr, mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_CLEAR(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_clear ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_and_explicit(a_addr, ~mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_FLIP(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_flip ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_xor_explicit(a_addr, mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_ASSIGN(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_assign ## size(uint ## size ## _t *addr, \ + unsigned int nr, bool value, \ + int memory_order) \ + { \ + if (value) \ + __rte_bit_atomic_set ## size(addr, nr, memory_order); \ + else \ + __rte_bit_atomic_clear ## size(addr, nr, \ + memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_SET(size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_test_and_set ## size(uint ## size ## _t *addr, \ + unsigned int nr, \ + int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + uint ## size ## _t prev; \ + \ + prev = rte_atomic_fetch_or_explicit(a_addr, mask, \ + memory_order); \ + \ + return prev & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_CLEAR(size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_test_and_clear ## size(uint ## size ## _t *addr, \ + unsigned int nr, \ + int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + uint ## size ## _t prev; \ + \ + prev = rte_atomic_fetch_and_explicit(a_addr, ~mask, \ + memory_order); \ + \ + return prev & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_test_and_assign ## size(uint ## size ## _t *addr, \ + unsigned int nr, \ + bool value, \ + int memory_order) \ + { \ + if (value) \ + return __rte_bit_atomic_test_and_set ## size(addr, nr, \ + memory_order); \ + else \ + return __rte_bit_atomic_test_and_clear ## size(addr, nr, \ + memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_OPS(size) \ + __RTE_GEN_BIT_ATOMIC_TEST(size) \ + __RTE_GEN_BIT_ATOMIC_SET(size) \ + __RTE_GEN_BIT_ATOMIC_CLEAR(size) \ + __RTE_GEN_BIT_ATOMIC_ASSIGN(size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_SET(size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_CLEAR(size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size) \ + __RTE_GEN_BIT_ATOMIC_FLIP(size) + +__RTE_GEN_BIT_ATOMIC_OPS(32) +__RTE_GEN_BIT_ATOMIC_OPS(64) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** @@ -1188,6 +1526,14 @@ rte_log2_u64(uint64_t v) #undef rte_bit_once_assign #undef rte_bit_once_flip +#undef rte_bit_atomic_test +#undef rte_bit_atomic_set +#undef rte_bit_atomic_clear +#undef rte_bit_atomic_assign +#undef rte_bit_atomic_test_and_set +#undef rte_bit_atomic_test_and_clear +#undef rte_bit_atomic_test_and_assign + #define __RTE_BIT_OVERLOAD_SZ_2(fun, qualifier, size, arg1_type, arg1_name) \ static inline void \ rte_bit_ ## fun(qualifier uint ## size ## _t *addr, \ @@ -1231,6 +1577,59 @@ rte_log2_u64(uint64_t v) __RTE_BIT_OVERLOAD_SZ_3(fun, qualifier, 64, arg1_type, arg1_name, \ arg2_type, arg2_name) +#define __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, size, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) \ + static inline ret_type \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name) \ + { \ + return __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name); \ + } + +#define __RTE_BIT_OVERLOAD_3R(fun, qualifier, ret_type, arg1_type, arg1_name, \ + arg2_type, arg2_name) \ + __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, 32, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) \ + __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, 64, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) + +#define __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, size, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + static inline void \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name, arg3_type arg3_name) \ + { \ + __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name, \ + arg3_name); \ + } + +#define __RTE_BIT_OVERLOAD_4(fun, qualifier, arg1_type, arg1_name, arg2_type, \ + arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, 32, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, 64, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) + +#define __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, size, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) \ + static inline ret_type \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name, arg3_type arg3_name) \ + { \ + return __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name, \ + arg3_name); \ + } + +#define __RTE_BIT_OVERLOAD_4R(fun, qualifier, ret_type, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, 32, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, 64, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) + __RTE_BIT_OVERLOAD_2R(test, const, bool, unsigned int, nr) __RTE_BIT_OVERLOAD_2(set,, unsigned int, nr) __RTE_BIT_OVERLOAD_2(clear,, unsigned int, nr) @@ -1243,6 +1642,20 @@ __RTE_BIT_OVERLOAD_2(once_clear, volatile, unsigned int, nr) __RTE_BIT_OVERLOAD_3(once_assign, volatile, unsigned int, nr, bool, value) __RTE_BIT_OVERLOAD_2(once_flip, volatile, unsigned int, nr) +__RTE_BIT_OVERLOAD_3R(atomic_test, const, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_set,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_clear,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_4(atomic_assign,, unsigned int, nr, bool, value, + int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_flip,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_3R(atomic_test_and_set,, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_3R(atomic_test_and_clear,, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_4R(atomic_test_and_assign,, bool, unsigned int, nr, + bool, value, int, memory_order) + #endif #endif /* _RTE_BITOPS_H_ */