From patchwork Thu Oct 10 08:30:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 145576 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9F80145B00; Thu, 10 Oct 2024 10:30:42 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7D3864060B; Thu, 10 Oct 2024 10:30:42 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id C7E17400D6 for ; Thu, 10 Oct 2024 10:30:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728549040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HuLxUkZzdkgBgJp6p2gadBeuE1BiWpQxGfL2mdlM9IU=; b=EfLhSYPjfMlmN0wYm75B18RBZbp0R4UnW+YxA+6Pot0s7p8n3YkkAbahgI9YuXepyvMEeQ t4NBh8WLmvk/0LGlmSzRJBxkmFxH+hC/MIfMvaVY+W2abiJEb+YIaYTt41B3EDr4kWdcIs JugEr3T7/h3gC1ka8JlCGe81+1/Wf+o= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-393-5s1mwNgDP6yaxGiKxqKfgQ-1; Thu, 10 Oct 2024 04:30:38 -0400 X-MC-Unique: 5s1mwNgDP6yaxGiKxqKfgQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B99631977023; Thu, 10 Oct 2024 08:30:36 +0000 (UTC) Received: from dmarchan.redhat.com (unknown [10.45.224.68]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A76BB19560AE; Thu, 10 Oct 2024 08:30:33 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: mattias.ronnblom@ericsson.com, mb@smartsharesystems.com, stephen@networkplumber.org, harry.van.haaren@intel.com, roretzla@linux.microsoft.com, Thomas Monjalon Subject: [PATCH v2 1/4] eal: add bitset type Date: Thu, 10 Oct 2024 10:30:19 +0200 Message-ID: <20241010083022.3437380-2-david.marchand@redhat.com> In-Reply-To: <20241010083022.3437380-1-david.marchand@redhat.com> References: <20240809201440.590464-1-mattias.ronnblom@ericsson.com> <20241010083022.3437380-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Mattias Rönnblom Introduce a set of functions and macros that operate on sets of bits, kept in arrays of 64-bit words. RTE bitset is designed for bitsets which are larger than what fits in a single machine word (i.e., 64 bits). For very large bitsets, the API may be a more appropriate choice. Signed-off-by: Mattias Rönnblom Acked-by: Morten Brørup Acked-by: Tyler Retzlaff --- MAINTAINERS | 6 + app/test/meson.build | 1 + app/test/test_bitset.c | 861 +++++++++++++++++++++ doc/api/doxy-api-index.md | 1 + doc/guides/rel_notes/release_24_11.rst | 9 + lib/eal/common/meson.build | 1 + lib/eal/common/rte_bitset.c | 30 + lib/eal/include/meson.build | 1 + lib/eal/include/rte_bitset.h | 998 +++++++++++++++++++++++++ lib/eal/version.map | 3 + 10 files changed, 1911 insertions(+) create mode 100644 app/test/test_bitset.c create mode 100644 lib/eal/common/rte_bitset.c create mode 100644 lib/eal/include/rte_bitset.h diff --git a/MAINTAINERS b/MAINTAINERS index 812463fe9f..00da3dac29 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -255,6 +255,12 @@ M: Jack Bond-Preston F: lib/eal/include/rte_bitops.h F: app/test/test_bitops.c +Bitset +M: Mattias Rönnblom +F: lib/eal/include/rte_bitset.h +F: lib/eal/common/rte_bitset.c +F: app/test/test_bitset.c + Bitmap M: Cristian Dumitrescu F: lib/eal/include/rte_bitmap.h diff --git a/app/test/meson.build b/app/test/meson.build index e29258e6ec..fe248b786c 100644 --- a/app/test/meson.build +++ b/app/test/meson.build @@ -33,6 +33,7 @@ source_file_deps = { 'test_bitcount.c': [], 'test_bitmap.c': [], 'test_bitops.c': [], + 'test_bitset.c': [], 'test_bitratestats.c': ['metrics', 'bitratestats', 'ethdev'] + sample_packet_forward_deps, 'test_bpf.c': ['bpf', 'net'], 'test_byteorder.c': [], diff --git a/app/test/test_bitset.c b/app/test/test_bitset.c new file mode 100644 index 0000000000..fd3e50f396 --- /dev/null +++ b/app/test/test_bitset.c @@ -0,0 +1,861 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2023 Ericsson AB + */ + +#include +#include + +#include +#include + +#include "test.h" + +#define MAGIC UINT64_C(0xdeadbeefdeadbeef) + +static void +rand_buf(void *buf, size_t n) +{ + size_t i; + + for (i = 0; i < n; i++) + ((unsigned char *)buf)[i] = rte_rand(); +} + +static uint64_t * +alloc_bitset(size_t size) +{ + uint64_t *p; + + p = malloc(RTE_BITSET_SIZE(size) + 2 * sizeof(uint64_t)); + if (p == NULL) + rte_panic("Unable to allocate memory\n"); + + rand_buf(&p[0], RTE_BITSET_SIZE(size)); + + p[0] = MAGIC; + p[RTE_BITSET_NUM_WORDS(size) + 1] = MAGIC; + + return p + 1; +} + + +static int +free_bitset(uint64_t *bitset, size_t size) +{ + uint64_t *p; + + p = bitset - 1; + + if (p[0] != MAGIC) + return TEST_FAILED; + + if (p[RTE_BITSET_NUM_WORDS(size) + 1] != MAGIC) + return TEST_FAILED; + + free(p); + + return TEST_SUCCESS; +} + +static bool +rand_bool(void) +{ + return rte_rand_max(2); +} + +static void +rand_bool_ary(bool *ary, size_t len) +{ + size_t i; + + for (i = 0; i < len; i++) + ary[i] = rand_bool(); +} + +static void +rand_unused_bits(uint64_t *bitset, size_t size) +{ + uint64_t bits = rte_rand() & ~__RTE_BITSET_USED_MASK(size); + + bitset[RTE_BITSET_NUM_WORDS(size) - 1] |= bits; +} + +static void +rand_bitset(uint64_t *bitset, size_t size) +{ + size_t i; + + rte_bitset_init(bitset, size); + + for (i = 0; i < size; i++) + rte_bitset_assign(bitset, i, rand_bool()); + + rand_unused_bits(bitset, size); +} + +typedef bool test_fun(const uint64_t *bitset, size_t bit_num); +typedef void set_fun(uint64_t *bitset, size_t bit_num); +typedef void clear_fun(uint64_t *bitset, size_t bit_num); +typedef void assign_fun(uint64_t *bitset, size_t bit_num, bool value); +typedef void flip_fun(uint64_t *bitset, size_t bit_num); + +static int +test_set_clear_size(test_fun test_fun, set_fun set_fun, clear_fun clear_fun, size_t size) +{ + size_t i; + bool reference[size]; + uint64_t *bitset; + + rand_bool_ary(reference, size); + + bitset = alloc_bitset(size); + + TEST_ASSERT(bitset != NULL, "Failed to allocate memory"); + + rte_bitset_init(bitset, size); + + for (i = 0; i < size; i++) { + if (reference[i]) + set_fun(bitset, i); + else + clear_fun(bitset, i); + } + + for (i = 0; i < size; i++) + if (reference[i] != test_fun(bitset, i)) + return TEST_FAILED; + + TEST_ASSERT_EQUAL(free_bitset(bitset, size), TEST_SUCCESS, + "Buffer over- or underrun detected"); + + return TEST_SUCCESS; +} + +#define RAND_ITERATIONS (10000) +#define RAND_SET_MAX_SIZE (1000) + +static int +test_set_clear_fun(test_fun test_fun, set_fun set_fun, clear_fun clear_fun) +{ + size_t i; + + for (i = 0; i < RAND_ITERATIONS; i++) { + size_t size = 1 + rte_rand_max(RAND_SET_MAX_SIZE - 1); + + if (test_set_clear_size(test_fun, set_fun, clear_fun, size) != TEST_SUCCESS) + return TEST_FAILED; + } + + return TEST_SUCCESS; +} + +static int +test_set_clear(void) +{ + return test_set_clear_fun(rte_bitset_test, rte_bitset_set, rte_bitset_clear); +} + +static int +test_flip_size(test_fun test_fun, assign_fun assign_fun, flip_fun flip_fun, size_t size) +{ + size_t i; + uint64_t *bitset; + + bitset = alloc_bitset(size); + + TEST_ASSERT(bitset != NULL, "Failed to allocate memory"); + + rand_bitset(bitset, size); + + for (i = 0; i < size; i++) { + RTE_BITSET_DECLARE(reference, size); + + rte_bitset_copy(reference, bitset, size); + + bool value = test_fun(bitset, i); + + flip_fun(bitset, i); + + TEST_ASSERT(test_fun(bitset, i) != value, "Bit %zd was not flipped", i); + + assign_fun(reference, i, !value); + + TEST_ASSERT(rte_bitset_equal(bitset, reference, size), + "Not only the target bit %zd was flipped", i); + + + } + + TEST_ASSERT_EQUAL(free_bitset(bitset, size), TEST_SUCCESS, + "Buffer over- or underrun detected"); + + return TEST_SUCCESS; +} + +static int +test_flip_fun(test_fun test_fun, assign_fun assign_fun, flip_fun flip_fun) +{ + size_t i; + + for (i = 0; i < RAND_ITERATIONS; i++) { + size_t size = 1 + rte_rand_max(RAND_SET_MAX_SIZE - 1); + + if (test_flip_size(test_fun, assign_fun, flip_fun, size) != TEST_SUCCESS) + return TEST_FAILED; + } + + return TEST_SUCCESS; +} + +static int +test_flip(void) +{ + return test_flip_fun(rte_bitset_test, rte_bitset_assign, rte_bitset_flip); +} + +static ssize_t +find(const bool *ary, size_t num_bools, size_t start, size_t len, bool set) +{ + size_t i; + + for (i = 0; i < len; i++) { + ssize_t idx = (start + i) % num_bools; + + if (ary[idx] == set) + return idx; + } + + return -1; +} + +static ssize_t +find_set(const bool *ary, size_t num_bools, size_t start, size_t len) +{ + return find(ary, num_bools, start, len, true); +} + +static ssize_t +find_clear(const bool *ary, size_t num_bools, size_t start, size_t len) +{ + return find(ary, num_bools, start, len, false); +} + +#define FFS_ITERATIONS (100) + +static int +test_find_size(size_t size, bool set) +{ + uint64_t *bitset; + bool reference[size]; + size_t i; + + bitset = alloc_bitset(size); + + TEST_ASSERT(bitset != NULL, "Failed to allocate memory"); + + rte_bitset_init(bitset, size); + + for (i = 0; i < size; i++) { + bool bit = rand_bool(); + reference[i] = bit; + + if (bit) + rte_bitset_set(bitset, i); + else /* redundant, still useful for testing */ + rte_bitset_clear(bitset, i); + } + + for (i = 0; i < FFS_ITERATIONS; i++) { + size_t start_bit = rte_rand_max(size); + size_t len = rte_rand_max(size + 1); + bool full_range = len == size && start_bit == 0; + bool wraps = start_bit + len > size; + ssize_t rc; + + if (set) { + if (full_range && rand_bool()) + rc = rte_bitset_find_first_set(bitset, size); + else if (wraps || rand_bool()) + rc = rte_bitset_find_set_wrap(bitset, size, start_bit, len); + else + rc = rte_bitset_find_set(bitset, size, start_bit, len); + + if (rc != find_set(reference, size, start_bit, len)) + return TEST_FAILED; + } else { + if (full_range && rand_bool()) + rc = rte_bitset_find_first_clear(bitset, size); + else if (wraps || rand_bool()) + rc = rte_bitset_find_clear_wrap(bitset, size, start_bit, len); + else + rc = rte_bitset_find_clear(bitset, size, start_bit, len); + + if (rc != find_clear(reference, size, start_bit, len)) + return TEST_FAILED; + } + + } + + TEST_ASSERT_EQUAL(free_bitset(bitset, size), TEST_SUCCESS, + "Buffer over- or underrun detected"); + + return TEST_SUCCESS; +} + +static int +test_find_set_size(size_t size) +{ + return test_find_size(size, true); +} + +static int +test_find_clear_size(size_t size) +{ + return test_find_size(size, false); +} + +static int +test_find(void) +{ + size_t i; + + for (i = 0; i < RAND_ITERATIONS; i++) { + size_t size = 2 + rte_rand_max(RAND_SET_MAX_SIZE - 2); + + if (test_find_set_size(size) != TEST_SUCCESS) + return TEST_FAILED; + + if (test_find_clear_size(size) != TEST_SUCCESS) + return TEST_FAILED; + } + + return TEST_SUCCESS; +} + +static int +record_match(ssize_t match_idx, size_t size, int *calls) +{ + if (match_idx < 0 || (size_t)match_idx >= size) + return TEST_FAILED; + + calls[match_idx]++; + + return TEST_SUCCESS; +} + +static int +test_foreach_size(ssize_t size, bool may_wrap, bool set) +{ + bool reference[size]; + int calls[size]; + uint64_t *bitset; + ssize_t i; + ssize_t start_bit; + ssize_t len; + bool full_range; + size_t total_calls = 0; + + rand_bool_ary(reference, size); + + bitset = alloc_bitset(size); + + TEST_ASSERT(bitset != NULL, "Failed to allocate memory"); + + memset(calls, 0, sizeof(calls)); + + start_bit = rte_rand_max(size); + len = may_wrap ? rte_rand_max(size + 1) : + rte_rand_max(size - start_bit + 1); + + rte_bitset_init(bitset, size); + + /* random data in the unused bits should not matter */ + rand_buf(bitset, RTE_BITSET_SIZE(size)); + + for (i = start_bit; i < start_bit + len; i++) { + size_t idx = i % size; + + if (reference[idx]) + rte_bitset_set(bitset, idx); + else + rte_bitset_clear(bitset, idx); + + if (rte_bitset_test(bitset, idx) != reference[idx]) + return TEST_FAILED; + } + + full_range = (len == size && start_bit == 0); + + /* XXX: verify iteration order as well */ + if (set) { + if (full_range && rand_bool()) { + RTE_BITSET_FOREACH_SET(i, bitset, size) { + if (record_match(i, size, calls) != TEST_SUCCESS) + return TEST_FAILED; + } + } else if (may_wrap) { + RTE_BITSET_FOREACH_SET_WRAP(i, bitset, size, start_bit, len) { + if (record_match(i, size, calls) != TEST_SUCCESS) { + printf("failed\n"); + return TEST_FAILED; + } + } + } else { + RTE_BITSET_FOREACH_SET_RANGE(i, bitset, size, start_bit, len) { + if (record_match(i, size, calls) != TEST_SUCCESS) + return TEST_FAILED; + } + } + } else { + if (full_range && rand_bool()) { + RTE_BITSET_FOREACH_CLEAR(i, bitset, size) + if (record_match(i, size, calls) != TEST_SUCCESS) + return TEST_FAILED; + } else if (may_wrap) { + RTE_BITSET_FOREACH_CLEAR_WRAP(i, bitset, size, start_bit, len) { + if (record_match(i, size, calls) != TEST_SUCCESS) + return TEST_FAILED; + } + } else { + RTE_BITSET_FOREACH_CLEAR_RANGE(i, bitset, size, start_bit, len) + if (record_match(i, size, calls) != TEST_SUCCESS) + return TEST_FAILED; + } + } + + for (i = 0; i < len; i++) { + size_t idx = (start_bit + i) % size; + + if (reference[idx] == set && calls[idx] != 1) { + printf("bit %zd shouldn't have been found %d times\n", idx, calls[idx]); + return TEST_FAILED; + } + + if (reference[idx] != set && calls[idx] != 0) { + puts("bar"); + return TEST_FAILED; + } + + total_calls += calls[idx]; + } + + if (full_range) { + size_t count; + + count = set ? rte_bitset_count_set(bitset, size) : + rte_bitset_count_clear(bitset, size); + + if (count != total_calls) + return TEST_FAILED; + } + + TEST_ASSERT_EQUAL(free_bitset(bitset, size), TEST_SUCCESS, + "Buffer over- or underrun detected"); + + return TEST_SUCCESS; +} + +static int +test_foreach(void) +{ + size_t i; + + for (i = 0; i < RAND_ITERATIONS; i++) { + size_t size = 1 + rte_rand_max(RAND_SET_MAX_SIZE - 1); + + if (test_foreach_size(size, false, true) != TEST_SUCCESS) + return TEST_FAILED; + + if (test_foreach_size(size, false, false) != TEST_SUCCESS) + return TEST_FAILED; + + if (test_foreach_size(size, true, true) != TEST_SUCCESS) + return TEST_FAILED; + + if (test_foreach_size(size, true, false) != TEST_SUCCESS) + return TEST_FAILED; + } + + return TEST_SUCCESS; +} + +static int +test_count_size(size_t size) +{ + uint64_t *bitset; + + bitset = alloc_bitset(size); + + TEST_ASSERT(bitset != NULL, "Failed to allocate memory"); + + rte_bitset_init(bitset, size); + + rand_unused_bits(bitset, size); + + if (rte_bitset_count_set(bitset, size) != 0) + return TEST_FAILED; + + if (rte_bitset_count_clear(bitset, size) != size) + return TEST_FAILED; + + rte_bitset_set_all(bitset, size); + + if (rte_bitset_count_set(bitset, size) != size) + return TEST_FAILED; + + if (rte_bitset_count_clear(bitset, size) != 0) + return TEST_FAILED; + + rte_bitset_clear_all(bitset, size); + + if (rte_bitset_count_set(bitset, size) != 0) + return TEST_FAILED; + + if (rte_bitset_count_clear(bitset, size) != size) + return TEST_FAILED; + + rte_bitset_set(bitset, rte_rand_max(size)); + + if (rte_bitset_count_set(bitset, size) != 1) + return TEST_FAILED; + + if (rte_bitset_count_clear(bitset, size) != (size - 1)) + return TEST_FAILED; + + rte_bitset_clear_all(bitset, size); + if (rte_bitset_count_set(bitset, size) != 0) + return TEST_FAILED; + if (rte_bitset_count_clear(bitset, size) != size) + return TEST_FAILED; + + rte_bitset_set_all(bitset, size); + if (rte_bitset_count_set(bitset, size) != size) + return TEST_FAILED; + if (rte_bitset_count_clear(bitset, size) != 0) + return TEST_FAILED; + + TEST_ASSERT_EQUAL(free_bitset(bitset, size), TEST_SUCCESS, + "Buffer over- or underrun detected"); + + return TEST_SUCCESS; +} + +static int +test_count(void) +{ + size_t i; + + if (test_count_size(128) != TEST_SUCCESS) + return TEST_FAILED; + if (test_count_size(1) != TEST_SUCCESS) + return TEST_FAILED; + if (test_count_size(63) != TEST_SUCCESS) + return TEST_FAILED; + if (test_count_size(64) != TEST_SUCCESS) + return TEST_FAILED; + if (test_count_size(65) != TEST_SUCCESS) + return TEST_FAILED; + + for (i = 0; i < RAND_ITERATIONS; i++) { + size_t size = 1 + rte_rand_max(RAND_SET_MAX_SIZE - 1); + + if (test_count_size(size) != TEST_SUCCESS) + return TEST_FAILED; + } + + return TEST_SUCCESS; +} + +#define GEN_DECLARE(size) \ +{ \ + RTE_BITSET_DECLARE(bitset, size); \ + size_t idx = rte_rand_max(size); \ + rte_bitset_init(bitset, size); \ + rte_bitset_set(bitset, idx); \ + if (!rte_bitset_test(bitset, idx)) \ + return TEST_FAILED; \ + if (rte_bitset_count_set(bitset, size) != 1) \ + return TEST_FAILED; \ + return TEST_SUCCESS; \ +} + +static int +test_define(void) +{ + GEN_DECLARE(1); + GEN_DECLARE(64); + GEN_DECLARE(65); + GEN_DECLARE(4097); +} + +typedef void bitset_op(uint64_t *dst, const uint64_t *a, const uint64_t *b, size_t bit_num); +typedef bool bool_op(bool a, bool b); + +static int +test_logic_op(bitset_op bitset_op, bool_op bool_op) +{ + const size_t size = 1 + rte_rand_max(200); + RTE_BITSET_DECLARE(bitset_a, size); + RTE_BITSET_DECLARE(bitset_b, size); + RTE_BITSET_DECLARE(bitset_d, size); + + bool ary_a[size]; + bool ary_b[size]; + bool ary_d[size]; + + rand_bool_ary(ary_a, size); + rand_bool_ary(ary_b, size); + + size_t i; + for (i = 0; i < size; i++) { + rte_bitset_assign(bitset_a, i, ary_a[i]); + rte_bitset_assign(bitset_b, i, ary_b[i]); + ary_d[i] = bool_op(ary_a[i], ary_b[i]); + } + + bitset_op(bitset_d, bitset_a, bitset_b, size); + + for (i = 0; i < size; i++) + TEST_ASSERT_EQUAL(rte_bitset_test(bitset_d, i), ary_d[i], + "Unexpected value of bit %zd", i); + + return TEST_SUCCESS; +} + +static bool +bool_or(bool a, bool b) +{ + return a || b; +} + +static int +test_or(void) +{ + return test_logic_op(rte_bitset_or, bool_or); +} + +static bool +bool_and(bool a, bool b) +{ + return a && b; +} + +static int +test_and(void) +{ + return test_logic_op(rte_bitset_and, bool_and); +} + +static bool +bool_xor(bool a, bool b) +{ + return a != b; +} + +static int +test_xor(void) +{ + return test_logic_op(rte_bitset_xor, bool_xor); +} + +static int +test_complement(void) +{ + int i; + + for (i = 0; i < RAND_ITERATIONS; i++) { + const size_t size = 1 + rte_rand_max(RAND_SET_MAX_SIZE - 1); + + RTE_BITSET_DECLARE(src, size); + + rand_bitset(src, size); + + bool bit_idx = rte_rand_max(size); + bool bit_value = rte_bitset_test(src, bit_idx); + + RTE_BITSET_DECLARE(dst, size); + + rte_bitset_complement(dst, src, size); + + TEST_ASSERT(bit_value != rte_bitset_test(dst, bit_idx), + "Bit %d was not flipped", bit_idx); + } + + return TEST_SUCCESS; +} + +static int +test_shift(bool right) +{ + int i; + + const char *direction = right ? "right" : "left"; + + for (i = 0; i < 10000; i++) { + const int size = 1 + (int)rte_rand_max(500); + const int shift_count = (int)rte_rand_max(1.5 * size); + int src_idx; + + RTE_BITSET_DECLARE(src, size); + RTE_BITSET_DECLARE(reference, size); + + rte_bitset_init(src, size); + rte_bitset_init(reference, size); + + rand_unused_bits(src, size); + rand_unused_bits(reference, size); + + for (src_idx = 0; src_idx < size; src_idx++) { + bool value = rand_bool(); + + rte_bitset_assign(src, src_idx, value); + + int dst_idx = right ? src_idx - shift_count : src_idx + shift_count; + + if (dst_idx >= 0 && dst_idx < size) + rte_bitset_assign(reference, dst_idx, value); + } + + uint64_t *dst = alloc_bitset(size); + + if (right) + rte_bitset_shift_right(dst, src, size, shift_count); + else + rte_bitset_shift_left(dst, src, size, shift_count); + + TEST_ASSERT(rte_bitset_equal(dst, reference, size), + "Unexpected result from shifting bitset of size %d bits %d bits %s", + size, shift_count, direction); + + TEST_ASSERT_EQUAL(free_bitset(dst, size), TEST_SUCCESS, + "Shift %s operation overwrote buffer", direction); + } + + return TEST_SUCCESS; +} + +static int +test_shift_right(void) +{ + return test_shift(true); +} + +static int +test_shift_left(void) +{ + return test_shift(false); +} + +static int +test_equal(void) +{ + const size_t size = 100; + RTE_BITSET_DECLARE(bitset_a, size); + RTE_BITSET_DECLARE(bitset_b, size); + + rand_buf(bitset_a, RTE_BITSET_SIZE(size)); + rand_buf(bitset_b, RTE_BITSET_SIZE(size)); + + rte_bitset_init(bitset_a, size); + rte_bitset_init(bitset_b, size); + + rte_bitset_set(bitset_a, 9); + rte_bitset_set(bitset_b, 9); + rte_bitset_set(bitset_a, 90); + rte_bitset_set(bitset_b, 90); + + if (!rte_bitset_equal(bitset_a, bitset_b, size)) + return TEST_FAILED; + + /* set unused bit, which should be ignored */ + rte_bitset_set(&bitset_a[1], 60); + + if (!rte_bitset_equal(bitset_a, bitset_b, size)) + return TEST_FAILED; + + return TEST_SUCCESS; +} + +static int +test_copy(void) +{ + const size_t size = 100; + RTE_BITSET_DECLARE(bitset_a, size); + RTE_BITSET_DECLARE(bitset_b, size); + + rand_buf(bitset_a, RTE_BITSET_SIZE(size)); + rand_buf(bitset_b, RTE_BITSET_SIZE(size)); + + rte_bitset_copy(bitset_a, bitset_b, size); + + if (!rte_bitset_equal(bitset_a, bitset_b, size)) + return TEST_FAILED; + + return TEST_SUCCESS; +} + +static int +test_to_str(void) +{ + char buf[1024]; + RTE_BITSET_DECLARE(bitset, 128); + + rte_bitset_init(bitset, 128); + rte_bitset_set(bitset, 1); + + if (rte_bitset_to_str(bitset, 2, buf, 3) != 3) + return TEST_FAILED; + if (strcmp(buf, "10") != 0) + return TEST_FAILED; + + rte_bitset_set(bitset, 0); + + if (rte_bitset_to_str(bitset, 1, buf, sizeof(buf)) != 2) + return TEST_FAILED; + if (strcmp(buf, "1") != 0) + return TEST_FAILED; + + rte_bitset_init(bitset, 99); + rte_bitset_set(bitset, 98); + + if (rte_bitset_to_str(bitset, 99, buf, sizeof(buf)) != 100) + return TEST_FAILED; + + if (buf[0] != '1' || strchr(&buf[1], '1') != NULL) + return TEST_FAILED; + + if (rte_bitset_to_str(bitset, 128, buf, 64) != -EINVAL) + return TEST_FAILED; + + return TEST_SUCCESS; +} + +static struct unit_test_suite bitset_tests = { + .suite_name = "bitset test suite", + .unit_test_cases = { + TEST_CASE_ST(NULL, NULL, test_set_clear), + TEST_CASE_ST(NULL, NULL, test_flip), + TEST_CASE_ST(NULL, NULL, test_find), + TEST_CASE_ST(NULL, NULL, test_foreach), + TEST_CASE_ST(NULL, NULL, test_count), + TEST_CASE_ST(NULL, NULL, test_define), + TEST_CASE_ST(NULL, NULL, test_or), + TEST_CASE_ST(NULL, NULL, test_and), + TEST_CASE_ST(NULL, NULL, test_xor), + TEST_CASE_ST(NULL, NULL, test_complement), + TEST_CASE_ST(NULL, NULL, test_shift_right), + TEST_CASE_ST(NULL, NULL, test_shift_left), + TEST_CASE_ST(NULL, NULL, test_equal), + TEST_CASE_ST(NULL, NULL, test_copy), + TEST_CASE_ST(NULL, NULL, test_to_str), + TEST_CASES_END() + } +}; + +static int +test_bitset(void) +{ + return unit_test_suite_runner(&bitset_tests); +} + +REGISTER_FAST_TEST(bitset_autotest, true, true, test_bitset); diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index f9f0300126..abd44b1861 100644 --- a/doc/api/doxy-api-index.md +++ b/doc/api/doxy-api-index.md @@ -174,6 +174,7 @@ The public API headers are grouped by topics: [ring](@ref rte_ring.h), [stack](@ref rte_stack.h), [tailq](@ref rte_tailq.h), + [bitset](@ref rte_bitset.h), [bitmap](@ref rte_bitmap.h) - **packet framework**: diff --git a/doc/guides/rel_notes/release_24_11.rst b/doc/guides/rel_notes/release_24_11.rst index 2f78f2d125..f3fd8517ed 100644 --- a/doc/guides/rel_notes/release_24_11.rst +++ b/doc/guides/rel_notes/release_24_11.rst @@ -70,6 +70,15 @@ New Features The new public API elements are polymorphic, using the _Generic-based macros (for C) and function overloading (in C++ translation units). +* **Added multi-word bitset API.** + + A new multi-word bitset API has been introduced in the EAL. + The RTE bitset is optimized for scenarios where the bitset size exceeds the + capacity of a single word (e.g., larger than 64 bits), but is not large + enough to justify the overhead and complexity of the more scalable, + yet slower, API. This addition provides an efficient and + straightforward alternative for handling bitsets of intermediate sizes. + * **Extended service cores statistics.** Two new per-service counters are added to the service cores framework. diff --git a/lib/eal/common/meson.build b/lib/eal/common/meson.build index 22a626ba6f..c1bbf26654 100644 --- a/lib/eal/common/meson.build +++ b/lib/eal/common/meson.build @@ -31,6 +31,7 @@ sources += files( 'eal_common_uuid.c', 'malloc_elem.c', 'malloc_heap.c', + 'rte_bitset.c', 'rte_malloc.c', 'rte_random.c', 'rte_reciprocal.c', diff --git a/lib/eal/common/rte_bitset.c b/lib/eal/common/rte_bitset.c new file mode 100644 index 0000000000..e25008519e --- /dev/null +++ b/lib/eal/common/rte_bitset.c @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2023 Ericsson AB + */ + +#include +#include +#include +#include + +#include "rte_bitset.h" + +ssize_t +rte_bitset_to_str(const uint64_t *bitset, size_t num_bits, char *buf, size_t capacity) +{ + size_t i; + + if (capacity < (num_bits + 1)) + return -EINVAL; + + for (i = 0; i < num_bits; i++) { + bool value; + + value = rte_bitset_test(bitset, num_bits - 1 - i); + buf[i] = value ? '1' : '0'; + } + + buf[num_bits] = '\0'; + + return num_bits + 1; +} diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build index e94b056d46..474097f211 100644 --- a/lib/eal/include/meson.build +++ b/lib/eal/include/meson.build @@ -7,6 +7,7 @@ headers += files( 'rte_alarm.h', 'rte_bitmap.h', 'rte_bitops.h', + 'rte_bitset.h', 'rte_branch_prediction.h', 'rte_bus.h', 'rte_class.h', diff --git a/lib/eal/include/rte_bitset.h b/lib/eal/include/rte_bitset.h new file mode 100644 index 0000000000..b2d71aa7c6 --- /dev/null +++ b/lib/eal/include/rte_bitset.h @@ -0,0 +1,998 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2023 Ericsson AB + */ + +#ifndef _RTE_BITSET_H_ +#define _RTE_BITSET_H_ + +/** + * @file + * RTE Bitset + * + * This file provides functions and macros for querying and + * manipulating sets of bits kept in arrays of @c uint64_t-sized + * elements. + * + * The bits in a bitset are numbered from 0 to @c size - 1, with the + * lowest index being the least significant bit. + * + * The bitset array must be properly aligned. + * + * For optimal performance, the @c size parameter, required by + * many of the API's functions, should be a compile-time constant. + * + * For large bitsets, the rte_bitmap.h API may be more appropriate. + * + * @warning + * All functions modifying a bitset may overwrite any unused bits of + * the last word. Such unused bits are ignored by all functions reading + * bits. + * + */ + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * The size (in bytes) of each element in the array used to represent + * a bitset. + */ +#define RTE_BITSET_WORD_SIZE (sizeof(uint64_t)) + +/** + * The size (in bits) of each element in the array used to represent + * a bitset. + */ +#define RTE_BITSET_WORD_BITS (RTE_BITSET_WORD_SIZE * CHAR_BIT) + +/** + * Computes the number of words required to store @c size bits. + */ +#define RTE_BITSET_NUM_WORDS(size) \ + ((size + RTE_BITSET_WORD_BITS - 1) / RTE_BITSET_WORD_BITS) + +/** + * Computes the amount of memory (in bytes) required to fit a bitset + * holding @c size bits. + */ +#define RTE_BITSET_SIZE(size) \ + ((size_t)(RTE_BITSET_NUM_WORDS(size) * RTE_BITSET_WORD_SIZE)) + +#define __RTE_BITSET_WORD_IDX(bit_num) ((bit_num) / RTE_BITSET_WORD_BITS) +#define __RTE_BITSET_BIT_OFFSET(bit_num) ((bit_num) % RTE_BITSET_WORD_BITS) +#define __RTE_BITSET_UNUSED(size) \ + ((RTE_BITSET_NUM_WORDS(size) * RTE_BITSET_WORD_BITS) - (size)) +#define __RTE_BITSET_USED_MASK(size) \ + (UINT64_MAX >> __RTE_BITSET_UNUSED(size)) + +#define __RTE_BITSET_DELEGATE_N(fun, bitset, bit_num, ...) \ + fun(&(bitset)[__RTE_BITSET_WORD_IDX(bit_num)], __RTE_BITSET_BIT_OFFSET(bit_num), \ + __VA_ARGS__) + +/* MSVC doesn't have ##__VA_ARGS__, so argument-less -> special case */ +#define __RTE_BITSET_DELEGATE(fun, bitset, bit_num) \ + fun(&(bitset)[__RTE_BITSET_WORD_IDX(bit_num)], __RTE_BITSET_BIT_OFFSET(bit_num)) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Declare a bitset. + * + * Declare (e.g., as a struct field) or define (e.g., as a stack + * variable) a bitset of the specified size. + * + * @param size + * The number of bits the bitset must be able to represent. Must be + * a compile-time constant. + * @param name + * The field or variable name of the resulting definition. + */ +#define RTE_BITSET_DECLARE(name, size) \ + uint64_t name[RTE_BITSET_NUM_WORDS(size)] + +#define __RTE_BITSET_FOREACH_LEFT(var, size, start_bit, len) \ + ((len) - 1 - ((var) >= (start_bit) ? (var) - (start_bit) : (size) - (start_bit) + (var))) + +#define __RTE_BITSET_FOREACH(var, bitset, size, start_bit, len, flags) \ + for ((var) = __rte_bitset_find(bitset, size, start_bit, len, flags); \ + (var) != -1; \ + (var) = __RTE_BITSET_FOREACH_LEFT(var, size, start_bit, len) > 0 ? \ + __rte_bitset_find(bitset, size, ((var) + 1) % (size), \ + __RTE_BITSET_FOREACH_LEFT(var, size, start_bit, len), flags) : -1) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Iterate over all bits set. + * + * This macro iterates over all bits set (i.e., all ones) in the + * bitset, in the forward direction (i.e., starting with the least + * significant '1'). + * + * @param var + * An iterator variable of type @c ssize_t. For each successive + * iteration, this variable will hold the bit index of a set bit. + * @param bitset + * A const uint64_t * pointer to the bitset array. + * @param size + * The size of the bitset (in bits). + */ +#define RTE_BITSET_FOREACH_SET(var, bitset, size) \ + __RTE_BITSET_FOREACH(var, bitset, size, 0, size, 0) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Iterate over all bits cleared. + * + * This macro iterates over all bits cleared in the bitset, in the + * forward direction (i.e., starting with the lowest-indexed set bit). + * + * @param var + * An iterator variable of type @c ssize_t. For each successive iteration, + * this variable will hold the bit index of a cleared bit. + * @param bitset + * A const uint64_t * pointer to the bitset array. + * @param size + * The size of the bitset (in bits). + */ +#define RTE_BITSET_FOREACH_CLEAR(var, bitset, size) \ + __RTE_BITSET_FOREACH(var, bitset, size, 0, size, __RTE_BITSET_FIND_FLAG_FIND_CLEAR) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Iterate over all bits set within a range. + * + * This macro iterates over all bits set (i.e., all ones) in the + * specified range, in the forward direction (i.e., starting with the + * least significant '1'). + * + * @param var + * An iterator variable of type @c ssize_t. For each successive iteration, + * this variable will hold the bit index of a set bit. + * @param bitset + * A const uint64_t * pointer to the bitset array. + * @param size + * The size of the bitset (in bits). + * @param start_bit + * The index of the first bit to check. Must be less than @c size. + * @param len + * The length (in bits) of the range. @c start_bit + @c len must be less + * than or equal to @c size. + */ +#define RTE_BITSET_FOREACH_SET_RANGE(var, bitset, size, start_bit, len) \ + __RTE_BITSET_FOREACH(var, bitset, size, start_bit, len, 0) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Iterate over all cleared bits within a range. + * + * This macro iterates over all bits cleared (i.e., all zeroes) in the + * specified range, in the forward direction (i.e., starting with the + * least significant '0'). + * + * @param var + * An iterator variable of type @c ssize_t. For each successive iteration, + * this variable will hold the bit index of a set bit. + * @param bitset + * A const uint64_t * pointer to the bitset array. + * @param size + * The size of the bitset (in bits). + * @param start_bit + * The index of the first bit to check. Must be less than @c size. + * @param len + * The length (in bits) of the range. @c start_bit + @c len must be less + * than or equal to @c size. + */ +#define RTE_BITSET_FOREACH_CLEAR_RANGE(var, bitset, size, start_bit, len) \ + __RTE_BITSET_FOREACH(var, bitset, size, start_bit, len, __RTE_BITSET_FIND_FLAG_FIND_CLEAR) + +#define RTE_BITSET_FOREACH_SET_WRAP(var, bitset, size, start_bit, len) \ + __RTE_BITSET_FOREACH(var, bitset, size, start_bit, len, __RTE_BITSET_FIND_FLAG_WRAP) + +#define RTE_BITSET_FOREACH_CLEAR_WRAP(var, bitset, size, start_bit, len) \ + __RTE_BITSET_FOREACH(var, bitset, size, start_bit, len, \ + __RTE_BITSET_FIND_FLAG_WRAP | __RTE_BITSET_FIND_FLAG_FIND_CLEAR) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Initializes a bitset. + * + * All bits are cleared. + * + * In case all words in the bitset array are already set to zero by + * other means (e.g., at the time of memory allocation), this function + * need not be called. + * + * @param bitset + * A pointer to the array of bitset 64-bit words. + * @param size + * The size of the bitset (in bits). + */ +__rte_experimental +static inline void +rte_bitset_init(uint64_t *bitset, size_t size) +{ + memset(bitset, 0, RTE_BITSET_SIZE(size)); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Test if a bit is set. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * Index of the bit to test. Index 0 is the least significant bit. + * @return + * Returns true if the bit is '1', and false if the bit is '0'. + */ +__rte_experimental +static inline bool +rte_bitset_test(const uint64_t *bitset, size_t bit_num) +{ + return __RTE_BITSET_DELEGATE(rte_bit_test, bitset, bit_num); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Set a bit in the bitset. + * + * Bits are numbered from 0 to (size - 1) (inclusive). + * + * The operation is not guaranteed to be atomic. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be set. + */ +__rte_experimental +static inline void +rte_bitset_set(uint64_t *bitset, size_t bit_num) +{ + __RTE_BITSET_DELEGATE(rte_bit_set, bitset, bit_num); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Clear a bit in the bitset. + * + * Bits are numbered 0 to (size - 1) (inclusive). + * + * The operation is not guaranteed to be atomic. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be cleared. + */ +__rte_experimental +static inline void +rte_bitset_clear(uint64_t *bitset, size_t bit_num) +{ + __RTE_BITSET_DELEGATE(rte_bit_clear, bitset, bit_num); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Set or clear a bit in the bitset. + * + * Bits are numbered 0 to (size - 1) (inclusive). + * + * The operation is not guaranteed to be atomic. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be set or cleared. + * @param bit_value + * Control if the bit should be set or cleared. + */ +__rte_experimental +static inline void +rte_bitset_assign(uint64_t *bitset, size_t bit_num, bool bit_value) +{ + __RTE_BITSET_DELEGATE_N(rte_bit_assign, bitset, bit_num, bit_value); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Change the value of a bit in the bitset. + * + * Bits are numbered 0 to (size - 1) (inclusive). + * + * The operation is not guaranteed to be atomic. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be flipped. + */ +__rte_experimental +static inline void +rte_bitset_flip(uint64_t *bitset, size_t bit_num) +{ + __RTE_BITSET_DELEGATE(rte_bit_flip, bitset, bit_num); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Set all bits in the bitset. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + */ +__rte_experimental +static inline void +rte_bitset_set_all(uint64_t *bitset, size_t size) +{ + memset(bitset, 0xFF, RTE_BITSET_SIZE(size)); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Clear all bits in the bitset. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + */ +__rte_experimental +static inline void +rte_bitset_clear_all(uint64_t *bitset, size_t size) +{ + rte_bitset_init(bitset, size); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Count all set bits (also known as the @e weight). + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @return + * Returns the number of '1' bits in the bitset. + */ +__rte_experimental +static inline size_t +rte_bitset_count_set(const uint64_t *bitset, size_t size) +{ + size_t i; + size_t total = 0; + + /* + * Unused bits in a rte_bitset are always '0', and thus are + * not included in this count. + */ + for (i = 0; i < RTE_BITSET_NUM_WORDS(size) - 1; i++) + total += rte_popcount64(bitset[i]); + + total += rte_popcount64(bitset[i] & __RTE_BITSET_USED_MASK(size)); + + return total; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Count all cleared bits. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @return + * Returns the number of '0' bits in the bitset. + */ +__rte_experimental +static inline size_t +rte_bitset_count_clear(const uint64_t *bitset, size_t size) +{ + return size - rte_bitset_count_set(bitset, size); +} + +#define __RTE_BITSET_FIND_FLAG_FIND_CLEAR (1U << 0) +#define __RTE_BITSET_FIND_FLAG_WRAP (1U << 1) + +__rte_experimental +static inline ssize_t +__rte_bitset_find_nowrap(const uint64_t *bitset, size_t __rte_unused size, size_t start_bit, + size_t len, bool find_clear) +{ + size_t word_idx; + size_t offset; + size_t end_bit = start_bit + len; + + RTE_ASSERT(end_bit <= size); + + if (unlikely(len == 0)) + return -1; + + word_idx = __RTE_BITSET_WORD_IDX(start_bit); + offset = __RTE_BITSET_BIT_OFFSET(start_bit); + + while (word_idx <= __RTE_BITSET_WORD_IDX(end_bit - 1)) { + uint64_t word; + int word_ffs; + + word = bitset[word_idx]; + if (find_clear) + word = ~word; + + word >>= offset; + + word_ffs = __builtin_ffsll(word); + + if (word_ffs != 0) { + ssize_t ffs = start_bit + word_ffs - 1; + + /* + * Check if set bit were among the last, + * unused bits, in the last word. + */ + if (unlikely(ffs >= (ssize_t)end_bit)) + return -1; + + return ffs; + } + + start_bit += (RTE_BITSET_WORD_BITS - offset); + word_idx++; + offset = 0; + } + + return -1; + +} + +__rte_experimental +static inline ssize_t +__rte_bitset_find(const uint64_t *bitset, size_t size, size_t start_bit, size_t len, + unsigned int flags) +{ + bool find_clear = flags & __RTE_BITSET_FIND_FLAG_FIND_CLEAR; + bool may_wrap = flags & __RTE_BITSET_FIND_FLAG_WRAP; + bool does_wrap = (start_bit + len) > size; + ssize_t rc; + + RTE_ASSERT(len <= size); + if (!may_wrap) + RTE_ASSERT(!does_wrap); + + if (may_wrap && does_wrap) { + size_t len0 = size - start_bit; + size_t len1 = len - len0; + + rc = __rte_bitset_find_nowrap(bitset, size, start_bit, len0, find_clear); + if (rc < 0) + rc = __rte_bitset_find_nowrap(bitset, size, 0, len1, find_clear); + } else + rc = __rte_bitset_find_nowrap(bitset, size, start_bit, len, find_clear); + + return rc; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Find first bit set. + * + * Scans the bitset in the forward direction (i.e., starting at the + * least significant bit), and returns the index of the first '1'. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @return + * Returns the index of the least significant '1', or -1 if all + * bits are '0'. + */ +__rte_experimental +static inline ssize_t +rte_bitset_find_first_set(const uint64_t *bitset, size_t size) +{ + return __rte_bitset_find(bitset, size, 0, size, 0); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Find first bit set at offset. + * + * Scans the bitset in the forward direction (i.e., starting at the + * least significant bit), starting at an offset @c start_bit into the + * bitset, and returns the index of the first '1' encountered. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @param start_bit + * The index of the first bit to check. Must be less than @c size. + * @param len + * The number of bits to scan. @c start_bit + @c len must be less + * than or equal to @c size. + * @return + * Returns the index of the least significant '1', or -1 if all + * bits are '0'. + */ +__rte_experimental +static inline ssize_t +rte_bitset_find_set(const uint64_t *bitset, size_t size, size_t start_bit, size_t len) +{ + return __rte_bitset_find(bitset, size, start_bit, len, 0); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Find first bit set at offset, with wrap-around. + * + * Scans the bitset in the forward direction (i.e., starting at the + * least significant bit), starting at an offset @c start_bit into the + * bitset. If no '1' is encountered before the end of the bitset, the search + * will continue at index 0. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @param start_bit + * The index of the first bit to check. Must be less than @c size. + * @param len + * The number of bits to scan. @c start_bit + @c len must be less + * than or equal to @c size. + * @return + * Returns the index of the least significant '1', or -1 if all + * bits are '0'. + */ +__rte_experimental +static inline ssize_t +rte_bitset_find_set_wrap(const uint64_t *bitset, size_t size, size_t start_bit, size_t len) +{ + return __rte_bitset_find(bitset, size, start_bit, len, __RTE_BITSET_FIND_FLAG_WRAP); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Find first cleared bit. + * + * Scans the bitset in the forward direction (i.e., starting at the + * least significant bit), and returns the index of the first '0'. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @return + * Returns the index of the least significant '0', or -1 if all + * bits are '1'. + */ +__rte_experimental +static inline ssize_t +rte_bitset_find_first_clear(const uint64_t *bitset, size_t size) +{ + return __rte_bitset_find(bitset, size, 0, size, __RTE_BITSET_FIND_FLAG_FIND_CLEAR); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Find first cleared bit at offset. + * + * Scans the bitset in the forward direction (i.e., starting at the + * least significant bit), starting at an offset @c start_bit into the + * bitset, and returns the index of the first '0' encountered. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @param start_bit + * The index of the first bit to check. Must be less than @c size. + * @param len + * The number of bits to scan. @c start_bit + @c len must be less + * than or equal to @c size. + * @return + * Returns the index of the least significant '0', or -1 if all + * bits are '1'. + */ +__rte_experimental +static inline ssize_t +rte_bitset_find_clear(const uint64_t *bitset, size_t size, size_t start_bit, size_t len) +{ + return __rte_bitset_find(bitset, size, start_bit, len, __RTE_BITSET_FIND_FLAG_FIND_CLEAR); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Find first cleared bit at offset, with wrap-around. + * + * Scans the bitset in the forward direction (i.e., starting at the + * least significant bit), starting at an offset @c start_bit into the + * bitset. If no '0' is encountered before the end of the bitset, the + * search will continue at index 0. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitset (in bits). + * @param start_bit + * The index of the first bit to check. Must be less than @c size. + * @param len + * The number of bits to scan. @c start_bit + @c len must be less + * than or equal to @c size. + * @return + * Returns the index of the least significant '0', or -1 if all + * bits are '1'. + */ +__rte_experimental +static inline ssize_t +rte_bitset_find_clear_wrap(const uint64_t *bitset, size_t size, size_t start_bit, size_t len) +{ + return __rte_bitset_find(bitset, size, start_bit, len, + __RTE_BITSET_FIND_FLAG_FIND_CLEAR | __RTE_BITSET_FIND_FLAG_WRAP); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Copy bitset. + * + * Copy the bits of the @c src_bitset to the @c dst_bitset. + * + * The bitsets may not overlap and must be of equal size. + * + * @param dst_bitset + * A pointer to the array of words making up the bitset. + * @param src_bitset + * A pointer to the array of words making up the bitset. + * @param size + * The size of the bitsets (in bits). + */ +__rte_experimental +static inline void +rte_bitset_copy(uint64_t *__rte_restrict dst_bitset, const uint64_t *__rte_restrict src_bitset, + size_t size) +{ + rte_memcpy(dst_bitset, src_bitset, RTE_BITSET_SIZE(size)); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Bitwise or two bitsets. + * + * Perform a bitwise OR operation on all bits in the two equal-size + * bitsets @c src_bitset0 and @c src_bitset1, and store the results in + * @c dst_bitset. + * + * @param dst_bitset + * A pointer to the destination bitset. + * @param src_bitset0 + * A pointer to the first source bitset. + * @param src_bitset1 + * A pointer to the second source bitset. + * @param size + * The size of the bitsets (in bits). + */ +__rte_experimental +static inline void +rte_bitset_or(uint64_t *dst_bitset, const uint64_t *src_bitset0, const uint64_t *src_bitset1, + size_t size) +{ + size_t i; + + for (i = 0; i < RTE_BITSET_NUM_WORDS(size); i++) + dst_bitset[i] = src_bitset0[i] | src_bitset1[i]; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Bitwise and two bitsets. + * + * Perform a bitwise AND operation on all bits in the two equal-size + * bitsets @c src_bitset0 and @c src_bitset1, and store the result in + * @c dst_bitset. + * + * @param dst_bitset + * A pointer to the destination bitset. + * @param src_bitset0 + * A pointer to the first source bitset. + * @param src_bitset1 + * A pointer to the second source bitset. + * @param size + * The size of the bitsets (in bits). + */ +__rte_experimental +static inline void +rte_bitset_and(uint64_t *dst_bitset, const uint64_t *src_bitset0, const uint64_t *src_bitset1, + size_t size) +{ + size_t i; + + for (i = 0; i < RTE_BITSET_NUM_WORDS(size); i++) + dst_bitset[i] = src_bitset0[i] & src_bitset1[i]; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Bitwise xor two bitsets. + * + * Perform a bitwise XOR operation on all bits in the two equal-size + * bitsets @c src_bitset0 and @c src_bitset1, and store the result in + * @c dst_bitset. + * + * @param dst_bitset + * A pointer to the destination bitset. + * @param src_bitset0 + * A pointer to the first source bitset. + * @param src_bitset1 + * A pointer to the second source bitset. + * @param size + * The size of the bitsets (in bits). + */ +__rte_experimental +static inline void +rte_bitset_xor(uint64_t *dst_bitset, const uint64_t *src_bitset0, const uint64_t *src_bitset1, + size_t size) +{ + size_t i; + + for (i = 0; i < RTE_BITSET_NUM_WORDS(size); i++) + dst_bitset[i] = src_bitset0[i] ^ src_bitset1[i]; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Compute the bitwise complement of a bitset. + * + * Flip every bit in the @c src_bitset, and store the result in @c + * dst_bitset. + * + * @param dst_bitset + * A pointer to the destination bitset. + * @param src_bitset + * A pointer to the source bitset. + * @param size + * The size of the bitsets (in bits). + */ +__rte_experimental +static inline void +rte_bitset_complement(uint64_t *dst_bitset, const uint64_t *src_bitset, size_t size) +{ + size_t i; + + for (i = 0; i < RTE_BITSET_NUM_WORDS(size); i++) + dst_bitset[i] = ~src_bitset[i]; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Shift bitset left. + * + * Perform a logical shift left of (multiply) @c src_bitset, and store + * the result in @c dst_bitset. + * + * @param dst_bitset + * A pointer to the destination bitset. + * @param src_bitset + * A pointer to the source bitset. + * @param size + * The size of the bitsets (in bits). + * @param shift_bits + * The number of bits to shift the bitset. + */ +__rte_experimental +static inline void +rte_bitset_shift_left(uint64_t *dst_bitset, const uint64_t *src_bitset, size_t size, + size_t shift_bits) +{ + const int src_word_offset = shift_bits / RTE_BITSET_WORD_BITS; + const int src_bit_offset = shift_bits % RTE_BITSET_WORD_BITS; + unsigned int dst_idx; + + for (dst_idx = 0; dst_idx < RTE_BITSET_NUM_WORDS(size); dst_idx++) { + int src_high_idx = dst_idx - src_word_offset; + uint64_t low_bits = 0; + uint64_t high_bits = 0; + + if (src_high_idx >= 0) { + int src_low_idx = src_high_idx - 1; + + high_bits = src_bitset[src_high_idx] << src_bit_offset; + + if (src_bit_offset > 0 && src_low_idx >= 0) + low_bits = src_bitset[src_low_idx] >> + (RTE_BITSET_WORD_BITS - src_bit_offset); + } + dst_bitset[dst_idx] = low_bits | high_bits; + } +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Shift bitset right. + * + * Perform a logical shift right of (divide) @c src_bitset, and store + * the result in @c dst_bitset. + * + * @param dst_bitset + * A pointer to the destination bitset. + * @param src_bitset + * A pointer to the source bitset. + * @param size + * The size of the bitsets (in bits). + * @param shift_bits + * The number of bits to shift the bitset. + */ +__rte_experimental +static inline void +rte_bitset_shift_right(uint64_t *dst_bitset, const uint64_t *src_bitset, size_t size, + size_t shift_bits) +{ + const int num_words = RTE_BITSET_NUM_WORDS(size); + const uint64_t used_mask = __RTE_BITSET_USED_MASK(size); + const int src_word_offset = shift_bits / RTE_BITSET_WORD_BITS; + const int src_bit_offset = shift_bits % RTE_BITSET_WORD_BITS; + int dst_idx; + + for (dst_idx = 0; dst_idx < num_words; dst_idx++) { + int src_low_idx = src_word_offset + dst_idx; + int src_high_idx = src_low_idx + 1; + uint64_t src_low_word_bits = 0; + uint64_t src_high_word_bits = 0; + + if (src_low_idx < num_words) { + src_low_word_bits = src_bitset[src_low_idx]; + + if (src_low_idx == (num_words - 1)) + src_low_word_bits &= used_mask; + + src_low_word_bits >>= src_bit_offset; + + if (src_bit_offset > 0 && src_high_idx < num_words) { + src_high_word_bits = src_bitset[src_high_idx]; + + if (src_high_idx == (num_words - 1)) + src_high_word_bits &= used_mask; + + src_high_word_bits <<= (RTE_BITSET_WORD_BITS - src_bit_offset); + } + } + dst_bitset[dst_idx] = src_low_word_bits | src_high_word_bits; + } +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Compare two bitsets. + * + * Compare two bitsets for equality. + * + * @param bitset_a + * A pointer to the destination bitset. + * @param bitset_b + * A pointer to the source bitset. + * @param size + * The size of the bitsets (in bits). + */ +__rte_experimental +static inline bool +rte_bitset_equal(const uint64_t *bitset_a, const uint64_t *bitset_b, size_t size) +{ + size_t i; + uint64_t last_a, last_b; + + for (i = 0; i < RTE_BITSET_NUM_WORDS(size) - 1; i++) + if (bitset_a[i] != bitset_b[i]) + return false; + + last_a = bitset_a[i] << __RTE_BITSET_UNUSED(size); + last_b = bitset_b[i] << __RTE_BITSET_UNUSED(size); + + return last_a == last_b; +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Converts a bitset to a string. + * + * This function prints a string representation of the bitstring to + * the supplied buffer. + * + * Each bit is represented either by '0' or '1' in the output, with + * the first (left-most) character in the output being the most + * significant bit. The resulting string is NUL terminated. + * + * @param bitset + * A pointer to the array of bitset 64-bit words. + * @param size + * The number of bits the bitset represent. + * @param buf + * A buffer to hold the output. + * @param capacity + * The size of the buffer. Must be @c size + 1 or larger. + * @return + * Returns the number of bytes written (i.e., @c size + 1), or -EINVAL + * in case the buffer capacity was too small. + */ +__rte_experimental +ssize_t +rte_bitset_to_str(const uint64_t *bitset, size_t size, char *buf, size_t capacity); + +#ifdef __cplusplus +} +#endif + +#endif /* _RTE_BITSET_H_ */ diff --git a/lib/eal/version.map b/lib/eal/version.map index e3ff412683..f493cd1ca7 100644 --- a/lib/eal/version.map +++ b/lib/eal/version.map @@ -396,6 +396,9 @@ EXPERIMENTAL { # added in 24.03 rte_vfio_get_device_info; # WINDOWS_NO_EXPORT + + # added in 24.11 + rte_bitset_to_str; }; INTERNAL { From patchwork Thu Oct 10 08:30:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 145577 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 19D3545B00; Thu, 10 Oct 2024 10:30:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2A4A140616; Thu, 10 Oct 2024 10:30:47 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 31A8F40651 for ; Thu, 10 Oct 2024 10:30:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728549044; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WxgyyuPcTZWRiEif+bPDSQv1GmL0OC/9jQKm5AQ6Ri4=; b=WGBG+hBO+9Rtnl+kRLYTBbnE2N0HTAhNCHp00ImjuLkUKbwxZ+6/4mPRbHrkNVquONYY9s aV6Z3oWmNZIW2H4Fq+k2aDpy9Yekhi5ITiysC0vulrCDLm0Ks6iGEggQmFAotgFGQ1Rawj dkaG8YPIP1yPN5wP68LcErFT8eLvbWg= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-357-DJ_BX5g-NVmRO36_v-YAYQ-1; Thu, 10 Oct 2024 04:30:43 -0400 X-MC-Unique: DJ_BX5g-NVmRO36_v-YAYQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EF27B1956095; Thu, 10 Oct 2024 08:30:41 +0000 (UTC) Received: from dmarchan.redhat.com (unknown [10.45.224.68]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 95CCF19560A2; Thu, 10 Oct 2024 08:30:39 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: mattias.ronnblom@ericsson.com, mb@smartsharesystems.com, stephen@networkplumber.org, harry.van.haaren@intel.com, roretzla@linux.microsoft.com Subject: [PATCH v2 2/4] bitset: add atomic functions Date: Thu, 10 Oct 2024 10:30:20 +0200 Message-ID: <20241010083022.3437380-3-david.marchand@redhat.com> In-Reply-To: <20241010083022.3437380-1-david.marchand@redhat.com> References: <20240809201440.590464-1-mattias.ronnblom@ericsson.com> <20241010083022.3437380-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Mattias Rönnblom Extend the bitset API with atomic versions of the most basic bitset operations. Signed-off-by: Mattias Rönnblom Acked-by: Tyler Retzlaff --- app/test/test_bitset.c | 44 +++++++++++ lib/eal/include/rte_bitset.h | 143 +++++++++++++++++++++++++++++++++++ 2 files changed, 187 insertions(+) diff --git a/app/test/test_bitset.c b/app/test/test_bitset.c index fd3e50f396..50b8bf0da4 100644 --- a/app/test/test_bitset.c +++ b/app/test/test_bitset.c @@ -213,6 +213,48 @@ test_flip(void) return test_flip_fun(rte_bitset_test, rte_bitset_assign, rte_bitset_flip); } +static bool +bitset_atomic_test(const uint64_t *bitset, size_t bit_num) +{ + return rte_bitset_atomic_test(bitset, bit_num, rte_memory_order_relaxed); +} + +static void +bitset_atomic_set(uint64_t *bitset, size_t bit_num) +{ + rte_bitset_atomic_set(bitset, bit_num, rte_memory_order_relaxed); +} + +static void +bitset_atomic_clear(uint64_t *bitset, size_t bit_num) +{ + rte_bitset_atomic_clear(bitset, bit_num, rte_memory_order_relaxed); +} + +static void +bitset_atomic_flip(uint64_t *bitset, size_t bit_num) +{ + rte_bitset_atomic_flip(bitset, bit_num, rte_memory_order_relaxed); +} + +static void +bitset_atomic_assign(uint64_t *bitset, size_t bit_num, bool bit_value) +{ + rte_bitset_atomic_assign(bitset, bit_num, bit_value, rte_memory_order_relaxed); +} + +static int +test_atomic_set_clear(void) +{ + return test_set_clear_fun(bitset_atomic_test, bitset_atomic_set, bitset_atomic_clear); +} + +static int +test_atomic_flip(void) +{ + return test_flip_fun(bitset_atomic_test, bitset_atomic_assign, bitset_atomic_flip); +} + static ssize_t find(const bool *ary, size_t num_bools, size_t start, size_t len, bool set) { @@ -835,6 +877,8 @@ static struct unit_test_suite bitset_tests = { .unit_test_cases = { TEST_CASE_ST(NULL, NULL, test_set_clear), TEST_CASE_ST(NULL, NULL, test_flip), + TEST_CASE_ST(NULL, NULL, test_atomic_set_clear), + TEST_CASE_ST(NULL, NULL, test_atomic_flip), TEST_CASE_ST(NULL, NULL, test_find), TEST_CASE_ST(NULL, NULL, test_foreach), TEST_CASE_ST(NULL, NULL, test_count), diff --git a/lib/eal/include/rte_bitset.h b/lib/eal/include/rte_bitset.h index b2d71aa7c6..74c643a72a 100644 --- a/lib/eal/include/rte_bitset.h +++ b/lib/eal/include/rte_bitset.h @@ -348,6 +348,149 @@ rte_bitset_flip(uint64_t *bitset, size_t bit_num) __RTE_BITSET_DELEGATE(rte_bit_flip, bitset, bit_num); } +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test if a bit is set. + * + * Atomically test if a bit in a bitset is set with the specified + * memory ordering. + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * Index of the bit to test. Index 0 is the least significant bit. + * @param memory_order + * The memory order to use. + * @return + * Returns true if the bit is '1', and false if the bit is '0'. + */ +__rte_experimental +static inline bool +rte_bitset_atomic_test(const uint64_t *bitset, size_t bit_num, int memory_order) +{ + return __RTE_BITSET_DELEGATE_N(rte_bit_atomic_test, bitset, bit_num, memory_order); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically set a bit in the bitset. + * + * Set a bit in a bitset as an atomic operation, with the specified + * memory ordering. + * + * rte_bitset_atomic_set() is multi-thread safe, provided all threads + * acting in parallel on the same bitset does so through + * @c rte_bitset_atomic_*() functions. + * + * Bits are numbered from 0 to (size - 1) (inclusive). + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be set. + * @param memory_order + * The memory order to use. + */ +__rte_experimental +static inline void +rte_bitset_atomic_set(uint64_t *bitset, size_t bit_num, int memory_order) +{ + __RTE_BITSET_DELEGATE_N(rte_bit_atomic_set, bitset, bit_num, memory_order); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically clear a bit in the bitset. + * + * Clear a bit in a bitset as an atomic operation, with the specified + * memory ordering. + * + * rte_bitset_atomic_clear() is multi-thread safe, provided all + * threads acting in parallel on the same bitset does so through @c + * rte_bitset_atomic_*() functions. + * + * Bits are numbered from 0 to (size - 1) (inclusive). + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be cleared. + * @param memory_order + * The memory order to use. + */ +__rte_experimental +static inline void +rte_bitset_atomic_clear(uint64_t *bitset, size_t bit_num, int memory_order) +{ + __RTE_BITSET_DELEGATE_N(rte_bit_atomic_clear, bitset, bit_num, memory_order); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically set or clear a bit in the bitset. + * + * Assign a value to a bit in a bitset as an atomic operation, with + * the specified memory ordering. + * + * rte_bitset_atomic_assign() is multi-thread safe, provided all + * threads acting in parallel on the same bitset does so through + * @c rte_bitset_atomic_*() functions. + * + * Bits are numbered from 0 to (size - 1) (inclusive). + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be set or cleared. + * @param bit_value + * Control if the bit should be set or cleared. + * @param memory_order + * The memory order to use. + */ +__rte_experimental +static inline void +rte_bitset_atomic_assign(uint64_t *bitset, size_t bit_num, bool bit_value, int memory_order) +{ + __RTE_BITSET_DELEGATE_N(rte_bit_atomic_assign, bitset, bit_num, bit_value, memory_order); +} + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically change the value of a bit in the bitset. + * + * Flip a bit in a bitset as an atomic operation, with the specified + * memory ordering. + * + * rte_bitset_atomic_flip() is multi-thread safe, provided all threads + * acting in parallel on the same bitset does so through + * @c rte_bitset_atomic_*() functions. + * + * Bits are numbered from 0 to (size - 1) (inclusive). + * + * @param bitset + * A pointer to the array of words making up the bitset. + * @param bit_num + * The index of the bit to be flipped. + * @param memory_order + * The memory order to use. + */ +__rte_experimental +static inline void +rte_bitset_atomic_flip(uint64_t *bitset, size_t bit_num, int memory_order) +{ + __RTE_BITSET_DELEGATE_N(rte_bit_atomic_flip, bitset, bit_num, memory_order); +} + /** * @warning * @b EXPERIMENTAL: this API may change without prior notice. From patchwork Thu Oct 10 08:30:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 145578 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C9FB945B00; Thu, 10 Oct 2024 10:30:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 36FE94042C; Thu, 10 Oct 2024 10:30:54 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id AF225402E8 for ; Thu, 10 Oct 2024 10:30:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728549052; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4r0tWWAQhiSUvJB2a5fSgVaJJpiK+YlDLrFLP2bd2hc=; b=WLkTz7lSWalYUFvT4DUXcwDa54zrlt2sS3aXYXt7WAwQUPU0+fXUplFnrWwCLZl8lhZhbi +UBDohYsuCO2ogNWTFGSVVYztDh8ELze1Fh4fMjKZi7V8Rtree+aqwKv7nNrN6dnmNZftI H7D4FFzaYmrPlOvX+VUhZfHWSZsBM20= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-75-7z2Qo9K9OdCSS0JKCiT76g-1; Thu, 10 Oct 2024 04:30:48 -0400 X-MC-Unique: 7z2Qo9K9OdCSS0JKCiT76g-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8133A19560B6; Thu, 10 Oct 2024 08:30:47 +0000 (UTC) Received: from dmarchan.redhat.com (unknown [10.45.224.68]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7213E19560A3; Thu, 10 Oct 2024 08:30:45 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: mattias.ronnblom@ericsson.com, mb@smartsharesystems.com, stephen@networkplumber.org, harry.van.haaren@intel.com, roretzla@linux.microsoft.com Subject: [PATCH v2 3/4] service: use multi-word bitset to represent service flags Date: Thu, 10 Oct 2024 10:30:21 +0200 Message-ID: <20241010083022.3437380-4-david.marchand@redhat.com> In-Reply-To: <20241010083022.3437380-1-david.marchand@redhat.com> References: <20240809201440.590464-1-mattias.ronnblom@ericsson.com> <20241010083022.3437380-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Mattias Rönnblom Use a multi-word bitset to track which services are mapped to which lcores, allowing the RTE_SERVICE_NUM_MAX compile-time constant to be > 64. Replace array-of-bytes service-currently-active flags with a more compact multi-word bitset-based representation, reducing memory footprint somewhat. Signed-off-by: Mattias Rönnblom Acked-by: Tyler Retzlaff --- lib/eal/common/rte_service.c | 70 ++++++++++++++---------------------- 1 file changed, 27 insertions(+), 43 deletions(-) diff --git a/lib/eal/common/rte_service.c b/lib/eal/common/rte_service.c index a38c594ce4..324471e897 100644 --- a/lib/eal/common/rte_service.c +++ b/lib/eal/common/rte_service.c @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -65,11 +66,11 @@ struct service_stats { /* the internal values of a service core */ struct __rte_cache_aligned core_state { /* map of services IDs are run on this core */ - uint64_t service_mask; + RTE_BITSET_DECLARE(mapped_services, RTE_SERVICE_NUM_MAX); RTE_ATOMIC(uint8_t) runstate; /* running or stopped */ RTE_ATOMIC(uint8_t) thread_active; /* indicates when thread is in service_run() */ uint8_t is_service_core; /* set if core is currently a service core */ - uint8_t service_active_on_lcore[RTE_SERVICE_NUM_MAX]; + RTE_BITSET_DECLARE(service_active_on_lcore, RTE_SERVICE_NUM_MAX); RTE_ATOMIC(uint64_t) loops; RTE_ATOMIC(uint64_t) cycles; struct service_stats service_stats[RTE_SERVICE_NUM_MAX]; @@ -83,11 +84,6 @@ static uint32_t rte_service_library_initialized; int32_t rte_service_init(void) { - /* Hard limit due to the use of an uint64_t-based bitmask (and the - * clzl intrinsic). - */ - RTE_BUILD_BUG_ON(RTE_SERVICE_NUM_MAX > 64); - if (rte_service_library_initialized) { EAL_LOG(NOTICE, "service library init() called, init flag %d", @@ -298,7 +294,7 @@ rte_service_component_unregister(uint32_t id) /* clear the run-bit in all cores */ for (i = 0; i < RTE_MAX_LCORE; i++) - lcore_states[i].service_mask &= ~(UINT64_C(1) << id); + rte_bitset_clear(lcore_states[i].mapped_services, id); memset(&rte_services[id], 0, sizeof(struct rte_service_spec_impl)); @@ -423,7 +419,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s, /* Expects the service 's' is valid. */ static int32_t -service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, +service_run(uint32_t i, struct core_state *cs, const uint64_t *mapped_services, struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe) { if (!s) @@ -437,12 +433,12 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, RUNSTATE_RUNNING || rte_atomic_load_explicit(&s->app_runstate, rte_memory_order_acquire) != RUNSTATE_RUNNING || - !(service_mask & (UINT64_C(1) << i))) { - cs->service_active_on_lcore[i] = 0; + !rte_bitset_test(mapped_services, i)) { + rte_bitset_clear(cs->service_active_on_lcore, i); return -ENOEXEC; } - cs->service_active_on_lcore[i] = 1; + rte_bitset_set(cs->service_active_on_lcore, i); if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) { if (!rte_spinlock_trylock(&s->execute_lock)) @@ -467,7 +463,7 @@ rte_service_may_be_active(uint32_t id) return -EINVAL; for (i = 0; i < lcore_count; i++) { - if (lcore_states[ids[i]].service_active_on_lcore[id]) + if (rte_bitset_test(lcore_states[ids[i]].service_active_on_lcore, id)) return 1; } @@ -487,7 +483,9 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) */ rte_atomic_fetch_add_explicit(&s->num_mapped_cores, 1, rte_memory_order_relaxed); - int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe); + RTE_BITSET_DECLARE(all_services, RTE_SERVICE_NUM_MAX); + rte_bitset_set_all(all_services, RTE_SERVICE_NUM_MAX); + int ret = service_run(id, cs, all_services, s, serialize_mt_unsafe); rte_atomic_fetch_sub_explicit(&s->num_mapped_cores, 1, rte_memory_order_relaxed); @@ -498,7 +496,6 @@ static int32_t service_runner_func(void *arg) { RTE_SET_USED(arg); - uint8_t i; const int lcore = rte_lcore_id(); struct core_state *cs = &lcore_states[lcore]; @@ -510,20 +507,11 @@ service_runner_func(void *arg) */ while (rte_atomic_load_explicit(&cs->runstate, rte_memory_order_acquire) == RUNSTATE_RUNNING) { + ssize_t id; - const uint64_t service_mask = cs->service_mask; - uint8_t start_id; - uint8_t end_id; - - if (service_mask == 0) - continue; - - start_id = rte_ctz64(service_mask); - end_id = 64 - rte_clz64(service_mask); - - for (i = start_id; i < end_id; i++) { + RTE_BITSET_FOREACH_SET(id, cs->mapped_services, RTE_SERVICE_NUM_MAX) { /* return value ignored as no change to code flow */ - service_run(i, cs, service_mask, service_get(i), 1); + service_run(id, cs, cs->mapped_services, service_get(id), 1); } rte_atomic_store_explicit(&cs->loops, cs->loops + 1, rte_memory_order_relaxed); @@ -532,8 +520,7 @@ service_runner_func(void *arg) /* Switch off this core for all services, to ensure that future * calls to may_be_active() know this core is switched off. */ - for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) - cs->service_active_on_lcore[i] = 0; + rte_bitset_clear_all(cs->service_active_on_lcore, RTE_SERVICE_NUM_MAX); /* Use SEQ CST memory ordering to avoid any re-ordering around * this store, ensuring that once this store is visible, the service @@ -599,7 +586,7 @@ rte_service_lcore_count_services(uint32_t lcore) if (!cs->is_service_core) return -ENOTSUP; - return rte_popcount64(cs->service_mask); + return rte_bitset_count_set(cs->mapped_services, RTE_SERVICE_NUM_MAX); } int32_t @@ -652,25 +639,23 @@ service_update(uint32_t sid, uint32_t lcore, uint32_t *set, uint32_t *enabled) !lcore_states[lcore].is_service_core) return -EINVAL; - uint64_t sid_mask = UINT64_C(1) << sid; if (set) { - uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask; + uint64_t lcore_mapped = rte_bitset_test(lcore_states[lcore].mapped_services, sid); if (*set && !lcore_mapped) { - lcore_states[lcore].service_mask |= sid_mask; + rte_bitset_set(lcore_states[lcore].mapped_services, sid); rte_atomic_fetch_add_explicit(&rte_services[sid].num_mapped_cores, 1, rte_memory_order_relaxed); } if (!*set && lcore_mapped) { - lcore_states[lcore].service_mask &= ~(sid_mask); + rte_bitset_clear(lcore_states[lcore].mapped_services, sid); rte_atomic_fetch_sub_explicit(&rte_services[sid].num_mapped_cores, 1, rte_memory_order_relaxed); } } if (enabled) - *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); + *enabled = rte_bitset_test(lcore_states[lcore].mapped_services, sid); return 0; } @@ -712,11 +697,11 @@ set_lcore_state(uint32_t lcore, int32_t state) int32_t rte_service_lcore_reset_all(void) { - /* loop over cores, reset all to mask 0 */ + /* loop over cores, reset all mapped services */ uint32_t i; for (i = 0; i < RTE_MAX_LCORE; i++) { if (lcore_states[i].is_service_core) { - lcore_states[i].service_mask = 0; + rte_bitset_clear_all(lcore_states[i].mapped_services, RTE_SERVICE_NUM_MAX); set_lcore_state(i, ROLE_RTE); /* runstate act as guard variable Use * store-release memory order here to synchronize @@ -744,7 +729,7 @@ rte_service_lcore_add(uint32_t lcore) set_lcore_state(lcore, ROLE_SERVICE); /* ensure that after adding a core the mask and state are defaults */ - lcore_states[lcore].service_mask = 0; + rte_bitset_clear_all(lcore_states[lcore].mapped_services, RTE_SERVICE_NUM_MAX); /* Use store-release memory order here to synchronize with * load-acquire in runstate read functions. */ @@ -827,12 +812,11 @@ rte_service_lcore_stop(uint32_t lcore) uint32_t i; struct core_state *cs = &lcore_states[lcore]; - uint64_t service_mask = cs->service_mask; for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { - int32_t enabled = service_mask & (UINT64_C(1) << i); - int32_t service_running = rte_service_runstate_get(i); - int32_t only_core = (1 == + bool enabled = rte_bitset_test(cs->mapped_services, i); + bool service_running = rte_service_runstate_get(i); + bool only_core = (1 == rte_atomic_load_explicit(&rte_services[i].num_mapped_cores, rte_memory_order_relaxed)); From patchwork Thu Oct 10 08:30:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 145579 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 965A545B00; Thu, 10 Oct 2024 10:31:04 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 778F64065A; Thu, 10 Oct 2024 10:31:04 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 780994065D for ; Thu, 10 Oct 2024 10:31:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728549062; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RGDPolikfXt2Se17+JlJF2CPnxYtihqTuJPT8p+Qz8o=; b=XBcPrnlZulOuU/CojsnuN+HZeXlhqXccnTJw33erbV37b+OJ1nn+vZYX2xQcB7UR24xYT3 Xf6MIK+lcI8f0UaabTE+SWSwLlymib7sR61JRtdI603YER7AqkcYOEWZJZ+m+MPrCna8tO huWCbENMw2TnUkTiS0agAHQkJtrw1cU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-561-jDJ5LpcVMbe5bY4E1Y5xzQ-1; Thu, 10 Oct 2024 04:30:53 -0400 X-MC-Unique: jDJ5LpcVMbe5bY4E1Y5xzQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 97BF419560A2; Thu, 10 Oct 2024 08:30:52 +0000 (UTC) Received: from dmarchan.redhat.com (unknown [10.45.224.68]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4114B19560AE; Thu, 10 Oct 2024 08:30:50 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: mattias.ronnblom@ericsson.com, mb@smartsharesystems.com, stephen@networkplumber.org, harry.van.haaren@intel.com, roretzla@linux.microsoft.com Subject: [PATCH v2 4/4] event/dsw: add support for larger port count Date: Thu, 10 Oct 2024 10:30:22 +0200 Message-ID: <20241010083022.3437380-5-david.marchand@redhat.com> In-Reply-To: <20241010083022.3437380-1-david.marchand@redhat.com> References: <20240809201440.590464-1-mattias.ronnblom@ericsson.com> <20241010083022.3437380-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Mattias Rönnblom Switch from using an open-coded, single-word bitset to using to represent which event ports are linked to a particular event queue. Besides the cleaner code, this also allow the user to extend the maximum port count beyond 64, by means of changing an "event_dev.h" Signed-off-by: Mattias Rönnblom Acked-by: Tyler Retzlaff --- drivers/event/dsw/dsw_evdev.c | 19 +++++++------------ drivers/event/dsw/dsw_evdev.h | 3 ++- drivers/event/dsw/dsw_event.c | 7 ++++--- 3 files changed, 13 insertions(+), 16 deletions(-) diff --git a/drivers/event/dsw/dsw_evdev.c b/drivers/event/dsw/dsw_evdev.c index 8a1a2db8ac..bf95cb39cd 100644 --- a/drivers/event/dsw/dsw_evdev.c +++ b/drivers/event/dsw/dsw_evdev.c @@ -123,6 +123,7 @@ dsw_queue_setup(struct rte_eventdev *dev, uint8_t queue_id, queue->schedule_type = conf->schedule_type; } + rte_bitset_init(queue->serving_ports, DSW_MAX_PORTS); queue->num_serving_ports = 0; return 0; @@ -149,20 +150,16 @@ dsw_queue_release(struct rte_eventdev *dev __rte_unused, static void queue_add_port(struct dsw_queue *queue, uint16_t port_id) { - uint64_t port_mask = UINT64_C(1) << port_id; - - queue->serving_ports |= port_mask; + rte_bitset_set(queue->serving_ports, port_id); queue->num_serving_ports++; } static bool queue_remove_port(struct dsw_queue *queue, uint16_t port_id) { - uint64_t port_mask = UINT64_C(1) << port_id; - - if (queue->serving_ports & port_mask) { + if (rte_bitset_test(queue->serving_ports, port_id)) { queue->num_serving_ports--; - queue->serving_ports ^= port_mask; + rte_bitset_clear(queue->serving_ports, port_id); return true; } @@ -263,14 +260,12 @@ initial_flow_to_port_assignment(struct dsw_evdev *dsw) struct dsw_queue *queue = &dsw->queues[queue_id]; uint16_t flow_hash; for (flow_hash = 0; flow_hash < DSW_MAX_FLOWS; flow_hash++) { - uint8_t skip = - rte_rand_max(queue->num_serving_ports); + uint8_t skip = rte_rand_max(queue->num_serving_ports); uint8_t port_id; for (port_id = 0;; port_id++) { - uint64_t port_mask = UINT64_C(1) << port_id; - - if (queue->serving_ports & port_mask) { + if (rte_bitset_test(queue->serving_ports, + port_id)) { if (skip == 0) break; skip--; diff --git a/drivers/event/dsw/dsw_evdev.h b/drivers/event/dsw/dsw_evdev.h index 3f29e53929..e05cee0e6a 100644 --- a/drivers/event/dsw/dsw_evdev.h +++ b/drivers/event/dsw/dsw_evdev.h @@ -7,6 +7,7 @@ #include +#include #include #include @@ -237,7 +238,7 @@ struct __rte_cache_aligned dsw_port { struct dsw_queue { uint8_t schedule_type; - uint64_t serving_ports; + RTE_BITSET_DECLARE(serving_ports, DSW_MAX_PORTS); uint16_t num_serving_ports; alignas(RTE_CACHE_LINE_SIZE) uint8_t flow_to_port_map[DSW_MAX_FLOWS]; diff --git a/drivers/event/dsw/dsw_event.c b/drivers/event/dsw/dsw_event.c index 67e630ffcb..6eeeb6da93 100644 --- a/drivers/event/dsw/dsw_event.c +++ b/drivers/event/dsw/dsw_event.c @@ -457,9 +457,8 @@ static bool dsw_is_serving_port(struct dsw_evdev *dsw, uint8_t port_id, uint8_t queue_id) { struct dsw_queue *queue = &dsw->queues[queue_id]; - uint64_t port_mask = UINT64_C(1) << port_id; - return queue->serving_ports & port_mask; + return rte_bitset_test(queue->serving_ports, port_id); } static bool @@ -583,7 +582,9 @@ dsw_schedule(struct dsw_evdev *dsw, uint8_t queue_id, uint16_t flow_hash) /* A single-link queue, or atomic/ordered/parallel but * with just a single serving port. */ - port_id = rte_bsf64(queue->serving_ports); + port_id = (uint8_t)rte_bitset_find_first_set( + queue->serving_ports, DSW_MAX_PORTS + ); DSW_LOG_DP_LINE(DEBUG, "Event with queue_id %d flow_hash %d is scheduled " "to port %d.", queue_id, flow_hash, port_id);