From patchwork Mon Mar 11 14:47:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Szczepanek X-Patchwork-Id: 138153 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1099F43C88; Mon, 11 Mar 2024 15:47:36 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C3622406B7; Mon, 11 Mar 2024 15:47:31 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id B0EBA4027C for ; Mon, 11 Mar 2024 15:47:29 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EAF57153B; Mon, 11 Mar 2024 07:48:05 -0700 (PDT) Received: from ampere-altra-2-1.usa.Arm.com (ampere-altra-2-1.usa.arm.com [10.118.91.158]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DED863F844; Mon, 11 Mar 2024 07:47:28 -0700 (PDT) From: Paul Szczepanek To: dev@dpdk.org Cc: bruce.richardson@intel.com, Paul Szczepanek , Honnappa Nagarahalli Subject: [PATCH v9 1/5] lib: allow libraries with no sources Date: Mon, 11 Mar 2024 14:47:02 +0000 Message-Id: <20240311144706.204831-2-paul.szczepanek@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240311144706.204831-1-paul.szczepanek@arm.com> References: <20230927150854.3670391-2-paul.szczepanek@arm.com> <20240311144706.204831-1-paul.szczepanek@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Allow header only libraries. Signed-off-by: Paul Szczepanek Reviewed-by: Honnappa Nagarahalli Acked-by: Bruce Richardson --- lib/meson.build | 14 ++++++++++++++ 1 file changed, 14 insertions(+) -- 2.25.1 diff --git a/lib/meson.build b/lib/meson.build index 179a272932..e4e31f7ecf 100644 --- a/lib/meson.build +++ b/lib/meson.build @@ -222,6 +222,20 @@ foreach l:libraries includes += include_directories(l) dpdk_includes += include_directories(l) + # special case for header only libraries + if sources.length() == 0 + header_lib_dep = declare_dependency(include_directories: includes, + dependencies: shared_deps) + set_variable('shared_rte_' + name, header_lib_dep) + set_variable('static_rte_' + name, header_lib_dep) + dpdk_shared_lib_deps += header_lib_dep + dpdk_static_lib_deps += header_lib_dep + if developer_mode + message('lib/@0@: Defining dependency "@1@"'.format(l, name)) + endif + continue + endif + if developer_mode and is_windows and use_function_versioning message('@0@: Function versioning is not supported by Windows.'.format(name)) endif From patchwork Mon Mar 11 14:47:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Szczepanek X-Patchwork-Id: 138154 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D06C043C88; Mon, 11 Mar 2024 15:47:42 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 108E540A67; Mon, 11 Mar 2024 15:47:33 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id F3197402DC for ; Mon, 11 Mar 2024 15:47:29 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 37AC71007; Mon, 11 Mar 2024 07:48:06 -0700 (PDT) Received: from ampere-altra-2-1.usa.Arm.com (ampere-altra-2-1.usa.arm.com [10.118.91.158]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 27A9F3F844; Mon, 11 Mar 2024 07:47:29 -0700 (PDT) From: Paul Szczepanek To: dev@dpdk.org Cc: bruce.richardson@intel.com, Paul Szczepanek , Honnappa Nagarahalli , Kamalakshitha Aligeri , Nathan Brown Subject: [PATCH v9 2/5] ptr_compress: add pointer compression library Date: Mon, 11 Mar 2024 14:47:03 +0000 Message-Id: <20240311144706.204831-3-paul.szczepanek@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240311144706.204831-1-paul.szczepanek@arm.com> References: <20230927150854.3670391-2-paul.szczepanek@arm.com> <20240311144706.204831-1-paul.szczepanek@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add a new utility header for compressing pointers. The provided functions can store pointers in 32-bit or 16-bit offsets. The compression takes advantage of the fact that pointers are usually located in a limited memory region (like a mempool). We can compress them by converting them to offsets from a base memory address. Offsets can be stored in fewer bytes (dictated by the memory region size and alignment of the pointer). For example: an 8 byte aligned pointer which is part of a 32GB memory pool can be stored in 4 bytes. Suggested-by: Honnappa Nagarahalli Signed-off-by: Paul Szczepanek Signed-off-by: Kamalakshitha Aligeri Reviewed-by: Honnappa Nagarahalli Reviewed-by: Nathan Brown --- MAINTAINERS | 4 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/rel_notes/release_24_03.rst | 6 + lib/meson.build | 1 + lib/ptr_compress/meson.build | 4 + lib/ptr_compress/rte_ptr_compress.h | 266 +++++++++++++++++++++++++ 7 files changed, 283 insertions(+) create mode 100644 lib/ptr_compress/meson.build create mode 100644 lib/ptr_compress/rte_ptr_compress.h -- 2.25.1 diff --git a/MAINTAINERS b/MAINTAINERS index 4755a68274..6f703b1b13 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1685,6 +1685,10 @@ M: Chenbo Xia M: Gaetan Rivet F: lib/pci/ +Pointer Compression +M: Paul Szczepanek +F: lib/ptr_compress/ + Power management M: Anatoly Burakov M: David Hunt diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md index 8c1eb8fafa..f9283154f8 100644 --- a/doc/api/doxy-api-index.md +++ b/doc/api/doxy-api-index.md @@ -222,6 +222,7 @@ The public API headers are grouped by topics: [config file](@ref rte_cfgfile.h), [key/value args](@ref rte_kvargs.h), [argument parsing](@ref rte_argparse.h), + [ptr_compress](@ref rte_ptr_compress.h), [string](@ref rte_string_fns.h), [thread](@ref rte_thread.h) diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index 27afec8b3b..a8823c046f 100644 --- a/doc/api/doxy-api.conf.in +++ b/doc/api/doxy-api.conf.in @@ -71,6 +71,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/pipeline \ @TOPDIR@/lib/port \ @TOPDIR@/lib/power \ + @TOPDIR@/lib/ptr_compress \ @TOPDIR@/lib/rawdev \ @TOPDIR@/lib/rcu \ @TOPDIR@/lib/regexdev \ diff --git a/doc/guides/rel_notes/release_24_03.rst b/doc/guides/rel_notes/release_24_03.rst index 932688ca4d..b82b8c5c0b 100644 --- a/doc/guides/rel_notes/release_24_03.rst +++ b/doc/guides/rel_notes/release_24_03.rst @@ -176,6 +176,12 @@ New Features * Added power-saving during polling within the ``rte_event_dequeue_burst()`` API. * Added support for DMA adapter. +* **Introduced pointer compression library.** + + Library provides functions to compress and decompress arrays of pointers + which can improve application performance under certain conditions. + Performance test was added to help users evaluate performance on their setup. + Removed Items ------------- diff --git a/lib/meson.build b/lib/meson.build index e4e31f7ecf..fe43d137d7 100644 --- a/lib/meson.build +++ b/lib/meson.build @@ -14,6 +14,7 @@ libraries = [ 'argparse', 'telemetry', # basic info querying 'eal', # everything depends on eal + 'ptr_compress', 'ring', 'rcu', # rcu depends on ring 'mempool', diff --git a/lib/ptr_compress/meson.build b/lib/ptr_compress/meson.build new file mode 100644 index 0000000000..e92706a45f --- /dev/null +++ b/lib/ptr_compress/meson.build @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2024 Arm Limited + +headers = files('rte_ptr_compress.h') diff --git a/lib/ptr_compress/rte_ptr_compress.h b/lib/ptr_compress/rte_ptr_compress.h new file mode 100644 index 0000000000..97c084003d --- /dev/null +++ b/lib/ptr_compress/rte_ptr_compress.h @@ -0,0 +1,266 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2024 Arm Limited + */ + +#ifndef RTE_PTR_COMPRESS_H +#define RTE_PTR_COMPRESS_H + +/** + * @file + * Pointer compression and decompression functions. + * + * When passing arrays full of pointers between threads, memory containing + * the pointers is copied multiple times which is especially costly between + * cores. These functions allow us to compress the pointers. + * + * Compression takes advantage of the fact that pointers are usually located in + * a limited memory region (like a mempool). We compress them by converting them + * to offsets from a base memory address. Offsets can be stored in fewer bytes. + * + * The compression functions come in two varieties: 32-bit and 16-bit. + * + * To determine how many bits are needed to compress the pointer calculate + * the biggest offset possible (highest value pointer - base pointer) + * and shift the value right according to alignment (shift by exponent of the + * power of 2 of alignment: aligned by 4 - shift by 2, aligned by 8 - shift by + * 3, etc.). The resulting value must fit in either 32 or 16 bits. + * + * For usage example and further explanation please see "Pointer Compression" in + * doc/guides/prog_guide/ptr_compress_lib.rst + */ + +#include +#include + +#include +#include +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/** + * Compress pointers into 32-bit offsets from base pointer. + * + * @note It is programmer's responsibility to ensure the resulting offsets fit + * into 32 bits. Alignment of the structures pointed to by the pointers allows + * us to drop bits from the offsets. This is controlled by the bit_shift + * parameter. This means that if structures are aligned by 8 bytes they must be + * within 32GB of the base pointer. If there is no such alignment guarantee they + * must be within 4GB. + * + * @param ptr_base + * A pointer used to calculate offsets of pointers in src_table. + * @param src_table + * A pointer to an array of pointers. + * @param dest_table + * A pointer to an array of compressed pointers returned by this function. + * @param n + * The number of objects to compress, must be strictly positive. + * @param bit_shift + * Byte alignment of memory pointed to by the pointers allows for + * bits to be dropped from the offset and hence widen the memory region that + * can be covered. This controls how many bits are right shifted. + **/ +static __rte_always_inline void +rte_ptr_compress_32(void *ptr_base, void **src_table, + uint32_t *dest_table, size_t n, uint8_t bit_shift) +{ + unsigned int i = 0; +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32 + svuint64_t v_ptr_table; + svbool_t pg = svwhilelt_b64(i, n); + do { + v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i); + v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base); + v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift); + svst1w(pg, &dest_table[i], v_ptr_table); + i += svcntd(); + pg = svwhilelt_b64(i, n); + } while (i < n); +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32 + uint64_t ptr_diff; + uint64x2_t v_ptr_table; + /* right shift is done by left shifting by negative int */ + int64x2_t v_shift = vdupq_n_s64(-bit_shift); + uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base); + for (; i < (n & ~0x1); i += 2) { + v_ptr_table = vld1q_u64((const uint64_t *)src_table + i); + v_ptr_table = vsubq_u64(v_ptr_table, v_ptr_base); + v_ptr_table = vshlq_u64(v_ptr_table, v_shift); + vst1_u32(dest_table + i, vqmovn_u64(v_ptr_table)); + } + /* process leftover single item in case of odd number of n */ + if (unlikely(n & 0x1)) { + ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base); + dest_table[i] = (uint32_t) (ptr_diff >> bit_shift); + } +#else + uintptr_t ptr_diff; + for (; i < n; i++) { + ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base); + ptr_diff = ptr_diff >> bit_shift; + RTE_ASSERT(ptr_diff <= UINT32_MAX); + dest_table[i] = (uint32_t) ptr_diff; + } +#endif +} + +/** + * Decompress pointers from 32-bit offsets from base pointer. + * + * @param ptr_base + * A pointer which was used to calculate offsets in src_table. + * @param src_table + * A pointer to an array to compressed pointers. + * @param dest_table + * A pointer to an array of decompressed pointers returned by this function. + * @param n + * The number of objects to decompress, must be strictly positive. + * @param bit_shift + * Byte alignment of memory pointed to by the pointers allows for + * bits to be dropped from the offset and hence widen the memory region that + * can be covered. This controls how many bits are left shifted when pointers + * are recovered from the offsets. + **/ +static __rte_always_inline void +rte_ptr_decompress_32(void *ptr_base, uint32_t *src_table, + void **dest_table, size_t n, uint8_t bit_shift) +{ + unsigned int i = 0; +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32 + svuint64_t v_ptr_table; + svbool_t pg = svwhilelt_b64(i, n); + do { + v_ptr_table = svld1uw_u64(pg, &src_table[i]); + v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift); + v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base); + svst1(pg, (uint64_t *)dest_table + i, v_ptr_table); + i += svcntd(); + pg = svwhilelt_b64(i, n); + } while (i < n); +#elif defined __ARM_NEON && !defined RTE_ARCH_ARMv8_AARCH32 + uint64_t ptr_diff; + uint64x2_t v_ptr_table; + int64x2_t v_shift = vdupq_n_s64(bit_shift); + uint64x2_t v_ptr_base = vdupq_n_u64((uint64_t)ptr_base); + for (; i < (n & ~0x1); i += 2) { + v_ptr_table = vmovl_u32(vld1_u32(src_table + i)); + v_ptr_table = vshlq_u64(v_ptr_table, v_shift); + v_ptr_table = vaddq_u64(v_ptr_table, v_ptr_base); + vst1q_u64((uint64_t *)dest_table + i, v_ptr_table); + } + /* process leftover single item in case of odd number of n */ + if (unlikely(n & 0x1)) { + ptr_diff = ((uint64_t) src_table[i]) << bit_shift; + dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff); + } +#else + uintptr_t ptr_diff; + for (; i < n; i++) { + ptr_diff = ((uintptr_t) src_table[i]) << bit_shift; + dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff); + } +#endif +} + +/** + * Compress pointers into 16-bit offsets from base pointer. + * + * @note It is programmer's responsibility to ensure the resulting offsets fit + * into 16 bits. Alignment of the structures pointed to by the pointers allows + * us to drop bits from the offsets. This is controlled by the bit_shift + * parameter. This means that if structures are aligned by 8 bytes they must be + * within 256KB of the base pointer. If there is no such alignment guarantee + * they must be within 64KB. + * + * @param ptr_base + * A pointer used to calculate offsets of pointers in src_table. + * @param src_table + * A pointer to an array of pointers. + * @param dest_table + * A pointer to an array of compressed pointers returned by this function. + * @param n + * The number of objects to compress, must be strictly positive. + * @param bit_shift + * Byte alignment of memory pointed to by the pointers allows for + * bits to be dropped from the offset and hence widen the memory region that + * can be covered. This controls how many bits are right shifted. + **/ +static __rte_always_inline void +rte_ptr_compress_16(void *ptr_base, void **src_table, + uint16_t *dest_table, size_t n, uint8_t bit_shift) +{ + + unsigned int i = 0; +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32 + svuint64_t v_ptr_table; + svbool_t pg = svwhilelt_b64(i, n); + do { + v_ptr_table = svld1_u64(pg, (uint64_t *)src_table + i); + v_ptr_table = svsub_x(pg, v_ptr_table, (uint64_t)ptr_base); + v_ptr_table = svlsr_x(pg, v_ptr_table, bit_shift); + svst1h(pg, &dest_table[i], v_ptr_table); + i += svcntd(); + pg = svwhilelt_b64(i, n); + } while (i < n); +#else + uintptr_t ptr_diff; + for (; i < n; i++) { + ptr_diff = RTE_PTR_DIFF(src_table[i], ptr_base); + ptr_diff = ptr_diff >> bit_shift; + RTE_ASSERT(ptr_diff <= UINT16_MAX); + dest_table[i] = (uint16_t) ptr_diff; + } +#endif +} + +/** + * Decompress pointers from 16-bit offsets from base pointer. + * + * @param ptr_base + * A pointer which was used to calculate offsets in src_table. + * @param src_table + * A pointer to an array to compressed pointers. + * @param dest_table + * A pointer to an array of decompressed pointers returned by this function. + * @param n + * The number of objects to decompress, must be strictly positive. + * @param bit_shift + * Byte alignment of memory pointed to by the pointers allows for + * bits to be dropped from the offset and hence widen the memory region that + * can be covered. This controls how many bits are left shifted when pointers + * are recovered from the offsets. + **/ +static __rte_always_inline void +rte_ptr_decompress_16(void *ptr_base, uint16_t *src_table, + void **dest_table, size_t n, uint8_t bit_shift) +{ + unsigned int i = 0; +#if defined RTE_HAS_SVE_ACLE && !defined RTE_ARCH_ARMv8_AARCH32 + svuint64_t v_ptr_table; + svbool_t pg = svwhilelt_b64(i, n); + do { + v_ptr_table = svld1uh_u64(pg, &src_table[i]); + v_ptr_table = svlsl_x(pg, v_ptr_table, bit_shift); + v_ptr_table = svadd_x(pg, v_ptr_table, (uint64_t)ptr_base); + svst1(pg, (uint64_t *)dest_table + i, v_ptr_table); + i += svcntd(); + pg = svwhilelt_b64(i, n); + } while (i < n); +#else + uintptr_t ptr_diff; + for (; i < n; i++) { + ptr_diff = ((uintptr_t) src_table[i]) << bit_shift; + dest_table[i] = RTE_PTR_ADD(ptr_base, ptr_diff); + } +#endif +} + +#ifdef __cplusplus +} +#endif + +#endif /* RTE_PTR_COMPRESS_H */ From patchwork Mon Mar 11 14:47:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Szczepanek X-Patchwork-Id: 138155 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8EC2743C88; Mon, 11 Mar 2024 15:47:51 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DCFF940A7A; Mon, 11 Mar 2024 15:47:34 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 44FBD4026B for ; Mon, 11 Mar 2024 15:47:30 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 789A51570; Mon, 11 Mar 2024 07:48:06 -0700 (PDT) Received: from ampere-altra-2-1.usa.Arm.com (ampere-altra-2-1.usa.arm.com [10.118.91.158]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 68B473F762; Mon, 11 Mar 2024 07:47:29 -0700 (PDT) From: Paul Szczepanek To: dev@dpdk.org Cc: bruce.richardson@intel.com, Paul Szczepanek , Honnappa Nagarahalli , Nathan Brown Subject: [PATCH v9 3/5] test: add pointer compress tests to ring perf test Date: Mon, 11 Mar 2024 14:47:04 +0000 Message-Id: <20240311144706.204831-4-paul.szczepanek@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240311144706.204831-1-paul.szczepanek@arm.com> References: <20230927150854.3670391-2-paul.szczepanek@arm.com> <20240311144706.204831-1-paul.szczepanek@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add a test that runs a zero copy burst enqueue and dequeue on a ring of raw pointers and compressed pointers at different burst sizes to showcase performance benefits of newly added pointer compression APIs. Refactored threading code to pass more parameters to threads to reuse existing code. Added more bulk sizes to showcase their effects on compression. Adjusted loop iteration numbers to take into account bulk sizes to keep runtime constant (instead of number of operations). Adjusted old printfs to match new ones which have aligned numbers. Signed-off-by: Paul Szczepanek Reviewed-by: Honnappa Nagarahalli Reviewed-by: Nathan Brown --- app/test/meson.build | 20 +-- app/test/test_ring.h | 92 ++++++++++ app/test/test_ring_perf.c | 352 +++++++++++++++++++++++++------------- 3 files changed, 332 insertions(+), 132 deletions(-) -- 2.25.1 diff --git a/app/test/meson.build b/app/test/meson.build index 7d909039ae..df8cc00730 100644 --- a/app/test/meson.build +++ b/app/test/meson.build @@ -156,16 +156,16 @@ source_file_deps = { # 'test_resource.c': [], 'test_rib.c': ['net', 'rib'], 'test_rib6.c': ['net', 'rib'], - 'test_ring.c': [], - 'test_ring_hts_stress.c': [], - 'test_ring_mpmc_stress.c': [], - 'test_ring_mt_peek_stress.c': [], - 'test_ring_mt_peek_stress_zc.c': [], - 'test_ring_perf.c': [], - 'test_ring_rts_stress.c': [], - 'test_ring_st_peek_stress.c': [], - 'test_ring_st_peek_stress_zc.c': [], - 'test_ring_stress.c': [], + 'test_ring.c': ['ptr_compress'], + 'test_ring_hts_stress.c': ['ptr_compress'], + 'test_ring_mpmc_stress.c': ['ptr_compress'], + 'test_ring_mt_peek_stress.c': ['ptr_compress'], + 'test_ring_mt_peek_stress_zc.c': ['ptr_compress'], + 'test_ring_perf.c': ['ptr_compress'], + 'test_ring_rts_stress.c': ['ptr_compress'], + 'test_ring_st_peek_stress.c': ['ptr_compress'], + 'test_ring_st_peek_stress_zc.c': ['ptr_compress'], + 'test_ring_stress.c': ['ptr_compress'], 'test_rwlock.c': [], 'test_sched.c': ['net', 'sched'], 'test_security.c': ['net', 'security'], diff --git a/app/test/test_ring.h b/app/test/test_ring.h index 45c263f3ff..f90662818c 100644 --- a/app/test/test_ring.h +++ b/app/test/test_ring.h @@ -5,6 +5,8 @@ #include #include #include +#include +#include /* API type to call * rte_ring__enqueue_ @@ -25,6 +27,10 @@ #define TEST_RING_ELEM_BULK 16 #define TEST_RING_ELEM_BURST 32 +#define TEST_RING_ELEM_BURST_ZC 64 +#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 128 +#define TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32 256 + #define TEST_RING_IGNORE_API_TYPE ~0U /* This function is placed here as it is required for both @@ -101,6 +107,9 @@ static inline unsigned int test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n, unsigned int api_type) { + unsigned int ret; + struct rte_ring_zc_data zcd = {0}; + /* Legacy queue APIs? */ if (esize == -1) switch (api_type) { @@ -152,6 +161,46 @@ test_ring_enqueue(struct rte_ring *r, void **obj, int esize, unsigned int n, case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST): return rte_ring_mp_enqueue_burst_elem(r, obj, esize, n, NULL); + case (TEST_RING_ELEM_BURST_ZC): + ret = rte_ring_enqueue_zc_burst_elem_start( + r, esize, n, &zcd, NULL); + if (unlikely(ret == 0)) + return 0; + rte_memcpy(zcd.ptr1, (char *)obj, zcd.n1 * esize); + if (unlikely(zcd.ptr2 != NULL)) + rte_memcpy(zcd.ptr2, + (char *)obj + zcd.n1 * esize, + (ret - zcd.n1) * esize); + rte_ring_enqueue_zc_finish(r, ret); + return ret; + case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16): + /* rings cannot store uint16_t so we use a uint32_t + * and half the requested number of elements + * and compensate by doubling the returned numbers + */ + ret = rte_ring_enqueue_zc_burst_elem_start( + r, sizeof(uint32_t), n / 2, &zcd, NULL); + if (unlikely(ret == 0)) + return 0; + rte_ptr_compress_16(0, obj, zcd.ptr1, zcd.n1 * 2, 3); + if (unlikely(zcd.ptr2 != NULL)) + rte_ptr_compress_16(0, + obj + (zcd.n1 * 2), + zcd.ptr2, + (ret - zcd.n1) * 2, 3); + rte_ring_enqueue_zc_finish(r, ret); + return ret * 2; + case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32): + ret = rte_ring_enqueue_zc_burst_elem_start( + r, sizeof(uint32_t), n, &zcd, NULL); + if (unlikely(ret == 0)) + return 0; + rte_ptr_compress_32(0, obj, zcd.ptr1, zcd.n1, 3); + if (unlikely(zcd.ptr2 != NULL)) + rte_ptr_compress_32(0, obj + zcd.n1, + zcd.ptr2, ret - zcd.n1, 3); + rte_ring_enqueue_zc_finish(r, ret); + return ret; default: printf("Invalid API type\n"); return 0; @@ -162,6 +211,9 @@ static inline unsigned int test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n, unsigned int api_type) { + unsigned int ret; + struct rte_ring_zc_data zcd = {0}; + /* Legacy queue APIs? */ if (esize == -1) switch (api_type) { @@ -213,6 +265,46 @@ test_ring_dequeue(struct rte_ring *r, void **obj, int esize, unsigned int n, case (TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BURST): return rte_ring_mc_dequeue_burst_elem(r, obj, esize, n, NULL); + case (TEST_RING_ELEM_BURST_ZC): + ret = rte_ring_dequeue_zc_burst_elem_start( + r, esize, n, &zcd, NULL); + if (unlikely(ret == 0)) + return 0; + rte_memcpy((char *)obj, zcd.ptr1, zcd.n1 * esize); + if (unlikely(zcd.ptr2 != NULL)) + rte_memcpy((char *)obj + zcd.n1 * esize, + zcd.ptr2, + (ret - zcd.n1) * esize); + rte_ring_dequeue_zc_finish(r, ret); + return ret; + case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16): + /* rings cannot store uint16_t so we use a uint32_t + * and half the requested number of elements + * and compensate by doubling the returned numbers + */ + ret = rte_ring_dequeue_zc_burst_elem_start( + r, sizeof(uint32_t), n / 2, &zcd, NULL); + if (unlikely(ret == 0)) + return 0; + rte_ptr_decompress_16(0, zcd.ptr1, obj, zcd.n1 * 2, 3); + if (unlikely(zcd.ptr2 != NULL)) + rte_ptr_decompress_16(0, zcd.ptr2, + obj + zcd.n1, + (ret - zcd.n1) * 2, + 3); + rte_ring_dequeue_zc_finish(r, ret); + return ret * 2; + case (TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32): + ret = rte_ring_dequeue_zc_burst_elem_start( + r, sizeof(uint32_t), n, &zcd, NULL); + if (unlikely(ret == 0)) + return 0; + rte_ptr_decompress_32(0, zcd.ptr1, obj, zcd.n1, 3); + if (unlikely(zcd.ptr2 != NULL)) + rte_ptr_decompress_32(0, zcd.ptr2, + obj + zcd.n1, ret - zcd.n1, 3); + rte_ring_dequeue_zc_finish(r, ret); + return ret; default: printf("Invalid API type\n"); return 0; diff --git a/app/test/test_ring_perf.c b/app/test/test_ring_perf.c index d7c5a4c30b..367670fd09 100644 --- a/app/test/test_ring_perf.c +++ b/app/test/test_ring_perf.c @@ -22,13 +22,13 @@ #define RING_NAME "RING_PERF" #define RING_SIZE 4096 -#define MAX_BURST 32 +#define MAX_BURST 256 /* * the sizes to enqueue and dequeue in testing * (marked volatile so they won't be seen as compile-time constants) */ -static const volatile unsigned bulk_sizes[] = { 8, 32 }; +static const volatile unsigned int bulk_sizes[] = { 8, 32, 64, 128, 256 }; struct lcore_pair { unsigned c1, c2; @@ -43,26 +43,30 @@ test_ring_print_test_string(unsigned int api_type, int esize, if (esize == -1) printf("legacy APIs"); else - printf("elem APIs: element size %dB", esize); + printf("elem APIs (size:%2dB)", esize); if (api_type == TEST_RING_IGNORE_API_TYPE) return; if ((api_type & TEST_RING_THREAD_DEF) == TEST_RING_THREAD_DEF) - printf(": default enqueue/dequeue: "); + printf(" - default enqueue/dequeue"); else if ((api_type & TEST_RING_THREAD_SPSC) == TEST_RING_THREAD_SPSC) - printf(": SP/SC: "); + printf(" - SP/SC"); else if ((api_type & TEST_RING_THREAD_MPMC) == TEST_RING_THREAD_MPMC) - printf(": MP/MC: "); + printf(" - MP/MC"); if ((api_type & TEST_RING_ELEM_SINGLE) == TEST_RING_ELEM_SINGLE) - printf("single: "); + printf(" - single - "); else if ((api_type & TEST_RING_ELEM_BULK) == TEST_RING_ELEM_BULK) - printf("bulk (size: %u): ", bsz); + printf(" - bulk (n:%-3u) - ", bsz); else if ((api_type & TEST_RING_ELEM_BURST) == TEST_RING_ELEM_BURST) - printf("burst (size: %u): ", bsz); + printf(" - burst (n:%-3u) - ", bsz); + else if ((api_type & (TEST_RING_ELEM_BURST_ZC | + TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16 | + TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32)) != 0) + printf(" - burst zero copy (n:%-3u) - ", bsz); - printf("%.2F\n", value); + printf("cycles per elem: %.3F\n", value); } /**** Functions to analyse our core mask to get cores for different tests ***/ @@ -144,28 +148,33 @@ static void test_empty_dequeue(struct rte_ring *r, const int esize, const unsigned int api_type) { - const unsigned int iter_shift = 26; + const unsigned int iter_shift = 29; const unsigned int iterations = 1 << iter_shift; unsigned int i = 0; void *burst[MAX_BURST]; + const unsigned int bulk_iterations = iterations / bulk_sizes[0]; const uint64_t start = rte_rdtsc(); - for (i = 0; i < iterations; i++) + for (i = 0; i < bulk_iterations; i++) test_ring_dequeue(r, burst, esize, bulk_sizes[0], api_type); const uint64_t end = rte_rdtsc(); test_ring_print_test_string(api_type, esize, bulk_sizes[0], - ((double)(end - start)) / iterations); + ((double)end - start) / iterations); } -/* - * for the separate enqueue and dequeue threads they take in one param - * and return two. Input = burst size, output = cycle average for sp/sc & mp/mc - */ -struct thread_params { +/* describes the ring used by the enqueue and dequeue thread */ +struct ring_params { struct rte_ring *r; - unsigned size; /* input value, the burst size */ - double spsc, mpmc; /* output value, the single or multi timings */ + unsigned int elem_size; + unsigned int bulk_sizes_i; /* index into bulk_size array */ + unsigned int ring_flags; /* flags for test_ring_enqueue/dequeue */ +}; + +/* Used to specify enqueue and dequeue ring operations and their results */ +struct thread_params { + struct ring_params *ring_params; + double *results; /* result array size must be equal to bulk_sizes */ }; /* @@ -174,16 +183,15 @@ struct thread_params { * flag == 1 -> dequeue */ static __rte_always_inline int -enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize, - struct thread_params *p) +enqueue_dequeue_bulk_helper(const unsigned int flag, struct thread_params *p) { int ret; - const unsigned int iter_shift = 15; + const unsigned int iter_shift = 22; const unsigned int iterations = 1 << iter_shift; - struct rte_ring *r = p->r; - unsigned int bsize = p->size; unsigned int i; void *burst = NULL; + unsigned int n_remaining; + const unsigned int bulk_n = bulk_sizes[p->ring_params->bulk_sizes_i]; #ifdef RTE_USE_C11_MEM_MODEL if (__atomic_fetch_add(&lcore_count, 1, __ATOMIC_RELAXED) + 1 != 2) @@ -193,44 +201,38 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize, while(lcore_count != 2) rte_pause(); - burst = test_ring_calloc(MAX_BURST, esize); + burst = test_ring_calloc(MAX_BURST, p->ring_params->elem_size); if (burst == NULL) return -1; const uint64_t sp_start = rte_rdtsc(); - for (i = 0; i < iterations; i++) + const unsigned int bulk_iterations = iterations / bulk_n; + for (i = 0; i < bulk_iterations; i++) { + n_remaining = bulk_n; do { if (flag == 0) - ret = test_ring_enqueue(r, burst, esize, bsize, - TEST_RING_THREAD_SPSC | - TEST_RING_ELEM_BULK); + ret = test_ring_enqueue(p->ring_params->r, + burst, + p->ring_params->elem_size, + n_remaining, + p->ring_params->ring_flags); else if (flag == 1) - ret = test_ring_dequeue(r, burst, esize, bsize, - TEST_RING_THREAD_SPSC | - TEST_RING_ELEM_BULK); + ret = test_ring_dequeue(p->ring_params->r, + burst, + p->ring_params->elem_size, + n_remaining, + p->ring_params->ring_flags); if (ret == 0) rte_pause(); - } while (!ret); + else + n_remaining -= ret; + } while (n_remaining > 0); + } const uint64_t sp_end = rte_rdtsc(); - const uint64_t mp_start = rte_rdtsc(); - for (i = 0; i < iterations; i++) - do { - if (flag == 0) - ret = test_ring_enqueue(r, burst, esize, bsize, - TEST_RING_THREAD_MPMC | - TEST_RING_ELEM_BULK); - else if (flag == 1) - ret = test_ring_dequeue(r, burst, esize, bsize, - TEST_RING_THREAD_MPMC | - TEST_RING_ELEM_BULK); - if (ret == 0) - rte_pause(); - } while (!ret); - const uint64_t mp_end = rte_rdtsc(); + p->results[p->ring_params->bulk_sizes_i] = + ((double)sp_end - sp_start) / iterations; - p->spsc = ((double)(sp_end - sp_start))/(iterations * bsize); - p->mpmc = ((double)(mp_end - mp_start))/(iterations * bsize); return 0; } @@ -243,15 +245,7 @@ enqueue_bulk(void *p) { struct thread_params *params = p; - return enqueue_dequeue_bulk_helper(0, -1, params); -} - -static int -enqueue_bulk_16B(void *p) -{ - struct thread_params *params = p; - - return enqueue_dequeue_bulk_helper(0, 16, params); + return enqueue_dequeue_bulk_helper(0, params); } /* @@ -263,15 +257,7 @@ dequeue_bulk(void *p) { struct thread_params *params = p; - return enqueue_dequeue_bulk_helper(1, -1, params); -} - -static int -dequeue_bulk_16B(void *p) -{ - struct thread_params *params = p; - - return enqueue_dequeue_bulk_helper(1, 16, params); + return enqueue_dequeue_bulk_helper(1, params); } /* @@ -279,42 +265,32 @@ dequeue_bulk_16B(void *p) * used to measure ring perf between hyperthreads, cores and sockets. */ static int -run_on_core_pair(struct lcore_pair *cores, struct rte_ring *r, const int esize) +run_on_core_pair(struct lcore_pair *cores, + struct thread_params *param1, struct thread_params *param2) { - lcore_function_t *f1, *f2; - struct thread_params param1 = {0}, param2 = {0}; unsigned i; - - if (esize == -1) { - f1 = enqueue_bulk; - f2 = dequeue_bulk; - } else { - f1 = enqueue_bulk_16B; - f2 = dequeue_bulk_16B; - } + struct ring_params *ring_params = param1->ring_params; for (i = 0; i < RTE_DIM(bulk_sizes); i++) { lcore_count = 0; - param1.size = param2.size = bulk_sizes[i]; - param1.r = param2.r = r; + ring_params->bulk_sizes_i = i; if (cores->c1 == rte_get_main_lcore()) { - rte_eal_remote_launch(f2, ¶m2, cores->c2); - f1(¶m1); + rte_eal_remote_launch(dequeue_bulk, param2, cores->c2); + enqueue_bulk(param1); rte_eal_wait_lcore(cores->c2); } else { - rte_eal_remote_launch(f1, ¶m1, cores->c1); - rte_eal_remote_launch(f2, ¶m2, cores->c2); + rte_eal_remote_launch(enqueue_bulk, param1, cores->c1); + rte_eal_remote_launch(dequeue_bulk, param2, cores->c2); if (rte_eal_wait_lcore(cores->c1) < 0) return -1; if (rte_eal_wait_lcore(cores->c2) < 0) return -1; } test_ring_print_test_string( - TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK, - esize, bulk_sizes[i], param1.spsc + param2.spsc); - test_ring_print_test_string( - TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK, - esize, bulk_sizes[i], param1.mpmc + param2.mpmc); + ring_params->ring_flags, + ring_params->elem_size, + bulk_sizes[i], + param1->results[i] + param2->results[i]); } return 0; @@ -333,7 +309,7 @@ load_loop_fn_helper(struct thread_params *p, const int esize) uint64_t hz = rte_get_timer_hz(); uint64_t lcount = 0; const unsigned int lcore = rte_lcore_id(); - struct thread_params *params = p; + struct ring_params *ring_params = p->ring_params; void *burst = NULL; burst = test_ring_calloc(MAX_BURST, esize); @@ -346,9 +322,11 @@ load_loop_fn_helper(struct thread_params *p, const int esize) begin = rte_get_timer_cycles(); while (time_diff < hz * TIME_MS / 1000) { - test_ring_enqueue(params->r, burst, esize, params->size, + test_ring_enqueue(ring_params->r, burst, esize, + ring_params->elem_size, TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK); - test_ring_dequeue(params->r, burst, esize, params->size, + test_ring_dequeue(ring_params->r, burst, esize, + ring_params->elem_size, TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK); lcount++; time_diff = rte_get_timer_cycles() - begin; @@ -380,7 +358,8 @@ static int run_on_all_cores(struct rte_ring *r, const int esize) { uint64_t total; - struct thread_params param; + struct ring_params ring_params = {0}; + struct thread_params params = { .ring_params = &ring_params }; lcore_function_t *lcore_f; unsigned int i, c; @@ -389,21 +368,20 @@ run_on_all_cores(struct rte_ring *r, const int esize) else lcore_f = load_loop_fn_16B; - memset(¶m, 0, sizeof(struct thread_params)); for (i = 0; i < RTE_DIM(bulk_sizes); i++) { total = 0; printf("\nBulk enq/dequeue count on size %u\n", bulk_sizes[i]); - param.size = bulk_sizes[i]; - param.r = r; + params.ring_params->bulk_sizes_i = i; + params.ring_params->r = r; /* clear synchro and start workers */ __atomic_store_n(&synchro, 0, __ATOMIC_RELAXED); - if (rte_eal_mp_remote_launch(lcore_f, ¶m, SKIP_MAIN) < 0) + if (rte_eal_mp_remote_launch(lcore_f, ¶ms, SKIP_MAIN) < 0) return -1; /* start synchro and launch test on main */ __atomic_store_n(&synchro, 1, __ATOMIC_RELAXED); - lcore_f(¶m); + lcore_f(¶ms); rte_eal_mp_wait_lcore(); @@ -462,9 +440,9 @@ static int test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize, const unsigned int api_type) { - const unsigned int iter_shift = 23; + const unsigned int iter_shift = 26; const unsigned int iterations = 1 << iter_shift; - unsigned int sz, i = 0; + unsigned int sz, i; void **burst = NULL; burst = test_ring_calloc(MAX_BURST, esize); @@ -472,17 +450,18 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize, return -1; for (sz = 0; sz < RTE_DIM(bulk_sizes); sz++) { + const unsigned int n = iterations / bulk_sizes[sz]; const uint64_t start = rte_rdtsc(); - for (i = 0; i < iterations; i++) { + for (i = 0; i < n; i++) { test_ring_enqueue(r, burst, esize, bulk_sizes[sz], - api_type); + api_type); test_ring_dequeue(r, burst, esize, bulk_sizes[sz], - api_type); + api_type); } const uint64_t end = rte_rdtsc(); test_ring_print_test_string(api_type, esize, bulk_sizes[sz], - ((double)(end - start)) / iterations); + ((double)end - start) / iterations); } rte_free(burst); @@ -490,12 +469,43 @@ test_burst_bulk_enqueue_dequeue(struct rte_ring *r, const int esize, return 0; } +static __rte_always_inline int +test_ring_perf_esize_run_on_two_cores( + struct thread_params *param1, struct thread_params *param2) +{ + struct lcore_pair cores; + + if (get_two_hyperthreads(&cores) == 0) { + printf("\n### Testing using two hyperthreads ###\n"); + if (run_on_core_pair(&cores, param1, param2) < 0) + return -1; + } + if (get_two_cores(&cores) == 0) { + printf("\n### Testing using two physical cores ###\n"); + if (run_on_core_pair(&cores, param1, param2) < 0) + return -1; + } + if (get_two_sockets(&cores) == 0) { + printf("\n### Testing using two NUMA nodes ###\n"); + if (run_on_core_pair(&cores, param1, param2) < 0) + return -1; + } + return 0; +} + /* Run all tests for a given element size */ static __rte_always_inline int test_ring_perf_esize(const int esize) { - struct lcore_pair cores; struct rte_ring *r = NULL; + double results_enq[RTE_DIM(bulk_sizes)]; + double results_deq[RTE_DIM(bulk_sizes)]; + struct ring_params ring_params = { + .elem_size = esize, .ring_flags = TEST_RING_ELEM_BULK }; + struct thread_params param1 = { + .ring_params = &ring_params, .results = results_enq }; + struct thread_params param2 = { + .ring_params = &ring_params, .results = results_deq }; /* * Performance test for legacy/_elem APIs @@ -535,22 +545,13 @@ test_ring_perf_esize(const int esize) test_empty_dequeue(r, esize, TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK); - if (get_two_hyperthreads(&cores) == 0) { - printf("\n### Testing using two hyperthreads ###\n"); - if (run_on_core_pair(&cores, r, esize) < 0) - goto test_fail; - } + ring_params.r = r; - if (get_two_cores(&cores) == 0) { - printf("\n### Testing using two physical cores ###\n"); - if (run_on_core_pair(&cores, r, esize) < 0) - goto test_fail; - } - if (get_two_sockets(&cores) == 0) { - printf("\n### Testing using two NUMA nodes ###\n"); - if (run_on_core_pair(&cores, r, esize) < 0) - goto test_fail; - } + ring_params.ring_flags = TEST_RING_THREAD_SPSC | TEST_RING_ELEM_BULK; + test_ring_perf_esize_run_on_two_cores(¶m1, ¶m2); + + ring_params.ring_flags = TEST_RING_THREAD_MPMC | TEST_RING_ELEM_BULK; + test_ring_perf_esize_run_on_two_cores(¶m1, ¶m2); printf("\n### Testing using all worker nodes ###\n"); if (run_on_all_cores(r, esize) < 0) @@ -566,6 +567,109 @@ test_ring_perf_esize(const int esize) return -1; } + +static __rte_always_inline int +test_ring_perf_compression(void) +{ + double results1[RTE_DIM(bulk_sizes)]; + double results2[RTE_DIM(bulk_sizes)]; + double results1_comp[2][RTE_DIM(bulk_sizes)]; + double results2_comp[2][RTE_DIM(bulk_sizes)]; + + struct lcore_pair cores; + int ret = -1; + unsigned int i, j; + struct ring_params ring_params = { .elem_size = sizeof(void *) }; + struct thread_params param1 = { + .ring_params = &ring_params, .results = results1 }; + struct thread_params param2 = { + .ring_params = &ring_params, .results = results2 }; + + printf("\n### Testing compression gain ###"); + + ring_params.r = rte_ring_create_elem( + RING_NAME, sizeof(void *), + RING_SIZE, rte_socket_id(), + RING_F_SP_ENQ | RING_F_SC_DEQ); + + if (ring_params.r == NULL) + return -1; + + if (get_two_cores(&cores) == 0) { + printf("\n### Testing zero copy ###\n"); + ring_params.ring_flags = TEST_RING_ELEM_BURST_ZC; + ret = run_on_core_pair(&cores, ¶m1, ¶m2); + } + + rte_ring_free(ring_params.r); + + if (ret != 0) + return ret; + + /* rings allow only multiples of 4 as sizes, + * we allocate size 4 despite only using 2 bytes + * and use half of RING_SIZE as the number of elements + */ + ring_params.r = rte_ring_create_elem( + RING_NAME, sizeof(uint32_t), + RING_SIZE / 2, rte_socket_id(), + RING_F_SP_ENQ | RING_F_SC_DEQ); + + if (ring_params.r == NULL) + return -1; + + param1.results = results1_comp[0]; + param2.results = results2_comp[0]; + + if (get_two_cores(&cores) == 0) { + printf("\n### Testing zero copy with compression (16b) ###\n"); + ring_params.ring_flags = + TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_16; + ret = run_on_core_pair(&cores, ¶m1, ¶m2); + } + + rte_ring_free(ring_params.r); + + if (ret != 0) + return ret; + + ring_params.r = rte_ring_create_elem( + RING_NAME, sizeof(uint32_t), + RING_SIZE, rte_socket_id(), + RING_F_SP_ENQ | RING_F_SC_DEQ); + + if (ring_params.r == NULL) + return -1; + + param1.results = results1_comp[1]; + param2.results = results2_comp[1]; + + if (get_two_cores(&cores) == 0) { + printf("\n### Testing zero copy with compression (32b) ###\n"); + ring_params.ring_flags = + TEST_RING_ELEM_BURST_ZC_COMPRESS_PTR_32; + ret = run_on_core_pair(&cores, ¶m1, ¶m2); + } + + rte_ring_free(ring_params.r); + + for (j = 0; j < 2; j++) { + printf("\n### Potential gain from compression (%d-bit offsets) " + "###\n", (j + 1) * 16); + for (i = 0; i < RTE_DIM(bulk_sizes); i++) { + const double result = results1[i] + results2[i]; + const double result_comp = results1_comp[j][i] + + results2_comp[j][i]; + const double gain = 100 - (result_comp / result) * 100; + + printf("Gain of %5.1F%% for burst of %-3u elems\n", + gain, bulk_sizes[i]); + } + } + + return ret; +} + static int test_ring_perf(void) { @@ -576,6 +680,10 @@ test_ring_perf(void) if (test_ring_perf_esize(16) == -1) return -1; + /* Test for performance gain of compression */ + if (test_ring_perf_compression() == -1) + return -1; + return 0; } From patchwork Mon Mar 11 14:47:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Szczepanek X-Patchwork-Id: 138156 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C85EF43C88; Mon, 11 Mar 2024 15:47:57 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6BE7840A6C; Mon, 11 Mar 2024 15:47:36 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id 630384027C for ; Mon, 11 Mar 2024 15:47:30 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B320F1576; Mon, 11 Mar 2024 07:48:06 -0700 (PDT) Received: from ampere-altra-2-1.usa.Arm.com (ampere-altra-2-1.usa.arm.com [10.118.91.158]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A94BD3F762; Mon, 11 Mar 2024 07:47:29 -0700 (PDT) From: Paul Szczepanek To: dev@dpdk.org Cc: bruce.richardson@intel.com, Paul Szczepanek , Honnappa Nagarahalli , Nathan Brown Subject: [PATCH v9 4/5] docs: add pointer compression guide Date: Mon, 11 Mar 2024 14:47:05 +0000 Message-Id: <20240311144706.204831-5-paul.szczepanek@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240311144706.204831-1-paul.szczepanek@arm.com> References: <20230927150854.3670391-2-paul.szczepanek@arm.com> <20240311144706.204831-1-paul.szczepanek@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Documentation added in the prog guide for the new utility functions for pointer compression showing example code and potential usecases. Signed-off-by: Paul Szczepanek Reviewed-by: Honnappa Nagarahalli Reviewed-by: Nathan Brown --- MAINTAINERS | 1 + doc/guides/prog_guide/index.rst | 1 + doc/guides/prog_guide/ptr_compress_lib.rst | 142 +++++++++++++++++++++ 3 files changed, 144 insertions(+) create mode 100644 doc/guides/prog_guide/ptr_compress_lib.rst -- 2.25.1 diff --git a/MAINTAINERS b/MAINTAINERS index 6f703b1b13..e70f92cdc4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1688,6 +1688,7 @@ F: lib/pci/ Pointer Compression M: Paul Szczepanek F: lib/ptr_compress/ +F: doc/guides/prog_guide/ptr_compress_lib.rst Power management M: Anatoly Burakov diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index d09d958e6c..6366849eb0 100644 --- a/doc/guides/prog_guide/index.rst +++ b/doc/guides/prog_guide/index.rst @@ -73,6 +73,7 @@ Programmer's Guide telemetry_lib bpf_lib graph_lib + ptr_compress_lib build-sdk-meson meson_ut build_app diff --git a/doc/guides/prog_guide/ptr_compress_lib.rst b/doc/guides/prog_guide/ptr_compress_lib.rst new file mode 100644 index 0000000000..1f9ef24da7 --- /dev/null +++ b/doc/guides/prog_guide/ptr_compress_lib.rst @@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2024 Arm Limited. + +Pointer Compression Library +=========================== + +Use ``rte_ptr_compress_16()`` and ``rte_ptr_decompress_16()`` to compress and +decompress pointers into 16-bit offsets. Use ``rte_ptr_compress_32()`` and +``rte_ptr_decompress_32()`` to compress and decompress pointers into 32-bit +offsets. + +Compression takes advantage of the fact that pointers are usually located in a +limited memory region (like a mempool). By converting them to offsets from a +base memory address they can be stored in fewer bytes. How many bytes are needed +to store the offset is dictated by the memory region size and alignment of +objects the pointers point to. + +For example, a pointer which is part of a 4GB memory pool can be stored as 32 +bit offset. If the pointer points to memory that is 8 bytes aligned then 3 bits +can be dropped from the offset and a 32GB memory pool can now fit in 32 bits. + +For performance reasons these requirements are not enforced programmatically. +The programmer is responsible for ensuring that the combination of distance +from the base pointer and memory alignment allow for storing of the offset in +the number of bits indicated by the function name (16 or 32). Start of mempool +memory would be a good candidate for the base pointer. Otherwise any pointer +that precedes all pointers, is close enough and has the same alignment as the +pointers being compressed will work. + +.. note:: + + Performance gains depend on the batch size of pointers and CPU capabilities + such as vector extensions. It's important to measure the performance + increase on target hardware. A test called ``ring_perf_autotest`` in + ``dpdk-test`` can provide the measurements. + +Example usage +~~~~~~~~~~~~~ + +In this example we send pointers between two cores through a ring. While this +is a realistic use case the code is simplified for demonstration purposes and +does not have error handling. + +.. code-block:: c + + #include + #include + #include + #include + + #define ITEMS_ARRAY_SIZE (1024) + #define BATCH_SIZE (128) + #define ALIGN_EXPONENT (3) + #define ITEM_ALIGN (1< X-Patchwork-Id: 138157 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5BB0443C88; Mon, 11 Mar 2024 15:48:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A3BA740A8B; Mon, 11 Mar 2024 15:47:37 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mails.dpdk.org (Postfix) with ESMTP id B3640406B4 for ; Mon, 11 Mar 2024 15:47:30 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ED2671595; Mon, 11 Mar 2024 07:48:06 -0700 (PDT) Received: from ampere-altra-2-1.usa.Arm.com (ampere-altra-2-1.usa.arm.com [10.118.91.158]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E38AD3F762; Mon, 11 Mar 2024 07:47:29 -0700 (PDT) From: Paul Szczepanek To: dev@dpdk.org Cc: bruce.richardson@intel.com, Paul Szczepanek , Honnappa Nagarahalli , Nathan Brown Subject: [PATCH v9 5/5] test: add unit test for ptr compression Date: Mon, 11 Mar 2024 14:47:06 +0000 Message-Id: <20240311144706.204831-6-paul.szczepanek@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240311144706.204831-1-paul.szczepanek@arm.com> References: <20230927150854.3670391-2-paul.szczepanek@arm.com> <20240311144706.204831-1-paul.szczepanek@arm.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Test compresses and decompresses pointers with various combinations of memory regions and alignments and verify the pointers are recovered correctly. Signed-off-by: Paul Szczepanek Reviewed-by: Honnappa Nagarahalli Reviewed-by: Nathan Brown --- MAINTAINERS | 1 + app/test/meson.build | 1 + app/test/test_ptr_compress.c | 108 +++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) create mode 100644 app/test/test_ptr_compress.c -- 2.25.1 diff --git a/MAINTAINERS b/MAINTAINERS index e70f92cdc4..77ab8a2959 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1688,6 +1688,7 @@ F: lib/pci/ Pointer Compression M: Paul Szczepanek F: lib/ptr_compress/ +F: app/test/test_ptr_compress.c F: doc/guides/prog_guide/ptr_compress_lib.rst Power management diff --git a/app/test/meson.build b/app/test/meson.build index df8cc00730..e29258e6ec 100644 --- a/app/test/meson.build +++ b/app/test/meson.build @@ -144,6 +144,7 @@ source_file_deps = { 'test_power_intel_uncore.c': ['power'], 'test_power_kvm_vm.c': ['power'], 'test_prefetch.c': [], + 'test_ptr_compress.c': ['ptr_compress'], 'test_rand_perf.c': [], 'test_rawdev.c': ['rawdev', 'bus_vdev'], 'test_rcu_qsbr.c': ['rcu', 'hash'], diff --git a/app/test/test_ptr_compress.c b/app/test/test_ptr_compress.c new file mode 100644 index 0000000000..972f832776 --- /dev/null +++ b/app/test/test_ptr_compress.c @@ -0,0 +1,108 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2024 Arm Limited + */ + +#include "test.h" +#include +#include + +#include + +#define MAX_ALIGN_EXPONENT 3 +#define MAX_PTRS 16 +#define NUM_BASES 2 +#define NUM_REGIONS 4 +#define MAX_32BIT_REGION ((uint64_t)UINT32_MAX + 1) +#define MAX_16BIT_REGION (UINT16_MAX + 1) + +static int +test_ptr_compress_params( + void *base, + uint64_t mem_sz, + unsigned int align_exp, + unsigned int num_ptrs, + bool use_32_bit) +{ + unsigned int i; + unsigned int align = 1 << align_exp; + void *ptrs[MAX_PTRS] = {0}; + void *ptrs_out[MAX_PTRS] = {0}; + uint32_t offsets32[MAX_PTRS] = {0}; + uint16_t offsets16[MAX_PTRS] = {0}; + + for (i = 0; i < num_ptrs; i++) { + /* make pointers point at memory in steps of align */ + /* alternate steps from the start and end of memory region */ + if ((i & 1) == 1) + ptrs[i] = (char *)base + mem_sz - i * align; + else + ptrs[i] = (char *)base + i * align; + } + + if (use_32_bit) { + rte_ptr_compress_32(base, ptrs, offsets32, num_ptrs, align_exp); + rte_ptr_decompress_32(base, offsets32, ptrs_out, num_ptrs, + align_exp); + } else { + rte_ptr_compress_16(base, ptrs, offsets16, num_ptrs, align_exp); + rte_ptr_decompress_16(base, offsets16, ptrs_out, num_ptrs, + align_exp); + } + + TEST_ASSERT_BUFFERS_ARE_EQUAL(ptrs, ptrs_out, sizeof(void *) * num_ptrs, + "Decompressed pointers corrupted\nbase pointer: %p, " + "memory region size: %" PRIu64 ", alignment exponent: %u, " + "num of pointers: %u, using %s offsets", + base, mem_sz, align_exp, num_ptrs, + use_32_bit ? "32-bit" : "16-bit"); + + return 0; +} + +static int +test_ptr_compress(void) +{ + unsigned int j, k, n; + int ret = 0; + void * const bases[NUM_BASES] = { (void *)0, (void *)UINT16_MAX }; + /* maximum size for pointers aligned by consecutive powers of 2 */ + const uint64_t region_sizes_16[NUM_REGIONS] = { + MAX_16BIT_REGION, + MAX_16BIT_REGION * 2, + MAX_16BIT_REGION * 4, + MAX_16BIT_REGION * 8, + }; + const uint64_t region_sizes_32[NUM_REGIONS] = { + MAX_32BIT_REGION, + MAX_32BIT_REGION * 2, + MAX_32BIT_REGION * 4, + MAX_32BIT_REGION * 8, + }; + + for (j = 0; j < NUM_REGIONS; j++) { + for (k = 0; k < NUM_BASES; k++) { + for (n = 1; n < MAX_PTRS; n++) { + ret |= test_ptr_compress_params( + bases[k], + region_sizes_16[j], + j /* exponent of alignment */, + n, + false + ); + ret |= test_ptr_compress_params( + bases[k], + region_sizes_32[j], + j /* exponent of alignment */, + n, + true + ); + if (ret != 0) + return ret; + } + } + } + + return ret; +} + +REGISTER_FAST_TEST(ptr_compress_autotest, true, true, test_ptr_compress);