From patchwork Tue Feb 14 17:40:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 123904 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0EF3441C97; Tue, 14 Feb 2023 18:40:58 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A767842F84; Tue, 14 Feb 2023 18:40:55 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2082.outbound.protection.outlook.com [40.107.244.82]) by mails.dpdk.org (Postfix) with ESMTP id 3BCF342DC6 for ; Tue, 14 Feb 2023 18:40:53 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cunh1AR3XvLjXlmkghakkp8gsOBi0id3ujhXvH5wDum3rzNkgija+dIIVaSio7g25iKFjaf7O1QtD68xdVdEjmkTRJhJ6y9NZr/6m9A5NRuwbvRshKNR5BhIGR8MVIkmIE/ioWFftC2bKthXfY7nFIp3DXYosMz0gsCrWsBuo6S3mFQYYYQ5AEYSFjwoOCwbgrJd6flQWs/jOFNmGm3ZYHD6wa6VdjB8os8jXbJk5/jBTGua1dvBLmaBu8EX62zzxJzT/4ME5o3pdkJ9SkNtyrioj3RoDdryYPvGBBlyFk1fryg1P5XLfCoGuy9G42p0zfih7uvETWB18UbACCwElg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DhYNMQhh/EtmW1h0Fe7cS6KE4TzacO8+wTY+HzGA/a4=; b=VqhhKVWfvDdXbkJfFD8jfybTt90ZJbTJ4gHQnPvc068Olccq/tVbJRTYdq8UIb9QKHa1t6WUNLdAqaRCxbzOVJsZ0L0YFZq46H1QnCCb33yKSezrWcNx/53Drbo+xlyhMza1BwDx+VX2IG5x7tguDJWaZ21izRN7LdWZo9yI03m8t+TSAUP1+EMA4gjODBRpX4hUKv+LNsrElFrjXpQPjcqbcBXSPaq1xlVaQKIsjN0ghmkwTMf2RrxTx6C9dJWlhJepG7zD3OxBqUO9/HwEv6STlm9Igqj4jJxheTr4XGP6JwDV7xC+5NKOrdul0QGGZcHCxZyZwqpDP9Yr9bWgXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DhYNMQhh/EtmW1h0Fe7cS6KE4TzacO8+wTY+HzGA/a4=; b=dm0S5Tdjwq933+YLj4GaRWdXmXf5yOJHWNc1kZnYB9fYw3Rm6e/ifmNj4yNAePZgwsNsxrLwxIV8SF5ji8MVsX/l/CIXxSSqlgY0LaDeWjXuTqO/D0F/KfDI5PNJ7ZTFMxG1Pi7aWILUMUSW02Ukoqhfu1gDvtK28+KFbSxDViN7zQWjp0NrmT9/es8GT0WQdFjFbR5SvV1Ey7mdCgrVdX06MJnL25q7JB8EIb+oCqbH4tCAVkWNqBrMM3N8fAjan3dpeqjxre99FYOaDWdSHWsCdZ+bO36otlNtHfhJuLCDXGO+cIfecM24c7HjlIkXfQ6xEfbN6puwTIs0eYZYVQ== Received: from DM6PR06CA0005.namprd06.prod.outlook.com (2603:10b6:5:120::18) by SN7PR12MB6765.namprd12.prod.outlook.com (2603:10b6:806:26b::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24; Tue, 14 Feb 2023 17:40:51 +0000 Received: from DM6NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:5:120:cafe::c1) by DM6PR06CA0005.outlook.office365.com (2603:10b6:5:120::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.26 via Frontend Transport; Tue, 14 Feb 2023 17:40:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT018.mail.protection.outlook.com (10.13.172.110) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24 via Frontend Transport; Tue, 14 Feb 2023 17:40:50 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 09:40:37 -0800 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 14 Feb 2023 09:40:37 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36 via Frontend Transport; Tue, 14 Feb 2023 09:40:35 -0800 From: Michael Baum To: CC: Matan Azrad , Akhil Goyal , "Ashish Gupta" , Fiona Trahe , "Thomas Monjalon" Subject: [PATCH v4 1/4] compressdev: add LZ4 algorithm support Date: Tue, 14 Feb 2023 19:40:26 +0200 Message-ID: <20230214174029.2397125-2-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230214174029.2397125-1-michaelba@nvidia.com> References: <20230213061140.2157499-1-michaelba@nvidia.com> <20230214174029.2397125-1-michaelba@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT018:EE_|SN7PR12MB6765:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d6d7286-bbee-4ee3-c389-08db0eb29d4f X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yVyAF0Hq+5rywspiHAt98Y55tPvHjFUDjYV7mELHj0cL0IHZLw1+3ZsXFjdTh8ZoseRP8T0FIIWCvhwiyOqpkFaDwDDCncNJtqECtxawzWaa+mQcWVKLGXxkNAtDJdxrBkP69q8Dp5xQ7uidjP4Vtu2FVRMsOulY75wBfA6vstV6pj+jFrWBCq2YxPDvp3e+gIx+A7qlOHpCN23pXH4HMejd7b7iSrGyy6oaPBAjMxMi5D1qGWtg7O1YSdJcJBNKjXEmnvpW2HFB0t0fIzSvtcCSI12xLUaP7Slo57xGHqWbqxbQKC3/UlOm9VGC/Umtcom6xkZMkip5IsD2cvejU/+NnvhNL5sVIBZv7HpmgG+QqaCOsNlPaCLDkGwsVPJ1N+kVMrXHAPM7Biclsii6UkcBqscppuiVUqxD8cq+Gu1/tubJVwQFPZDUbwV+PRE95PEPD+4e17hXRO6auSbYge16HMTigGI+bmAtlC+RsdQxyIDc2DF1cZJIwfkjq65NronMSLQwvf3tVNpx5Sf2UAPEVBF2DADI17n1HcrEevbvDyD2FsnaUjEAe0MdJf4ERbcn57mTvWR4or7z5CMbAb+F7Xzsj1461clZXSLUdm/ID4JFej1zJPN1Ss+uNoOEu9ydcKoznM+ioSrzbgykfcTlVmYzAeGPlaGPLqBoS/6p6r2HubTEXHm3OqalrheCPmFPXI6CTB4UFz7X/jm7hh9jKX8hTguk0+KYe4J0ckTHDOe66O/CthlS9wZDsygpAhzDIuGOoUT+4cv7XglGNA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(346002)(136003)(396003)(376002)(39860400002)(451199018)(46966006)(36840700001)(40470700004)(82740400003)(36860700001)(41300700001)(70206006)(7696005)(966005)(5660300002)(8676002)(4326008)(6916009)(70586007)(8936002)(7636003)(356005)(86362001)(83380400001)(1076003)(6666004)(6286002)(186003)(26005)(40460700003)(478600001)(426003)(55016003)(47076005)(2616005)(336012)(82310400005)(36756003)(40480700001)(54906003)(316002)(2906002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2023 17:40:50.7868 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d6d7286-bbee-4ee3-c389-08db0eb29d4f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6765 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for LZ4 algorithm: - Add Lz4 param structure to XFORM structures. - Add capabilities flags for LZ4 params. - Add xxHash-32 checksum and capabilities flag. Signed-off-by: Michael Baum --- doc/guides/compressdevs/features/default.ini | 7 ++ doc/guides/rel_notes/release_23_03.rst | 7 ++ lib/compressdev/rte_comp.c | 12 +++ lib/compressdev/rte_comp.h | 79 +++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-) diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini index e1419ee8db..2f178c5efd 100644 --- a/doc/guides/compressdevs/features/default.ini +++ b/doc/guides/compressdevs/features/default.ini @@ -20,8 +20,15 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Dictionary ID = +LZ4 Content Checksum = +LZ4 Content Size = +LZ4 Block Checksum = +LZ4 Block Independence = diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst index 7527c6d57f..790c2a5eef 100644 --- a/doc/guides/rel_notes/release_23_03.rst +++ b/doc/guides/rel_notes/release_23_03.rst @@ -105,6 +105,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items ------------- diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c index 320c6dab92..f060c68557 100644 --- a/lib/compressdev/rte_comp.c +++ b/lib/compressdev/rte_comp.c @@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag) return "HUFFMAN_FIXED"; case RTE_COMP_FF_HUFFMAN_DYNAMIC: return "HUFFMAN_DYNAMIC"; + case RTE_COMP_FF_XXHASH32_CHECKSUM: + return "XXHASH32_CHECKSUM"; + case RTE_COMP_FF_LZ4_DICT_ID: + return "LZ4_DICT_ID"; + case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM: + return "LZ4_CONTENT_WITH_CHECKSUM"; + case RTE_COMP_FF_LZ4_CONTENT_SIZE: + return "LZ4_CONTENT_SIZE"; + case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE: + return "LZ4_BLOCK_INDEPENDENCE"; + case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM: + return "LZ4_BLOCK_WITH_CHECKSUM"; default: return NULL; } diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h index 5bd711fda1..2096fb2407 100644 --- a/lib/compressdev/rte_comp.h +++ b/lib/compressdev/rte_comp.h @@ -67,6 +67,18 @@ extern "C" { /**< Fixed huffman encoding is supported */ #define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14) /**< Dynamic huffman encoding is supported */ +#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15) +/**< xxHash-32 Checksum is supported */ +#define RTE_COMP_FF_LZ4_DICT_ID (1ULL << 16) +/**< LZ4 dictionary ID is supported */ +#define RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM (1ULL << 17) +/**< LZ4 content with checksum is supported */ +#define RTE_COMP_FF_LZ4_CONTENT_SIZE (1ULL << 18) +/**< LZ4 content size is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 19) +/**< LZ4 block independent is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 20) +/**< LZ4 block with checksum is supported */ /** Status of comp operation */ enum rte_comp_op_status { @@ -109,6 +121,10 @@ enum rte_comp_algorithm { /**< LZS compression algorithm * https://tools.ietf.org/html/rfc2395 */ + RTE_COMP_ALGO_LZ4, + /**< LZ4 compression algorithm + * https://github.com/lz4/lz4 + */ }; /** Compression Hash Algorithms */ @@ -147,9 +163,12 @@ enum rte_comp_checksum_type { /**< Generates both Adler-32 and CRC32 checksums, concatenated. * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits. */ + RTE_COMP_CHECKSUM_XXHASH32, + /**< Generates a xxHash-32 checksum, as used by lz4. + * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md + */ }; - /** Compression Huffman Type - used by DEFLATE algorithm */ enum rte_comp_huffman { RTE_COMP_HUFFMAN_DEFAULT, @@ -206,13 +225,63 @@ enum rte_comp_op_type { */ }; - /** Parameters specific to the deflate algorithm */ struct rte_comp_deflate_params { enum rte_comp_huffman huffman; /**< Compression huffman encoding type */ }; +/** + * Dictionary ID flag + * If this flag is set, a 4-bytes Dict-ID field will be present, after the + * descriptor flags and the Content Size. + */ +#define RTE_COMP_LZ4_FLAG_DICT_ID (1 << 0) + +/** + * Content Checksum flag + * If this flag is set, a 32-bits content checksum will be appended after the + * EndMark. + */ +#define RTE_COMP_LZ4_FLAG_CONTENT_CHECKSUM (1 << 2) + +/** + * Content Size flag + * If this flag is set, the uncompressed size of data included within the frame + * will be present as an 8 bytes unsigned little-endian value, after the flags. + * Content Size usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_CONTENT_SIZE (1 << 3) + +/** + * Block Checksum flag. + * If this flag is set, each data block will be followed by a 4-bytes checksum, + * calculated by using the xxHash-32 algorithm on the raw (compressed) data + * block. The intention is to detect data corruption (storage or transmission + * errors) immediately, before decoding. Block checksum usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4) + +/** + * Block Independence flag. + * If this flag is set to 1, blocks are independent. + * If this flag is set to 0, each block depends on previous ones (up to LZ4 + * window size, which is 64 KB). In such case, it is necessary to decode all + * blocks in sequence. + * Block dependency improves compression ratio, especially for small blocks. On + * the other hand, it makes random access or multi-threaded decoding impossible. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5) + +/** Parameters specific to the LZ4 algorithm */ +struct rte_comp_lz4_params { + uint8_t flags; + /**< Compression LZ4 parameter flags. + * Based on LZ4 standard flags: + * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor + */ +}; + /** Setup Data for compression */ struct rte_comp_compress_xform { enum rte_comp_algorithm algo; @@ -220,6 +289,8 @@ struct rte_comp_compress_xform { union { struct rte_comp_deflate_params deflate; /**< Parameters specific to the deflate algorithm */ + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ }; /**< Algorithm specific parameters */ int level; /**< Compression level */ @@ -249,6 +320,10 @@ struct rte_comp_decompress_xform { * compressed data. If window size can't be supported by the PMD then * setup of stream or private_xform should fail. */ + union { + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ + }; /**< Algorithm specific parameters */ enum rte_comp_hash_algorithm hash_algo; /**< Hash algorithm to be used with decompress operation. Hash is always * done on plaintext.