From patchwork Mon Jan 9 07:45:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 121709 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7A33E4237B; Mon, 9 Jan 2023 08:45:59 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7269640E50; Mon, 9 Jan 2023 08:45:59 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2066.outbound.protection.outlook.com [40.107.244.66]) by mails.dpdk.org (Postfix) with ESMTP id 6A8DC4067C for ; Mon, 9 Jan 2023 08:45:58 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WCrusGv9WwdQoQCtmKjZdY5ZnKXhMpSCaf5saRGavdnek3AwmO7xjieI46skL22I2+OjTJw3lj/wOP9ITRthAKzbocK0d4dTRd20E7Lt7cT6RoH07t0drcbwVc9P1fn9UcqxPTy5gK8VKr32sZm46KUEJ2gHUg5JASDqUjDgFhv7wI4V8Ui7coDsrB0VQmHS4+Uoy956+l5r+Bfv3tHxdjhhX7ycYSc6sl/ud8tsdnfrvcUMBIpg1BVVCx9OzviVWmSf0LO8Q3lWpSG/oBEjJw8ZIggOUEBMU1JS+Js92hJCw3wA7r4zid6DTugy1up+8JnDWOqrHQ04H6m1Zt9wbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3NbvY5+nM7km6OC3DiAdgkeQZ10yQoMTwVaf5TDhFso=; b=GAiATn6/iyqXYRFz8d6GYDaTVNW/jmvelKUE9l0G+0vvzn+C4DKKTWdzL/fL3nFefmmzqS/JoKH+nC9nYH859vX8e1yUJn2oxEKClwLaOKVuhzQbDM/r/lbRMMwa1itnuQkum9NBqGUjcwD4L6g+H5PaPpTHNUUCAtVrF6nu3H4sYEqdDQ1Ia6zjkBJq12qkzj9Tpe9Ve/7fZfobPxo7qCk0h0urcQI/iUVHdpbdUjjsiUrnIXRMcZ7M6kA285UbuIkVBy++44Tw9UtPbTLpwIJb6HvQogWXsVFANos93Ok4H+bkk/lc7Y9foVSwoEX3kXzq8lJn79qfYLzwgIL0Ng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3NbvY5+nM7km6OC3DiAdgkeQZ10yQoMTwVaf5TDhFso=; b=nCwZjuxtYr3/kCcKsbqK2Encx6HD8C+9CxloARAiLxkjIuWnNEnf26tUVeQatoJpJEeq6gfqDLc3U2vHqXq7MG3ws9A7CfxkEO02w025zvM4nsw3aGFwmVoFNJYTvGyo+XHG/eChFwlv2rlWf8jGpnHxpLM/EsDM5nV8k9R1g53udbRM0gSXLWsUSDv0AR1lD5Z0s3qVSAOcxuDW10O/Wt3UuN9KWU+g/31h+c01L8mQhFHeBkH1pr4ch+7GQ5ZLEFP1ZDTLIRQIDkWqSs5IZRrMnAZJ+xV4zDRg1H1Gqz7sv56/aIdzPQZ5yQMR0jsrGmBBOHbxIAmKWdlXNKzwUA== Received: from DM6PR03CA0031.namprd03.prod.outlook.com (2603:10b6:5:40::44) by LV2PR12MB6015.namprd12.prod.outlook.com (2603:10b6:408:14f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18; Mon, 9 Jan 2023 07:45:55 +0000 Received: from DM6NAM11FT026.eop-nam11.prod.protection.outlook.com (2603:10b6:5:40:cafe::d6) by DM6PR03CA0031.outlook.office365.com (2603:10b6:5:40::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18 via Frontend Transport; Mon, 9 Jan 2023 07:45:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT026.mail.protection.outlook.com (10.13.172.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18 via Frontend Transport; Mon, 9 Jan 2023 07:45:54 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 8 Jan 2023 23:45:40 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 8 Jan 2023 23:45:40 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36 via Frontend Transport; Sun, 8 Jan 2023 23:45:38 -0800 From: Michael Baum To: CC: Matan Azrad , Akhil Goyal , "Ashish Gupta" , Fiona Trahe , "Thomas Monjalon" Subject: [PATCH 1/4] compressdev: add LZ4 algorithm support Date: Mon, 9 Jan 2023 09:45:23 +0200 Message-ID: <20230109074526.2507344-2-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230109074526.2507344-1-michaelba@nvidia.com> References: <20230109074526.2507344-1-michaelba@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT026:EE_|LV2PR12MB6015:EE_ X-MS-Office365-Filtering-Correlation-Id: cd939282-db4a-4b8a-b34a-08daf21589e3 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aeeEp6F+/v/xwLFvfAyp8TJKFbLHnLX0ZBSQJDGUT1sgqrvjAtSm9QWeybvcLShWcCJZUcFpbPv6xxgSkjUkDIIZTE19HD6mgzEH2mfBppF3TEkx33V18yKaa9gwLauqZ7bdEKZkbxW+VZQTs3ECrQN30BtDiQg26ebtUazqv+2uyFbk2iV/GEEOh62SaWQZd98ztUavDXr4xRKmbVYwg0iL4rBHLPGapLlfvMwZziAY+pciGBdssW/qVSbihc12RAaSFUVaq7Ppu1pbamvedGTkyUYTxzLD+K2i1oRo38m4pU8oi3CK0ew7Bw1vIlQs/P8IX2sNO60ak5xJg6uGysirxD4KKIYqNz+BuJGMVehmrg0gUwP//z2xk057OYRWHEDBL+jwc8/BOZmvJu+D4+muwNUPlj+XJCobKpI5KKIER70eE1evYKBFSqUzG/wK7zLIN3ofhLM2mHT3qqeSxt9mnClC3XEVEB+aPgxo3ihW7XONBC/coiUqV0Wpq9XiYcJIC3p0f53GkfpVH4Ib31y2ggCtmn2Smi1M/40c4/H7ACCTReIOYE8H1m0CV2qMM26Uv1DNpEilufdSL2NBtijHxYdljLbhybY3FRsD0GDqdFMERbG5yD6DQXIaxc1lU520iwRPfKp4hDy4kGYXKev+sGSWncvdZ9++OffOjH1LtV6n0XBQYEhSLwQ0WF24mTPVSomSmty8VVP4d7A6WQ0VzYta3thI3I8hLsh9HwABuIHpNiZbEdl0i9+r8StPASLyAcNbTWzQu8vdhELBSvYHfUIbOequvSzKtCqqJKnlf/Wg+xIDJhelQOsZPBw09s5YkWm7P6tL/6SaiQ3n8g== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(346002)(396003)(376002)(136003)(39860400002)(451199015)(36840700001)(40470700004)(46966006)(36860700001)(82740400003)(6666004)(7636003)(356005)(2906002)(478600001)(966005)(2616005)(1076003)(7696005)(6286002)(26005)(186003)(5660300002)(316002)(40480700001)(83380400001)(82310400005)(8936002)(55016003)(36756003)(86362001)(426003)(41300700001)(40460700003)(47076005)(70586007)(70206006)(8676002)(4326008)(6916009)(336012)(54906003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Jan 2023 07:45:54.6420 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cd939282-db4a-4b8a-b34a-08daf21589e3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT026.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6015 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for LZ4 algorithm: - Add Lz4 param structure to XFORM structures. - Add capabilities flags for LZ4 params. - Add xxHash-32 checksum and capabilities flag. Signed-off-by: Michael Baum --- doc/guides/compressdevs/features/default.ini | 4 ++ doc/guides/rel_notes/release_23_03.rst | 7 +++ lib/compressdev/rte_comp.c | 6 +++ lib/compressdev/rte_comp.h | 51 +++++++++++++++++++- 4 files changed, 66 insertions(+), 2 deletions(-) diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini index e1419ee8db..208b8591cf 100644 --- a/doc/guides/compressdevs/features/default.ini +++ b/doc/guides/compressdevs/features/default.ini @@ -20,8 +20,12 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Block Checksum = +LZ4 Block Independence = diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst index b8c5b68d6c..3bcf638544 100644 --- a/doc/guides/rel_notes/release_23_03.rst +++ b/doc/guides/rel_notes/release_23_03.rst @@ -55,6 +55,13 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items ------------- diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c index 320c6dab92..2ad05dbf01 100644 --- a/lib/compressdev/rte_comp.c +++ b/lib/compressdev/rte_comp.c @@ -39,6 +39,12 @@ rte_comp_get_feature_name(uint64_t flag) return "HUFFMAN_FIXED"; case RTE_COMP_FF_HUFFMAN_DYNAMIC: return "HUFFMAN_DYNAMIC"; + case RTE_COMP_FF_XXHASH32_CHECKSUM: + return "XXHASH32_CHECKSUM"; + case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE: + return "LZ4_BLOCK_INDEPENDENCE"; + case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM: + return "LZ4_BLOCK_WITH_CHECKSUM"; default: return NULL; } diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h index a8f398b57b..ec206d17cf 100644 --- a/lib/compressdev/rte_comp.h +++ b/lib/compressdev/rte_comp.h @@ -67,6 +67,12 @@ extern "C" { /**< Fixed huffman encoding is supported */ #define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14) /**< Dynamic huffman encoding is supported */ +#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15) +/**< xxHash-32 Checksum is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 16) +/**< LZ4 block independent is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 17) +/**< LZ4 block with checksum is supported */ /** Status of comp operation */ enum rte_comp_op_status { @@ -109,6 +115,10 @@ enum rte_comp_algorithm { /**< LZS compression algorithm * https://tools.ietf.org/html/rfc2395 */ + RTE_COMP_ALGO_LZ4, + /**< LZ4 compression algorithm + * https://github.com/lz4/lz4 + */ RTE_COMP_ALGO_LIST_END }; @@ -149,9 +159,12 @@ enum rte_comp_checksum_type { /**< Generates both Adler-32 and CRC32 checksums, concatenated. * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits. */ + RTE_COMP_CHECKSUM_XXHASH32, + /**< Generates a xxHash-32 checksum, as used by lz4. + * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md + */ }; - /** Compression Huffman Type - used by DEFLATE algorithm */ enum rte_comp_huffman { RTE_COMP_HUFFMAN_DEFAULT, @@ -208,13 +221,41 @@ enum rte_comp_op_type { */ }; - /** Parameters specific to the deflate algorithm */ struct rte_comp_deflate_params { enum rte_comp_huffman huffman; /**< Compression huffman encoding type */ }; +/** + * Block checksum flag. + * If this flag is set, each data block will be followed by a 4-bytes checksum, + * calculated by using the xxHash-32 algorithm on the raw (compressed) data + * block. The intention is to detect data corruption (storage or transmission + * errors) immediately, before decoding. Block checksum usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4) + +/** + * Block Independence flag. + * If this flag is set to 1, blocks are independent. + * If this flag is set to 0, each block depends on previous ones (up to LZ4 + * window size, which is 64 KB). In such case, it is necessary to decode all + * blocks in sequence. + * Block dependency improves compression ratio, especially for small blocks. On + * the other hand, it makes random access or multi-threaded decoding impossible. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5) + +/** Parameters specific to the LZ4 algorithm */ +struct rte_comp_lz4_params { + uint8_t flags; + /**< Compression LZ4 parameter flags. + * Based on LZ4 standard flags: + * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor + */ +}; + /** Setup Data for compression */ struct rte_comp_compress_xform { enum rte_comp_algorithm algo; @@ -222,6 +263,8 @@ struct rte_comp_compress_xform { union { struct rte_comp_deflate_params deflate; /**< Parameters specific to the deflate algorithm */ + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ }; /**< Algorithm specific parameters */ int level; /**< Compression level */ @@ -251,6 +294,10 @@ struct rte_comp_decompress_xform { * compressed data. If window size can't be supported by the PMD then * setup of stream or private_xform should fail. */ + union { + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ + }; /**< Algorithm specific parameters */ enum rte_comp_hash_algorithm hash_algo; /**< Hash algorithm to be used with decompress operation. Hash is always * done on plaintext.