From patchwork Thu Feb 2 09:13:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 122888 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9C57941BAB; Thu, 2 Feb 2023 10:14:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5D53242D44; Thu, 2 Feb 2023 10:13:59 +0100 (CET) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2042.outbound.protection.outlook.com [40.107.92.42]) by mails.dpdk.org (Postfix) with ESMTP id 4204140689 for ; Thu, 2 Feb 2023 10:13:56 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b6oLW8RBASnM91PVVFjKq82HqWEIsvUK7xbFVMz1WEUjk+C/2AIidVQisFUcSREB5ui+ZxogQVBTnT4upxfKofYmqpMOYiNO291JxiDkUxq8AgrXjwmTKlAnNeCsATlL/RIIb1AVnrgY70bX5FOpNkW3CgKPYUc88pjRnodEDDgrpXNFkG6JRA9qm+tHfEC1uePYPu0W520QP8BG58qspSsP3igvM6CuO/on8wFxFeMOX22MJh+FpvLgQNeNlhErG0ZfdJpob1T9I5tqvVPJ9iWPpJExnDh9Bv8u0E9Mrv5dtOh6BXyxmQU09x42stdWTheWRRhh0uEn5knk2poV3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CmebHCPFoD0g7iW23UtegGjIEgKkYliN82KvAgo+9XA=; b=AeM41ku3gh/WbjZ0zpwq1dE3IEa3gVreYQfISaFuDVDPqvXw7zwMD44I4/x8A1nsVBjwuLCIgGVKhp5ld/zxM7sh0BkCaPzrz/fgnHW+2tVcU/sn13EQtQ7vpb9yyD5jarSGLIvhkELhr8LYYf7UmxhctojIalb70wdFd0rHQpVRtAmiBxZhh7TK8h8A8dCi+J90E4ICW8xc1RmBokqMUHmWq/jfk7Ykj/YB871Qy3Op0jJkdtwbm7Cxlj3+4E8zUynql3FTHBsBDFnLwDpEXUyXwwM61hSPFySs8IZZc9l9G20xigy67fjV3HBuoGfR0wW7NgQwhmWliFiv2k0RlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CmebHCPFoD0g7iW23UtegGjIEgKkYliN82KvAgo+9XA=; b=OvCOD/Gn9H1cwgLwY8NPcnPoBzM9XUPT2KfNu5swSEG+bsTbH9d3yXLjIMQFzEGyMBTSPfFX0qOMo0DxJcIR7e5VEbnwn4mHl1tSYbdC6db7f/aWTxKM7vZ+hJjKUzzh/Audx+N69ZocSwYEJRkya7WAxVRL95dll9X+bk3d2wxVyVWTGGOYt9lX5sdExyYQTc5T3NuUt7MQfoqDsDtWvk3Eclpqod/UCc3iAE3ZCmTPEDho8qen6on5bDN5MpMc7+CUB+JTkAnxMdX/3DjuJgt4/AAEAKYcA/S+HM17UHiy1FBi2i10D+ryOF+Uu43IPjFNFurIN31rr1kHQAlqGw== Received: from BN9PR03CA0264.namprd03.prod.outlook.com (2603:10b6:408:ff::29) by PH0PR12MB8049.namprd12.prod.outlook.com (2603:10b6:510:28f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.27; Thu, 2 Feb 2023 09:13:54 +0000 Received: from BN8NAM11FT060.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ff:cafe::3c) by BN9PR03CA0264.outlook.office365.com (2603:10b6:408:ff::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.27 via Frontend Transport; Thu, 2 Feb 2023 09:13:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN8NAM11FT060.mail.protection.outlook.com (10.13.177.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.27 via Frontend Transport; Thu, 2 Feb 2023 09:13:53 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 2 Feb 2023 01:13:40 -0800 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 2 Feb 2023 01:13:40 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36 via Frontend Transport; Thu, 2 Feb 2023 01:13:38 -0800 From: Michael Baum To: CC: Matan Azrad , Akhil Goyal , "Ashish Gupta" , Fiona Trahe , "Thomas Monjalon" Subject: [PATCH v2 1/4] compressdev: add LZ4 algorithm support Date: Thu, 2 Feb 2023 11:13:29 +0200 Message-ID: <20230202091332.1037078-2-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230202091332.1037078-1-michaelba@nvidia.com> References: <20230109074526.2507344-1-michaelba@nvidia.com> <20230202091332.1037078-1-michaelba@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT060:EE_|PH0PR12MB8049:EE_ X-MS-Office365-Filtering-Correlation-Id: 78b8f023-94c7-4dae-a7f0-08db04fdce85 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QsLwG9qIWG/5tFOT/O5GMR+LPuG1juvpnNGGsSsRJEBg1CsbrRlIiXNdQpiN3rCsKEzT6ld0Fa95eiG0D4AWgNreZ+DZGWzFZpZdB3S5QAmHli8TRSljMmDnOH9nYtfkyRSQPPTKtnbsabYzhz+ZBcMCnCMyx7/QY+K49htVbtJCG4AfLpPGQz7D83/7GSgaGBtB4z0aIdTKmzimmAMgqxFl8rkZU1wcJVDDrkG0ynxtSTesn9lXpuysFP7FDv60gTkKnMjoR3ks1jT6PkQUpycwkrI50xFDFSOxXSvHj4rHegqMkFGVckMuxnX7VXSqsGuCx9+C1c3AcAP5+OYwiKL8/A7AQrbCAHmnlZf/EK7qCYTDVM8sXjjhl6TYBctELZ+9O0MwzytwlD5qn+mpkBFlIvP8MxbS7xVwxT6yb2ZRuGqX97Qis9QKKFmf0TgHWjD/LENbfc7XN6pD1/weZexno+P0GW2lkbESByP8qjBfstQ8N1OptrbbV8fcuY6wqymcSXnEotpENBJLEEekxHikHKPUr6hLwSI0HSWWsAblM+sDGeT9pD3AYqqG/L2O4DVC4AQEEXIGIoPI6uA3TdgxTnohZykx5FIvTT/8x1w0rCnFv4kkPxr8mMKpM7jRRBF2dOnMoqwOoxuG9DZW8R/yTqc5cI1FXqH6Bpxg8znWiINDVDf7NNFH0/irfjxzbqtCN3eChAC9DhVsWlDGQ+feK9o0x1NGblQkhUNliCbk19QGH50koOp0x5lNGF0a4L+TwF/PtQ0LOzz4OfADzTj0S8ImBYJa6PSksHFRcFF9esrxXD+XZKnxewGjV1vB X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(136003)(396003)(39860400002)(346002)(376002)(451199018)(46966006)(36840700001)(40470700004)(478600001)(966005)(47076005)(7696005)(6286002)(2616005)(186003)(1076003)(6666004)(83380400001)(426003)(40460700003)(336012)(26005)(5660300002)(2906002)(82310400005)(36756003)(316002)(54906003)(8676002)(55016003)(40480700001)(41300700001)(8936002)(6916009)(70206006)(70586007)(4326008)(7636003)(82740400003)(356005)(36860700001)(86362001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2023 09:13:53.9285 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 78b8f023-94c7-4dae-a7f0-08db04fdce85 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT060.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8049 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for LZ4 algorithm: - Add Lz4 param structure to XFORM structures. - Add capabilities flags for LZ4 params. - Add xxHash-32 checksum and capabilities flag. Signed-off-by: Michael Baum --- doc/guides/compressdevs/features/default.ini | 7 ++ doc/guides/rel_notes/release_23_03.rst | 7 ++ lib/compressdev/rte_comp.c | 12 +++ lib/compressdev/rte_comp.h | 79 +++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-) diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini index e1419ee8db..2f178c5efd 100644 --- a/doc/guides/compressdevs/features/default.ini +++ b/doc/guides/compressdevs/features/default.ini @@ -20,8 +20,15 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Dictionary ID = +LZ4 Content Checksum = +LZ4 Content Size = +LZ4 Block Checksum = +LZ4 Block Independence = diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst index 73f5d94e14..2722296180 100644 --- a/doc/guides/rel_notes/release_23_03.rst +++ b/doc/guides/rel_notes/release_23_03.rst @@ -78,6 +78,13 @@ New Features ``rte_event_dev_config::nb_single_link_event_port_queues`` parameter required for eth_rx, eth_tx, crypto and timer eventdev adapters. +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items ------------- diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c index 320c6dab92..f060c68557 100644 --- a/lib/compressdev/rte_comp.c +++ b/lib/compressdev/rte_comp.c @@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag) return "HUFFMAN_FIXED"; case RTE_COMP_FF_HUFFMAN_DYNAMIC: return "HUFFMAN_DYNAMIC"; + case RTE_COMP_FF_XXHASH32_CHECKSUM: + return "XXHASH32_CHECKSUM"; + case RTE_COMP_FF_LZ4_DICT_ID: + return "LZ4_DICT_ID"; + case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM: + return "LZ4_CONTENT_WITH_CHECKSUM"; + case RTE_COMP_FF_LZ4_CONTENT_SIZE: + return "LZ4_CONTENT_SIZE"; + case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE: + return "LZ4_BLOCK_INDEPENDENCE"; + case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM: + return "LZ4_BLOCK_WITH_CHECKSUM"; default: return NULL; } diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h index 5bd711fda1..2096fb2407 100644 --- a/lib/compressdev/rte_comp.h +++ b/lib/compressdev/rte_comp.h @@ -67,6 +67,18 @@ extern "C" { /**< Fixed huffman encoding is supported */ #define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14) /**< Dynamic huffman encoding is supported */ +#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15) +/**< xxHash-32 Checksum is supported */ +#define RTE_COMP_FF_LZ4_DICT_ID (1ULL << 16) +/**< LZ4 dictionary ID is supported */ +#define RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM (1ULL << 17) +/**< LZ4 content with checksum is supported */ +#define RTE_COMP_FF_LZ4_CONTENT_SIZE (1ULL << 18) +/**< LZ4 content size is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 19) +/**< LZ4 block independent is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 20) +/**< LZ4 block with checksum is supported */ /** Status of comp operation */ enum rte_comp_op_status { @@ -109,6 +121,10 @@ enum rte_comp_algorithm { /**< LZS compression algorithm * https://tools.ietf.org/html/rfc2395 */ + RTE_COMP_ALGO_LZ4, + /**< LZ4 compression algorithm + * https://github.com/lz4/lz4 + */ }; /** Compression Hash Algorithms */ @@ -147,9 +163,12 @@ enum rte_comp_checksum_type { /**< Generates both Adler-32 and CRC32 checksums, concatenated. * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits. */ + RTE_COMP_CHECKSUM_XXHASH32, + /**< Generates a xxHash-32 checksum, as used by lz4. + * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md + */ }; - /** Compression Huffman Type - used by DEFLATE algorithm */ enum rte_comp_huffman { RTE_COMP_HUFFMAN_DEFAULT, @@ -206,13 +225,63 @@ enum rte_comp_op_type { */ }; - /** Parameters specific to the deflate algorithm */ struct rte_comp_deflate_params { enum rte_comp_huffman huffman; /**< Compression huffman encoding type */ }; +/** + * Dictionary ID flag + * If this flag is set, a 4-bytes Dict-ID field will be present, after the + * descriptor flags and the Content Size. + */ +#define RTE_COMP_LZ4_FLAG_DICT_ID (1 << 0) + +/** + * Content Checksum flag + * If this flag is set, a 32-bits content checksum will be appended after the + * EndMark. + */ +#define RTE_COMP_LZ4_FLAG_CONTENT_CHECKSUM (1 << 2) + +/** + * Content Size flag + * If this flag is set, the uncompressed size of data included within the frame + * will be present as an 8 bytes unsigned little-endian value, after the flags. + * Content Size usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_CONTENT_SIZE (1 << 3) + +/** + * Block Checksum flag. + * If this flag is set, each data block will be followed by a 4-bytes checksum, + * calculated by using the xxHash-32 algorithm on the raw (compressed) data + * block. The intention is to detect data corruption (storage or transmission + * errors) immediately, before decoding. Block checksum usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4) + +/** + * Block Independence flag. + * If this flag is set to 1, blocks are independent. + * If this flag is set to 0, each block depends on previous ones (up to LZ4 + * window size, which is 64 KB). In such case, it is necessary to decode all + * blocks in sequence. + * Block dependency improves compression ratio, especially for small blocks. On + * the other hand, it makes random access or multi-threaded decoding impossible. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5) + +/** Parameters specific to the LZ4 algorithm */ +struct rte_comp_lz4_params { + uint8_t flags; + /**< Compression LZ4 parameter flags. + * Based on LZ4 standard flags: + * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor + */ +}; + /** Setup Data for compression */ struct rte_comp_compress_xform { enum rte_comp_algorithm algo; @@ -220,6 +289,8 @@ struct rte_comp_compress_xform { union { struct rte_comp_deflate_params deflate; /**< Parameters specific to the deflate algorithm */ + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ }; /**< Algorithm specific parameters */ int level; /**< Compression level */ @@ -249,6 +320,10 @@ struct rte_comp_decompress_xform { * compressed data. If window size can't be supported by the PMD then * setup of stream or private_xform should fail. */ + union { + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ + }; /**< Algorithm specific parameters */ enum rte_comp_hash_algorithm hash_algo; /**< Hash algorithm to be used with decompress operation. Hash is always * done on plaintext.