From patchwork Mon Feb 13 06:11:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 123758 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2610141C82; Mon, 13 Feb 2023 07:11:54 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 02D7A40DDB; Mon, 13 Feb 2023 07:11:54 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2083.outbound.protection.outlook.com [40.107.220.83]) by mails.dpdk.org (Postfix) with ESMTP id 5282240A81 for ; Mon, 13 Feb 2023 07:11:52 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TXIVjJXnSfP7foAbDB/Stkstedg/r5tBfEt7PUJzko2TvBosx7/LrTGAc/HDDn1xwJKNY5DBqu/5PBFC7U3hVCVB0V9HSi3D6EPSiUFXUbeh23OOPat9k02oeVtiXvKD2m9Sz676R4xlD/kqVNmClAD0NhCZl8Zn49kEyDK923bTztOLuzF72zTJ0vI16IzqFw7WeCMC4/78v/vu/cDI5i6fias4Fq5/jVWBvgZgTGXdqSgvOuZpmKn/RVNX6lGPX+jeI/YYLjHMuCRrEObfRWR8E1PJPgSRdeFItg1V7MfbWCzqiCnlOkgelo3ZJ6a14SZ+E5B2Tf9SB+vQYhu++w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DhYNMQhh/EtmW1h0Fe7cS6KE4TzacO8+wTY+HzGA/a4=; b=Kz0sLHoHjbYK3MOMYo6ZLcW91PhzYcRxINC8qbY61QEnV/KpBm/x8Z+7xdHp17wTwusafGLaK0RDXQ9MeUm7L8Wy+eTTeo3evZdjGeYHKJ1mlnX0NkHjSpWoEoZLR3SSKpMabUb2NdC0mFxOI/ug2nOi4MxljtsUmCWhxsfUWO0M4EWhNYRvx3cB0WjgXZXBC1HNBzTifWyEKI+dW7qAshxXyYD+OKbsDv1H49aozelKynyYEV5OFMJUN5X1sWnVH4pO8XiwF/1kRXIe4nQCVzVQYHFwqQnZDiANOVCd9RRYjN3w1bGa2wR0Hwr4pBqhR11s/UvGhhdkbhAl4LOhcQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DhYNMQhh/EtmW1h0Fe7cS6KE4TzacO8+wTY+HzGA/a4=; b=nxY070Rx1WZeCMi0WHeH191FnvU2fREuO+qf+744McB0cgHHVu8t5Z9Pgus95j354tRmVtBH7pi1nNNRp4Q3abN79XuXFPPzTwCZ0kFYBqoQA4f0LbD1Q1oMyK2sUmA+FOFF3YLHJWYleBhcYfm/OmN9tQDtPT6hqVjRG9QNcbl75mGLk0qEydDCwm5PPOiIhUEzmhzj2OEs5SvEIwZuQyT07TVHDMAdUKHWa7OTVoqlYvl9MuyqGSykrQEY+Hx2+6fTs17vcfF3GYJxVS8MaeVxMesXD9/D4OesZn48YYgRvDerOpqZKrEUKKeq+es7/KIqaKC2MD7u9wqdbad4oQ== Received: from MW4PR03CA0042.namprd03.prod.outlook.com (2603:10b6:303:8e::17) by PH8PR12MB7229.namprd12.prod.outlook.com (2603:10b6:510:227::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.23; Mon, 13 Feb 2023 06:11:50 +0000 Received: from CO1NAM11FT021.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8e:cafe::a1) by MW4PR03CA0042.outlook.office365.com (2603:10b6:303:8e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24 via Frontend Transport; Mon, 13 Feb 2023 06:11:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT021.mail.protection.outlook.com (10.13.175.51) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.24 via Frontend Transport; Mon, 13 Feb 2023 06:11:50 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 12 Feb 2023 22:11:49 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Sun, 12 Feb 2023 22:11:48 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36 via Frontend Transport; Sun, 12 Feb 2023 22:11:47 -0800 From: Michael Baum To: CC: Matan Azrad , Akhil Goyal , "Ashish Gupta" , Fiona Trahe , "Thomas Monjalon" Subject: [PATCH v3 1/4] compressdev: add LZ4 algorithm support Date: Mon, 13 Feb 2023 08:11:37 +0200 Message-ID: <20230213061140.2157499-2-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230213061140.2157499-1-michaelba@nvidia.com> References: <20230202091332.1037078-1-michaelba@nvidia.com> <20230213061140.2157499-1-michaelba@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT021:EE_|PH8PR12MB7229:EE_ X-MS-Office365-Filtering-Correlation-Id: 8b697baf-7602-4888-85ae-08db0d8931db X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: j0mSzIyoxVzY6QP52r6fej/iDsRGGRIiq+XjQUaXxTomorXJd8huRyWfothCciuE0l4LLAHDSZLDoGFIam3ynYj0WtMfNhyJCOP8pLM78L9cNd8oflm+khuiJsshsJSMUeCAsg/j6xpem0ITP3wDW1f+kj9GcyLQMK0EdKEmkhkwPAxSsRNewdxPIlmQE+vhwGKI57JIgCPaKEmKUuhze7aDCF56mAJHLdjPKMNgLLlWy6LHoa1xsNQb/DtNkRplXUrsvfoe08fG3YMMLqRpqglUnnUJAUxqHiS4pdUaRmvFn6PKJDttiCh+T2wpLky9QIilCT7ljYuw9e1aw4+DHaVJLpJ9HLzdPw8tG6eq0qGb/7benUZMe233ec5qEjSewBeep4ltMIiJfaTDy1bATv9vChxVDyXM3HbGOfLMtQteQOAUvp7iurGPpChkkjCq2rXRaqR6a9ASez6SBeeZf/FUmlMcimCaLCk6bznY7hSBpbGqQyC1/vF2blSQm9f2sSfqAwEaspa6RXdu9XgjS5UTeLdv91dERc8poMnbPThzieFE9JHTg3YKStPJG8DYu8gNLmZBPp4PjzfL8BsNo8tBGZDRoCTGyrBvhoBlHAWaYLOFaJdry2OX/tC0eaYwq++cBMgpkdCu2wIgZf2k7uNuhAVX0Wcbb1JPxXmgmJVn2iJIg+ZNpX8GMthypHCqY5PRNAgw8nVl28H+H19m59Pi9mshrvqm/l/+qFXF/xcMN70j7txL0TunYT+eVeiB/oeT1EqJ2zsW/a6LZsOj8ib3za96AOmmIH1/M5XIKvwgkZQyIEo2SFphiE3Pfdre X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(396003)(346002)(376002)(136003)(451199018)(36840700001)(40470700004)(46966006)(83380400001)(426003)(47076005)(82310400005)(86362001)(356005)(36756003)(40460700003)(55016003)(40480700001)(7636003)(36860700001)(82740400003)(5660300002)(2906002)(7696005)(41300700001)(8676002)(6916009)(4326008)(70206006)(70586007)(8936002)(54906003)(316002)(2616005)(336012)(966005)(26005)(1076003)(186003)(478600001)(6286002)(6666004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2023 06:11:50.0270 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8b697baf-7602-4888-85ae-08db0d8931db X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT021.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7229 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for LZ4 algorithm: - Add Lz4 param structure to XFORM structures. - Add capabilities flags for LZ4 params. - Add xxHash-32 checksum and capabilities flag. Signed-off-by: Michael Baum --- doc/guides/compressdevs/features/default.ini | 7 ++ doc/guides/rel_notes/release_23_03.rst | 7 ++ lib/compressdev/rte_comp.c | 12 +++ lib/compressdev/rte_comp.h | 79 +++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-) diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini index e1419ee8db..2f178c5efd 100644 --- a/doc/guides/compressdevs/features/default.ini +++ b/doc/guides/compressdevs/features/default.ini @@ -20,8 +20,15 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Dictionary ID = +LZ4 Content Checksum = +LZ4 Content Size = +LZ4 Block Checksum = +LZ4 Block Independence = diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst index 7527c6d57f..790c2a5eef 100644 --- a/doc/guides/rel_notes/release_23_03.rst +++ b/doc/guides/rel_notes/release_23_03.rst @@ -105,6 +105,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items ------------- diff --git a/lib/compressdev/rte_comp.c b/lib/compressdev/rte_comp.c index 320c6dab92..f060c68557 100644 --- a/lib/compressdev/rte_comp.c +++ b/lib/compressdev/rte_comp.c @@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag) return "HUFFMAN_FIXED"; case RTE_COMP_FF_HUFFMAN_DYNAMIC: return "HUFFMAN_DYNAMIC"; + case RTE_COMP_FF_XXHASH32_CHECKSUM: + return "XXHASH32_CHECKSUM"; + case RTE_COMP_FF_LZ4_DICT_ID: + return "LZ4_DICT_ID"; + case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM: + return "LZ4_CONTENT_WITH_CHECKSUM"; + case RTE_COMP_FF_LZ4_CONTENT_SIZE: + return "LZ4_CONTENT_SIZE"; + case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE: + return "LZ4_BLOCK_INDEPENDENCE"; + case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM: + return "LZ4_BLOCK_WITH_CHECKSUM"; default: return NULL; } diff --git a/lib/compressdev/rte_comp.h b/lib/compressdev/rte_comp.h index 5bd711fda1..2096fb2407 100644 --- a/lib/compressdev/rte_comp.h +++ b/lib/compressdev/rte_comp.h @@ -67,6 +67,18 @@ extern "C" { /**< Fixed huffman encoding is supported */ #define RTE_COMP_FF_HUFFMAN_DYNAMIC (1ULL << 14) /**< Dynamic huffman encoding is supported */ +#define RTE_COMP_FF_XXHASH32_CHECKSUM (1ULL << 15) +/**< xxHash-32 Checksum is supported */ +#define RTE_COMP_FF_LZ4_DICT_ID (1ULL << 16) +/**< LZ4 dictionary ID is supported */ +#define RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM (1ULL << 17) +/**< LZ4 content with checksum is supported */ +#define RTE_COMP_FF_LZ4_CONTENT_SIZE (1ULL << 18) +/**< LZ4 content size is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE (1ULL << 19) +/**< LZ4 block independent is supported */ +#define RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM (1ULL << 20) +/**< LZ4 block with checksum is supported */ /** Status of comp operation */ enum rte_comp_op_status { @@ -109,6 +121,10 @@ enum rte_comp_algorithm { /**< LZS compression algorithm * https://tools.ietf.org/html/rfc2395 */ + RTE_COMP_ALGO_LZ4, + /**< LZ4 compression algorithm + * https://github.com/lz4/lz4 + */ }; /** Compression Hash Algorithms */ @@ -147,9 +163,12 @@ enum rte_comp_checksum_type { /**< Generates both Adler-32 and CRC32 checksums, concatenated. * CRC32 is in the lower 32bits, Adler-32 in the upper 32 bits. */ + RTE_COMP_CHECKSUM_XXHASH32, + /**< Generates a xxHash-32 checksum, as used by lz4. + * https://github.com/Cyan4973/xxHash/blob/dev/doc/xxhash_spec.md + */ }; - /** Compression Huffman Type - used by DEFLATE algorithm */ enum rte_comp_huffman { RTE_COMP_HUFFMAN_DEFAULT, @@ -206,13 +225,63 @@ enum rte_comp_op_type { */ }; - /** Parameters specific to the deflate algorithm */ struct rte_comp_deflate_params { enum rte_comp_huffman huffman; /**< Compression huffman encoding type */ }; +/** + * Dictionary ID flag + * If this flag is set, a 4-bytes Dict-ID field will be present, after the + * descriptor flags and the Content Size. + */ +#define RTE_COMP_LZ4_FLAG_DICT_ID (1 << 0) + +/** + * Content Checksum flag + * If this flag is set, a 32-bits content checksum will be appended after the + * EndMark. + */ +#define RTE_COMP_LZ4_FLAG_CONTENT_CHECKSUM (1 << 2) + +/** + * Content Size flag + * If this flag is set, the uncompressed size of data included within the frame + * will be present as an 8 bytes unsigned little-endian value, after the flags. + * Content Size usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_CONTENT_SIZE (1 << 3) + +/** + * Block Checksum flag. + * If this flag is set, each data block will be followed by a 4-bytes checksum, + * calculated by using the xxHash-32 algorithm on the raw (compressed) data + * block. The intention is to detect data corruption (storage or transmission + * errors) immediately, before decoding. Block checksum usage is optional. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4) + +/** + * Block Independence flag. + * If this flag is set to 1, blocks are independent. + * If this flag is set to 0, each block depends on previous ones (up to LZ4 + * window size, which is 64 KB). In such case, it is necessary to decode all + * blocks in sequence. + * Block dependency improves compression ratio, especially for small blocks. On + * the other hand, it makes random access or multi-threaded decoding impossible. + */ +#define RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE (1 << 5) + +/** Parameters specific to the LZ4 algorithm */ +struct rte_comp_lz4_params { + uint8_t flags; + /**< Compression LZ4 parameter flags. + * Based on LZ4 standard flags: + * https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor + */ +}; + /** Setup Data for compression */ struct rte_comp_compress_xform { enum rte_comp_algorithm algo; @@ -220,6 +289,8 @@ struct rte_comp_compress_xform { union { struct rte_comp_deflate_params deflate; /**< Parameters specific to the deflate algorithm */ + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ }; /**< Algorithm specific parameters */ int level; /**< Compression level */ @@ -249,6 +320,10 @@ struct rte_comp_decompress_xform { * compressed data. If window size can't be supported by the PMD then * setup of stream or private_xform should fail. */ + union { + struct rte_comp_lz4_params lz4; + /**< Parameters specific to the LZ4 algorithm */ + }; /**< Algorithm specific parameters */ enum rte_comp_hash_algorithm hash_algo; /**< Hash algorithm to be used with decompress operation. Hash is always * done on plaintext.