From patchwork Tue Feb 28 16:43:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kozyrev X-Patchwork-Id: 124578 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D51FA41D9E; Tue, 28 Feb 2023 17:44:06 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6D6A941611; Tue, 28 Feb 2023 17:43:55 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2072.outbound.protection.outlook.com [40.107.93.72]) by mails.dpdk.org (Postfix) with ESMTP id E394541611 for ; Tue, 28 Feb 2023 17:43:54 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rfsy0QqVHR5mLB0AA60C7LXJRGw9hBV0DOO7zkB1HIAhuMGv57d3Xr4P40wGGn6IT41HIONPjm9u4/LQu/p8NA1Qx7s+W4gb+NHkTdfc67vJlR4b/fciX0Hn9MvHHVgT4Rq0SAwekmSDivH+ic81neXTAIUTpM23WtUlR+F0yPEpvbMHnlf/MdJsapJnvz08veQ5AsVLFt2WHJjKrNx9DpnQP8BqhdxtfmAJ3mOp1RhUVULVkEmH37q3Zd/1/ggTTZAjM9dBSp1z1emnvMk5d5BEmKeUt78qhxuMlcTUFQogL8amU/a3HXfPZGwlAYs2xk4/3CjBrBeO+ykhJrB+8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bijbOXvRpwd4nc12X0TjtZQlclNuxchkeFb+molrNgQ=; b=SOTG61oYr3QSDi0WSyeoqwAFhZMoXXH6S/i08kiSVyRQXVPZile9LQRRKwwyY0HaDRZQpAgBYwwMne21BriVMSzxgfReLRy+3CdToY0jJHiYsuL7gXc/zuVJPJsvXwAnCOTbTSkFJLXtQoUyceNBJ/k/glRAkvzNIEXg9a1pbTDCS0fmE/urelVSGQxoE03Rk4Z1z/pmjYTBPmjEYeHei4XFqWSpcHe6+uV/4n2NWCfe9JYpAWdl5GO1eIb25ybsEejyUQK4T0AiPcy2sCvTmlpud/RPt5iIWOyyG4t5IQsmPJ5UwyVd0kXdbZ6fEHuteATmjq2ImnYApEuNZ2mdEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bijbOXvRpwd4nc12X0TjtZQlclNuxchkeFb+molrNgQ=; b=oPqPjZN7povCcWRhJ5jlFkDrXVDx0GH9JS/RPGwAjkeV4JSnwZGmwjzh20bQbsoMe5wC/18WEl+LFcqvW+c9htzzI0z41Zd/lqN8ylGV5cLi2vXpK2r5Xm1f6k6p8zXOYPwZR58+MGv6wXiWYDQ3e6uRp6CdEn0TJoSDiqzHDzeZsRRwjhETVobIYESJWJ9wSlBvzOjywfa5cZJkU5g+SCU1W/NSrDibT3vI+hbDKFDJZaYVYtuAP6XwdIPiXlKXyHkQHIzZ+J3q8IckN9dYcEPye9eN58kWONoGSSr9XT2JYscvaHWXn5f7q79dKhmj8amW177KKJ0ErTHS0UerGw== Received: from DM6PR21CA0012.namprd21.prod.outlook.com (2603:10b6:5:174::22) by MN2PR12MB4254.namprd12.prod.outlook.com (2603:10b6:208:1d0::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.30; Tue, 28 Feb 2023 16:43:44 +0000 Received: from DS1PEPF0000E63D.namprd02.prod.outlook.com (2603:10b6:5:174:cafe::25) by DM6PR21CA0012.outlook.office365.com (2603:10b6:5:174::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.4 via Frontend Transport; Tue, 28 Feb 2023 16:43:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS1PEPF0000E63D.mail.protection.outlook.com (10.167.17.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.12 via Frontend Transport; Tue, 28 Feb 2023 16:43:44 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Tue, 28 Feb 2023 08:43:35 -0800 Received: from pegasus01.mtr.labs.mlnx (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Tue, 28 Feb 2023 08:43:33 -0800 From: Alexander Kozyrev To: CC: , , Subject: [PATCH 5/5] net/mlx5: enable enhanced CQE compression Date: Tue, 28 Feb 2023 18:43:10 +0200 Message-ID: <20230228164310.807594-6-akozyrev@nvidia.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20230228164310.807594-1-akozyrev@nvidia.com> References: <20230228164310.807594-1-akozyrev@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF0000E63D:EE_|MN2PR12MB4254:EE_ X-MS-Office365-Filtering-Correlation-Id: 3f96e588-d710-49ab-3e04-08db19aaf4c1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Qut6UGIG4HEWmTJ37N5hAbDBU550CxAMFoc5fi8JlJMwMjmt6JsI8I/8KXQP9T2EcNnTykxi+PoO5NHbW9NkQoWJcn8mXrLT3VcNIFvOyJbzIvWh+LqXt8+5Ah8kgMcsQb+6qEJxlv+3R+1j80l4mDigwQddAbGcXshSThSKtlWpWsa+IToWW9MFJKalhOuyAO6xSuUpAEA/ezDyZLqmG/MNNb6cKR4ZeyyP7AFeCGjZUwgdCQ7TeRJUV84U09l7UAD8uVeW3TE9iWsdSYGbGVXSs6ELmmYyFNP9QnVKUwOGGEkBQOf/yI5N1uwsPSQ3o2JG/ZEHsXPtm1kpR0PB41gvSLRI2o4SzFbb1VJ3oo2tVy1k1pEEIopyA2wYqyTlGK4DrhUcv3V1JxvfV9KPJGpxkRFHt1r+B1/Vf3Gx64lhty6hBNbq1Hj8RjSBitaKN7YcRpWLHQCq2jX9Gh8aYzy4BGMtesX+sfBXUJg1jo0zKaJdHYMdIP5I/HFb6oYS+4S4p/QGG0B7BhEkVNJTXJDsj3hIZCq4QZc2eFE9v4iEiEH7RVZ2q9cOu4n7LpVVd6dyQtXTKJqfNSr27oSPQlxCu+/rhjPnAgQ0Ge5ybirp5H0vTNdqcl9ps+sHTqqT7Lal7+4nDoru9XOLCWalvT/6FmJgs+uZR3zNjr6TMCx/KEPnp3SdlMrkR7Etv+bPKY/PFzArs07GSdQyv9LLwBBsJvd2993VHIGsGrKG5cs= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(346002)(136003)(39860400002)(376002)(396003)(451199018)(36840700001)(40470700004)(46966006)(86362001)(36756003)(6916009)(5660300002)(8936002)(4326008)(2906002)(40480700001)(41300700001)(36860700001)(356005)(8676002)(82740400003)(7636003)(6666004)(107886003)(478600001)(54906003)(316002)(70586007)(70206006)(82310400005)(40460700003)(83380400001)(47076005)(426003)(26005)(2616005)(16526019)(186003)(336012)(1076003)(309714004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2023 16:43:44.2621 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3f96e588-d710-49ab-3e04-08db19aaf4c1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF0000E63D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4254 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Extend rxq_cqe_comp_en devarg to allow the Enhanced CQE Compression layout to be enabled by a user. Setting the 8th bit turns it on. For example, rxq_cqe_comp_en=0x84 means the L3/L4 Header miniCQE format and the Enhanced CQE Compression layout. Enhanced CQE Compression can be enabled only if it is supported by FW. Create CQ with the proper CQE compression layout based on capabilities. Signed-off-by: Alexander Kozyrev Acked-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 14 ++++++++++---- doc/guides/rel_notes/release_23_03.rst | 1 + drivers/net/mlx5/mlx5.c | 17 ++++++++++++++--- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_devx.c | 2 ++ 5 files changed, 28 insertions(+), 7 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 0929f3ead0..29eedd7a35 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -681,14 +681,20 @@ for an additional list of options shared with other mlx5 drivers. Multi-Packet Rx queue configuration: Hash RSS format is used in case MPRQ is disabled, Checksum format is used in case MPRQ is enabled. - Specifying 2 as a ``rxq_cqe_comp_en`` value selects Flow Tag format for - better compression rate in case of RTE Flow Mark traffic. - Specifying 3 as a ``rxq_cqe_comp_en`` value selects Checksum format. - Specifying 4 as a ``rxq_cqe_comp_en`` value selects L3/L4 Header format for + The lower 3 bits define the CQE compression format: + Specifying 2 in these bits of the ``rxq_cqe_comp_en`` parameter selects + Flow Tag format for better compression rate in case of RTE Flow Mark traffic. + Specifying 3 in these bits selects Checksum format. + Specifying 4 in these bits selects L3/L4 Header format for better compression rate in case of mixed TCP/UDP and IPv4/IPv6 traffic. CQE compression format selection requires DevX to be enabled. If there is no DevX enabled/supported the value is reset to 1 by default. + 8th bit defines the CQE compression layout. + Setting this bit to 1 turns Enhanced CQE Compression Layout on. + Enhanced CQE Compression is designed for better latency and SW utilization. + This bit is ignored if the Basic CQE compression layout is only supported. + Supported on: - x86_64 with ConnectX-4, ConnectX-4 Lx, ConnectX-5, ConnectX-6, ConnectX-6 Dx, diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst index 49c18617a5..de151b2b2f 100644 --- a/doc/guides/rel_notes/release_23_03.rst +++ b/doc/guides/rel_notes/release_23_03.rst @@ -155,6 +155,7 @@ New Features * **Updated NVIDIA mlx5 driver.** * Added support for matching on ICMPv6 ID and sequence fields. + * Added support for Enhanced CQE Compression layout. * **Updated Wangxun ngbe driver.** diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6bf522ae9d..41b1b12b91 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -384,6 +384,8 @@ static const struct mlx5_indexed_pool_config mlx5_ipool_cfg[] = { #define MLX5_FLOW_TABLE_HLIST_ARRAY_SIZE 1024 +#define MLX5_RXQ_ENH_CQE_COMP_MASK 0x80 + /** * Decide whether representor ID is a HPF(host PF) port on BF2. * @@ -2461,14 +2463,16 @@ mlx5_port_args_check_handler(const char *key, const char *val, void *opaque) return -rte_errno; } if (strcmp(MLX5_RXQ_CQE_COMP_EN, key) == 0) { - if (tmp > MLX5_CQE_RESP_FORMAT_L34H_STRIDX) { + if ((tmp & ~MLX5_RXQ_ENH_CQE_COMP_MASK) > + MLX5_CQE_RESP_FORMAT_L34H_STRIDX) { DRV_LOG(ERR, "invalid CQE compression " "format parameter"); rte_errno = EINVAL; return -rte_errno; } config->cqe_comp = !!tmp; - config->cqe_comp_fmt = tmp; + config->cqe_comp_fmt = tmp & ~MLX5_RXQ_ENH_CQE_COMP_MASK; + config->enh_cqe_comp = !!(tmp & MLX5_RXQ_ENH_CQE_COMP_MASK); } else if (strcmp(MLX5_RXQ_PKT_PAD_EN, key) == 0) { config->hw_padding = !!tmp; } else if (strcmp(MLX5_RX_MPRQ_EN, key) == 0) { @@ -2640,7 +2644,13 @@ mlx5_port_args_config(struct mlx5_priv *priv, struct mlx5_kvargs_ctrl *mkvlist, "L3/L4 Header CQE compression format isn't supported."); config->cqe_comp = 0; } - DRV_LOG(DEBUG, "Rx CQE compression is %ssupported.", + if (config->enh_cqe_comp && !hca_attr->enhanced_cqe_compression) { + DRV_LOG(WARNING, + "Enhanced CQE compression isn't supported."); + config->enh_cqe_comp = 0; + } + DRV_LOG(DEBUG, "%sRx CQE compression is %ssupported.", + config->enh_cqe_comp ? "Enhanced " : "", config->cqe_comp ? "" : "not "); if ((config->std_delay_drop || config->hp_delay_drop) && !dev_cap->rq_delay_drop_en) { @@ -2662,6 +2672,7 @@ mlx5_port_args_config(struct mlx5_priv *priv, struct mlx5_kvargs_ctrl *mkvlist, DRV_LOG(DEBUG, "\"rxq_pkt_pad_en\" is %u.", config->hw_padding); DRV_LOG(DEBUG, "\"rxq_cqe_comp_en\" is %u.", config->cqe_comp); DRV_LOG(DEBUG, "\"cqe_comp_fmt\" is %u.", config->cqe_comp_fmt); + DRV_LOG(DEBUG, "\"enh_cqe_comp\" is %u.", config->enh_cqe_comp); DRV_LOG(DEBUG, "\"rx_vec_en\" is %u.", config->rx_vec_en); DRV_LOG(DEBUG, "Standard \"delay_drop\" is %u.", config->std_delay_drop); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 4d1af3089e..29e12cf4a7 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -271,6 +271,7 @@ struct mlx5_port_config { unsigned int hw_vlan_insert:1; /* VLAN insertion in WQE is supported. */ unsigned int hw_padding:1; /* End alignment padding is supported. */ unsigned int cqe_comp:1; /* CQE compression is enabled. */ + unsigned int enh_cqe_comp:1; /* Enhanced CQE compression is enabled. */ unsigned int cqe_comp_fmt:3; /* CQE compression format. */ unsigned int rx_vec_en:1; /* Rx vector is enabled. */ unsigned int std_delay_drop:1; /* Enable standard Rxq delay drop. */ diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index d02cedb202..4369d2557e 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -372,6 +372,8 @@ mlx5_rxq_create_devx_cq_resources(struct mlx5_rxq_priv *rxq) if (priv->config.cqe_comp && !rxq_data->hw_timestamp && !rxq_data->lro) { cq_attr.cqe_comp_en = 1u; + cq_attr.cqe_comp_layout = priv->config.enh_cqe_comp; + rxq_data->cqe_comp_layout = cq_attr.cqe_comp_layout; rxq_data->mcqe_format = priv->config.cqe_comp_fmt; rxq_data->byte_mask = UINT32_MAX; switch (priv->config.cqe_comp_fmt) {