From patchwork Wed Dec 21 07:39:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rongwei Liu X-Patchwork-Id: 121168 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 78302A034C; Wed, 21 Dec 2022 08:40:39 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5A26F42D3E; Wed, 21 Dec 2022 08:40:09 +0100 (CET) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2067.outbound.protection.outlook.com [40.107.92.67]) by mails.dpdk.org (Postfix) with ESMTP id 0E9AF40684 for ; Wed, 21 Dec 2022 08:40:07 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mlLB+qOLgj0thn1wOjSTUYRltj4qQhllwZwhxSDdTgbfs3fFes9jcA/9Y1Esw5g72fAUzexdpMmsuwsPLNOzbn6aDZ1eeCcuYXj4kQASfthp2yf2OY5FUVkqXACWHzPO7IpUFiyj5EzkV/cbaIGgRsHxCeaimawgHUDnr38nUxyduygEHPIr2BVsWoW69JPWMQF5hM36XbWmer3LTM4EMxmwDTmB1KaPYA0mgRHjlO3lwlnD0/w9MtzMt8AdzoSZjQoYPVWrpeTcW/aKncItTnfxfM9Q67SgPDhUlJK67ToRtmnceQL57XohaY8LiGkbtaz22NiWaWAOUUglhaAbsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HLZ5E5FPu6PW6834tLYv+FhD0mSp/55NbvrpPImXnZs=; b=jTxQkeY2Qgii7szeOTy4OX9YpaFQhd/W/5YiQkCsGLBl4cipPSeQVnwP8XgLD/Vkqz1XD6nGKd1bk97LTUw7Gc19N9OuwXopYWp1nW/bTnz2nRm0dyiOxtamr1UwCaUxa2lWy4cxzZ3Pz/K0GRVQJ4TLzNXahpT2dEkaNSCD6Zx0749o6UcDo5vUF7FZAQTD0+teprvI9RE9Up7+zCZ0AeLPgykj7t+1VwOiwXaSWAjhuXfG+QOMFM5Xgb3f1fEV/acgtS/2l6c2BqcSuo1ftFYmBFnaXG77uJiVMUXrMFQk3wLEWyUubyrgY1T0K1rnxnUnK9j9XdYARrgxg/Dhlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HLZ5E5FPu6PW6834tLYv+FhD0mSp/55NbvrpPImXnZs=; b=kCdCcxbh4zjX7OCzHYKaCbwDUhd26LzvBM7ZgA3AOs85FCKDopmQq/X8f/hiqXew1tbXjU5lPODP/r2tJb+XkL+NEIZ0NWEgyH/9Qo80E7bySt/DlkQuZjrO+PWXbledoMWxQPpyyLHjtrNEJui/AP2i9ZJ2RSqwRGQNoLeC/5zS5mQs9voFMTE2jDURxU+FpHaQvc58Z+uERkuqPE4irNArzErJKIhcPL195G6XsZ0HJR5fpLX7uGodt8hnvsO1DMRur1g1MeMHQE9FgUtN/Omr8BJdS/1H9xbLPGODF2BdeE0+EEIecvnfLObS6fG/T5KkpFhG9zONHERnaObUCQ== Received: from BN9PR03CA0702.namprd03.prod.outlook.com (2603:10b6:408:ef::17) by LV2PR12MB5749.namprd12.prod.outlook.com (2603:10b6:408:17f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5924.16; Wed, 21 Dec 2022 07:40:05 +0000 Received: from BN8NAM11FT048.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ef:cafe::5e) by BN9PR03CA0702.outlook.office365.com (2603:10b6:408:ef::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5924.21 via Frontend Transport; Wed, 21 Dec 2022 07:40:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN8NAM11FT048.mail.protection.outlook.com (10.13.177.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.6 via Frontend Transport; Wed, 21 Dec 2022 07:40:05 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 20 Dec 2022 23:39:53 -0800 Received: from nvidia.com (10.126.231.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Tue, 20 Dec 2022 23:39:51 -0800 From: Rongwei Liu To: , , , CC: , Subject: [RFC 7/9] net/mlx5/hws: add flex item modify field implementation Date: Wed, 21 Dec 2022 09:39:16 +0200 Message-ID: <20221221073918.3581151-8-rongweil@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221221073918.3581151-1-rongweil@nvidia.com> References: <20221221073918.3581151-1-rongweil@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.37] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT048:EE_|LV2PR12MB5749:EE_ X-MS-Office365-Filtering-Correlation-Id: 13d79a84-648f-46cc-2dfb-08dae32693cd X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oBhpfasmZ3WV+kxJAaQV0FZEXYl8Qc2joi5SOHtOKpWLkFwWDS86ZLSuWVmxliAvVUsrowcQTkrA/D5AICNYjktEAHwsyHaWskV9/vSN3478NelGmi/UURMkFmihgP7/FS2iOG7ezYFKEZS19zyg0UhvYcDSsVXacBrHGaABb/G/XQMzeWrOWGZs6l47kVUP2ToX4srCPnINt/kRtN0OTSh9Yt7jnm0E7DGP5uJcGRXMasqF6BMoZNQM4JqF47NpfGk1m6Ab8UPU3iW/mslmBWplBRTzrDTO92le4X7z0IiKBJLeiIEi6mRjd39tbfsWaWeyr5olqT2FBkBQI6nKCi2RKgMUltuorJIJf75yqjwBVliNuMElgoPiYkdEiAP9a0PLTr4OPxm9HJw7lZtLybaH73vYfWG4ttqJ0V1aSkXnmLjb7bxn7McXlGySZYkma+LFfUbMVJMpPXEUUKoV2ZB3P+02MiRYhW/blDMNTnmYLTuhtga7lwt455QjhQlvLj90/7jQSImF9W/Grh50agXNDJDf7wYej/RRbsKbcgWFRs/XMCKzPhXPLchop0kOB2XejnubnsSxnGAfHxHN17U1JAVnedbfsbIC3XPTnGxweHorSSoViorIxCgZq4bC2Xjhfjx0EEP+sZLvyoNJ5ACiY7R3QhD5ZlAF62AyzBLYhvK0O5Gh6V05S6J2D0+4UIIWUKSnzOIffkys9u0MLg== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(136003)(39860400002)(396003)(376002)(346002)(451199015)(40470700004)(46966006)(36840700001)(8936002)(6286002)(5660300002)(4326008)(83380400001)(7636003)(30864003)(7696005)(8676002)(356005)(70206006)(40460700003)(70586007)(41300700001)(426003)(16526019)(2906002)(86362001)(82310400005)(54906003)(82740400003)(336012)(55016003)(2616005)(1076003)(186003)(47076005)(26005)(40480700001)(107886003)(110136005)(6666004)(478600001)(36756003)(36860700001)(316002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Dec 2022 07:40:05.2275 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 13d79a84-648f-46cc-2dfb-08dae32693cd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT048.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5749 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add flex item modify field HWS implementation. The minimum modify boundary is one byte. Signed-off-by: Rongwei Liu --- doc/guides/nics/mlx5.rst | 1 + drivers/common/mlx5/mlx5_prm.h | 1 + drivers/net/mlx5/mlx5_flow.h | 3 + drivers/net/mlx5/mlx5_flow_dv.c | 164 +++++++++++++++++++++++++++++--- drivers/net/mlx5/mlx5_flow_hw.c | 14 ++- 5 files changed, 170 insertions(+), 13 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index e442f9c015..9f95ff9112 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -107,6 +107,7 @@ Features - Sub-Function. - Matching on represented port. - Matching on flex item with specific pattern. +- Modify flex item field. Limitations diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 97bc1eac21..6c730ff85f 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -760,6 +760,7 @@ enum mlx5_modification_field { MLX5_MODI_TUNNEL_HDR_DW_1 = 0x75, MLX5_MODI_GTPU_FIRST_EXT_DW_0 = 0x76, MLX5_MODI_HASH_RESULT = 0x81, + MLX5_MODI_INVALID = INT_MAX, }; /* Total number of metadata reg_c's. */ diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 82b5a4a81f..531f34c281 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -1134,6 +1134,8 @@ struct field_modify_info { uint32_t size; /* Size of field in protocol header, in bytes. */ uint32_t offset; /* Offset of field in protocol header, in bytes. */ enum mlx5_modification_field id; + uint32_t shift; + uint8_t is_flex; /* Temporary indicator for flex item modify filed WA. */ }; /* HW steering flow attributes. */ @@ -1299,6 +1301,7 @@ struct rte_flow_actions_template { uint16_t mhdr_off; /* Offset of DR modify header action. */ uint32_t refcnt; /* Reference counter. */ uint16_t rx_cpy_pos; /* Action position of Rx metadata to be copied. */ + uint8_t flex_item; /* flex item index. */ }; /* Jump action struct. */ diff --git a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c index 310fb7c5c3..501a190c5e 100644 --- a/drivers/net/mlx5/mlx5_flow_dv.c +++ b/drivers/net/mlx5/mlx5_flow_dv.c @@ -424,10 +424,15 @@ flow_dv_convert_modify_action(struct rte_flow_item *item, ++field; continue; } - /* Deduce actual data width in bits from mask value. */ - off_b = rte_bsf32(mask) + carry_b; - size_b = sizeof(uint32_t) * CHAR_BIT - - off_b - __builtin_clz(mask); + if (type == MLX5_MODIFICATION_TYPE_COPY && field->is_flex) { + off_b = 32 - field->shift + carry_b - field->size * CHAR_BIT; + size_b = field->size * CHAR_BIT - carry_b; + } else { + /* Deduce actual data width in bits from mask value. */ + off_b = rte_bsf32(mask) + carry_b; + size_b = sizeof(uint32_t) * CHAR_BIT - + off_b - __builtin_clz(mask); + } MLX5_ASSERT(size_b); actions[i] = (struct mlx5_modification_cmd) { .action_type = type, @@ -447,40 +452,46 @@ flow_dv_convert_modify_action(struct rte_flow_item *item, * Destination field overflow. Copy leftovers of * a source field to the next destination field. */ - carry_b = 0; if ((size_b > dcopy->size * CHAR_BIT - dcopy->offset) && dcopy->size != 0) { actions[i].length = dcopy->size * CHAR_BIT - dcopy->offset; - carry_b = actions[i].length; + carry_b += actions[i].length; next_field = false; + } else { + carry_b = 0; } /* * Not enough bits in a source filed to fill a * destination field. Switch to the next source. */ if ((size_b < dcopy->size * CHAR_BIT - dcopy->offset) && - (size_b == field->size * CHAR_BIT - off_b)) { - actions[i].length = - field->size * CHAR_BIT - off_b; + ((size_b == field->size * CHAR_BIT - off_b) || + field->is_flex)) { + actions[i].length = size_b; dcopy->offset += actions[i].length; next_dcopy = false; } - if (next_dcopy) - ++dcopy; } else { MLX5_ASSERT(item->spec); data = flow_dv_fetch_field((const uint8_t *)item->spec + field->offset, field->size); /* Shift out the trailing masked bits from data. */ data = (data & mask) >> off_b; + if (field->is_flex) + actions[i].offset = 32 - field->shift - field->size * CHAR_BIT; actions[i].data1 = rte_cpu_to_be_32(data); } /* Convert entire record to expected big-endian format. */ actions[i].data0 = rte_cpu_to_be_32(actions[i].data0); + if ((type != MLX5_MODIFICATION_TYPE_COPY || + dcopy->id != (enum mlx5_modification_field)UINT32_MAX) && + field->id != (enum mlx5_modification_field)UINT32_MAX) + ++i; + if (next_dcopy && type == MLX5_MODIFICATION_TYPE_COPY) + ++dcopy; if (next_field) ++field; - ++i; } while (field->size); if (resource->actions_num == i) return rte_flow_error_set(error, EINVAL, @@ -1476,6 +1487,130 @@ flow_modify_info_mask_32_masked(uint32_t length, uint32_t off, uint32_t post_mas return rte_cpu_to_be_32(mask & post_mask); } +static void +mlx5_modify_flex_item(const struct rte_eth_dev *dev, + const struct mlx5_flex_item *flex, + const struct rte_flow_action_modify_data *data, + struct field_modify_info *info, + uint32_t *mask, uint32_t width) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_hca_flex_attr *attr = &priv->sh->cdev->config.hca_attr.flex; + uint32_t i, j; + int id = 0; + uint32_t pos = 0; + const struct mlx5_flex_pattern_field *map; + uint32_t offset = data->offset; + uint32_t width_left = width; + uint32_t def; + uint32_t cur_width = 0; + uint32_t tmp_ofs; + uint32_t idx = 0; + struct field_modify_info tmp; + int tmp_id; + + if (!attr->ext_sample_id) { + DRV_LOG(ERR, "FW doesn't support modify field with flex item."); + return; + } + /* + * search for the mapping instance until Accumulated width is no + * less than data->offset. + */ + for (i = 0; i < flex->mapnum; i++) { + if (flex->map[i].width + pos > data->offset) + break; + pos += flex->map[i].width; + } + if (i >= flex->mapnum) + return; + tmp_ofs = pos < data->offset ? data->offset - pos : 0; + for (j = i; i < flex->mapnum && width_left > 0; ) { + map = flex->map + i; + id = mlx5_flex_get_sample_id(flex, i, &pos, false, &def); + if (id == -1) { + i++; + /* All left length is dummy */ + if (pos >= data->offset + width) + return; + cur_width = map->width; + /* One mapping instance covers the whole width. */ + } else if (pos + map->width >= (data->offset + width)) { + cur_width = width_left; + } else { + cur_width = cur_width + map->width - tmp_ofs; + pos += map->width; + /* + * Continue to search next until: + * 1. Another flex parser ID. + * 2. Width has been covered. + */ + for (j = i + 1; j < flex->mapnum; j++) { + tmp_id = mlx5_flex_get_sample_id(flex, j, &pos, false, &def); + if (tmp_id == -1) { + i = j; + pos -= flex->map[j].width; + break; + } + if (id >= (int)flex->devx_fp->num_samples || + id >= MLX5_GRAPH_NODE_SAMPLE_NUM || + tmp_id >= (int)flex->devx_fp->num_samples || + tmp_id >= MLX5_GRAPH_NODE_SAMPLE_NUM) + return; + if (flex->devx_fp->sample_ids[id].id != + flex->devx_fp->sample_ids[tmp_id].id || + flex->map[j].shift != flex->map[j - 1].width + + flex->map[j - 1].shift) { + i = j; + break; + } + if ((pos + flex->map[j].width) >= (data->offset + width)) { + cur_width = width_left; + break; + } + pos += flex->map[j].width; + cur_width += flex->map[j].width; + } + } + if (cur_width > width_left) + cur_width = width_left; + else if (cur_width < width_left && (j == flex->mapnum || i == flex->mapnum)) + return; + + MLX5_ASSERT(id < (int)flex->devx_fp->num_samples); + if (id >= (int)flex->devx_fp->num_samples || id >= MLX5_GRAPH_NODE_SAMPLE_NUM) + return; + /* Use invalid entry as placeholder for DUMMY mapping. */ + info[idx] = (struct field_modify_info){cur_width / CHAR_BIT, offset / CHAR_BIT, + id == -1 ? MLX5_MODI_INVALID : + (enum mlx5_modification_field)flex->devx_fp->sample_ids[id].modify_field_id, + map->shift + tmp_ofs, 1}; + offset += cur_width; + width_left -= cur_width; + if (!mask) { + info[idx].offset = (32 - cur_width - map->shift - tmp_ofs); + info[idx].size = cur_width / CHAR_BIT + info[idx].offset / CHAR_BIT; + } + cur_width = 0; + tmp_ofs = 0; + idx++; + } + if (unlikely(width_left > 0)) { + MLX5_ASSERT(false); + return; + } + if (mask) + memset(mask, 0xff, data->offset / CHAR_BIT + width / CHAR_BIT); + /* Re-order the info to follow IPv6 address. */ + for (i = 0; i < idx / 2; i++) { + tmp = info[i]; + MLX5_ASSERT(info[i].id); + MLX5_ASSERT(info[idx - 1 - i].id); + info[i] = info[idx - 1 - i]; + info[idx - 1 - i] = tmp; + } +} + void mlx5_flow_field_id_to_modify_info (const struct rte_flow_action_modify_data *data, @@ -1950,6 +2085,11 @@ mlx5_flow_field_id_to_modify_info else info[idx].offset = off_be; break; + case RTE_FLOW_FIELD_FLEX_ITEM: + MLX5_ASSERT(data->flex_handle != NULL && !(data->offset & 0x7)); + mlx5_modify_flex_item(dev, (const struct mlx5_flex_item *)data->flex_handle, + data, info, mask, width); + break; case RTE_FLOW_FIELD_POINTER: case RTE_FLOW_FIELD_VALUE: default: diff --git a/drivers/net/mlx5/mlx5_flow_hw.c b/drivers/net/mlx5/mlx5_flow_hw.c index 56f1e699fa..042f8cdc7f 100644 --- a/drivers/net/mlx5/mlx5_flow_hw.c +++ b/drivers/net/mlx5/mlx5_flow_hw.c @@ -4850,6 +4850,17 @@ flow_hw_actions_template_create(struct rte_eth_dev *dev, at->actions[i].conf = actions->conf; at->masks[i].conf = masks->conf; } + if (actions->type == RTE_FLOW_ACTION_TYPE_MODIFY_FIELD) { + const struct rte_flow_action_modify_field *info = actions->conf; + + if ((info->dst.field == RTE_FLOW_FIELD_FLEX_ITEM && + flow_hw_flex_item_acquire(dev, info->dst.flex_handle, + &at->flex_item)) || + (info->src.field == RTE_FLOW_FIELD_FLEX_ITEM && + flow_hw_flex_item_acquire(dev, info->src.flex_handle, + &at->flex_item))) + goto error; + } } at->tmpl = flow_hw_dr_actions_template_create(at); if (!at->tmpl) @@ -4881,7 +4892,7 @@ flow_hw_actions_template_create(struct rte_eth_dev *dev, * 0 on success, a negative errno value otherwise and rte_errno is set. */ static int -flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused, +flow_hw_actions_template_destroy(struct rte_eth_dev *dev, struct rte_flow_actions_template *template, struct rte_flow_error *error __rte_unused) { @@ -4894,6 +4905,7 @@ flow_hw_actions_template_destroy(struct rte_eth_dev *dev __rte_unused, "action template in using"); } LIST_REMOVE(template, next); + flow_hw_flex_item_release(dev, &template->flex_item); if (template->tmpl) mlx5dr_action_template_destroy(template->tmpl); mlx5_free(template);