From patchwork Fri Jun 2 06:34:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumara Parameshwaran X-Patchwork-Id: 127947 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 555AE42C0E; Fri, 2 Jun 2023 08:34:33 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D9729406B8; Fri, 2 Jun 2023 08:34:32 +0200 (CEST) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mails.dpdk.org (Postfix) with ESMTP id 3F56840695 for ; Fri, 2 Jun 2023 08:34:31 +0200 (CEST) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-64d3bc502ddso2083130b3a.0 for ; Thu, 01 Jun 2023 23:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685687670; x=1688279670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tooszPnTTb5yTgnPyN/m53HHsdHS7KjEKRGyjVi55rM=; b=jyy13fGw1YO76Q/AQ/4zaoKi9l4mKwc85z0CQJC0rEXNhc0/aqtkmjPzceLjGPSAy8 tPX40YfLqI41XeDCv3XgHjCEm65e6z7eeva3UmsRvLBbfDDuw+mMzg1J6JI4VNwudxxX 8juOvdw38OvAYgM/JXVllUKIdAGPDRND9Iyz5OqesoObJ23gvniykrxH7pCrO6LUoVWI GAZF6YhaJwxsf+GBMmaX8aZvE4OJb1MZSzm2StqyoZl7u/1YNU5e75dKRN46ka1ksrWc f9bc5InZeRP6C6unYNq5KhKdSfOo0G+1YAyhBeYOZxM8kTFZhugGMHY/ohZHfkOBIz2f 3jLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685687670; x=1688279670; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tooszPnTTb5yTgnPyN/m53HHsdHS7KjEKRGyjVi55rM=; b=lBJeUR1N+a+uv9au7g7pes1OIs5a5hPGkIAiqIZzxJHs+znDXULkSBOPo9scJPlDGB s3jqL6723DQUlC5WYXofZI/hRA/6LwQnHYi7W+8iDLZcFevlVA99x9T9mOF4b6ysz3ZK F5Lea0rOQVGUo8lkhksDg+HWaDzj5/tYlt8PqAn0igfwOl5fvsNh33dBsHv7slE3nrtg IwX4txp31L4R0ZJCULSApOXOvojNnP+zLQ8FqMh7KGoTRlkXyFzpg8O2HqQ75jDjiDKY WUfRTM5DR6PtwlHFfd0GgFZdwLC7+89Ny7jpXytzfuep69vHE6JLTDo/Rk/z7AsNL6fu JdgA== X-Gm-Message-State: AC+VfDwma2wEy8QSaRPeFAROSuhLquInV7D130Bi19p3MwTFDNVD2PHF EWqcSXtHDK2UyZeMZBgAMWf29I5Xc98= X-Google-Smtp-Source: ACHHUZ69v6/umS3ajxQIUhHZLAkeqCWLwnVPgZy6798NzWuP/0uiMv1f/o/yI0SuxMiJqmVrUoeDNA== X-Received: by 2002:a05:6a20:ae10:b0:110:2e00:3d59 with SMTP id dp16-20020a056a20ae1000b001102e003d59mr9515139pzb.23.1685687670010; Thu, 01 Jun 2023 23:34:30 -0700 (PDT) Received: from kparameshw7KFWX.vmware.com.com ([49.207.243.222]) by smtp.gmail.com with ESMTPSA id x42-20020a056a000bea00b0064fd8b3dd10sm313264pfu.109.2023.06.01.23.34.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 01 Jun 2023 23:34:29 -0700 (PDT) From: Kumara Parameshwaran To: jiayu.hu@intel.com Cc: dev@dpdk.org, Kumara Parameshwaran Subject: [PATCH v3] gro : ipv6 changes to support GRO for TCP/ipv6 Date: Fri, 2 Jun 2023 12:04:23 +0530 Message-Id: <20230602063423.30312-1-kumaraparamesh92@gmail.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20221020181425.48006-1-kumaraparmesh92@gmail.com> References: <20221020181425.48006-1-kumaraparmesh92@gmail.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The patch adds GRO support for TCP/ipv6 packets. This does not include the support for vxlan, udp ipv6 packets. Signed-off-by: Kumara Parameshwaran --- v1: * Changes to support GRO for TCP/ipv6 packets. This does not include vxlan changes. * The GRO is performed only for ipv6 packets that does not contain extension headers. * The logic for the TCP coalescing remains the same, in ipv6 header the source address, destination address, flow label, version fields are expected to be the same. * Re-organised the code to reuse certain tcp functions for both ipv4 and ipv6 flows. v2: * Fix comments in gro_tcp6.h header file. v3: * Adderess review comments to fix code duplication for v4 and v6 lib/gro/gro_tcp.c | 160 ++++++++++++++++++++++++ lib/gro/gro_tcp.h | 63 ++++++++++ lib/gro/gro_tcp4.c | 255 ++++++++++++--------------------------- lib/gro/gro_tcp4.h | 18 +-- lib/gro/gro_tcp6.c | 243 ++++++++++--------------------------- lib/gro/gro_tcp6.h | 31 +++-- lib/gro/gro_vxlan_tcp4.c | 18 +-- lib/gro/meson.build | 1 + 8 files changed, 396 insertions(+), 393 deletions(-) create mode 100644 lib/gro/gro_tcp.c diff --git a/lib/gro/gro_tcp.c b/lib/gro/gro_tcp.c new file mode 100644 index 0000000000..6a5aaada58 --- /dev/null +++ b/lib/gro/gro_tcp.c @@ -0,0 +1,160 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2017 Intel Corporation + */ +#include +#include +#include + +#include "gro_tcp.h" + +static inline uint32_t +find_an_empty_item(struct gro_tcp_item *items, + uint32_t table_size) +{ + uint32_t i; + + for (i = 0; i < table_size; i++) + if (items[i].firstseg == NULL) + return i; + return INVALID_ARRAY_INDEX; +} + +uint32_t +insert_new_tcp_item(struct rte_mbuf *pkt, + struct gro_tcp_item *items, + uint32_t *item_num, + uint32_t table_size, + uint64_t start_time, + uint32_t prev_idx, + uint32_t sent_seq, + uint16_t ip_id, + uint8_t is_atomic) +{ + uint32_t item_idx; + + item_idx = find_an_empty_item(items, table_size); + if (item_idx == INVALID_ARRAY_INDEX) + return INVALID_ARRAY_INDEX; + + items[item_idx].firstseg = pkt; + items[item_idx].lastseg = rte_pktmbuf_lastseg(pkt); + items[item_idx].start_time = start_time; + items[item_idx].next_pkt_idx = INVALID_ARRAY_INDEX; + items[item_idx].sent_seq = sent_seq; + items[item_idx].ip_id = ip_id; + items[item_idx].nb_merged = 1; + items[item_idx].is_atomic = is_atomic; + (*item_num) += 1; + + /* if the previous packet exists, chain them together. */ + if (prev_idx != INVALID_ARRAY_INDEX) { + items[item_idx].next_pkt_idx = + items[prev_idx].next_pkt_idx; + items[prev_idx].next_pkt_idx = item_idx; + } + + return item_idx; +} + +uint32_t +delete_tcp_item(struct gro_tcp_item *items, uint32_t item_idx, + uint32_t *item_num, + uint32_t prev_item_idx) +{ + uint32_t next_idx = items[item_idx].next_pkt_idx; + + /* NULL indicates an empty item */ + items[item_idx].firstseg = NULL; + (*item_num) -= 1; + if (prev_item_idx != INVALID_ARRAY_INDEX) + items[prev_item_idx].next_pkt_idx = next_idx; + + return next_idx; +} + +int32_t +gro_tcp_reassemble(struct rte_mbuf *pkt, + void *tbl, + void *key, + int32_t tcp_dl, + struct gro_tcp_flow_ops *ops, + struct gro_tcp_item *items, + uint32_t *item_num, + uint32_t table_size, + uint16_t ip_id, + uint8_t is_atomic, + uint64_t start_time) +{ + uint32_t item_idx; + uint32_t cur_idx; + uint32_t prev_idx; + struct rte_tcp_hdr *tcp_hdr; + int cmp; + uint32_t sent_seq; + + tcp_hdr = rte_pktmbuf_mtod_offset(pkt, struct rte_tcp_hdr *, pkt->l2_len + pkt->l3_len); + /* + * Don't process the packet which has FIN, SYN, RST, PSH, URG, ECE + * or CWR set. + */ + if (tcp_hdr->tcp_flags != RTE_TCP_ACK_FLAG) + return -1; + sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); + + ops->tcp_flow_key_init(key, tcp_hdr); + + item_idx = ops->tcp_flow_lookup(tbl, key); + if (item_idx == INVALID_ARRAY_INDEX) { + item_idx = insert_new_tcp_item(pkt, items, item_num, table_size, start_time, + INVALID_ARRAY_INDEX, sent_seq, ip_id, + is_atomic); + if (item_idx == INVALID_ARRAY_INDEX) + return -1; + if (ops->tcp_flow_insert(tbl, key, item_idx) == + INVALID_ARRAY_INDEX) { + /* + * Fail to insert a new flow, so delete the + * stored packet. + */ + delete_tcp_item(items, item_idx, item_num, INVALID_ARRAY_INDEX); + return -1; + } + return 0; + } + /* + * Check all packets in the flow and try to find a neighbor for + * the input packet. + */ + cur_idx = item_idx; + prev_idx = cur_idx; + do { + cmp = check_seq_option(&items[cur_idx], tcp_hdr, + sent_seq, ip_id, pkt->l4_len, tcp_dl, 0, + is_atomic); + if (cmp) { + if (merge_two_tcp_packets(&items[cur_idx], + pkt, cmp, sent_seq, ip_id, 0)) + return 1; + /* + * Fail to merge the two packets, as the packet + * length is greater than the max value. Store + * the packet into the flow. + */ + if (insert_new_tcp_item(pkt, items, item_num, table_size, start_time, cur_idx, + sent_seq, ip_id, is_atomic) == + INVALID_ARRAY_INDEX) + return -1; + return 0; + } + prev_idx = cur_idx; + cur_idx = items[cur_idx].next_pkt_idx; + } while (cur_idx != INVALID_ARRAY_INDEX); + + /* Fail to find a neighbor, so store the packet into the flow. */ + if (insert_new_tcp_item(pkt, items, item_num, table_size, start_time, prev_idx, sent_seq, + ip_id, is_atomic) == INVALID_ARRAY_INDEX) + return -1; + + return 0; + +} diff --git a/lib/gro/gro_tcp.h b/lib/gro/gro_tcp.h index c5d248a022..202f485c18 100644 --- a/lib/gro/gro_tcp.h +++ b/lib/gro/gro_tcp.h @@ -1,6 +1,8 @@ #ifndef _GRO_TCP_H_ #define _GRO_TCP_H_ +#define INVALID_ARRAY_INDEX 0xffffffffUL + #include /* @@ -14,6 +16,31 @@ #define INVALID_TCP_HDRLEN(len) \ (((len) < sizeof(struct rte_tcp_hdr)) || ((len) > MAX_TCP_HLEN)) +struct gro_tcp_flow { + struct rte_ether_addr eth_saddr; + struct rte_ether_addr eth_daddr; + uint32_t recv_ack; + uint16_t src_port; + uint16_t dst_port; +}; + +#define ASSIGN_TCP_FLOW_KEY(k1, k2) \ + rte_ether_addr_copy(&(k1->eth_saddr), &(k2->eth_saddr)); \ + rte_ether_addr_copy(&(k1->eth_daddr), &(k2->eth_daddr)); \ + k2->recv_ack = k1->recv_ack; \ + k2->src_port = k1->src_port; \ + k2->dst_port = k1->dst_port; + +typedef uint32_t (*gro_tcp_flow_lookup)(void *table, void *key); +typedef uint32_t (*gro_tcp_flow_insert)(void *table, void *key, uint32_t item_idx); +typedef void (*gro_tcp_flow_key_init)(void *key, struct rte_tcp_hdr *tcp_hdr); + +struct gro_tcp_flow_ops { + gro_tcp_flow_lookup tcp_flow_lookup; + gro_tcp_flow_insert tcp_flow_insert; + gro_tcp_flow_key_init tcp_flow_key_init; +}; + struct gro_tcp_item { /* * The first MBUF segment of the packet. If the value @@ -44,6 +71,36 @@ struct gro_tcp_item { uint8_t is_atomic; }; +uint32_t +insert_new_tcp_item(struct rte_mbuf *pkt, + struct gro_tcp_item *items, + uint32_t *item_num, + uint32_t table_size, + uint64_t start_time, + uint32_t prev_idx, + uint32_t sent_seq, + uint16_t ip_id, + uint8_t is_atomic); + +uint32_t +delete_tcp_item(struct gro_tcp_item *items, + uint32_t item_idx, + uint32_t *item_num, + uint32_t prev_item_idx); + +int32_t +gro_tcp_reassemble(struct rte_mbuf *pkt, + void *tbl, + void *key, + int32_t tcp_dl, + struct gro_tcp_flow_ops *ops, + struct gro_tcp_item *items, + uint32_t *item_num, + uint32_t table_size, + uint16_t ip_id, + uint8_t is_atomic, + uint64_t start_time); + /* * Merge two TCP packets without updating checksums. * If cmp is larger than 0, append the new packet to the @@ -152,4 +209,10 @@ check_seq_option(struct gro_tcp_item *item, return 0; } +static inline int +is_same_tcp_flow(struct gro_tcp_flow *k1, struct gro_tcp_flow *k2) +{ + return (!memcmp(k1, k2, sizeof(struct gro_tcp_flow))); +} + #endif diff --git a/lib/gro/gro_tcp4.c b/lib/gro/gro_tcp4.c index 2cfb524689..cd510d3a7c 100644 --- a/lib/gro/gro_tcp4.c +++ b/lib/gro/gro_tcp4.c @@ -72,18 +72,6 @@ gro_tcp4_tbl_destroy(void *tbl) rte_free(tcp_tbl); } -static inline uint32_t -find_an_empty_item(struct gro_tcp4_tbl *tbl) -{ - uint32_t i; - uint32_t max_item_num = tbl->max_item_num; - - for (i = 0; i < max_item_num; i++) - if (tbl->items[i].firstseg == NULL) - return i; - return INVALID_ARRAY_INDEX; -} - static inline uint32_t find_an_empty_flow(struct gro_tcp4_tbl *tbl) { @@ -97,98 +85,76 @@ find_an_empty_flow(struct gro_tcp4_tbl *tbl) } static inline uint32_t -insert_new_item(struct gro_tcp4_tbl *tbl, - struct rte_mbuf *pkt, - uint64_t start_time, - uint32_t prev_idx, - uint32_t sent_seq, - uint16_t ip_id, - uint8_t is_atomic) +gro_tcp4_insert_flow(void *tbl, + void *key, + uint32_t item_idx) { - uint32_t item_idx; + struct gro_tcp4_tbl *tcp4_tbl = (struct gro_tcp4_tbl *)tbl; + struct tcp4_flow_key *src_key = (struct tcp4_flow_key *)key; + struct tcp4_flow_key *dst_key; + struct gro_tcp_flow *src_flow; + struct gro_tcp_flow *dst_flow; + uint32_t flow_idx; - item_idx = find_an_empty_item(tbl); - if (item_idx == INVALID_ARRAY_INDEX) + flow_idx = find_an_empty_flow(tcp4_tbl); + if (unlikely(flow_idx == INVALID_ARRAY_INDEX)) return INVALID_ARRAY_INDEX; - tbl->items[item_idx].firstseg = pkt; - tbl->items[item_idx].lastseg = rte_pktmbuf_lastseg(pkt); - tbl->items[item_idx].start_time = start_time; - tbl->items[item_idx].next_pkt_idx = INVALID_ARRAY_INDEX; - tbl->items[item_idx].sent_seq = sent_seq; - tbl->items[item_idx].ip_id = ip_id; - tbl->items[item_idx].nb_merged = 1; - tbl->items[item_idx].is_atomic = is_atomic; - tbl->item_num++; - - /* if the previous packet exists, chain them together. */ - if (prev_idx != INVALID_ARRAY_INDEX) { - tbl->items[item_idx].next_pkt_idx = - tbl->items[prev_idx].next_pkt_idx; - tbl->items[prev_idx].next_pkt_idx = item_idx; - } + dst_key = &(tcp4_tbl->flows[flow_idx].key); + dst_flow = &dst_key->tcp_flow; + src_flow = &src_key->tcp_flow; - return item_idx; -} + ASSIGN_TCP_FLOW_KEY(src_flow, dst_flow) -static inline uint32_t -delete_item(struct gro_tcp4_tbl *tbl, uint32_t item_idx, - uint32_t prev_item_idx) -{ - uint32_t next_idx = tbl->items[item_idx].next_pkt_idx; + dst_key->ip_src_addr = src_key->ip_src_addr; + dst_key->ip_dst_addr = src_key->ip_dst_addr; - /* NULL indicates an empty item */ - tbl->items[item_idx].firstseg = NULL; - tbl->item_num--; - if (prev_item_idx != INVALID_ARRAY_INDEX) - tbl->items[prev_item_idx].next_pkt_idx = next_idx; + tcp4_tbl->flows[flow_idx].start_index = item_idx; + tcp4_tbl->flow_num++; - return next_idx; + return flow_idx; } -static inline uint32_t -insert_new_flow(struct gro_tcp4_tbl *tbl, - struct tcp4_flow_key *src, - uint32_t item_idx) -{ - struct tcp4_flow_key *dst; - uint32_t flow_idx; - - flow_idx = find_an_empty_flow(tbl); - if (unlikely(flow_idx == INVALID_ARRAY_INDEX)) - return INVALID_ARRAY_INDEX; - - dst = &(tbl->flows[flow_idx].key); - - rte_ether_addr_copy(&(src->eth_saddr), &(dst->eth_saddr)); - rte_ether_addr_copy(&(src->eth_daddr), &(dst->eth_daddr)); - dst->ip_src_addr = src->ip_src_addr; - dst->ip_dst_addr = src->ip_dst_addr; - dst->recv_ack = src->recv_ack; - dst->src_port = src->src_port; - dst->dst_port = src->dst_port; - tbl->flows[flow_idx].start_index = item_idx; - tbl->flow_num++; +static uint32_t +gro_tcp4_flow_lookup(void *tbl, void *key) +{ + struct gro_tcp4_tbl *tcp4_tbl = (struct gro_tcp4_tbl *)tbl; + struct tcp4_flow_key *tcp4_key = (struct tcp4_flow_key *)key; + uint32_t max_flow_num = tcp4_tbl->max_flow_num; + uint32_t remaining_flow_num = tcp4_tbl->flow_num; + uint32_t i; - return flow_idx; + /* Search for a matched flow. */ + for (i = 0; i < max_flow_num && remaining_flow_num; i++) { + if (tcp4_tbl->flows[i].start_index != INVALID_ARRAY_INDEX) { + if (is_same_tcp4_flow(tcp4_tbl->flows[i].key, *tcp4_key)) { + return i; + } + remaining_flow_num--; + } + } + return INVALID_ARRAY_INDEX; } -/* - * update the packet length for the flushed packet. - */ -static inline void -update_header(struct gro_tcp_item *item) +static void +gro_tcp4_flow_key_init(void *key, struct rte_tcp_hdr *tcp_hdr) { - struct rte_ipv4_hdr *ipv4_hdr; - struct rte_mbuf *pkt = item->firstseg; + struct tcp4_flow_key *tcp4_key = (struct tcp4_flow_key *)key; + struct gro_tcp_flow *tcp_flow = &tcp4_key->tcp_flow; + + tcp_flow->src_port = tcp_hdr->src_port; + tcp_flow->dst_port = tcp_hdr->dst_port; + tcp_flow->recv_ack = tcp_hdr->recv_ack; - ipv4_hdr = (struct rte_ipv4_hdr *)(rte_pktmbuf_mtod(pkt, char *) + - pkt->l2_len); - ipv4_hdr->total_length = rte_cpu_to_be_16(pkt->pkt_len - - pkt->l2_len); } +struct gro_tcp_flow_ops gro_tcp4_flows_ops = { + .tcp_flow_lookup = gro_tcp4_flow_lookup, + .tcp_flow_insert = gro_tcp4_insert_flow, + .tcp_flow_key_init = gro_tcp4_flow_key_init +}; + int32_t gro_tcp4_reassemble(struct rte_mbuf *pkt, struct gro_tcp4_tbl *tbl, @@ -196,17 +162,10 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, { struct rte_ether_hdr *eth_hdr; struct rte_ipv4_hdr *ipv4_hdr; - struct rte_tcp_hdr *tcp_hdr; - uint32_t sent_seq; int32_t tcp_dl; uint16_t ip_id, hdr_len, frag_off, ip_tlen; uint8_t is_atomic; - struct tcp4_flow_key key; - uint32_t cur_idx, prev_idx, item_idx; - uint32_t i, max_flow_num, remaining_flow_num; - int cmp; - uint8_t find; /* * Don't process the packet whose TCP header length is greater @@ -217,21 +176,13 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, eth_hdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *); ipv4_hdr = (struct rte_ipv4_hdr *)((char *)eth_hdr + pkt->l2_len); - tcp_hdr = (struct rte_tcp_hdr *)((char *)ipv4_hdr + pkt->l3_len); - hdr_len = pkt->l2_len + pkt->l3_len + pkt->l4_len; - - /* - * Don't process the packet which has FIN, SYN, RST, PSH, URG, ECE - * or CWR set. - */ - if (tcp_hdr->tcp_flags != RTE_TCP_ACK_FLAG) - return -1; /* trim the tail padding bytes */ ip_tlen = rte_be_to_cpu_16(ipv4_hdr->total_length); if (pkt->pkt_len > (uint32_t)(ip_tlen + pkt->l2_len)) rte_pktmbuf_trim(pkt, pkt->pkt_len - ip_tlen - pkt->l2_len); + hdr_len = pkt->l2_len + pkt->l3_len + pkt->l4_len; /* * Don't process the packet whose payload length is less than or * equal to 0. @@ -240,6 +191,10 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, if (tcp_dl <= 0) return -1; + rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.tcp_flow.eth_saddr)); + rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.tcp_flow.eth_daddr)); + key.ip_src_addr = ipv4_hdr->src_addr; + key.ip_dst_addr = ipv4_hdr->dst_addr; /* * Save IPv4 ID for the packet whose DF bit is 0. For the packet * whose DF bit is 1, IPv4 ID is ignored. @@ -247,87 +202,25 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, frag_off = rte_be_to_cpu_16(ipv4_hdr->fragment_offset); is_atomic = (frag_off & RTE_IPV4_HDR_DF_FLAG) == RTE_IPV4_HDR_DF_FLAG; ip_id = is_atomic ? 0 : rte_be_to_cpu_16(ipv4_hdr->packet_id); - sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); - rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.eth_saddr)); - rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.eth_daddr)); - key.ip_src_addr = ipv4_hdr->src_addr; - key.ip_dst_addr = ipv4_hdr->dst_addr; - key.src_port = tcp_hdr->src_port; - key.dst_port = tcp_hdr->dst_port; - key.recv_ack = tcp_hdr->recv_ack; - - /* Search for a matched flow. */ - max_flow_num = tbl->max_flow_num; - remaining_flow_num = tbl->flow_num; - find = 0; - for (i = 0; i < max_flow_num && remaining_flow_num; i++) { - if (tbl->flows[i].start_index != INVALID_ARRAY_INDEX) { - if (is_same_tcp4_flow(tbl->flows[i].key, key)) { - find = 1; - break; - } - remaining_flow_num--; - } - } - - /* - * Fail to find a matched flow. Insert a new flow and store the - * packet into the flow. - */ - if (find == 0) { - item_idx = insert_new_item(tbl, pkt, start_time, - INVALID_ARRAY_INDEX, sent_seq, ip_id, - is_atomic); - if (item_idx == INVALID_ARRAY_INDEX) - return -1; - if (insert_new_flow(tbl, &key, item_idx) == - INVALID_ARRAY_INDEX) { - /* - * Fail to insert a new flow, so delete the - * stored packet. - */ - delete_item(tbl, item_idx, INVALID_ARRAY_INDEX); - return -1; - } - return 0; - } - - /* - * Check all packets in the flow and try to find a neighbor for - * the input packet. - */ - cur_idx = tbl->flows[i].start_index; - prev_idx = cur_idx; - do { - cmp = check_seq_option(&(tbl->items[cur_idx]), tcp_hdr, - sent_seq, ip_id, pkt->l4_len, tcp_dl, 0, - is_atomic); - if (cmp) { - if (merge_two_tcp_packets(&(tbl->items[cur_idx]), - pkt, cmp, sent_seq, ip_id, 0)) - return 1; - /* - * Fail to merge the two packets, as the packet - * length is greater than the max value. Store - * the packet into the flow. - */ - if (insert_new_item(tbl, pkt, start_time, cur_idx, - sent_seq, ip_id, is_atomic) == - INVALID_ARRAY_INDEX) - return -1; - return 0; - } - prev_idx = cur_idx; - cur_idx = tbl->items[cur_idx].next_pkt_idx; - } while (cur_idx != INVALID_ARRAY_INDEX); + return gro_tcp_reassemble(pkt, tbl, &key, tcp_dl, &gro_tcp4_flows_ops, tbl->items, + &tbl->item_num, tbl->max_item_num, + ip_id, is_atomic, start_time); +} - /* Fail to find a neighbor, so store the packet into the flow. */ - if (insert_new_item(tbl, pkt, start_time, prev_idx, sent_seq, - ip_id, is_atomic) == INVALID_ARRAY_INDEX) - return -1; +/* + * update the packet length for the flushed packet. + */ +static inline void +update_header(struct gro_tcp_item *item) +{ + struct rte_ipv4_hdr *ipv4_hdr; + struct rte_mbuf *pkt = item->firstseg; - return 0; + ipv4_hdr = (struct rte_ipv4_hdr *)(rte_pktmbuf_mtod(pkt, char *) + + pkt->l2_len); + ipv4_hdr->total_length = rte_cpu_to_be_16(pkt->pkt_len - + pkt->l2_len); } uint16_t @@ -354,7 +247,7 @@ gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl *tbl, * Delete the packet and get the next * packet in the flow. */ - j = delete_item(tbl, j, INVALID_ARRAY_INDEX); + j = delete_tcp_item(tbl->items, j, &tbl->item_num, INVALID_ARRAY_INDEX); tbl->flows[i].start_index = j; if (j == INVALID_ARRAY_INDEX) tbl->flow_num--; diff --git a/lib/gro/gro_tcp4.h b/lib/gro/gro_tcp4.h index 634a215b98..8ff1eed096 100644 --- a/lib/gro/gro_tcp4.h +++ b/lib/gro/gro_tcp4.h @@ -5,19 +5,15 @@ #ifndef _GRO_TCP4_H_ #define _GRO_TCP4_H_ -#define INVALID_ARRAY_INDEX 0xffffffffUL +#include + #define GRO_TCP4_TBL_MAX_ITEM_NUM (1024UL * 1024UL) /* Header fields representing a TCP/IPv4 flow */ struct tcp4_flow_key { - struct rte_ether_addr eth_saddr; - struct rte_ether_addr eth_daddr; + struct gro_tcp_flow tcp_flow; uint32_t ip_src_addr; uint32_t ip_dst_addr; - - uint32_t recv_ack; - uint16_t src_port; - uint16_t dst_port; }; struct gro_tcp4_flow { @@ -143,13 +139,9 @@ uint32_t gro_tcp4_tbl_pkt_count(void *tbl); static inline int is_same_tcp4_flow(struct tcp4_flow_key k1, struct tcp4_flow_key k2) { - return (rte_is_same_ether_addr(&k1.eth_saddr, &k2.eth_saddr) && - rte_is_same_ether_addr(&k1.eth_daddr, &k2.eth_daddr) && - (k1.ip_src_addr == k2.ip_src_addr) && + return ((k1.ip_src_addr == k2.ip_src_addr) && (k1.ip_dst_addr == k2.ip_dst_addr) && - (k1.recv_ack == k2.recv_ack) && - (k1.src_port == k2.src_port) && - (k1.dst_port == k2.dst_port)); + is_same_tcp_flow(&k1.tcp_flow, &k2.tcp_flow)); } #endif diff --git a/lib/gro/gro_tcp6.c b/lib/gro/gro_tcp6.c index a37d8e2512..da8d1f2252 100644 --- a/lib/gro/gro_tcp6.c +++ b/lib/gro/gro_tcp6.c @@ -72,18 +72,6 @@ gro_tcp6_tbl_destroy(void *tbl) rte_free(tcp_tbl); } -static inline uint32_t -find_an_empty_item(struct gro_tcp6_tbl *tbl) -{ - uint32_t i; - uint32_t max_item_num = tbl->max_item_num; - - for (i = 0; i < max_item_num; i++) - if (tbl->items[i].firstseg == NULL) - return i; - return INVALID_ARRAY_INDEX; -} - static inline uint32_t find_an_empty_flow(struct gro_tcp6_tbl *tbl) { @@ -97,79 +85,32 @@ find_an_empty_flow(struct gro_tcp6_tbl *tbl) } static inline uint32_t -insert_new_item(struct gro_tcp6_tbl *tbl, - struct rte_mbuf *pkt, - uint64_t start_time, - uint32_t prev_idx, - uint32_t sent_seq, - uint8_t is_atomic) -{ - uint32_t item_idx; - - item_idx = find_an_empty_item(tbl); - if (item_idx == INVALID_ARRAY_INDEX) - return INVALID_ARRAY_INDEX; - - tbl->items[item_idx].firstseg = pkt; - tbl->items[item_idx].lastseg = rte_pktmbuf_lastseg(pkt); - tbl->items[item_idx].start_time = start_time; - tbl->items[item_idx].next_pkt_idx = INVALID_ARRAY_INDEX; - tbl->items[item_idx].sent_seq = sent_seq; - tbl->items[item_idx].nb_merged = 1; - tbl->items[item_idx].is_atomic = is_atomic; - tbl->item_num++; - - /* if the previous packet exists, chain them together. */ - if (prev_idx != INVALID_ARRAY_INDEX) { - tbl->items[item_idx].next_pkt_idx = - tbl->items[prev_idx].next_pkt_idx; - tbl->items[prev_idx].next_pkt_idx = item_idx; - } - - return item_idx; -} - -static inline uint32_t -delete_item(struct gro_tcp6_tbl *tbl, uint32_t item_idx, - uint32_t prev_item_idx) -{ - uint32_t next_idx = tbl->items[item_idx].next_pkt_idx; - - /* NULL indicates an empty item */ - tbl->items[item_idx].firstseg = NULL; - tbl->item_num--; - if (prev_item_idx != INVALID_ARRAY_INDEX) - tbl->items[prev_item_idx].next_pkt_idx = next_idx; - - return next_idx; -} - -static inline uint32_t -insert_new_flow(struct gro_tcp6_tbl *tbl, - struct tcp6_flow_key *src, - rte_be32_t vtc_flow, +gro_tcp6_insert_flow(void *tbl, + void *key, uint32_t item_idx) { - struct tcp6_flow_key *dst; + struct gro_tcp6_tbl *tcp6_tbl = (struct gro_tcp6_tbl *)tbl; + struct tcp6_flow_key *dst_key; + struct tcp6_flow_key *src_key = (struct tcp6_flow_key *)key; uint32_t flow_idx; + struct gro_tcp_flow *src_flow; + struct gro_tcp_flow *dst_flow; - flow_idx = find_an_empty_flow(tbl); + flow_idx = find_an_empty_flow(tcp6_tbl); if (unlikely(flow_idx == INVALID_ARRAY_INDEX)) return INVALID_ARRAY_INDEX; - dst = &(tbl->flows[flow_idx].key); + dst_key = &(tcp6_tbl->flows[flow_idx].key); + dst_flow = &dst_key->tcp_flow; + src_flow = &src_key->tcp_flow; - rte_ether_addr_copy(&(src->eth_saddr), &(dst->eth_saddr)); - rte_ether_addr_copy(&(src->eth_daddr), &(dst->eth_daddr)); - memcpy(&dst->src_addr[0], &src->src_addr[0], sizeof(dst->src_addr)); - memcpy(&dst->dst_addr[0], &src->dst_addr[0], sizeof(dst->dst_addr)); - dst->recv_ack = src->recv_ack; - dst->src_port = src->src_port; - dst->dst_port = src->dst_port; + ASSIGN_TCP_FLOW_KEY(src_flow, dst_flow) + memcpy(&dst_key->src_addr[0], &src_key->src_addr[0], sizeof(dst_key->src_addr)); + memcpy(&dst_key->dst_addr[0], &src_key->dst_addr[0], sizeof(dst_key->dst_addr)); + dst_key->vtc_flow = src_key->vtc_flow; - tbl->flows[flow_idx].start_index = item_idx; - tbl->flow_num++; - tbl->flows->vtc_flow = vtc_flow; + tcp6_tbl->flows[flow_idx].start_index = item_idx; + tcp6_tbl->flow_num++; return flow_idx; } @@ -189,6 +130,47 @@ update_header(struct gro_tcp_item *item) pkt->l2_len - pkt->l3_len); } +static uint32_t +gro_tcp6_flow_lookup(void *tbl, void *key) +{ + struct gro_tcp6_tbl *tcp6_tbl = (struct gro_tcp6_tbl *)tbl; + struct tcp6_flow_key *tcp6_key = (struct tcp6_flow_key *)key; + uint32_t max_flow_num = tcp6_tbl->max_flow_num; + uint32_t remaining_flow_num = tcp6_tbl->flow_num; + uint32_t i; + + /* Search for a matched flow. */ + max_flow_num = tcp6_tbl->max_flow_num; + remaining_flow_num = tcp6_tbl->flow_num; + for (i = 0; i < max_flow_num && remaining_flow_num; i++) { + if (tcp6_tbl->flows[i].start_index != INVALID_ARRAY_INDEX) { + if (is_same_tcp6_flow(&tcp6_tbl->flows[i].key, tcp6_key)) { + return i; + } + remaining_flow_num--; + } + } + + return INVALID_ARRAY_INDEX; +} + +static void +gro_tcp6_flow_key_init(void *key, struct rte_tcp_hdr *tcp_hdr) +{ + struct tcp6_flow_key *tcp6_key = (struct tcp6_flow_key *)key; + struct gro_tcp_flow *tcp_flow = &tcp6_key->tcp_flow; + + tcp_flow->src_port = tcp_hdr->src_port; + tcp_flow->dst_port = tcp_hdr->dst_port; + tcp_flow->recv_ack = tcp_hdr->recv_ack; +} + +struct gro_tcp_flow_ops gro_tcp6_flows_ops = { + .tcp_flow_lookup = gro_tcp6_flow_lookup, + .tcp_flow_insert = gro_tcp6_insert_flow, + .tcp_flow_key_init = gro_tcp6_flow_key_init +}; + int32_t gro_tcp6_reassemble(struct rte_mbuf *pkt, struct gro_tcp6_tbl *tbl, @@ -196,16 +178,9 @@ gro_tcp6_reassemble(struct rte_mbuf *pkt, { struct rte_ether_hdr *eth_hdr; struct rte_ipv6_hdr *ipv6_hdr; - struct rte_tcp_hdr *tcp_hdr; - uint32_t sent_seq; int32_t tcp_dl; uint16_t ip_tlen; struct tcp6_flow_key key; - uint32_t cur_idx, prev_idx, item_idx; - uint32_t i, max_flow_num, remaining_flow_num; - int cmp; - uint8_t find; - rte_be32_t vtc_flow_diff; /* * Don't process the packet whose TCP header length is greater @@ -216,14 +191,6 @@ gro_tcp6_reassemble(struct rte_mbuf *pkt, eth_hdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *); ipv6_hdr = (struct rte_ipv6_hdr *)((char *)eth_hdr + pkt->l2_len); - tcp_hdr = (struct rte_tcp_hdr *)((char *)ipv6_hdr + pkt->l3_len); - - /* - * Don't process the packet which has FIN, SYN, RST, PSH, URG, ECE - * or CWR set. - */ - if (tcp_hdr->tcp_flags != RTE_TCP_ACK_FLAG) - return -1; ip_tlen = rte_be_to_cpu_16(ipv6_hdr->payload_len); /* @@ -242,95 +209,15 @@ gro_tcp6_reassemble(struct rte_mbuf *pkt, if (tcp_dl <= 0) return -1; - sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); - - rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.eth_saddr)); - rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.eth_daddr)); + rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.tcp_flow.eth_saddr)); + rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.tcp_flow.eth_daddr)); memcpy(&key.src_addr[0], &ipv6_hdr->src_addr, sizeof(key.src_addr)); memcpy(&key.dst_addr[0], &ipv6_hdr->dst_addr, sizeof(key.dst_addr)); - key.src_port = tcp_hdr->src_port; - key.dst_port = tcp_hdr->dst_port; - key.recv_ack = tcp_hdr->recv_ack; + key.vtc_flow = ipv6_hdr->vtc_flow; - /* Search for a matched flow. */ - max_flow_num = tbl->max_flow_num; - remaining_flow_num = tbl->flow_num; - find = 0; - for (i = 0; i < max_flow_num && remaining_flow_num; i++) { - if (tbl->flows[i].start_index != INVALID_ARRAY_INDEX) { - if (is_same_tcp6_flow(tbl->flows[i].key, key)) { - /* - * IP version (4) Traffic Class (8) Flow Label (20) - * All fields except Traffic class should be same - */ - vtc_flow_diff = (ipv6_hdr->vtc_flow ^ tbl->flows->vtc_flow); - if (vtc_flow_diff & htonl(0xF00FFFFF)) - continue; - find = 1; - break; - } - remaining_flow_num--; - } - } - - /* - * Fail to find a matched flow. Insert a new flow and store the - * packet into the flow. - */ - if (find == 0) { - item_idx = insert_new_item(tbl, pkt, start_time, - INVALID_ARRAY_INDEX, sent_seq, true); - if (item_idx == INVALID_ARRAY_INDEX) - return -1; - if (insert_new_flow(tbl, &key, ipv6_hdr->vtc_flow, item_idx) == - INVALID_ARRAY_INDEX) { - /* - * Fail to insert a new flow, so delete the - * stored packet. - */ - delete_item(tbl, item_idx, INVALID_ARRAY_INDEX); - return -1; - } - return 0; - } - - /* - * Check all packets in the flow and try to find a neighbor for - * the input packet. - */ - cur_idx = tbl->flows[i].start_index; - prev_idx = cur_idx; - do { - cmp = check_seq_option(&(tbl->items[cur_idx]), tcp_hdr, - sent_seq, 0, pkt->l4_len, tcp_dl, 0, - true); - if (cmp) { - if (merge_two_tcp_packets(&(tbl->items[cur_idx]), - pkt, cmp, sent_seq, 0, 0)) { - return 1; - } - - /* - * Fail to merge the two packets, as the packet - * length is greater than the max value. Store - * the packet into the flow. - */ - if (insert_new_item(tbl, pkt, start_time, cur_idx, - sent_seq, true) == - INVALID_ARRAY_INDEX) - return -1; - return 0; - } - prev_idx = cur_idx; - cur_idx = tbl->items[cur_idx].next_pkt_idx; - } while (cur_idx != INVALID_ARRAY_INDEX); - - /* Fail to find a neighbor, so store the packet into the flow. */ - if (insert_new_item(tbl, pkt, start_time, prev_idx, sent_seq, - true) == INVALID_ARRAY_INDEX) - return -1; - - return 0; + return gro_tcp_reassemble(pkt, tbl, &key, tcp_dl, &gro_tcp6_flows_ops, tbl->items, + &tbl->item_num, tbl->max_item_num, + 0, true, start_time); } uint16_t @@ -357,7 +244,7 @@ gro_tcp6_tbl_timeout_flush(struct gro_tcp6_tbl *tbl, * Delete the packet and get the next * packet in the flow. */ - j = delete_item(tbl, j, INVALID_ARRAY_INDEX); + j = delete_tcp_item(tbl->items, j, &tbl->item_num, INVALID_ARRAY_INDEX); tbl->flows[i].start_index = j; if (j == INVALID_ARRAY_INDEX) tbl->flow_num--; diff --git a/lib/gro/gro_tcp6.h b/lib/gro/gro_tcp6.h index 91325c0da2..9eb6be2dfe 100644 --- a/lib/gro/gro_tcp6.h +++ b/lib/gro/gro_tcp6.h @@ -5,24 +5,21 @@ #ifndef _GRO_TCP6_H_ #define _GRO_TCP6_H_ +#include + #define INVALID_ARRAY_INDEX 0xffffffffUL #define GRO_TCP6_TBL_MAX_ITEM_NUM (1024UL * 1024UL) /* Header fields representing a TCP/IPv6 flow */ struct tcp6_flow_key { - struct rte_ether_addr eth_saddr; - struct rte_ether_addr eth_daddr; + struct gro_tcp_flow tcp_flow; uint8_t src_addr[16]; uint8_t dst_addr[16]; - - uint32_t recv_ack; - uint16_t src_port; - uint16_t dst_port; + rte_be32_t vtc_flow; }; struct gro_tcp6_flow { struct tcp6_flow_key key; - rte_be32_t vtc_flow; /* * The index of the first packet in the flow. * INVALID_ARRAY_INDEX indicates an empty flow. @@ -142,9 +139,25 @@ uint32_t gro_tcp6_tbl_pkt_count(void *tbl); * Check if two TCP/IPv6 packets belong to the same flow. */ static inline int -is_same_tcp6_flow(struct tcp6_flow_key k1, struct tcp6_flow_key k2) +is_same_tcp6_flow(struct tcp6_flow_key *k1, struct tcp6_flow_key *k2) { - return (!memcmp(&k1, &k2, sizeof(struct tcp6_flow_key))); + rte_be32_t vtc_flow_diff; + + if (memcmp(&k1->src_addr, &k2->src_addr, 16)) { + return 0; + } + if (memcmp(&k1->dst_addr, &k2->dst_addr, 16)) { + return 0; + } + /* + * IP version (4) Traffic Class (8) Flow Label (20) + * All fields except Traffic class should be same + */ + vtc_flow_diff = (k1->vtc_flow ^ k2->vtc_flow); + if (vtc_flow_diff & htonl(0xF00FFFFF)) + return 0; + + return is_same_tcp_flow(&k1->tcp_flow, &k2->tcp_flow); } #endif diff --git a/lib/gro/gro_vxlan_tcp4.c b/lib/gro/gro_vxlan_tcp4.c index 56b30b8c98..16aea7049f 100644 --- a/lib/gro/gro_vxlan_tcp4.c +++ b/lib/gro/gro_vxlan_tcp4.c @@ -164,15 +164,9 @@ insert_new_flow(struct gro_vxlan_tcp4_tbl *tbl, dst = &(tbl->flows[flow_idx].key); - rte_ether_addr_copy(&(src->inner_key.eth_saddr), - &(dst->inner_key.eth_saddr)); - rte_ether_addr_copy(&(src->inner_key.eth_daddr), - &(dst->inner_key.eth_daddr)); + ASSIGN_TCP_FLOW_KEY((&(src->inner_key.tcp_flow)), (&(dst->inner_key.tcp_flow))); dst->inner_key.ip_src_addr = src->inner_key.ip_src_addr; dst->inner_key.ip_dst_addr = src->inner_key.ip_dst_addr; - dst->inner_key.recv_ack = src->inner_key.recv_ack; - dst->inner_key.src_port = src->inner_key.src_port; - dst->inner_key.dst_port = src->inner_key.dst_port; dst->vxlan_hdr.vx_flags = src->vxlan_hdr.vx_flags; dst->vxlan_hdr.vx_vni = src->vxlan_hdr.vx_vni; @@ -358,13 +352,13 @@ gro_vxlan_tcp4_reassemble(struct rte_mbuf *pkt, sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); - rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.inner_key.eth_saddr)); - rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.inner_key.eth_daddr)); + rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.inner_key.tcp_flow.eth_saddr)); + rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.inner_key.tcp_flow.eth_daddr)); key.inner_key.ip_src_addr = ipv4_hdr->src_addr; key.inner_key.ip_dst_addr = ipv4_hdr->dst_addr; - key.inner_key.recv_ack = tcp_hdr->recv_ack; - key.inner_key.src_port = tcp_hdr->src_port; - key.inner_key.dst_port = tcp_hdr->dst_port; + key.inner_key.tcp_flow.recv_ack = tcp_hdr->recv_ack; + key.inner_key.tcp_flow.src_port = tcp_hdr->src_port; + key.inner_key.tcp_flow.dst_port = tcp_hdr->dst_port; key.vxlan_hdr.vx_flags = vxlan_hdr->vx_flags; key.vxlan_hdr.vx_vni = vxlan_hdr->vx_vni; diff --git a/lib/gro/meson.build b/lib/gro/meson.build index dbce05220d..a7d73f21ec 100644 --- a/lib/gro/meson.build +++ b/lib/gro/meson.build @@ -3,6 +3,7 @@ sources = files( 'rte_gro.c', + 'gro_tcp.c', 'gro_tcp4.c', 'gro_tcp6.c', 'gro_udp4.c',