From patchwork Sun Sep 11 18:12:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavan Nikhilesh Bhagavatula X-Patchwork-Id: 116164 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 43850A00C3; Sun, 11 Sep 2022 20:13:03 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D529C400D4; Sun, 11 Sep 2022 20:13:02 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 329664003F; Sun, 11 Sep 2022 20:13:01 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28BI1ORK015394; Sun, 11 Sep 2022 11:12:59 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=5O9dBw1H8rozQBIEV+MREzmaC6p7M37Js0gCiJpe5mM=; b=FaWkcMTksP7tIEQQWi/KBn+yeUQqRbVi670Ux8KoqECbOHiJ9Eca2ooCdkg7HNZP4EdV TXoBVtp1YQPHp1Bf7iuvxGMSmy3aofc4g9e+wjpWL7L+CG183A+9iWF8cE2NzaIjqyu3 n+gZtBLK6MNob5jER+nxlho1aAjGBjrov0fcvNFf/xdR8wXMoBjBzBZXKXRCnY3TF87e 5Y+Vh5HrNgYeCB7NPKCplDv3tILAQmu3A1moXodiMnf8deMfNsUmtw0vxrquG5NXCWBr o6+UCG8SLTrdZ69bdS2tXVgkAJOEy+m/EbYgL+qmJFR3X6MQzWqeSmNigLC1xCkJk6Ev vQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3jgqxq45kb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Sun, 11 Sep 2022 11:12:59 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Sun, 11 Sep 2022 11:12:58 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Sun, 11 Sep 2022 11:12:58 -0700 Received: from MININT-80QBFE8.corp.innovium.com (unknown [10.193.65.138]) by maili.marvell.com (Postfix) with ESMTP id 4A2CE3F706D; Sun, 11 Sep 2022 11:12:56 -0700 (PDT) From: To: , David Christensen CC: , Pavan Nikhilesh , Subject: [PATCH v3 1/5] examples/l3fwd: fix port group mask generation Date: Sun, 11 Sep 2022 23:42:45 +0530 Message-ID: <20220911181250.2286-1-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220902091833.9074-1-pbhagavatula@marvell.com> References: <20220902091833.9074-1-pbhagavatula@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: C6Thru1lANXsacGw2m7HdN6l7Tvpq6gI X-Proofpoint-GUID: C6Thru1lANXsacGw2m7HdN6l7Tvpq6gI X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-11_08,2022-09-09_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Fix port group mask generation in altivec, vec_any_eq returns 0 or 1 while port_groupx4 expects comparison mask result. Fixes: 2193b7467f7a ("examples/l3fwd: optimize packet processing on powerpc") Cc: stable@dpdk.org Signed-off-by: Pavan Nikhilesh --- v3 Changes: - PPC optimize port mask generation. - Fix aarch32 compilation. v2 Changes: - Fix PPC, RISC-V, aarch32 compilation. examples/common/altivec/port_group.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) -- 2.25.1 diff --git a/examples/common/altivec/port_group.h b/examples/common/altivec/port_group.h index 5e209b02fa..1c05bc025a 100644 --- a/examples/common/altivec/port_group.h +++ b/examples/common/altivec/port_group.h @@ -26,12 +26,17 @@ port_groupx4(uint16_t pn[FWDSTEP + 1], uint16_t *lp, uint16_t u16[FWDSTEP + 1]; uint64_t u64; } *pnum = (void *)pn; - + __vector unsigned long long result; + const __vector unsigned int perm_mask = {0x00204060, 0x80808080, + 0x80808080, 0x80808080}; int32_t v; - v = vec_any_eq(dp1, dp2); - + dp1 = (__vector unsigned short)vec_cmpeq(dp1, dp2); + dp1 = vec_mergeh(dp1, dp1); + result = (__vector unsigned long long)vec_vbpermq( + (__vector unsigned char)dp1, (__vector unsigned char)perm_mask); + v = result[1]; /* update last port counter. */ lp[0] += gptbl[v].lpv; From patchwork Sun Sep 11 18:12:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavan Nikhilesh Bhagavatula X-Patchwork-Id: 116165 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 19DBDA00C3; Sun, 11 Sep 2022 20:13:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0C6E3410D2; Sun, 11 Sep 2022 20:13:09 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id DD9534003F for ; Sun, 11 Sep 2022 20:13:06 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28BFuFa4031971; Sun, 11 Sep 2022 11:13:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=/wyFidc59OvGdGuTslI+YEJs4K6yvXCfzEhEP846jbM=; b=Ctw03uYa9xgmn8Bnvhzq91HH+wAgivPyWdk6V0ZtZPOd+lGW/Mfx+qwaEvdXDrviouG3 AgvmXuLtu2V9zXkXjHZIl0NLgTdw4RigOFeCGe+oUA3t5lYO7efkqa5gX26t6XmmDEyc xN1ZG0t2QzxgpYFD2OXSmipcgCM9E2jqaHxv5l1bx+d5A6byXylue/Sw3sNLzCA4/kHZ d/KwXFzOr3Y1GHUiIj7um6LNxdrXxRh1PUZZDgpvil6UXmFYY9MiUeDkb+Dtic9eAw8m G2eAeeNchWsNQhQ1GJjNdrAPNuJHHKA7IGVABpkzaO25l88EmwurKPV/qYWpxPB11Wpv MQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3jgt3mutc2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Sun, 11 Sep 2022 11:13:03 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Sun, 11 Sep 2022 11:13:01 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Sun, 11 Sep 2022 11:13:01 -0700 Received: from MININT-80QBFE8.corp.innovium.com (unknown [10.193.65.138]) by maili.marvell.com (Postfix) with ESMTP id B76713F7070; Sun, 11 Sep 2022 11:12:58 -0700 (PDT) From: To: , David Christensen , "Ruifeng Wang" , Bruce Richardson , Konstantin Ananyev CC: , Pavan Nikhilesh Subject: [PATCH v3 2/5] examples/l3fwd: split processing and send stages Date: Sun, 11 Sep 2022 23:42:46 +0530 Message-ID: <20220911181250.2286-2-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220911181250.2286-1-pbhagavatula@marvell.com> References: <20220902091833.9074-1-pbhagavatula@marvell.com> <20220911181250.2286-1-pbhagavatula@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: KLEWhcAUvIgRgp-FA_Ym6CFMPPEkdzcX X-Proofpoint-GUID: KLEWhcAUvIgRgp-FA_Ym6CFMPPEkdzcX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-11_08,2022-09-09_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Split packet processing from packet send stage, as send stage is not common for poll and event mode. Signed-off-by: Pavan Nikhilesh --- examples/l3fwd/l3fwd_em_hlm.h | 39 +++++++++++++++++++----------- examples/l3fwd/l3fwd_lpm_altivec.h | 25 ++++++++++++++++--- examples/l3fwd/l3fwd_lpm_neon.h | 35 ++++++++++++++++++++------- examples/l3fwd/l3fwd_lpm_sse.h | 25 ++++++++++++++++--- 4 files changed, 95 insertions(+), 29 deletions(-) diff --git a/examples/l3fwd/l3fwd_em_hlm.h b/examples/l3fwd/l3fwd_em_hlm.h index e76f2760b0..12b997e477 100644 --- a/examples/l3fwd/l3fwd_em_hlm.h +++ b/examples/l3fwd/l3fwd_em_hlm.h @@ -177,16 +177,12 @@ em_get_dst_port(const struct lcore_conf *qconf, struct rte_mbuf *pkt, return portid; } -/* - * Buffer optimized handling of packets, invoked - * from main_loop. - */ static inline void -l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, - uint16_t portid, struct lcore_conf *qconf) +l3fwd_em_process_packets(int nb_rx, struct rte_mbuf **pkts_burst, + uint16_t *dst_port, uint16_t portid, + struct lcore_conf *qconf, const uint8_t do_step3) { int32_t i, j, pos; - uint16_t dst_port[MAX_PKT_BURST]; /* * Send nb_rx - nb_rx % EM_HASH_LOOKUP_COUNT packets @@ -233,13 +229,30 @@ l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, dst_port[j + i] = em_get_dst_port(qconf, pkts_burst[j + i], portid); } + + for (i = 0; i < EM_HASH_LOOKUP_COUNT && do_step3; i += FWDSTEP) + processx4_step3(&pkts_burst[j + i], &dst_port[j + i]); } - for (; j < nb_rx; j++) + for (; j < nb_rx; j++) { dst_port[j] = em_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &pkts_burst[j]->port); + } +} - send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); +/* + * Buffer optimized handling of packets, invoked + * from main_loop. + */ +static inline void +l3fwd_em_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, uint16_t portid, + struct lcore_conf *qconf) +{ + uint16_t dst_port[MAX_PKT_BURST]; + l3fwd_em_process_packets(nb_rx, pkts_burst, dst_port, portid, qconf, 0); + send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); } /* @@ -260,11 +273,8 @@ l3fwd_em_process_events(int nb_rx, struct rte_event **ev, */ int32_t n = RTE_ALIGN_FLOOR(nb_rx, EM_HASH_LOOKUP_COUNT); - for (j = 0; j < EM_HASH_LOOKUP_COUNT && j < nb_rx; j++) { + for (j = 0; j < nb_rx; j++) pkts_burst[j] = ev[j]->mbuf; - rte_prefetch0(rte_pktmbuf_mtod(pkts_burst[j], - struct rte_ether_hdr *) + 1); - } for (j = 0; j < n; j += EM_HASH_LOOKUP_COUNT) { @@ -305,7 +315,8 @@ l3fwd_em_process_events(int nb_rx, struct rte_event **ev, } continue; } - processx4_step3(&pkts_burst[j], &dst_port[j]); + for (i = 0; i < EM_HASH_LOOKUP_COUNT; i += FWDSTEP) + processx4_step3(&pkts_burst[j + i], &dst_port[j + i]); for (i = 0; i < EM_HASH_LOOKUP_COUNT; i++) pkts_burst[j + i]->port = dst_port[j + i]; diff --git a/examples/l3fwd/l3fwd_lpm_altivec.h b/examples/l3fwd/l3fwd_lpm_altivec.h index 0c6852a7bb..adb82f1478 100644 --- a/examples/l3fwd/l3fwd_lpm_altivec.h +++ b/examples/l3fwd/l3fwd_lpm_altivec.h @@ -96,11 +96,11 @@ processx4_step2(const struct lcore_conf *qconf, * from main_loop. */ static inline void -l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, - uint8_t portid, struct lcore_conf *qconf) +l3fwd_lpm_process_packets(int nb_rx, struct rte_mbuf **pkts_burst, + uint8_t portid, uint16_t *dst_port, + struct lcore_conf *qconf, const uint8_t do_step3) { int32_t j; - uint16_t dst_port[MAX_PKT_BURST]; __vector unsigned int dip[MAX_PKT_BURST / FWDSTEP]; uint32_t ipv4_flag[MAX_PKT_BURST / FWDSTEP]; const int32_t k = RTE_ALIGN_FLOOR(nb_rx, FWDSTEP); @@ -114,22 +114,41 @@ l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, ipv4_flag[j / FWDSTEP], portid, &pkts_burst[j], &dst_port[j]); + if (do_step3) + for (j = 0; j != k; j += FWDSTEP) + processx4_step3(&pkts_burst[j], &dst_port[j]); + /* Classify last up to 3 packets one by one */ switch (nb_rx % FWDSTEP) { case 3: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fall-through */ case 2: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fall-through */ case 1: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fall-through */ } +} + +static inline void +l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, uint8_t portid, + struct lcore_conf *qconf) +{ + uint16_t dst_port[MAX_PKT_BURST]; + l3fwd_lpm_process_packets(nb_rx, pkts_burst, portid, dst_port, qconf, + 0); send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); } diff --git a/examples/l3fwd/l3fwd_lpm_neon.h b/examples/l3fwd/l3fwd_lpm_neon.h index 78ee83b76c..2a68c4c15e 100644 --- a/examples/l3fwd/l3fwd_lpm_neon.h +++ b/examples/l3fwd/l3fwd_lpm_neon.h @@ -80,16 +80,12 @@ processx4_step2(const struct lcore_conf *qconf, } } -/* - * Buffer optimized handling of packets, invoked - * from main_loop. - */ static inline void -l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, - uint16_t portid, struct lcore_conf *qconf) +l3fwd_lpm_process_packets(int nb_rx, struct rte_mbuf **pkts_burst, + uint16_t portid, uint16_t *dst_port, + struct lcore_conf *qconf, const uint8_t do_step3) { int32_t i = 0, j = 0; - uint16_t dst_port[MAX_PKT_BURST]; int32x4_t dip; uint32_t ipv4_flag; const int32_t k = RTE_ALIGN_FLOOR(nb_rx, FWDSTEP); @@ -100,7 +96,6 @@ l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, rte_prefetch0(rte_pktmbuf_mtod(pkts_burst[i], void *)); } - for (j = 0; j != k - FWDSTEP; j += FWDSTEP) { for (i = 0; i < FWDSTEP; i++) { rte_prefetch0(rte_pktmbuf_mtod( @@ -111,11 +106,15 @@ l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, processx4_step1(&pkts_burst[j], &dip, &ipv4_flag); processx4_step2(qconf, dip, ipv4_flag, portid, &pkts_burst[j], &dst_port[j]); + if (do_step3) + processx4_step3(&pkts_burst[j], &dst_port[j]); } processx4_step1(&pkts_burst[j], &dip, &ipv4_flag); processx4_step2(qconf, dip, ipv4_flag, portid, &pkts_burst[j], &dst_port[j]); + if (do_step3) + processx4_step3(&pkts_burst[j], &dst_port[j]); j += FWDSTEP; } @@ -138,26 +137,44 @@ l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, void *)); j++; } - j -= m; /* Classify last up to 3 packets one by one */ switch (m) { case 3: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fallthrough */ case 2: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fallthrough */ case 1: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); } } +} + +/* + * Buffer optimized handling of packets, invoked + * from main_loop. + */ +static inline void +l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, uint16_t portid, + struct lcore_conf *qconf) +{ + uint16_t dst_port[MAX_PKT_BURST]; + l3fwd_lpm_process_packets(nb_rx, pkts_burst, portid, dst_port, qconf, + 0); send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); } diff --git a/examples/l3fwd/l3fwd_lpm_sse.h b/examples/l3fwd/l3fwd_lpm_sse.h index 3f637a23d1..db15030320 100644 --- a/examples/l3fwd/l3fwd_lpm_sse.h +++ b/examples/l3fwd/l3fwd_lpm_sse.h @@ -82,11 +82,11 @@ processx4_step2(const struct lcore_conf *qconf, * from main_loop. */ static inline void -l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, - uint16_t portid, struct lcore_conf *qconf) +l3fwd_lpm_process_packets(int nb_rx, struct rte_mbuf **pkts_burst, + uint16_t portid, uint16_t *dst_port, + struct lcore_conf *qconf, const uint8_t do_step3) { int32_t j; - uint16_t dst_port[MAX_PKT_BURST]; __m128i dip[MAX_PKT_BURST / FWDSTEP]; uint32_t ipv4_flag[MAX_PKT_BURST / FWDSTEP]; const int32_t k = RTE_ALIGN_FLOOR(nb_rx, FWDSTEP); @@ -99,21 +99,40 @@ l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, processx4_step2(qconf, dip[j / FWDSTEP], ipv4_flag[j / FWDSTEP], portid, &pkts_burst[j], &dst_port[j]); + if (do_step3) + for (j = 0; j != k; j += FWDSTEP) + processx4_step3(&pkts_burst[j], &dst_port[j]); + /* Classify last up to 3 packets one by one */ switch (nb_rx % FWDSTEP) { case 3: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fall-through */ case 2: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; /* fall-through */ case 1: dst_port[j] = lpm_get_dst_port(qconf, pkts_burst[j], portid); + if (do_step3) + process_packet(pkts_burst[j], &dst_port[j]); j++; } +} + +static inline void +l3fwd_lpm_send_packets(int nb_rx, struct rte_mbuf **pkts_burst, uint16_t portid, + struct lcore_conf *qconf) +{ + uint16_t dst_port[MAX_PKT_BURST]; + l3fwd_lpm_process_packets(nb_rx, pkts_burst, portid, dst_port, qconf, + 0); send_packets_multi(qconf, pkts_burst, dst_port, nb_rx); } From patchwork Sun Sep 11 18:12:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavan Nikhilesh Bhagavatula X-Patchwork-Id: 116167 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 815E8A00C3; Sun, 11 Sep 2022 20:13:20 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CCDDA4281A; Sun, 11 Sep 2022 20:13:10 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 1FAE2427F1 for ; Sun, 11 Sep 2022 20:13:09 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28BDtL4E002825; Sun, 11 Sep 2022 11:13:06 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=k9JhoVL4KHhh6TSq/jbG9PmW9bTDg+qnA0H6Opzy/nM=; b=g8gAWFmzEEmU/YKbfFQtCkfGEPfN4z7LlZhSyVQg91iZ+DVNCGIZdYFI34LIiOCpDUHU lXHVue61xI5yzjYOGdcGOCiTTppnBdfcsYz6p2E1fk1tchaiF1O7h8DmQocaeGIVmHPa XZ8E9zC3au0UHZwIeuhXyN+sZXxkgeoqiz4PRwcZh64m6NTENczEdAtADMOS1GYW84gJ czYFcCi4ix0VbN+HqCKhEF+fbNEMiq7WEKOhs2HH1/JMk3AG1MnDjZOUi6S7OTo5lSfN YhmXAYwAxDwctxXkZAnApiEQpZ2SPqnMZ2frIFo2YTFJtHmH3Zu3q3uOafvXJq/d7Y5l rg== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3jgt3mutc8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Sun, 11 Sep 2022 11:13:06 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 11 Sep 2022 11:13:04 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Sun, 11 Sep 2022 11:13:04 -0700 Received: from MININT-80QBFE8.corp.innovium.com (unknown [10.193.65.138]) by maili.marvell.com (Postfix) with ESMTP id B28FD3F7075; Sun, 11 Sep 2022 11:13:01 -0700 (PDT) From: To: , David Christensen , "Ruifeng Wang" , Bruce Richardson , Konstantin Ananyev CC: , Pavan Nikhilesh Subject: [PATCH v3 3/5] examples/l3fwd: use lpm vector path for event vector Date: Sun, 11 Sep 2022 23:42:47 +0530 Message-ID: <20220911181250.2286-3-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220911181250.2286-1-pbhagavatula@marvell.com> References: <20220902091833.9074-1-pbhagavatula@marvell.com> <20220911181250.2286-1-pbhagavatula@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: gUFZ0uuoX-OLNm4UPtXWA9d_oFgG_q97 X-Proofpoint-GUID: gUFZ0uuoX-OLNm4UPtXWA9d_oFgG_q97 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-11_08,2022-09-09_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Use lpm vector path to process event vector. Signed-off-by: Pavan Nikhilesh --- examples/l3fwd/l3fwd_altivec.h | 29 ++++++++++++++ examples/l3fwd/l3fwd_event.h | 71 ++++++++++++++++++++++++++++++++++ examples/l3fwd/l3fwd_lpm.c | 38 ++++++++++-------- examples/l3fwd/l3fwd_neon.h | 47 ++++++++++++++++++++++ examples/l3fwd/l3fwd_sse.h | 44 +++++++++++++++++++++ 5 files changed, 213 insertions(+), 16 deletions(-) diff --git a/examples/l3fwd/l3fwd_altivec.h b/examples/l3fwd/l3fwd_altivec.h index 87018f5dbe..e45e138e59 100644 --- a/examples/l3fwd/l3fwd_altivec.h +++ b/examples/l3fwd/l3fwd_altivec.h @@ -222,4 +222,33 @@ send_packets_multi(struct lcore_conf *qconf, struct rte_mbuf **pkts_burst, } } +static __rte_always_inline uint16_t +process_dst_port(uint16_t *dst_ports, uint16_t nb_elem) +{ + uint16_t i = 0, res; + + while (nb_elem > 7) { + __vector unsigned short dp1; + __vector unsigned short dp; + + dp = (__vector unsigned short)vec_splats((short)dst_ports[0]); + dp1 = *((__vector unsigned short *)&dst_ports[i]); + res = vec_all_eq(dp1, dp); + if (!res) + return BAD_PORT; + + nb_elem -= 8; + i += 8; + } + + while (nb_elem) { + if (dst_ports[i] != dst_ports[0]) + return BAD_PORT; + nb_elem--; + i++; + } + + return dst_ports[0]; +} + #endif /* _L3FWD_ALTIVEC_H_ */ diff --git a/examples/l3fwd/l3fwd_event.h b/examples/l3fwd/l3fwd_event.h index b93841a16f..3fe38aada0 100644 --- a/examples/l3fwd/l3fwd_event.h +++ b/examples/l3fwd/l3fwd_event.h @@ -82,6 +82,27 @@ struct l3fwd_event_resources { uint64_t vector_tmo_ns; }; +#if defined(RTE_ARCH_X86) +#include "l3fwd_sse.h" +#elif defined __ARM_NEON +#include "l3fwd_neon.h" +#elif defined(RTE_ARCH_PPC_64) +#include "l3fwd_altivec.h" +#else +static inline uint16_t +process_dst_port(uint16_t *dst_ports, uint16_t nb_elem) +{ + int i; + + for (i = 0; i < nb_elem; i++) { + if (dst_ports[i] != dst_ports[0]) + return BAD_PORT; + } + + return dst_ports[0]; +} +#endif + static inline void event_vector_attr_validate(struct rte_event_vector *vec, struct rte_mbuf *mbuf) { @@ -103,7 +124,57 @@ event_vector_txq_set(struct rte_event_vector *vec, uint16_t txq) } } +static inline uint16_t +filter_bad_packets(struct rte_mbuf **mbufs, uint16_t *dst_port, + uint16_t nb_pkts) +{ + uint16_t *des_pos, free = 0; + struct rte_mbuf **pos; + int i; + + /* Filter out and free bad packets */ + for (i = 0; i < nb_pkts; i++) { + if (dst_port[i] == BAD_PORT) { + rte_pktmbuf_free(mbufs[i]); + if (!free) { + pos = &mbufs[i]; + des_pos = &dst_port[i]; + } + free++; + continue; + } + + if (free) { + *pos = mbufs[i]; + pos++; + *des_pos = dst_port[i]; + des_pos++; + } + } + return nb_pkts - free; +} + +static inline void +process_event_vector(struct rte_event_vector *vec, uint16_t *dst_port) +{ + uint16_t port, i; + + vec->nb_elem = filter_bad_packets(vec->mbufs, dst_port, vec->nb_elem); + /* Verify destination array */ + port = process_dst_port(dst_port, vec->nb_elem); + if (port == BAD_PORT) { + vec->attr_valid = 0; + for (i = 0; i < vec->nb_elem; i++) { + vec->mbufs[i]->port = dst_port[i]; + rte_event_eth_tx_adapter_txq_set(vec->mbufs[i], 0); + } + } else { + vec->attr_valid = 1; + vec->port = port; + vec->queue = 0; + } +} struct l3fwd_event_resources *l3fwd_get_eventdev_rsrc(void); void l3fwd_event_resource_setup(struct rte_eth_conf *port_conf); diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c index d1b850dd5b..1652b7c470 100644 --- a/examples/l3fwd/l3fwd_lpm.c +++ b/examples/l3fwd/l3fwd_lpm.c @@ -425,24 +425,27 @@ lpm_event_main_loop_tx_q_burst(__rte_unused void *dummy) } static __rte_always_inline void -lpm_process_event_vector(struct rte_event_vector *vec, struct lcore_conf *lconf) +lpm_process_event_vector(struct rte_event_vector *vec, struct lcore_conf *lconf, + uint16_t *dst_port) { struct rte_mbuf **mbufs = vec->mbufs; int i; - /* Process first packet to init vector attributes */ - lpm_process_event_pkt(lconf, mbufs[0]); +#if defined RTE_ARCH_X86 || defined __ARM_NEON || defined RTE_ARCH_PPC_64 if (vec->attr_valid) { - if (mbufs[0]->port != BAD_PORT) - vec->port = mbufs[0]->port; - else - vec->attr_valid = 0; + l3fwd_lpm_process_packets(vec->nb_elem, mbufs, vec->port, + dst_port, lconf, 1); + } else { + for (i = 0; i < vec->nb_elem; i++) + l3fwd_lpm_process_packets(1, &mbufs[i], mbufs[i]->port, + &dst_port[i], lconf, 1); } +#else + for (i = 0; i < vec->nb_elem; i++) + dst_port[i] = lpm_process_event_pkt(lconf, mbufs[i]); +#endif - for (i = 1; i < vec->nb_elem; i++) { - lpm_process_event_pkt(lconf, mbufs[i]); - event_vector_attr_validate(vec, mbufs[i]); - } + process_event_vector(vec, dst_port); } /* Same eventdev loop for single and burst of vector */ @@ -458,6 +461,7 @@ lpm_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, struct rte_event events[MAX_PKT_BURST]; int i, nb_enq = 0, nb_deq = 0; struct lcore_conf *lconf; + uint16_t *dst_port_list; unsigned int lcore_id; if (event_p_id < 0) @@ -465,7 +469,11 @@ lpm_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, lcore_id = rte_lcore_id(); lconf = &lcore_conf[lcore_id]; - + dst_port_list = + rte_zmalloc("", sizeof(uint16_t) * evt_rsrc->vector_size, + RTE_CACHE_LINE_SIZE); + if (dst_port_list == NULL) + return; RTE_LOG(INFO, L3FWD, "entering %s on lcore %u\n", __func__, lcore_id); while (!force_quit) { @@ -483,10 +491,8 @@ lpm_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, events[i].op = RTE_EVENT_OP_FORWARD; } - lpm_process_event_vector(events[i].vec, lconf); - - if (flags & L3FWD_EVENT_TX_DIRECT) - event_vector_txq_set(events[i].vec, 0); + lpm_process_event_vector(events[i].vec, lconf, + dst_port_list); } if (flags & L3FWD_EVENT_TX_ENQ) { diff --git a/examples/l3fwd/l3fwd_neon.h b/examples/l3fwd/l3fwd_neon.h index ce515e0bc4..bf365341fb 100644 --- a/examples/l3fwd/l3fwd_neon.h +++ b/examples/l3fwd/l3fwd_neon.h @@ -194,4 +194,51 @@ send_packets_multi(struct lcore_conf *qconf, struct rte_mbuf **pkts_burst, } } +static __rte_always_inline uint16_t +process_dst_port(uint16_t *dst_ports, uint16_t nb_elem) +{ + uint16_t i = 0; + +#if defined(RTE_ARCH_ARM64) + uint16_t res; + + while (nb_elem > 7) { + uint16x8_t dp = vdupq_n_u16(dst_ports[0]); + uint16x8_t dp1; + + dp1 = vld1q_u16(&dst_ports[i]); + dp1 = vceqq_u16(dp1, dp); + res = vminvq_u16(dp1); + if (!res) + return BAD_PORT; + + nb_elem -= 8; + i += 8; + } + + while (nb_elem > 3) { + uint16x4_t dp = vdup_n_u16(dst_ports[0]); + uint16x4_t dp1; + + dp1 = vld1_u16(&dst_ports[i]); + dp1 = vceq_u16(dp1, dp); + res = vminv_u16(dp1); + if (!res) + return BAD_PORT; + + nb_elem -= 4; + i += 4; + } +#endif + + while (nb_elem) { + if (dst_ports[i] != dst_ports[0]) + return BAD_PORT; + nb_elem--; + i++; + } + + return dst_ports[0]; +} + #endif /* _L3FWD_NEON_H_ */ diff --git a/examples/l3fwd/l3fwd_sse.h b/examples/l3fwd/l3fwd_sse.h index 0f0d0323a2..083729cdef 100644 --- a/examples/l3fwd/l3fwd_sse.h +++ b/examples/l3fwd/l3fwd_sse.h @@ -194,4 +194,48 @@ send_packets_multi(struct lcore_conf *qconf, struct rte_mbuf **pkts_burst, } } +static __rte_always_inline uint16_t +process_dst_port(uint16_t *dst_ports, uint16_t nb_elem) +{ + uint16_t i = 0, res; + + while (nb_elem > 7) { + __m128i dp = _mm_set1_epi16(dst_ports[0]); + __m128i dp1; + + dp1 = _mm_loadu_si128((__m128i *)&dst_ports[i]); + dp1 = _mm_cmpeq_epi16(dp1, dp); + res = _mm_movemask_epi8(dp1); + if (res != 0xFFFF) + return BAD_PORT; + + nb_elem -= 8; + i += 8; + } + + while (nb_elem > 3) { + __m128i dp = _mm_set1_epi16(dst_ports[0]); + __m128i dp1; + + dp1 = _mm_loadu_si128((__m128i *)&dst_ports[i]); + dp1 = _mm_cmpeq_epi16(dp1, dp); + dp1 = _mm_unpacklo_epi16(dp1, dp1); + res = _mm_movemask_ps((__m128)dp1); + if (res != 0xF) + return BAD_PORT; + + nb_elem -= 4; + i += 4; + } + + while (nb_elem) { + if (dst_ports[i] != dst_ports[0]) + return BAD_PORT; + nb_elem--; + i++; + } + + return dst_ports[0]; +} + #endif /* _L3FWD_SSE_H_ */ From patchwork Sun Sep 11 18:12:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavan Nikhilesh Bhagavatula X-Patchwork-Id: 116166 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 08A34A00C3; Sun, 11 Sep 2022 20:13:15 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E9205427F6; Sun, 11 Sep 2022 20:13:09 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 541644003F for ; Sun, 11 Sep 2022 20:13:08 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28BFuFa6031971 for ; Sun, 11 Sep 2022 11:13:07 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=DhuOqJOoAbSuw+FoBlxIEgTz9Z0ExI44HOQlvcUC8Nc=; b=PVVvtfn9fC7XHf4I1Ex4E6fL3n531hzN6OSBtfmFmD/7yADjj3HXEtH3OlrZs0RaM4D5 7jIZIgFVfuOmOCzQSIzL7koFedFjfmiICyHBk6HAzb0pdo0YiWlkrPIPc1ce5fDHCArm GcJsvEjJkjqs8Ot9ETjoKK9g2n90L/q4G50OM7b0wbkRzygNDbuOGlzeq71tmVcyjNd4 6OkbRGY+GzBha6I3ui7ciKICZkF1PI0F/nv/D8x8wm4+xlUKCLOvUNj9WKVKLPYCF+kD WqrQwm13kdkdmMON+UCuVepaiqawH2X5dH9r7hgRxocjMfVNDCr3h4XFgwCTXxhKy1tR 9w== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3jgt3mutcd-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Sun, 11 Sep 2022 11:13:07 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Sun, 11 Sep 2022 11:13:06 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Sun, 11 Sep 2022 11:13:06 -0700 Received: from MININT-80QBFE8.corp.innovium.com (unknown [10.193.65.138]) by maili.marvell.com (Postfix) with ESMTP id A9B483F7070; Sun, 11 Sep 2022 11:13:04 -0700 (PDT) From: To: CC: , Pavan Nikhilesh Subject: [PATCH v3 4/5] examples/l3fwd: fix event vector processing in fib Date: Sun, 11 Sep 2022 23:42:48 +0530 Message-ID: <20220911181250.2286-4-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220911181250.2286-1-pbhagavatula@marvell.com> References: <20220902091833.9074-1-pbhagavatula@marvell.com> <20220911181250.2286-1-pbhagavatula@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 1LF-glCQSvDEQqrBKOS9a0m5RlyHvP-Q X-Proofpoint-GUID: 1LF-glCQSvDEQqrBKOS9a0m5RlyHvP-Q X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-11_08,2022-09-09_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Fix stack overflow when event vector size is greater than MAX_BURST_SIZE. Add missing mac swap and rfc1812 stage. Fixes: e8adca1951d4 ("examples/l3fwd: support event vector") Signed-off-by: Pavan Nikhilesh --- examples/l3fwd/l3fwd_fib.c | 123 ++++++++++++++++++++++++++----------- 1 file changed, 86 insertions(+), 37 deletions(-) diff --git a/examples/l3fwd/l3fwd_fib.c b/examples/l3fwd/l3fwd_fib.c index e02e4b3f5a..c4a45bc7f3 100644 --- a/examples/l3fwd/l3fwd_fib.c +++ b/examples/l3fwd/l3fwd_fib.c @@ -77,27 +77,37 @@ fib_parse_packet(struct rte_mbuf *mbuf, */ #if !defined FIB_SEND_MULTI static inline void -fib_send_single(int nb_tx, struct lcore_conf *qconf, - struct rte_mbuf **pkts_burst, uint16_t hops[nb_tx]) +process_packet(struct rte_mbuf *pkt, uint16_t *hop) { - int32_t j; struct rte_ether_hdr *eth_hdr; - for (j = 0; j < nb_tx; j++) { - /* Run rfc1812 if packet is ipv4 and checks enabled. */ + /* Run rfc1812 if packet is ipv4 and checks enabled. */ #if defined DO_RFC_1812_CHECKS - rfc1812_process((struct rte_ipv4_hdr *)(rte_pktmbuf_mtod( - pkts_burst[j], struct rte_ether_hdr *) + 1), - &hops[j], pkts_burst[j]->packet_type); + rfc1812_process( + (struct rte_ipv4_hdr *)(rte_pktmbuf_mtod( + pkt, struct rte_ether_hdr *) + + 1), + hop, pkt->packet_type); #endif - /* Set MAC addresses. */ - eth_hdr = rte_pktmbuf_mtod(pkts_burst[j], - struct rte_ether_hdr *); - *(uint64_t *)ð_hdr->dst_addr = dest_eth_addr[hops[j]]; - rte_ether_addr_copy(&ports_eth_addr[hops[j]], - ð_hdr->src_addr); + /* Set MAC addresses. */ + eth_hdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *); + *(uint64_t *)ð_hdr->dst_addr = dest_eth_addr[*hop]; + rte_ether_addr_copy(&ports_eth_addr[*hop], ð_hdr->src_addr); +} + +static inline void +fib_send_single(int nb_tx, struct lcore_conf *qconf, + struct rte_mbuf **pkts_burst, uint16_t hops[nb_tx]) +{ + int32_t j; + for (j = 0; j < nb_tx; j++) { + process_packet(pkts_burst[j], &hops[j]); + if (hops[j] == BAD_PORT) { + rte_pktmbuf_free(pkts_burst[j]); + continue; + } /* Send single packet. */ send_single_packet(qconf, pkts_burst[j], hops[j]); } @@ -261,7 +271,7 @@ fib_event_loop(struct l3fwd_event_resources *evt_rsrc, uint32_t ipv4_arr[MAX_PKT_BURST]; uint8_t ipv6_arr[MAX_PKT_BURST][RTE_FIB6_IPV6_ADDR_SIZE]; uint64_t hopsv4[MAX_PKT_BURST], hopsv6[MAX_PKT_BURST]; - uint16_t nh; + uint16_t nh, hops[MAX_PKT_BURST]; uint8_t type_arr[MAX_PKT_BURST]; uint32_t ipv4_cnt, ipv6_cnt; uint32_t ipv4_arr_assem, ipv6_arr_assem; @@ -350,7 +360,13 @@ fib_event_loop(struct l3fwd_event_resources *evt_rsrc, else nh = (uint16_t)hopsv6[ipv6_arr_assem++]; if (nh != FIB_DEFAULT_HOP) - events[i].mbuf->port = nh; + hops[i] = nh != FIB_DEFAULT_HOP ? + nh : + events[i].mbuf->port; + process_packet(events[i].mbuf, &hops[i]); + events[i].mbuf->port = hops[i] != BAD_PORT ? + hops[i] : + events[i].mbuf->port; } if (flags & L3FWD_EVENT_TX_ENQ) { @@ -418,14 +434,12 @@ fib_event_main_loop_tx_q_burst(__rte_unused void *dummy) } static __rte_always_inline void -fib_process_event_vector(struct rte_event_vector *vec) +fib_process_event_vector(struct rte_event_vector *vec, uint8_t *type_arr, + uint8_t **ipv6_arr, uint64_t *hopsv4, uint64_t *hopsv6, + uint32_t *ipv4_arr, uint16_t *hops) { - uint8_t ipv6_arr[MAX_PKT_BURST][RTE_FIB6_IPV6_ADDR_SIZE]; - uint64_t hopsv4[MAX_PKT_BURST], hopsv6[MAX_PKT_BURST]; uint32_t ipv4_arr_assem, ipv6_arr_assem; struct rte_mbuf **mbufs = vec->mbufs; - uint32_t ipv4_arr[MAX_PKT_BURST]; - uint8_t type_arr[MAX_PKT_BURST]; uint32_t ipv4_cnt, ipv6_cnt; struct lcore_conf *lconf; uint16_t nh; @@ -463,16 +477,10 @@ fib_process_event_vector(struct rte_event_vector *vec) /* Lookup IPv6 hops if IPv6 packets are present. */ if (ipv6_cnt > 0) - rte_fib6_lookup_bulk(lconf->ipv6_lookup_struct, ipv6_arr, - hopsv6, ipv6_cnt); - - if (vec->attr_valid) { - nh = type_arr[0] ? (uint16_t)hopsv4[0] : (uint16_t)hopsv6[0]; - if (nh != FIB_DEFAULT_HOP) - vec->port = nh; - else - vec->attr_valid = 0; - } + rte_fib6_lookup_bulk( + lconf->ipv6_lookup_struct, + (uint8_t(*)[RTE_FIB6_IPV6_ADDR_SIZE])ipv6_arr, hopsv6, + ipv6_cnt); /* Assign ports looked up in fib depending on IPv4 or IPv6 */ for (i = 0; i < vec->nb_elem; i++) { @@ -481,9 +489,26 @@ fib_process_event_vector(struct rte_event_vector *vec) else nh = (uint16_t)hopsv6[ipv6_arr_assem++]; if (nh != FIB_DEFAULT_HOP) - mbufs[i]->port = nh; - event_vector_attr_validate(vec, mbufs[i]); + hops[i] = nh; + else + hops[i] = vec->attr_valid ? vec->port : + vec->mbufs[i]->port; } + +#if defined FIB_SEND_MULTI + uint16_t k; + k = RTE_ALIGN_FLOOR(vec->nb_elem, FWDSTEP); + + for (i = 0; i != k; i += FWDSTEP) + processx4_step3(&vec->mbufs[i], &hops[i]); + for (; i < vec->nb_elem; i++) + process_packet(vec->mbufs[i], &hops[i]); +#else + for (i = 0; i < vec->nb_elem; i++) + process_packet(vec->mbufs[i], &hops[i]); +#endif + + process_event_vector(vec, hops); } static __rte_always_inline void @@ -496,7 +521,32 @@ fib_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, const uint8_t event_d_id = evt_rsrc->event_d_id; const uint16_t deq_len = evt_rsrc->deq_depth; struct rte_event events[MAX_PKT_BURST]; + uint8_t *type_arr, **ipv6_arr, *ptr; int nb_enq = 0, nb_deq = 0, i; + uint64_t *hopsv4, *hopsv6; + uint32_t *ipv4_arr; + uint16_t *hops; + uintptr_t mem; + + mem = (uintptr_t)rte_zmalloc( + "vector_fib", + (sizeof(uint32_t) + sizeof(uint8_t) + sizeof(uint64_t) + + sizeof(uint64_t) + sizeof(uint16_t) + sizeof(uint8_t *) + + (sizeof(uint8_t) * RTE_FIB6_IPV6_ADDR_SIZE)) * + evt_rsrc->vector_size, + RTE_CACHE_LINE_SIZE); + if (mem == 0) + return; + ipv4_arr = (uint32_t *)mem; + type_arr = (uint8_t *)&ipv4_arr[evt_rsrc->vector_size]; + hopsv4 = (uint64_t *)&type_arr[evt_rsrc->vector_size]; + hopsv6 = (uint64_t *)&hopsv4[evt_rsrc->vector_size]; + hops = (uint16_t *)&hopsv6[evt_rsrc->vector_size]; + ipv6_arr = (uint8_t **)&hops[evt_rsrc->vector_size]; + + ptr = (uint8_t *)&ipv6_arr[evt_rsrc->vector_size]; + for (i = 0; i < evt_rsrc->vector_size; i++) + ipv6_arr[i] = &ptr[RTE_FIB6_IPV6_ADDR_SIZE + i]; if (event_p_id < 0) return; @@ -519,10 +569,9 @@ fib_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, events[i].op = RTE_EVENT_OP_FORWARD; } - fib_process_event_vector(events[i].vec); - - if (flags & L3FWD_EVENT_TX_DIRECT) - event_vector_txq_set(events[i].vec, 0); + fib_process_event_vector(events[i].vec, type_arr, + ipv6_arr, hopsv4, hopsv6, + ipv4_arr, hops); } if (flags & L3FWD_EVENT_TX_ENQ) { From patchwork Sun Sep 11 18:12:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavan Nikhilesh Bhagavatula X-Patchwork-Id: 116168 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07A17A00C5; Sun, 11 Sep 2022 20:13:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 354224282E; Sun, 11 Sep 2022 20:13:13 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id BCDC442802 for ; Sun, 11 Sep 2022 20:13:10 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28B9s3V0018843 for ; Sun, 11 Sep 2022 11:13:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=LR2dSSlc22iyALls4fFE2QLRQryonixyhvCE2df+Phw=; b=XjNwI8O0XsSBsmu6MWRvnTelwqJdRr6go4x36xw41aC8F2jtcKkFcX89NTKtLRGBYjT6 GZvjY7ZjM9/7LG6DiISCDrGNg1Ujv2ibwGhXYq4oJfn4RnqGNVHS4TpV3csXDNVUuM6I D+GUJJ5fTm1Ag2Hw0V/Jaww/lB1kDOZ4eu5pN0JWCSBHorfs967sp9aX3aGYZL8NPjEp edk3JtJCybLOnjyGkK2Ndqj73JCsQRdSoc08DJ18mBabXqu/yQKu+V/HnoxvmFNW9eNM 4m70ddcaZ2WLvs67KPPThIWoYyl3ux7GKqmqEjCW/7TmaOQf0/PxZ2wo3R/IeVIgxVT3 vQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3jgt3mutcn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Sun, 11 Sep 2022 11:13:10 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 11 Sep 2022 11:13:07 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Sun, 11 Sep 2022 11:13:07 -0700 Received: from MININT-80QBFE8.corp.innovium.com (unknown [10.193.65.138]) by maili.marvell.com (Postfix) with ESMTP id 922573F706D; Sun, 11 Sep 2022 11:13:06 -0700 (PDT) From: To: CC: , Pavan Nikhilesh Subject: [PATCH v3 5/5] examples/l3fwd: use em vector path for event vector Date: Sun, 11 Sep 2022 23:42:49 +0530 Message-ID: <20220911181250.2286-5-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220911181250.2286-1-pbhagavatula@marvell.com> References: <20220902091833.9074-1-pbhagavatula@marvell.com> <20220911181250.2286-1-pbhagavatula@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 5LLlYgYvbaPw6zd0yNSqYPNF6ArmeY1G X-Proofpoint-GUID: 5LLlYgYvbaPw6zd0yNSqYPNF6ArmeY1G X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-11_08,2022-09-09_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Use em vector path to process event vector. Signed-off-by: Pavan Nikhilesh --- examples/l3fwd/l3fwd_em.c | 12 +++-- examples/l3fwd/l3fwd_em.h | 29 +++++------ examples/l3fwd/l3fwd_em_hlm.h | 72 +++++----------------------- examples/l3fwd/l3fwd_em_sequential.h | 25 ++++++---- examples/l3fwd/l3fwd_event.h | 21 -------- 5 files changed, 47 insertions(+), 112 deletions(-) diff --git a/examples/l3fwd/l3fwd_em.c b/examples/l3fwd/l3fwd_em.c index 10be24c61d..e7b35cfbd9 100644 --- a/examples/l3fwd/l3fwd_em.c +++ b/examples/l3fwd/l3fwd_em.c @@ -852,10 +852,15 @@ em_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, int i, nb_enq = 0, nb_deq = 0; struct lcore_conf *lconf; unsigned int lcore_id; + uint16_t *dst_ports; if (event_p_id < 0) return; + dst_ports = rte_zmalloc("", sizeof(uint16_t) * evt_rsrc->vector_size, + RTE_CACHE_LINE_SIZE); + if (dst_ports == NULL) + return; lcore_id = rte_lcore_id(); lconf = &lcore_conf[lcore_id]; @@ -877,13 +882,12 @@ em_event_loop_vector(struct l3fwd_event_resources *evt_rsrc, } #if defined RTE_ARCH_X86 || defined __ARM_NEON - l3fwd_em_process_event_vector(events[i].vec, lconf); + l3fwd_em_process_event_vector(events[i].vec, lconf, + dst_ports); #else l3fwd_em_no_opt_process_event_vector(events[i].vec, - lconf); + lconf, dst_ports); #endif - if (flags & L3FWD_EVENT_TX_DIRECT) - event_vector_txq_set(events[i].vec, 0); } if (flags & L3FWD_EVENT_TX_ENQ) { diff --git a/examples/l3fwd/l3fwd_em.h b/examples/l3fwd/l3fwd_em.h index fe2ee59f6a..7d051fc076 100644 --- a/examples/l3fwd/l3fwd_em.h +++ b/examples/l3fwd/l3fwd_em.h @@ -100,7 +100,7 @@ l3fwd_em_simple_forward(struct rte_mbuf *m, uint16_t portid, } } -static __rte_always_inline void +static __rte_always_inline uint16_t l3fwd_em_simple_process(struct rte_mbuf *m, struct lcore_conf *qconf) { struct rte_ether_hdr *eth_hdr; @@ -117,6 +117,8 @@ l3fwd_em_simple_process(struct rte_mbuf *m, struct lcore_conf *qconf) m->port = l3fwd_em_handle_ipv6(m, m->port, eth_hdr, qconf); else m->port = BAD_PORT; + + return m->port; } /* @@ -179,7 +181,8 @@ l3fwd_em_no_opt_process_events(int nb_rx, struct rte_event **events, static inline void l3fwd_em_no_opt_process_event_vector(struct rte_event_vector *vec, - struct lcore_conf *qconf) + struct lcore_conf *qconf, + uint16_t *dst_ports) { struct rte_mbuf **mbufs = vec->mbufs; int32_t i; @@ -188,30 +191,20 @@ l3fwd_em_no_opt_process_event_vector(struct rte_event_vector *vec, for (i = 0; i < PREFETCH_OFFSET && i < vec->nb_elem; i++) rte_prefetch0(rte_pktmbuf_mtod(mbufs[i], void *)); - /* Process first packet to init vector attributes */ - l3fwd_em_simple_process(mbufs[0], qconf); - if (vec->attr_valid) { - if (mbufs[0]->port != BAD_PORT) - vec->port = mbufs[0]->port; - else - vec->attr_valid = 0; - } - /* * Prefetch and forward already prefetched packets. */ - for (i = 1; i < (vec->nb_elem - PREFETCH_OFFSET); i++) { + for (i = 0; i < (vec->nb_elem - PREFETCH_OFFSET); i++) { rte_prefetch0( rte_pktmbuf_mtod(mbufs[i + PREFETCH_OFFSET], void *)); - l3fwd_em_simple_process(mbufs[i], qconf); - event_vector_attr_validate(vec, mbufs[i]); + dst_ports[i] = l3fwd_em_simple_process(mbufs[i], qconf); } /* Forward remaining prefetched packets */ - for (; i < vec->nb_elem; i++) { - l3fwd_em_simple_process(mbufs[i], qconf); - event_vector_attr_validate(vec, mbufs[i]); - } + for (; i < vec->nb_elem; i++) + dst_ports[i] = l3fwd_em_simple_process(mbufs[i], qconf); + + process_event_vector(vec, dst_ports); } #endif /* __L3FWD_EM_H__ */ diff --git a/examples/l3fwd/l3fwd_em_hlm.h b/examples/l3fwd/l3fwd_em_hlm.h index 12b997e477..2e11eefad7 100644 --- a/examples/l3fwd/l3fwd_em_hlm.h +++ b/examples/l3fwd/l3fwd_em_hlm.h @@ -332,70 +332,20 @@ l3fwd_em_process_events(int nb_rx, struct rte_event **ev, static inline void l3fwd_em_process_event_vector(struct rte_event_vector *vec, - struct lcore_conf *qconf) + struct lcore_conf *qconf, uint16_t *dst_port) { - struct rte_mbuf **mbufs = vec->mbufs; - uint16_t dst_port[MAX_PKT_BURST]; - int32_t i, j, n, pos; - - for (j = 0; j < EM_HASH_LOOKUP_COUNT && j < vec->nb_elem; j++) - rte_prefetch0( - rte_pktmbuf_mtod(mbufs[j], struct rte_ether_hdr *) + 1); + uint16_t i; if (vec->attr_valid) - vec->port = em_get_dst_port(qconf, mbufs[0], mbufs[0]->port); - - n = RTE_ALIGN_FLOOR(vec->nb_elem, EM_HASH_LOOKUP_COUNT); - for (j = 0; j < n; j += EM_HASH_LOOKUP_COUNT) { - uint32_t pkt_type = - RTE_PTYPE_L3_MASK | RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP; - uint32_t l3_type, tcp_or_udp; - - for (i = 0; i < EM_HASH_LOOKUP_COUNT; i++) - pkt_type &= mbufs[j + i]->packet_type; - - l3_type = pkt_type & RTE_PTYPE_L3_MASK; - tcp_or_udp = pkt_type & (RTE_PTYPE_L4_TCP | RTE_PTYPE_L4_UDP); - - for (i = 0, pos = j + EM_HASH_LOOKUP_COUNT; - i < EM_HASH_LOOKUP_COUNT && pos < vec->nb_elem; - i++, pos++) { - rte_prefetch0(rte_pktmbuf_mtod(mbufs[pos], - struct rte_ether_hdr *) + - 1); - } - - if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV4)) { - em_get_dst_port_ipv4xN_events(qconf, &mbufs[j], - &dst_port[j]); - } else if (tcp_or_udp && (l3_type == RTE_PTYPE_L3_IPV6)) { - em_get_dst_port_ipv6xN_events(qconf, &mbufs[j], - &dst_port[j]); - } else { - for (i = 0; i < EM_HASH_LOOKUP_COUNT; i++) { - mbufs[j + i]->port = - em_get_dst_port(qconf, mbufs[j + i], - mbufs[j + i]->port); - process_packet(mbufs[j + i], - &mbufs[j + i]->port); - event_vector_attr_validate(vec, mbufs[j + i]); - } - continue; - } - processx4_step3(&mbufs[j], &dst_port[j]); - - for (i = 0; i < EM_HASH_LOOKUP_COUNT; i++) { - mbufs[j + i]->port = dst_port[j + i]; - event_vector_attr_validate(vec, mbufs[j + i]); - } - } - - for (; j < vec->nb_elem; j++) { - mbufs[j]->port = - em_get_dst_port(qconf, mbufs[j], mbufs[j]->port); - process_packet(mbufs[j], &mbufs[j]->port); - event_vector_attr_validate(vec, mbufs[j]); - } + l3fwd_em_process_packets(vec->nb_elem, vec->mbufs, dst_port, + vec->port, qconf, 1); + else + for (i = 0; i < vec->nb_elem; i++) + l3fwd_em_process_packets(1, &vec->mbufs[i], + &dst_port[i], + vec->mbufs[i]->port, qconf, 1); + + process_event_vector(vec, dst_port); } #endif /* __L3FWD_EM_HLM_H__ */ diff --git a/examples/l3fwd/l3fwd_em_sequential.h b/examples/l3fwd/l3fwd_em_sequential.h index d2f75edb8a..067f23889a 100644 --- a/examples/l3fwd/l3fwd_em_sequential.h +++ b/examples/l3fwd/l3fwd_em_sequential.h @@ -113,39 +113,48 @@ l3fwd_em_process_events(int nb_rx, struct rte_event **events, for (i = 1, j = 0; j < nb_rx; i++, j++) { struct rte_mbuf *mbuf = events[j]->mbuf; + uint16_t port; if (i < nb_rx) { rte_prefetch0(rte_pktmbuf_mtod( events[i]->mbuf, struct rte_ether_hdr *) + 1); } + port = mbuf->port; mbuf->port = em_get_dst_port(qconf, mbuf, mbuf->port); process_packet(mbuf, &mbuf->port); + if (mbuf->port == BAD_PORT) + mbuf->port = port; } } static inline void l3fwd_em_process_event_vector(struct rte_event_vector *vec, - struct lcore_conf *qconf) + struct lcore_conf *qconf, uint16_t *dst_ports) { + const uint8_t attr_valid = vec->attr_valid; struct rte_mbuf **mbufs = vec->mbufs; int32_t i, j; rte_prefetch0(rte_pktmbuf_mtod(mbufs[0], struct rte_ether_hdr *) + 1); - if (vec->attr_valid) - vec->port = em_get_dst_port(qconf, mbufs[0], mbufs[0]->port); - for (i = 0, j = 1; i < vec->nb_elem; i++, j++) { if (j < vec->nb_elem) rte_prefetch0(rte_pktmbuf_mtod(mbufs[j], struct rte_ether_hdr *) + 1); - mbufs[i]->port = - em_get_dst_port(qconf, mbufs[i], mbufs[i]->port); - process_packet(mbufs[i], &mbufs[i]->port); - event_vector_attr_validate(vec, mbufs[i]); + dst_ports[i] = em_get_dst_port(qconf, mbufs[i], + attr_valid ? vec->port : + mbufs[i]->port); } + j = RTE_ALIGN_FLOOR(vec->nb_elem, FWDSTEP); + + for (i = 0; i != j; i += FWDSTEP) + processx4_step3(&vec->mbufs[i], &dst_ports[i]); + for (; i < vec->nb_elem; i++) + process_packet(vec->mbufs[i], &dst_ports[i]); + + process_event_vector(vec, dst_ports); } #endif /* __L3FWD_EM_SEQUENTIAL_H__ */ diff --git a/examples/l3fwd/l3fwd_event.h b/examples/l3fwd/l3fwd_event.h index 3fe38aada0..e21817c36b 100644 --- a/examples/l3fwd/l3fwd_event.h +++ b/examples/l3fwd/l3fwd_event.h @@ -103,27 +103,6 @@ process_dst_port(uint16_t *dst_ports, uint16_t nb_elem) } #endif -static inline void -event_vector_attr_validate(struct rte_event_vector *vec, struct rte_mbuf *mbuf) -{ - /* l3fwd application only changes mbuf port while processing */ - if (vec->attr_valid && (vec->port != mbuf->port)) - vec->attr_valid = 0; -} - -static inline void -event_vector_txq_set(struct rte_event_vector *vec, uint16_t txq) -{ - if (vec->attr_valid) { - vec->queue = txq; - } else { - int i; - - for (i = 0; i < vec->nb_elem; i++) - rte_event_eth_tx_adapter_txq_set(vec->mbufs[i], txq); - } -} - static inline uint16_t filter_bad_packets(struct rte_mbuf **mbufs, uint16_t *dst_port, uint16_t nb_pkts)