From patchwork Sun Feb 6 14:30:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 106923 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29F7BA034E; Sun, 6 Feb 2022 15:30:51 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1AE4A410FB; Sun, 6 Feb 2022 15:30:41 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 9684241140 for ; Sun, 6 Feb 2022 15:30:39 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 2169VxLJ016260; Sun, 6 Feb 2022 06:30:39 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=cwkUQ3GJDtJ5sXsrH58NO4sgVkdCXCMVPcZ1Nk2v+sU=; b=P4Ls2oKaKkmt5QdCa6NOKhln+6YsTuVNHndEE3LqXls9sAavTJarDQ4yPhToAy+7Ot+d kQTGCqvDNpEq22Y7gYIdoVAxUDCkIle0PBCssxPQ0/h84T7gf7ZcvfJ5mk5tD3PN22tV kfv2XOqIoS0xHtcuF/ZD/sdbfBo8j7VDQaQaDrOW7bnZerQJ1uzbL3deD7Qz5d3f8dV/ WHNVfCqk+VqaIr9nq+L0AezvhqqBILxIrHU8BWfKRwhC/lx/v42c5ROIN9gnjrd2r9o7 BgbFOJGQdrv1G9jVDaxsWAOl+3VeFuQRqXoizWKjv11UQ8RpiHN78J71x/DuY3jdfTXE 5g== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3e1smr2kx3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Sun, 06 Feb 2022 06:30:38 -0800 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 6 Feb 2022 06:30:36 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Sun, 6 Feb 2022 06:30:36 -0800 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 2966A3F7064; Sun, 6 Feb 2022 06:30:34 -0800 (PST) From: Nithin Dabilpuram To: , Radu Nicolau , Akhil Goyal CC: , Nithin Dabilpuram Subject: [PATCH 4/4] examples/ipsec-secgw: add per port pool and vector pool size Date: Sun, 6 Feb 2022 20:00:22 +0530 Message-ID: <20220206143022.13098-4-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20220206143022.13098-1-ndabilpuram@marvell.com> References: <20220206143022.13098-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: CD1n1dF3_-RjfcXquLlhHvfRVVKDxr0w X-Proofpoint-ORIG-GUID: CD1n1dF3_-RjfcXquLlhHvfRVVKDxr0w X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2022-02-06_04,2022-02-03_01,2021-12-02_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support to enable per port packet pool and also override vector pool size from command line args. This is useful on some HW to tune performance based on usecase. Signed-off-by: Nithin Dabilpuram --- examples/ipsec-secgw/event_helper.c | 17 ++++++-- examples/ipsec-secgw/event_helper.h | 2 + examples/ipsec-secgw/ipsec-secgw.c | 80 ++++++++++++++++++++++++++++--------- examples/ipsec-secgw/ipsec-secgw.h | 2 + examples/ipsec-secgw/ipsec.h | 2 +- 5 files changed, 80 insertions(+), 23 deletions(-) diff --git a/examples/ipsec-secgw/event_helper.c b/examples/ipsec-secgw/event_helper.c index 8947e41..172ab8e 100644 --- a/examples/ipsec-secgw/event_helper.c +++ b/examples/ipsec-secgw/event_helper.c @@ -792,8 +792,8 @@ eh_rx_adapter_configure(struct eventmode_conf *em_conf, uint32_t service_id, socket_id, nb_elem; struct rte_mempool *vector_pool = NULL; uint32_t lcore_id = rte_lcore_id(); + int ret, portid, nb_ports = 0; uint8_t eventdev_id; - int ret; int j; /* Get event dev ID */ @@ -806,10 +806,21 @@ eh_rx_adapter_configure(struct eventmode_conf *em_conf, return ret; } + RTE_ETH_FOREACH_DEV(portid) + if ((em_conf->eth_portmask & (1 << portid))) + nb_ports++; + if (em_conf->ext_params.event_vector) { socket_id = rte_lcore_to_socket_id(lcore_id); - nb_elem = (nb_bufs_in_pool / em_conf->ext_params.vector_size) - + 1; + + if (em_conf->vector_pool_sz) { + nb_elem = em_conf->vector_pool_sz; + } else { + nb_elem = (nb_bufs_in_pool / + em_conf->ext_params.vector_size) + 1; + if (per_port_pool) + nb_elem = nb_ports * nb_elem; + } vector_pool = rte_event_vector_pool_create( "vector_pool", nb_elem, 0, diff --git a/examples/ipsec-secgw/event_helper.h b/examples/ipsec-secgw/event_helper.h index 5be6c62..f3cbe57 100644 --- a/examples/ipsec-secgw/event_helper.h +++ b/examples/ipsec-secgw/event_helper.h @@ -183,6 +183,8 @@ struct eventmode_conf { /**< 64 bit field to specify extended params */ uint64_t vector_tmo_ns; /**< Max vector timeout in nanoseconds */ + uint64_t vector_pool_sz; + /**< Vector pool size */ }; /** diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c index d8a9bfa..96da776 100644 --- a/examples/ipsec-secgw/ipsec-secgw.c +++ b/examples/ipsec-secgw/ipsec-secgw.c @@ -118,6 +118,8 @@ struct flow_info flow_info_tbl[RTE_MAX_ETHPORTS]; #define CMD_LINE_OPT_EVENT_VECTOR "event-vector" #define CMD_LINE_OPT_VECTOR_SIZE "vector-size" #define CMD_LINE_OPT_VECTOR_TIMEOUT "vector-tmo" +#define CMD_LINE_OPT_VECTOR_POOL_SZ "vector-pool-sz" +#define CMD_LINE_OPT_PER_PORT_POOL "per-port-pool" #define CMD_LINE_ARG_EVENT "event" #define CMD_LINE_ARG_POLL "poll" @@ -145,6 +147,8 @@ enum { CMD_LINE_OPT_EVENT_VECTOR_NUM, CMD_LINE_OPT_VECTOR_SIZE_NUM, CMD_LINE_OPT_VECTOR_TIMEOUT_NUM, + CMD_LINE_OPT_VECTOR_POOL_SZ_NUM, + CMD_LINE_OPT_PER_PORT_POOL_NUM, }; static const struct option lgopts[] = { @@ -161,6 +165,8 @@ static const struct option lgopts[] = { {CMD_LINE_OPT_EVENT_VECTOR, 0, 0, CMD_LINE_OPT_EVENT_VECTOR_NUM}, {CMD_LINE_OPT_VECTOR_SIZE, 1, 0, CMD_LINE_OPT_VECTOR_SIZE_NUM}, {CMD_LINE_OPT_VECTOR_TIMEOUT, 1, 0, CMD_LINE_OPT_VECTOR_TIMEOUT_NUM}, + {CMD_LINE_OPT_VECTOR_POOL_SZ, 1, 0, CMD_LINE_OPT_VECTOR_POOL_SZ_NUM}, + {CMD_LINE_OPT_PER_PORT_POOL, 0, 0, CMD_LINE_OPT_PER_PORT_POOL_NUM}, {NULL, 0, 0, 0} }; @@ -234,7 +240,6 @@ struct lcore_conf { struct rt_ctx *rt6_ctx; struct { struct rte_ip_frag_tbl *tbl; - struct rte_mempool *pool_dir; struct rte_mempool *pool_indir; struct rte_ip_frag_death_row dr; } frag; @@ -262,6 +267,8 @@ static struct rte_eth_conf port_conf = { struct socket_ctx socket_ctx[NB_SOCKETS]; +bool per_port_pool; + /* * Determine is multi-segment support required: * - either frame buffer size is smaller then mtu @@ -630,12 +637,10 @@ send_fragment_packet(struct lcore_conf *qconf, struct rte_mbuf *m, if (proto == IPPROTO_IP) rc = rte_ipv4_fragment_packet(m, tbl->m_table + len, - n, mtu_size, qconf->frag.pool_dir, - qconf->frag.pool_indir); + n, mtu_size, m->pool, qconf->frag.pool_indir); else rc = rte_ipv6_fragment_packet(m, tbl->m_table + len, - n, mtu_size, qconf->frag.pool_dir, - qconf->frag.pool_indir); + n, mtu_size, m->pool, qconf->frag.pool_indir); if (rc >= 0) len += rc; @@ -1256,7 +1261,6 @@ ipsec_poll_mode_worker(void) qconf->outbound.session_pool = socket_ctx[socket_id].session_pool; qconf->outbound.session_priv_pool = socket_ctx[socket_id].session_priv_pool; - qconf->frag.pool_dir = socket_ctx[socket_id].mbuf_pool; qconf->frag.pool_indir = socket_ctx[socket_id].mbuf_pool_indir; rc = ipsec_sad_lcore_cache_init(app_sa_prm.cache_sz); @@ -1511,6 +1515,9 @@ print_usage(const char *prgname) " --vector-size Max vector size (default value: 16)\n" " --vector-tmo Max vector timeout in nanoseconds" " (default value: 102400)\n" + " --" CMD_LINE_OPT_PER_PORT_POOL " Enable per port mbuf pool\n" + " --" CMD_LINE_OPT_VECTOR_POOL_SZ " Vector pool size\n" + " (default value is based on mbuf count)\n" "\n", prgname); } @@ -1894,6 +1901,15 @@ parse_args(int32_t argc, char **argv, struct eh_conf *eh_conf) em_conf = eh_conf->mode_params; em_conf->vector_tmo_ns = ret; break; + case CMD_LINE_OPT_VECTOR_POOL_SZ_NUM: + ret = parse_decimal(optarg); + + em_conf = eh_conf->mode_params; + em_conf->vector_pool_sz = ret; + break; + case CMD_LINE_OPT_PER_PORT_POOL_NUM: + per_port_pool = 1; + break; default: print_usage(prgname); return -1; @@ -2378,6 +2394,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads) /* init RX queues */ for (queue = 0; queue < qconf->nb_rx_queue; ++queue) { struct rte_eth_rxconf rxq_conf; + struct rte_mempool *pool; if (portid != qconf->rx_queue_list[queue].port_id) continue; @@ -2389,9 +2406,14 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads) rxq_conf = dev_info.default_rxconf; rxq_conf.offloads = local_port_conf.rxmode.offloads; + + if (per_port_pool) + pool = socket_ctx[socket_id].mbuf_pool[portid]; + else + pool = socket_ctx[socket_id].mbuf_pool[0]; + ret = rte_eth_rx_queue_setup(portid, rx_queueid, - nb_rxd, socket_id, &rxq_conf, - socket_ctx[socket_id].mbuf_pool); + nb_rxd, socket_id, &rxq_conf, pool); if (ret < 0) rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup: err=%d, " @@ -2504,28 +2526,37 @@ session_priv_pool_init(struct socket_ctx *ctx, int32_t socket_id, } static void -pool_init(struct socket_ctx *ctx, int32_t socket_id, uint32_t nb_mbuf) +pool_init(struct socket_ctx *ctx, int32_t socket_id, int portid, + uint32_t nb_mbuf) { char s[64]; int32_t ms; - snprintf(s, sizeof(s), "mbuf_pool_%d", socket_id); - ctx->mbuf_pool = rte_pktmbuf_pool_create(s, nb_mbuf, - MEMPOOL_CACHE_SIZE, ipsec_metadata_size(), - frame_buf_size, socket_id); + + /* mbuf_pool is initialised by the pool_init() function*/ + if (socket_ctx[socket_id].mbuf_pool[portid]) + return; + + snprintf(s, sizeof(s), "mbuf_pool_%d_%d", socket_id, portid); + ctx->mbuf_pool[portid] = rte_pktmbuf_pool_create(s, nb_mbuf, + MEMPOOL_CACHE_SIZE, + ipsec_metadata_size(), + frame_buf_size, + socket_id); /* * if multi-segment support is enabled, then create a pool - * for indirect mbufs. + * for indirect mbufs. This is not per-port but global. */ ms = multi_seg_required(); - if (ms != 0) { + if (ms != 0 && !ctx->mbuf_pool_indir) { snprintf(s, sizeof(s), "mbuf_pool_indir_%d", socket_id); ctx->mbuf_pool_indir = rte_pktmbuf_pool_create(s, nb_mbuf, MEMPOOL_CACHE_SIZE, 0, 0, socket_id); } - if (ctx->mbuf_pool == NULL || (ms != 0 && ctx->mbuf_pool_indir == NULL)) + if (ctx->mbuf_pool[portid] == NULL || + (ms != 0 && ctx->mbuf_pool_indir == NULL)) rte_exit(EXIT_FAILURE, "Cannot init mbuf pool on socket %d\n", socket_id); else @@ -3338,11 +3369,22 @@ main(int32_t argc, char **argv) else socket_id = 0; - /* mbuf_pool is initialised by the pool_init() function*/ - if (socket_ctx[socket_id].mbuf_pool) + if (per_port_pool) { + RTE_ETH_FOREACH_DEV(portid) { + if ((enabled_port_mask & (1 << portid)) == 0) + continue; + + pool_init(&socket_ctx[socket_id], socket_id, + portid, nb_bufs_in_pool); + } + } else { + pool_init(&socket_ctx[socket_id], socket_id, 0, + nb_bufs_in_pool); + } + + if (socket_ctx[socket_id].session_pool) continue; - pool_init(&socket_ctx[socket_id], socket_id, nb_bufs_in_pool); session_pool_init(&socket_ctx[socket_id], socket_id, sess_sz); session_priv_pool_init(&socket_ctx[socket_id], socket_id, sess_sz); diff --git a/examples/ipsec-secgw/ipsec-secgw.h b/examples/ipsec-secgw/ipsec-secgw.h index ac4fa5e..24f11ad 100644 --- a/examples/ipsec-secgw/ipsec-secgw.h +++ b/examples/ipsec-secgw/ipsec-secgw.h @@ -134,6 +134,8 @@ extern volatile bool force_quit; extern uint32_t nb_bufs_in_pool; +extern bool per_port_pool; + static inline uint8_t is_unprotected_port(uint16_t port_id) { diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h index bc87b1a..ccfde8e 100644 --- a/examples/ipsec-secgw/ipsec.h +++ b/examples/ipsec-secgw/ipsec.h @@ -248,7 +248,7 @@ struct socket_ctx { struct sp_ctx *sp_ip6_out; struct rt_ctx *rt_ip4; struct rt_ctx *rt_ip6; - struct rte_mempool *mbuf_pool; + struct rte_mempool *mbuf_pool[RTE_MAX_ETHPORTS]; struct rte_mempool *mbuf_pool_indir; struct rte_mempool *session_pool; struct rte_mempool *session_priv_pool;