From patchwork Tue Feb 27 19:26:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amit Prakash Shukla X-Patchwork-Id: 137399 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4572443B80; Tue, 27 Feb 2024 20:26:14 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 36499409FA; Tue, 27 Feb 2024 20:26:14 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 5D6144026E for ; Tue, 27 Feb 2024 20:26:12 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 41RG2W6m023990; Tue, 27 Feb 2024 11:26:11 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s= pfpt0220; bh=C5H5VM3eKwiRTvhvJd3hkRNid9+DL2ERvQE2wmbiGl4=; b=gUg hM2Ieo8dhfjt/SI8mi3y0ng5Tpub+4Rn9uQ+3UkY3wq7d1Us442m0QhAkQYIglPE oQh1+idg5fAvGIPB+HTHor1AlptP+gaC7V4nqF7rLsBePL4u7qrpp07GwRLmQhtS eJEZEi2ZWV5M3Bt/4Deyhans8GHUccbG8oZTYNHpOXjtMFrnswzp+kMF+Kpv7uZO 4x/2DYzC1XGH55yB/NfatO2imHvQT4IkjfUeHSgAIM7INip6zuekr59jP4YlmyMH oehchhA7wOnYUkwRuPEp5ZqFrDLtgRz2Rn9xMUTDmvJEC5B6Gm9am0zgz3rQnPjB m1j2rYdIaJI26a3mtsg== Received: from dc6wp-exch02.marvell.com ([4.21.29.225]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3whjwdh277-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 27 Feb 2024 11:26:11 -0800 (PST) Received: from DC6WP-EXCH02.marvell.com (10.76.176.209) by DC6WP-EXCH02.marvell.com (10.76.176.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.12; Tue, 27 Feb 2024 11:26:09 -0800 Received: from maili.marvell.com (10.69.176.80) by DC6WP-EXCH02.marvell.com (10.76.176.209) with Microsoft SMTP Server id 15.2.1258.12 via Frontend Transport; Tue, 27 Feb 2024 11:26:09 -0800 Received: from cavium-OptiPlex-5090-BM14.. (unknown [10.28.36.157]) by maili.marvell.com (Postfix) with ESMTP id 7062B3F7099; Tue, 27 Feb 2024 11:26:06 -0800 (PST) From: Amit Prakash Shukla To: Cheng Jiang , Chengwen Feng CC: , , , Kevin Laatz , Bruce Richardson , "Pavan Nikhilesh" , Gowrishankar Muthukrishnan , Amit Prakash Shukla Subject: [PATCH v2] app/dma-perf: support bi-directional transfer Date: Wed, 28 Feb 2024 00:56:01 +0530 Message-ID: <20240227192601.3932913-1-amitprakashs@marvell.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240108082749.1016345-1-amitprakashs@marvell.com> References: <20240108082749.1016345-1-amitprakashs@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: jnU6_vSJMdDxIUb4BnRl7v2KJYSKA3nu X-Proofpoint-ORIG-GUID: jnU6_vSJMdDxIUb4BnRl7v2KJYSKA3nu X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-27_06,2024-02-27_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Adds bi-directional DMA transfer support to test performance. One DMA device on one core will do mem2dev transfer and another DMA device on another core will do dev2mem transfer. Depends-on: series-31252 ("PCI Dev and SG copy support") Signed-off-by: Amit Prakash Shukla --- v2: - Fixed depends on series. app/test-dma-perf/benchmark.c | 64 +++++++++++++++++++++++++++-------- app/test-dma-perf/config.ini | 5 +++ app/test-dma-perf/main.c | 18 +++++++++- app/test-dma-perf/main.h | 1 + 4 files changed, 73 insertions(+), 15 deletions(-) diff --git a/app/test-dma-perf/benchmark.c b/app/test-dma-perf/benchmark.c index 25ed6fa6d0..8a23944763 100644 --- a/app/test-dma-perf/benchmark.c +++ b/app/test-dma-perf/benchmark.c @@ -144,12 +144,19 @@ cache_flush_buf(__rte_unused struct rte_mbuf **array, static int vchan_data_populate(uint32_t dev_id, struct rte_dma_vchan_conf *qconf, - struct test_configure *cfg) + struct test_configure *cfg, uint16_t dev_num) { struct rte_dma_info info; qconf->direction = cfg->transfer_dir; + /* If its a bi-directional test, configure odd device for inbound dma + * transfer and even device for outbound dma transfer. + */ + if (cfg->is_bidir) + qconf->direction = (dev_num % 2) ? RTE_DMA_DIR_MEM_TO_DEV : + RTE_DMA_DIR_DEV_TO_MEM; + rte_dma_info_get(dev_id, &info); if (!(RTE_BIT64(qconf->direction) & info.dev_capa)) return -1; @@ -181,14 +188,15 @@ vchan_data_populate(uint32_t dev_id, struct rte_dma_vchan_conf *qconf, /* Configuration of device. */ static void -configure_dmadev_queue(uint32_t dev_id, struct test_configure *cfg, uint8_t ptrs_max) +configure_dmadev_queue(uint32_t dev_id, struct test_configure *cfg, uint8_t ptrs_max, + uint16_t dev_num) { uint16_t vchan = 0; struct rte_dma_info info; struct rte_dma_conf dev_config = { .nb_vchans = 1 }; struct rte_dma_vchan_conf qconf = { 0 }; - if (vchan_data_populate(dev_id, &qconf, cfg) != 0) + if (vchan_data_populate(dev_id, &qconf, cfg, dev_num) != 0) rte_exit(EXIT_FAILURE, "Error with vchan data populate.\n"); if (rte_dma_configure(dev_id, &dev_config) != 0) @@ -235,7 +243,7 @@ config_dmadevs(struct test_configure *cfg) } ldm->dma_ids[i] = dev_id; - configure_dmadev_queue(dev_id, cfg, ptrs_max); + configure_dmadev_queue(dev_id, cfg, ptrs_max, nb_dmadevs); ++nb_dmadevs; } @@ -504,7 +512,7 @@ setup_memory_env(struct test_configure *cfg, } } - if (cfg->transfer_dir == RTE_DMA_DIR_DEV_TO_MEM) { + if (cfg->transfer_dir == RTE_DMA_DIR_DEV_TO_MEM && !cfg->is_bidir) { ext_buf_info->free_cb = dummy_free_ext_buf; ext_buf_info->fcb_opaque = NULL; for (i = 0; i < nr_buf; i++) { @@ -516,7 +524,7 @@ setup_memory_env(struct test_configure *cfg, } } - if (cfg->transfer_dir == RTE_DMA_DIR_MEM_TO_DEV) { + if (cfg->transfer_dir == RTE_DMA_DIR_MEM_TO_DEV && !cfg->is_bidir) { ext_buf_info->free_cb = dummy_free_ext_buf; ext_buf_info->fcb_opaque = NULL; for (i = 0; i < nr_buf; i++) { @@ -528,6 +536,18 @@ setup_memory_env(struct test_configure *cfg, } } + if (cfg->is_bidir) { + ext_buf_info->free_cb = dummy_free_ext_buf; + ext_buf_info->fcb_opaque = NULL; + for (i = 0; i < nr_buf; i++) { + /* Using mbuf structure to hold remote iova address. */ + rte_pktmbuf_attach_extbuf((*srcs)[i], (void *)(cfg->vchan_dev.raddr + + (i * buf_size)), (rte_iova_t)(cfg->vchan_dev.raddr + + (i * buf_size)), 0, ext_buf_info); + rte_mbuf_ext_refcnt_update(ext_buf_info, 1); + } + } + if (cfg->is_sg) { uint8_t src_ptrs = cfg->src_ptrs; uint8_t dst_ptrs = cfg->dst_ptrs; @@ -649,16 +669,30 @@ mem_copy_benchmark(struct test_configure *cfg) lcores[i]->nr_buf = (uint32_t)(nr_buf / nb_workers); lcores[i]->buf_size = buf_size; lcores[i]->test_secs = test_secs; - lcores[i]->srcs = srcs + offset; - lcores[i]->dsts = dsts + offset; lcores[i]->scenario_id = cfg->scenario_id; lcores[i]->lcore_id = lcore_id; - if (cfg->is_sg) { - lcores[i]->src_ptrs = cfg->src_ptrs; - lcores[i]->dst_ptrs = cfg->dst_ptrs; - lcores[i]->src_sges = src_sges + (nr_sgsrc / nb_workers * i); - lcores[i]->dst_sges = dst_sges + (nr_sgdst / nb_workers * i); + /* Number of workers is equal to number of devices. In case of bi-directional + * dma, use 1 device for mem-to-dev and 1 device for dev-to-mem. + */ + if (cfg->is_dma && cfg->is_bidir && (i % 2 != 0)) { + lcores[i]->dsts = srcs + offset; + lcores[i]->srcs = dsts + offset; + if (cfg->is_sg) { + lcores[i]->dst_ptrs = cfg->src_ptrs; + lcores[i]->src_ptrs = cfg->dst_ptrs; + lcores[i]->dst_sges = src_sges + (nr_sgsrc / nb_workers * i); + lcores[i]->src_sges = dst_sges + (nr_sgdst / nb_workers * i); + } + } else { + lcores[i]->srcs = srcs + offset; + lcores[i]->dsts = dsts + offset; + if (cfg->is_sg) { + lcores[i]->src_ptrs = cfg->src_ptrs; + lcores[i]->dst_ptrs = cfg->dst_ptrs; + lcores[i]->src_sges = src_sges + (nr_sgsrc / nb_workers * i); + lcores[i]->dst_sges = dst_sges + (nr_sgdst / nb_workers * i); + } } if (cfg->is_dma) { @@ -759,6 +793,8 @@ mem_copy_benchmark(struct test_configure *cfg) calc_result(buf_size, nr_buf, nb_workers, test_secs, lcores[i]->worker_info.test_cpl, &memory, &avg_cycles, &bandwidth, &mops); + if (cfg->is_bidir) + printf("%s direction\n", i % 2 ? "MEM-to-DEV" : "DEV-to-MEM"); output_result(cfg, lcores[i], kick_batch, avg_cycles, buf_size, nr_buf / nb_workers, memory, bandwidth, mops); mops_total += mops; @@ -772,7 +808,7 @@ mem_copy_benchmark(struct test_configure *cfg) out: - if (cfg->transfer_dir == RTE_DMA_DIR_DEV_TO_MEM) + if (cfg->transfer_dir == RTE_DMA_DIR_DEV_TO_MEM || cfg->is_bidir) m = srcs; else if (cfg->transfer_dir == RTE_DMA_DIR_MEM_TO_DEV) m = dsts; diff --git a/app/test-dma-perf/config.ini b/app/test-dma-perf/config.ini index 28f6c9d1db..7814ddcdc3 100644 --- a/app/test-dma-perf/config.ini +++ b/app/test-dma-perf/config.ini @@ -61,6 +61,10 @@ ; "pfid" denotes PF-id to be used for data transfer ; "vfid" denotes VF-id of PF-id to be used for data transfer. +; "xfer_mode" denotes mode of data transfer. It can take 2 values: +; 0 - unidirection transfer based on direction configured (default). +; 1 - Bi-directional transfer based on direction configured (mem-to-dev and dev-to-mem). + ; =========== End of "mem2dev" and "dev2mem" config parameters. ============== [case1] @@ -95,6 +99,7 @@ eal_args=--in-memory --file-prefix=test skip=1 type=DMA_MEM_COPY direction=dev2mem +xfer_mode=0 vchan_dev=raddr=0x200000000,coreid=1,pfid=2,vfid=3 mem_size=10 buf_size=64,4096,2,MUL diff --git a/app/test-dma-perf/main.c b/app/test-dma-perf/main.c index a27e4c9429..4488890697 100644 --- a/app/test-dma-perf/main.c +++ b/app/test-dma-perf/main.c @@ -368,6 +368,7 @@ load_configs(const char *path) const char *skip; struct rte_kvargs *kvlist; const char *vchan_dev; + const char *xfer_mode; int args_nr, nb_vp; bool is_dma; @@ -421,6 +422,21 @@ load_configs(const char *path) test_case->transfer_dir = RTE_DMA_DIR_MEM_TO_MEM; } } + + xfer_mode = rte_cfgfile_get_entry(cfgfile, section_name, "xfer_mode"); + if (xfer_mode) { + int xmode = atoi(xfer_mode); + if (xmode == 1) { + if (test_case->transfer_dir == RTE_DMA_DIR_MEM_TO_MEM) { + printf("Error: Invalid configuration. For mem to" + " mem dma transfer bi-directional cannot be" + " configured.\n"); + test_case->is_valid = false; + continue; + } + test_case->is_bidir = true; + } + } is_dma = true; } else if (strcmp(case_type, CPU_MEM_COPY) == 0) { test_case->test_type = TEST_TYPE_CPU_MEM_COPY; @@ -433,7 +449,7 @@ load_configs(const char *path) } if (test_case->transfer_dir == RTE_DMA_DIR_MEM_TO_DEV || - test_case->transfer_dir == RTE_DMA_DIR_DEV_TO_MEM) { + test_case->transfer_dir == RTE_DMA_DIR_DEV_TO_MEM) { vchan_dev = rte_cfgfile_get_entry(cfgfile, section_name, "vchan_dev"); if (vchan_dev == NULL) { printf("Transfer direction mem2dev and dev2mem" diff --git a/app/test-dma-perf/main.h b/app/test-dma-perf/main.h index baf149b72b..70f6c393c2 100644 --- a/app/test-dma-perf/main.h +++ b/app/test-dma-perf/main.h @@ -67,6 +67,7 @@ struct test_configure { const char *eal_args; uint8_t scenario_id; struct test_vchan_dev_config vchan_dev; + bool is_bidir; }; int mem_copy_benchmark(struct test_configure *cfg);