From patchwork Thu Apr 20 07:20:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ling, WeiX" X-Patchwork-Id: 126288 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C470942993; Thu, 20 Apr 2023 09:21:41 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BF90241141; Thu, 20 Apr 2023 09:21:41 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 18D7F40687 for ; Thu, 20 Apr 2023 09:21:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681975300; x=1713511300; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rSn8sasq2j9Ph+K7iyahaQHtFp8ENMsLqBq/4VOyuic=; b=TSzBYR/9Y3ZZsNtcUM2C/pLqzY2Xngm4Nwvf0l/qxTg+3l3XdckbKgo/ WzkI+UO4jvvyeJxd7/vMQC/aPKy26hfDC/scIc/JMPlc2vUHLIdQr8Nmj IqGL7BFgdonEQsXqEyuqeBeDbT97UTxFtHoBtHDQNRXnSIYjHJZQKzI/R JtYDA/uKOV7qrIxlG40ToFJZbnGFqZSEKEMWo0qVoDCBDi+Hrcdi5UAPA JsB/wiKCEzEelcf457JO1qiJ1dgxTM5Yh2ZG9mn9RUvJgvrzHlE2wwOPR HUM3i9l3Y+bb/90W46oQE9ojPAZyKYSCs0Av8iU20s1Kbxknf4yl7HCej A==; X-IronPort-AV: E=McAfee;i="6600,9927,10685"; a="325985652" X-IronPort-AV: E=Sophos;i="5.99,211,1677571200"; d="scan'208";a="325985652" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 00:21:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10685"; a="756394677" X-IronPort-AV: E=Sophos;i="5.99,211,1677571200"; d="scan'208";a="756394677" Received: from unknown (HELO localhost.localdomain) ([10.239.252.222]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 00:21:38 -0700 From: Wei Ling To: dts@dpdk.org Cc: Wei Ling Subject: [dts][PATCH V1 1/3] test_plans/pvp_diff_qemu_version: add case3 to test virtio 1.1 path Date: Thu, 20 Apr 2023 15:20:24 +0800 Message-Id: <20230420072026.10999-2-weix.ling@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230420072026.10999-1-weix.ling@intel.com> References: <20230420072026.10999-1-weix.ling@intel.com> MIME-Version: 1.0 X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org 1.Add case3 to test virtio 1.1 path. 2.Modify the QEMU version as the version upgrade. Signed-off-by: Wei Ling --- .../pvp_diff_qemu_version_test_plan.rst | 51 ++++++++++++++++--- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/test_plans/pvp_diff_qemu_version_test_plan.rst b/test_plans/pvp_diff_qemu_version_test_plan.rst index 83a7b269..0514eeaa 100644 --- a/test_plans/pvp_diff_qemu_version_test_plan.rst +++ b/test_plans/pvp_diff_qemu_version_test_plan.rst @@ -8,7 +8,7 @@ vhost/virtio different qemu version test plan Description =========== -This test plan will test pvp different qemu version test cases, also cover virtio 1.0 and virtio 0.95. +This test plan will test pvp different qemu version test cases, also cover virtio 0.95, virtio 1.0 and virtio 1.1. Test flow ========= @@ -27,9 +27,9 @@ Test Case 1: PVP multi qemu version test with virtio 0.95 mergeable path testpmd>set fwd mac testpmd>start -2. Check dut machine already has installed different version qemu, includes [qemu_2.5, qemu_2.6, qemu_2.7, qemu_2.8, qemu_2.9, qemu_2.10, qemu_2.11, qemu_2.12, qemu_3.0]. +2. Check dut machine already has installed different version qemu, includes [qemu-5.2.0, qemu-6.2.0, qemu-7.0.0, qemu-7.1.0, qemu-7.2.0]. -3. Go to the absolute_path of different version qemu,then launch VM with different version qemu:: +3. Go to the absolute_path of different version qemu,then launch VM with different version qemu, note: we need add "disable-modern=true" to enable virtio 0.95:: qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 2 -m 4096 \ -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \ @@ -40,7 +40,7 @@ Test Case 1: PVP multi qemu version test with virtio 0.95 mergeable path -netdev user,id=netdev0,hostfwd=tcp:127.0.0.1:6002-:22 \ -chardev socket,id=char0,path=./vhost-net \ -netdev type=vhost-user,id=netdev1,chardev=char0,vhostforce \ - -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,mrg_rxbuf=on \ + -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,,disable-modern=true,mrg_rxbuf=on \ -vnc :10 4. On VM, bind virtio net to vfio-pci and run testpmd :: @@ -65,7 +65,7 @@ Test Case 2: PVP test with virtio 1.0 mergeable path testpmd>set fwd mac testpmd>start -2. Check dut machine already has installed different version qemu, includes [qemu_2.5, qemu_2.6, qemu_2.7, qemu_2.8, qemu_2.9, qemu_2.10, qemu_2.11, qemu_2.12, qemu_3.0]. +2. Check dut machine already has installed different version qemu, includes [qemu-5.2.0, qemu-6.2.0, qemu-7.0.0, qemu-7.1.0, qemu-7.2.0]. 3. Go to the absolute_path of different version qemu,then launch VM with different version qemu, note: we need add "disable-modern=false" to enable virtio 1.0:: @@ -90,4 +90,43 @@ Test Case 2: PVP test with virtio 1.0 mergeable path 4. Send packet by packet generator with different packet sizes(68,128,256,512,1024,1280,1518),repeat below command to get throughput 10 times,then calculate the average throughput:: - testpmd>show port stats all \ No newline at end of file + testpmd>show port stats all + +Test Case 3: PVP test with virtio 1.1 mergeable path +==================================================== + +1. Bind one port to vfio-pci, then launch testpmd by below command:: + + rm -rf vhost-net* + ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0xe -n 4 \ + --vdev 'eth_vhost0,iface=vhost-net,queues=1' -- \ + -i --nb-cores=1 --txd=1024 --rxd=1024 + testpmd>set fwd mac + testpmd>start + +2. Check dut machine already has installed different version qemu, includes [qemu-5.2.0, qemu-6.2.0, qemu-7.0.0, qemu-7.1.0, qemu-7.2.0]. + +3. Go to the absolute_path of different version qemu,then launch VM with different version qemu, note: we need add "disable-modern=false,packed_vq=on" to enable virtio 1.1:: + + qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 2 -m 4096 \ + -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \ + -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu16.img \ + -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \ + -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \ + -monitor unix:/tmp/vm2_monitor.sock,server,nowait -net nic,macaddr=00:00:00:08:e8:aa,addr=1f \ + -netdev user,id=netdev0,hostfwd=tcp:127.0.0.1:6002-:22 \ + -chardev socket,id=char0,path=./vhost-net \ + -netdev type=vhost-user,id=netdev1,chardev=char0,vhostforce \ + -device virtio-net-pci,netdev=netdev1,mac=52:54:00:00:00:01,disable-modern=false,mrg_rxbuf=on,packed_vq=on \ + -vnc :10 + +3. On VM, bind virtio net to vfio-pci and run testpmd:: + + ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0x3 -n 3 -- -i \ + --nb-cores=1 --txd=1024 --rxd=1024 + testpmd>set fwd mac + testpmd>start + +4. Send packet by packet generator with different packet sizes(68,128,256,512,1024,1280,1518),repeat below command to get throughput 10 times,then calculate the average throughput:: + + testpmd>show port stats all From patchwork Thu Apr 20 07:20:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ling, WeiX" X-Patchwork-Id: 126289 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F2CFA42993; Thu, 20 Apr 2023 09:21:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ECE1542C76; Thu, 20 Apr 2023 09:21:44 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 6885A40687 for ; Thu, 20 Apr 2023 09:21:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681975302; x=1713511302; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ok+iMQrC0hReE7LUCMoLm32stDYOuDkPHkzVGjzla4U=; b=lOtej5E2EOfPgZXN7LlnS+toU1tvTHBqxsgVN+oytljIiVtJfhweMxSm GeWN0YqzhMsxHQOfUh+D4FsIIcFcOUpIKn8qjv41WjI74hXXiNcySQ/kL oMhQPBKan4OjFZHzQOqZRqGlQEMAIsafSt0MQ6clg6LsbRrIhQIPT9UCv XyNWO+m8PqkgtMzBepVjOVAQ7FwcJ7nNs54CjUyVNyLiMHHNTy00tpX6x CiY7DfvADZw0WTNaBsRbyfWcoe6DvX5eralq6dvLSM+Zn+ba3rvGk5Zu+ yE/LsBfNibyt0dH6Z5PAVOJXsc6Ohu0tskmoe2+E4KFBRGZ77VzbxWzN7 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10685"; a="325985668" X-IronPort-AV: E=Sophos;i="5.99,211,1677571200"; d="scan'208";a="325985668" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 00:21:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10685"; a="756394696" X-IronPort-AV: E=Sophos;i="5.99,211,1677571200"; d="scan'208";a="756394696" Received: from unknown (HELO localhost.localdomain) ([10.239.252.222]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 00:21:40 -0700 From: Wei Ling To: dts@dpdk.org Cc: Wei Ling Subject: [dts][PATCH V1 2/3] tests/pvp_diff_qemu_version: add case3 to test virtio1.1 and optimize verify expected value Date: Thu, 20 Apr 2023 15:20:25 +0800 Message-Id: <20230420072026.10999-3-weix.ling@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230420072026.10999-1-weix.ling@intel.com> References: <20230420072026.10999-1-weix.ling@intel.com> MIME-Version: 1.0 X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org 1.Add case3 to test virtio 1.1 path. 2.Modify the casename with virtio protocol version. 3.Optimize the verify the expected value from the hard code to read from conf/pvp_diff_qemu_version.cfg to adapt different platform. Signed-off-by: Wei Ling --- tests/TestSuite_pvp_diff_qemu_version.py | 229 +++++++++++++---------- 1 file changed, 129 insertions(+), 100 deletions(-) diff --git a/tests/TestSuite_pvp_diff_qemu_version.py b/tests/TestSuite_pvp_diff_qemu_version.py index 02682815..70d546b3 100644 --- a/tests/TestSuite_pvp_diff_qemu_version.py +++ b/tests/TestSuite_pvp_diff_qemu_version.py @@ -2,23 +2,11 @@ # Copyright(c) 2019 Intel Corporation # -""" -DPDK Test suite. - -Vhost PVP performance using differnet Qemu test suite. -Can config the qemu version in config file like: -qemu = - path=qemu-2.5/bin/qemu-system-x86_64; - path=qemu-2.6/bin/qemu-system-x86_64; -""" import re import time -from scapy.utils import wrpcap - -import framework.utils as utils from framework.pktgen import PacketGeneratorHelper -from framework.settings import HEADER_SIZE +from framework.settings import HEADER_SIZE, UPDATE_EXPECTED, load_global_setting from framework.test_case import TestCase from framework.virt_common import VM @@ -34,7 +22,6 @@ class TestVhostPVPDiffQemuVersion(TestCase): self.cores_num = len( [n for n in self.dut.cores if int(n["socket"]) == self.socket] ) - self.verify(len(self.dut_ports) >= 1, "Insufficient ports for testing") self.verify( self.cores_num >= 3, "There has not enought cores to test this suite" @@ -42,7 +29,6 @@ class TestVhostPVPDiffQemuVersion(TestCase): self.cores = self.dut.get_core_list("1S/3C/1T", socket=self.socket) self.vm_dut = None self.packet_params_set() - self.logger.info( "You can config all the path of qemu version you want to" + " tested in the conf file %s.cfg" % self.suite_name @@ -53,7 +39,6 @@ class TestVhostPVPDiffQemuVersion(TestCase): ) res = self.verify_qemu_version_config() self.verify(res is True, "The path of qemu version in config file not right") - self.out_path = "/tmp" out = self.tester.send_expect("ls -d %s" % self.out_path, "# ") if "No such file or directory" in out: @@ -62,36 +47,37 @@ class TestVhostPVPDiffQemuVersion(TestCase): self.pktgen_helper = PacketGeneratorHelper() self.base_dir = self.dut.base_dir.replace("~", "/root") self.pci_info = self.dut.ports_info[0]["pci"] - self.number_of_ports = 1 + self.nb_ports = 1 self.path = self.dut.apps_name["test-pmd"] self.testpmd_name = self.path.split("/")[-1] + self.expected_throughput = self.get_suite_cfg()["expected_throughput"] + self.gap = self.get_suite_cfg()["accepted_tolerance"] + self.vhost_user = self.dut.new_session(suite="vhost-user") def set_up(self): """ Run before each test case. """ - self.vhost = self.dut.new_session(suite="vhost-user") self.dut.send_expect("rm -rf %s/vhost-net*" % self.base_dir, "#") self.dut.send_expect("killall -s INT %s" % self.testpmd_name, "#") self.dut.send_expect("killall -I qemu-system-x86_64", "#", 20) + self.throughput = dict() + self.test_result = [] + self.table_header = [ + "QemuVersion", + "FrameSize(B)", + "Throughput(Mpps)", + "LineRate(%)", + ] def packet_params_set(self): - self.frame_sizes = [64, 128, 256, 512, 1024, 1500] + self.frame_sizes = [64, 128, 256, 512, 1024, 1280, 1518] # get the frame_sizes from cfg file if "packet_sizes" in self.get_suite_cfg(): self.frame_sizes = self.get_suite_cfg()["packet_sizes"] - self.virtio1_mac = "52:54:00:00:00:01" self.src1 = "192.168.4.1" self.dst1 = "192.168.3.1" - self.header_row = [ - "QemuVersion", - "FrameSize(B)", - "Throughput(Mpps)", - "LineRate(%)", - "Cycle", - "Expected Throughput(Mpps)", - ] def get_qemu_list_from_config(self): """ @@ -122,7 +108,6 @@ class TestVhostPVPDiffQemuVersion(TestCase): qemu_num = len(self.qemu_list) for i in range(qemu_num): qemu_path = self.qemu_list[i]["path"] - out = self.dut.send_expect("ls %s" % qemu_path, "#") if "No such file or directory" in out: self.logger.error( @@ -137,13 +122,14 @@ class TestVhostPVPDiffQemuVersion(TestCase): % (qemu_path, self.dut.get_ip_address()) ) return False - out = self.dut.send_expect("%s --version" % qemu_path, "#") result = re.search("QEMU\s*emulator\s*version\s*(\d*.\d*)", out) version = result.group(1) # update the version info to self.qemu_list self.qemu_list[i].update({"version": "qemu-%s" % version}) - + self.qemu_versions = list() + for i in self.qemu_list: + self.qemu_versions.append(i["version"]) # print all the qemu version you config config_qemu_version = "" for i in range(len(self.qemu_list)): @@ -151,7 +137,6 @@ class TestVhostPVPDiffQemuVersion(TestCase): self.logger.info( "The suite will test the qemu version of: %s" % config_qemu_version ) - return True def rm_vm_qemu_path_config(self): @@ -167,7 +152,7 @@ class TestVhostPVPDiffQemuVersion(TestCase): self.verify(qemu_index < params_num, "Please config qemu path in conf gile") self.vm.params.pop(qemu_index) - def start_vm(self, path, modem): + def start_one_vm(self, qemu_path, qemu_mode): """ start vm """ @@ -176,15 +161,17 @@ class TestVhostPVPDiffQemuVersion(TestCase): vm_params["driver"] = "vhost-user" vm_params["opt_path"] = "%s/vhost-net" % self.base_dir vm_params["opt_mac"] = self.virtio1_mac - if modem == 1: - vm_params["opt_settings"] = "disable-modern=false,mrg_rxbuf=on" - elif modem == 0: + if qemu_mode == 0: vm_params["opt_settings"] = "disable-modern=true,mrg_rxbuf=on" + elif qemu_mode == 1: + vm_params["opt_settings"] = "disable-modern=false,mrg_rxbuf=on" + elif qemu_mode == 2: + vm_params["opt_settings"] = "disable-modern=false,mrg_rxbuf=on,packed_vq=on" self.vm.set_vm_device(**vm_params) self.vm.load_config() self.rm_vm_qemu_path_config() # set qemu version info - self.vm.set_qemu_emulator(path) + self.vm.set_qemu_emulator(qemu_path) # Due to we have change the params info before, # so need to start vm with load_config=False try: @@ -204,9 +191,9 @@ class TestVhostPVPDiffQemuVersion(TestCase): ) para = " -- -i --nb-cores=1 --txd=1024 --rxd=1024" command_line_client = self.path + eal_params + para - self.vhost.send_expect(command_line_client, "testpmd> ", 30) - self.vhost.send_expect("set fwd mac", "testpmd> ", 30) - self.vhost.send_expect("start", "testpmd> ", 30) + self.vhost_user.send_expect(command_line_client, "testpmd> ", 30) + self.vhost_user.send_expect("set fwd mac", "testpmd> ", 30) + self.vhost_user.send_expect("start", "testpmd> ", 30) def vm_testpmd_start(self): """ @@ -220,26 +207,9 @@ class TestVhostPVPDiffQemuVersion(TestCase): self.vm_dut.send_expect("set fwd mac", "testpmd> ", 20) self.vm_dut.send_expect("start", "testpmd> ") - @property - def check_value(self): - check_dict = dict.fromkeys(self.frame_sizes) - linerate = { - 64: 0.07, - 128: 0.10, - 256: 0.17, - 512: 0.18, - 1024: 0.35, - 1280: 0.40, - 1500: 0.45, - } - for size in self.frame_sizes: - speed = self.wirespeed(self.nic, size, self.number_of_ports) - check_dict[size] = round(speed * linerate[size], 2) - return check_dict - - def send_verify(self, qemu_version, vlan_id1=0, tag="Performance"): - self.result_table_create(self.header_row) - perf_result = {} + def perf_test(self, qemu_version, vlan_id1=0): + self.result_table_create(self.table_header) + self.throughput[qemu_version] = {} for frame_size in self.frame_sizes: info = "Running test %s, and %d frame size." % ( self.running_case, @@ -264,42 +234,30 @@ class TestVhostPVPDiffQemuVersion(TestCase): tgenInput.append( (port, port, "%s/pvp_diff_qemu_version.pcap" % self.out_path) ) - self.tester.pktgen.clear_streams() streams = self.pktgen_helper.prepare_stream_from_tginput( tgenInput, 100, None, self.tester.pktgen ) # set traffic option - traffic_opt = {"delay": 5, "duration": 20} + traffic_opt = { + "delay": 5, + "duration": self.get_suite_cfg()["test_duration"], + } _, pps = self.tester.pktgen.measure_throughput( stream_ids=streams, options=traffic_opt ) Mpps = pps / 1000000.0 line_rate = Mpps * 100 / float(self.wirespeed(self.nic, frame_size, 1)) + self.throughput[qemu_version][frame_size] = Mpps # update print table info data_row = [ qemu_version, frame_size, str(Mpps), str(line_rate), - tag, - self.check_value[frame_size], ] self.result_table_add(data_row) - perf_result[frame_size] = Mpps - self.result_table_print() - for frame_size in perf_result.keys(): - self.verify( - perf_result[frame_size] > self.check_value[frame_size], - "%s of frame size %d speed verify failed, expect %s, result %s" - % ( - self.running_case, - frame_size, - self.check_value[frame_size], - perf_result[frame_size], - ), - ) def close_testpmd_and_qemu(self): """ @@ -307,57 +265,128 @@ class TestVhostPVPDiffQemuVersion(TestCase): close the qemu """ self.vm_dut.send_expect("quit", "#", 20) - self.vhost.send_expect("quit", "#", 20) + self.vhost_user.send_expect("quit", "#", 20) self.vm.stop() self.dut.send_expect("killall -I %s" % self.testpmd_name, "#", 20) self.dut.send_expect("rm -rf %s/vhost-net*" % self.base_dir, "#") - def test_perf_vhost_pvp_diffrent_qemu_version_mergeable_mac(self): + def test_perf_vhost_pvp_diffrent_qemu_version_virtio95_mergeable_path(self): """ - Test the performance of one vm with virtio 0.95 on mergeable path + Test Case 1: PVP multi qemu version test with virtio 0.95 mergeable path """ for i in range(len(self.qemu_list)): - path = self.qemu_list[i]["path"] - version = self.qemu_list[i]["version"] + qemu_path = self.qemu_list[i]["path"] + qemu_version = self.qemu_list[i]["version"] self.start_vhost_testpmd() - self.start_vm(path, 0) - # Start testpmd in vm + self.logger.info("now testing the qemu path of %s" % qemu_path) + self.start_one_vm(qemu_path=qemu_path, qemu_mode=0) self.vm_testpmd_start() - self.logger.info("now testing the qemu path of %s" % path) - time.sleep(5) vlan_id1 = 1000 - self.send_verify(version, vlan_id1, "virtio-0.95, Mergeable") + self.perf_test(qemu_version, vlan_id1) self.close_testpmd_and_qemu() + self.handle_expected() + self.handle_results() - def test_perf_vhost_pvp_diffrent_qemu_version_modern_mergeable_mac(self): + def test_perf_vhost_pvp_diffrent_qemu_version_virtio10_mergeable_path(self): """ - Test the performance of one vm with virtio 1.0 on mergeable path + Test Case 2: PVP test with virtio 1.0 mergeable path """ for i in range(len(self.qemu_list)): - path = self.qemu_list[i]["path"] - version = self.qemu_list[i]["version"] + qemu_path = self.qemu_list[i]["path"] + qemu_version = self.qemu_list[i]["version"] self.start_vhost_testpmd() - self.start_vm(path, 1) - # Start testpmd in vm + self.logger.info("now testing the qemu path of %s" % qemu_path) + self.start_one_vm(qemu_path=qemu_path, qemu_mode=1) self.vm_testpmd_start() - self.logger.info("now testing the qemu path of %s" % path) - time.sleep(5) vlan_id1 = 1000 - self.send_verify(version, vlan_id1, "virtio-1.0, Mergeable") + self.perf_test(qemu_version, vlan_id1) self.close_testpmd_and_qemu() + self.handle_expected() + self.handle_results() + + def test_perf_vhost_pvp_diffrent_qemu_version_virtio11_mergeable_path(self): + """ + Test Case 3: PVP test with virtio 1.1 mergeable path + """ + for i in range(len(self.qemu_list)): + qemu_path = self.qemu_list[i]["path"] + qemu_version = self.qemu_list[i]["version"] + self.start_vhost_testpmd() + self.logger.info("now testing the qemu path of %s" % qemu_path) + self.start_one_vm(qemu_path=qemu_path, qemu_mode=2) + self.vm_testpmd_start() + vlan_id1 = 1000 + self.perf_test(qemu_version, vlan_id1) + self.close_testpmd_and_qemu() + self.handle_expected() + self.handle_results() + + def handle_expected(self): + """ + Update expected numbers to configurate file: $DTS_CFG_FOLDER/$suite_name.cfg + """ + if load_global_setting(UPDATE_EXPECTED) == "yes": + for qemu_version in self.qemu_versions: + for frame_size in self.frame_sizes: + self.expected_throughput[self.running_case][qemu_version][ + frame_size + ] = round(self.throughput[qemu_version][frame_size], 3) + + def handle_results(self): + """ + results handled process: + 1, save to self.test_results + 2, create test results table + """ + # save test results to self.test_result + header = self.table_header + header.append("Expected Throughput(Mpps)") + header.append("Status") + self.result_table_create(self.table_header) + for qv in self.qemu_list: + qemu_version = qv["version"] + for frame_size in self.frame_sizes: + wirespeed = self.wirespeed(self.nic, frame_size, self.nb_ports) + ret_data = {} + ret_data[header[0]] = qemu_version + ret_data[header[1]] = str(frame_size) + _real = float(self.throughput[qemu_version][frame_size]) + _exp = float( + self.expected_throughput[self.running_case][qemu_version][ + frame_size + ] + ) + ret_data[header[2]] = "{:.3f}".format(_real) + ret_data[header[3]] = "{:.3f}%".format(_real * 100 / wirespeed) + ret_data[header[4]] = "{:.3f}".format(_exp) + gap = _exp * -self.gap * 0.01 + if _real > _exp + gap: + ret_data[header[5]] = "PASS" + else: + ret_data[header[5]] = "FAIL" + self.test_result.append(ret_data) + + for test_result in self.test_result: + table_row = list() + for i in range(len(header)): + table_row.append(test_result[header[i]]) + self.result_table_add(table_row) + # present test results to screen + self.result_table_print() + self.verify( + "FAIL" not in self.test_result, + "Excessive gap between test results and expectations", + ) def tear_down(self): """ Run after each test case. - Clear qemu and testpmd to avoid blocking the following TCs """ - self.dut.close_session(self.vhost) self.dut.send_expect("killall -s INT qemu-system-x86_64", "#") self.dut.send_expect("killall -s INT %s" % self.testpmd_name, "#") - time.sleep(2) def tear_down_all(self): """ Run after each test suite. """ - pass + self.dut.close_session(self.vhost_user) From patchwork Thu Apr 20 07:20:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ling, WeiX" X-Patchwork-Id: 126290 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2881342993; Thu, 20 Apr 2023 09:21:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 234DD40EE3; Thu, 20 Apr 2023 09:21:47 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 2826940687 for ; Thu, 20 Apr 2023 09:21:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681975305; x=1713511305; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Kpy21i120uMpiKRJs6GJ+wCR/dParnkfRODivedMEyE=; b=ZIBwd3ozxAWLqFmC0lKgqY/Pt16Xjx/uSv9/seZkmu38Q3pEkDF9oF91 nEemAUGBcOGmSSlrjJaMwkdVn9kT7jaaP5JQ5NIZ7qeBcMFJ5jY5hmbZZ dhnSNN5kPcrBb+l2/1qRCmAvCgerQvOQoACDp0elkphbzITxBzaRovbps /yQVC/R9S60Knx6keAE4RFp4GVeJBcJQlWSTGei7E4GxPcfPBA9hg62Ph SXP2JoQP6YLCyiM+zNTdt8NTFEO5xHHbDZ/QZP+CmApjFAyvowqlnEZ7Y 9yn+a8cLLCtamJuGMn4CD0+6Sl8NLOoDke7OmmPaNtnewzJofCi/moxqA w==; X-IronPort-AV: E=McAfee;i="6600,9927,10685"; a="325985679" X-IronPort-AV: E=Sophos;i="5.99,211,1677571200"; d="scan'208";a="325985679" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 00:21:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10685"; a="756394722" X-IronPort-AV: E=Sophos;i="5.99,211,1677571200"; d="scan'208";a="756394722" Received: from unknown (HELO localhost.localdomain) ([10.239.252.222]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 00:21:42 -0700 From: Wei Ling To: dts@dpdk.org Cc: Wei Ling Subject: [dts][PATCH V1 3/3] conf/pvp_diff_qemu_version: add expected value to verify Date: Thu, 20 Apr 2023 15:20:26 +0800 Message-Id: <20230420072026.10999-4-weix.ling@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230420072026.10999-1-weix.ling@intel.com> References: <20230420072026.10999-1-weix.ling@intel.com> MIME-Version: 1.0 X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Add expected value to verify the running result. Signed-off-by: Wei Ling --- conf/pvp_diff_qemu_version.cfg | 44 +++++++++++++++++++++++++++------- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/conf/pvp_diff_qemu_version.cfg b/conf/pvp_diff_qemu_version.cfg index 0353c246..1e09731b 100644 --- a/conf/pvp_diff_qemu_version.cfg +++ b/conf/pvp_diff_qemu_version.cfg @@ -1,20 +1,46 @@ -# Configuration sample: vhost_sample.cfg [vm0] cpu = - model=host,number=4,cpupin=4 5; + model=host,number=8,cpupin=20 21 22 23 24 25 26 27; mem = - size=4096,hugepage=yes; + size=8192,hugepage=yes; disk = - file=/home/img/vm.img; + file=/home/image/ubuntu2004.img; login = user=root,password=tester; vnc = displayNum=4; +net = + type=user,opt_vlan=2; + type=nic,opt_vlan=2; daemon = enable=yes; -# config all the path of qemu version you want to tested qemu = - path=path/qemu-2.5/bin/qemu-system-x86_64; - path=path/qemu-2.6/bin/qemu-system-x86_64; - path=path/qemu-2.7/bin/qemu-system-x86_64; - path=path/qemu-2.8/bin/qemu-system-x86_64; + path=/home/QEMU/qemu-5.2.0/bin/qemu-system-x86_64; + path=/home/QEMU/qemu-6.2.0/bin/qemu-system-x86_64; + path=/home/QEMU/qemu-7.0.0/bin/qemu-system-x86_64; + path=/home/QEMU/qemu-7.1.0/bin/qemu-system-x86_64; + path=/home/QEMU/qemu-7.2.0/bin/qemu-system-x86_64; +[suite] +update_expected = True +packet_sizes = [64, 128, 256, 512, 1024, 1280, 1518] +test_duration = 60 +accepted_tolerance = 0.05 +expected_throughput = { + 'test_perf_vhost_pvp_diffrent_qemu_version_virtio95_mergeable_path': { + 'qemu-5.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-6.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.0': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.1': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}}, + 'test_perf_vhost_pvp_diffrent_qemu_version_virtio10_mergeable_path': { + 'qemu-5.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-6.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.0': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.1': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}}, + 'test_perf_vhost_pvp_diffrent_qemu_version_virtio11_mergeable_path': { + 'qemu-5.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-6.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.0': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.1': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}, + 'qemu-7.2': {64: 0.0, 128: 0.0, 256: 0.0, 512: 0.0, 1024: 0.0, 1280: 0.0, 1518: 0.0}}}