From patchwork Thu May 26 09:19:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Weiyuan Li X-Patchwork-Id: 111911 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CCA67A0555; Thu, 26 May 2022 11:19:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C594440E64; Thu, 26 May 2022 11:19:49 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 73F0940DF7 for ; Thu, 26 May 2022 11:19:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653556787; x=1685092787; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=4tIZrx9HmLZzII4/Zj+TYJuhT/SxFBSf4E3ZcecSODc=; b=G7M9CM8x/1470cPLni0viT83mlpdqZ/i+nyYOLrkmnORKqKlHTyc+dk2 dxzfzhHMPvEVyrKRgbkf7Anuxk6k3UXMoZGNWWeA6bS8/+mif15xmtV4Z rAG9Q1grFcKwDERUuBZ3eY5m1J1hDtMruvTKyyvSZ1UNGNBuiySSC995v Ybo6dR9mEplNXBqB6+LMmEBwm43jZzuDcmygJOdb4JWrD2QFssxhOJ1ZQ lVzKFC4kBdPhXFz8e9YfdFIDZbsx5AbKQ9FsQv4sX/z+CwYDIDPxXZvaL 2XczS3Hy3EzfWJ58nG7D6gheUVZX12dRkcyU/xY4Rnb5o7MwbgvmvAcUN w==; X-IronPort-AV: E=McAfee;i="6400,9594,10358"; a="360483832" X-IronPort-AV: E=Sophos;i="5.91,252,1647327600"; d="scan'208";a="360483832" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2022 02:19:46 -0700 X-IronPort-AV: E=Sophos;i="5.91,252,1647327600"; d="scan'208";a="677384555" Received: from shwdenpg560.ccr.corp.intel.com ([10.253.109.11]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2022 02:19:45 -0700 From: "Li, WeiyuanX" To: dts@dpdk.org, lijuan.tu@intel.com Cc: Weiyuan Li Subject: [dts][PATCH V3 1/2] tests/meson_tests: add new testsuite for meson tests Date: Thu, 26 May 2022 17:19:35 +0800 Message-Id: <20220526091937.1276-1-weiyuanx.li@intel.com> X-Mailer: git-send-email 2.34.1.windows.1 MIME-Version: 1.0 X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org From: Weiyuan Li Add new testsuite for meson tests. Signed-off-by: Weiyuan Li --- v2: -fix format issue. v3: -add verification that the test result is empty tests/TestSuite_meson_tests.py | 233 +++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 tests/TestSuite_meson_tests.py diff --git a/tests/TestSuite_meson_tests.py b/tests/TestSuite_meson_tests.py new file mode 100644 index 00000000..4b1f7df2 --- /dev/null +++ b/tests/TestSuite_meson_tests.py @@ -0,0 +1,233 @@ +# BSD LICENSE +# +# Copyright(c) 2022 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +""" +DPDK Test Suite. + +meson autotest +""" +import os +import re +from mimetypes import init +from sre_constants import SUCCESS + +import framework.utils as utils +from framework.test_case import TestCase + + +class TestMesonTests(TestCase): + def set_up_all(self): + """ + Run at the start of each test suite. + """ + # icc compilation cost long long time. + self.cores = self.dut.get_core_list("all") + self.dut_ip = self.dut.get_ip_address() + self.timeout = 600 + # -t 2 + self.ratio = 6 + self.dut_pathlog = "fast-test.log" + self.execute_wait_time = self.ratio * self.timeout * 10 + # skip scope + self.SKIP_SCOPE = "" + # Test log storage directory + self.base_output = os.path.join( + os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "output" + ) + + def set_up(self): + """ + Run before each test case. + """ + self.meson_auto_test = {} + + def delete_exists_files(self): + out = self.tester.send_command( + f"ls -l {self.base_output}/{self.dut_pathlog}", "# " + ) + if self.dut_pathlog in out: + self.tester.send_command(f"rm -rf {self.base_output}/{self.dut_pathlog}") + + def find_text(self, res): + pattern = r"(DPDK:fast-tests|DPDK:perf-tests|DPDK:debug-tests|DPDK:extra-tests|DPDK:driver-tests) / (\w+)\s+(\w+)\s+" + regex = re.compile(pattern) + mo = regex.search(res) + + if mo is None: + return 0 + case_name = mo.group(2) + test_result = mo.group(3) + + self.meson_auto_test[case_name] = test_result + + def check_sub_case(self): + success = [] + skip = [] + timeout = [] + failed = [] + + self.verify( + bool(self.meson_auto_test), + f"Test failed the meson no test results were obtained.", + ) + + for k, v in self.meson_auto_test.items(): + if v == "FAIL" and k not in self.SKIP_SCOPE: + failed.append(k) + elif v == "OK": + success.append(k) + elif v == "SKIP": + skip.append(k) + elif v == "TIMEOUT": + timeout.append(k) + else: + failed.append(k) + + for item in skip: + self.logger.debug(utils.RED(f"Skip sub case:{item}")) + + for item in timeout: + self.logger.debug(utils.RED(f"TimeOut sub case:{item}")) + + if (len(failed) + len(timeout)) > 0: + sub_fail = "" + sub_timeout = "" + if len(failed) > 0: + sub_fail = "[" + ",".join(failed) + "] sub case failed. " + if len(timeout) > 0: + sub_timeout = "[" + ",".join(timeout) + "] sub case time out. " + self.verify(False, f"Test failed. {sub_fail}{sub_timeout}") + + def check_scp_file_valid_between_dut(self): + out = self.tester.send_command( + f"scp root@{self.dut_ip}:/root/{self.dut_pathlog} {self.base_output}", + timeout=15, + ) + if "Are you sure you want to continue connecting" in out: + out = self.tester.send_command("yes", timeout=20) + for item in range(30): + if "password" in out: + self.tester.send_command(self.dut.get_password(), timeout=20) + break + + out = self.tester.send_command( + f"ls -l {self.base_output}/{self.dut_pathlog}", "# " + ) + self.verify( + "No such file or directory" not in out, "No test result log was obtained!" + ) + + def insmod_kni(self): + out = self.dut.send_expect("lsmod | grep rte_kni", "# ") + + if "rte_kni" in out: + self.dut.send_expect("rmmod rte_kni.ko", "# ") + + out = self.dut.send_expect( + "insmod ./%s/kmod/rte_kni.ko lo_mode=lo_mode_fifo" % (self.target), "# " + ) + self.verify("Error" not in out, "Error loading KNI module: " + out) + + def check_meson_test_result(self, res=None): + if not os.path.exists(f"{self.base_output}/{self.dut_pathlog}"): + self.verify(False, "meson tests log file is not exists!!") + if res is None: + with open(f"{self.base_output}/{self.dut_pathlog}", "r") as file_obj: + for files in file_obj: + self.find_text(files) + else: + self.find_text(res) + self.check_sub_case() + + def test_fasts(self): + # init file name + self.dut_pathlog = "fast-test.log" + self.delete_exists_files() + self.insmod_kni() + cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:fast-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}' + out = self.dut.send_expect(cmds, "# ", self.execute_wait_time) + self.logger.info(out) + self.check_scp_file_valid_between_dut() + self.check_meson_test_result() + + def test_driver(self): + # init file name + self.dut_pathlog = "driver-test.log" + self.delete_exists_files() + cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:driver-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}' + out = self.dut.send_expect(cmds, "# ", self.execute_wait_time) + self.logger.info(out) + self.check_scp_file_valid_between_dut() + self.check_meson_test_result() + + def test_debug(self): + self.logger.warning( + "Debug-tests all case failed commands are not registered, there is a bug https://bugs.dpdk.org/show_bug.cgi?id=1002" + ) + + def test_extra(self): + self.dut_pathlog = "extra-test.log" + # delete exists files + self.delete_exists_files() + cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:extra-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}' + out = self.dut.send_expect(cmds, "# ", self.execute_wait_time) + self.logger.info(out) + self.check_scp_file_valid_between_dut() + self.check_meson_test_result() + self.logger.warning( + "Extra-tests are know issues which are recorded in DPDK commit and meson.build (detail see test plan)" + ) + + def test_perf(self): + # init file name + self.dut_pathlog = "perf-test.log" + # delete exists files + self.delete_exists_files() + cmds = f'meson test -C x86_64-native-linuxapp-gcc/ --suite DPDK:perf-tests -t {self.ratio} --test-args="-c 0xff" |tee /root/{self.dut_pathlog}' + out = self.dut.send_expect(cmds, "# ", self.execute_wait_time) + self.logger.info(out) + self.check_scp_file_valid_between_dut() + self.check_meson_test_result() + + def tear_down(self): + """ + Run after each test case. + """ + out = self.dut.send_expect("lsmod | grep rte_kni", "# ") + + if "rte_kni" in out: + self.dut.send_expect("rmmod rte_kni.ko", "# ") + + def tear_down_all(self): + """ + Run after each test suite. + """ + pass From patchwork Thu May 26 09:19:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Weiyuan Li X-Patchwork-Id: 111912 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ED85BA0555; Thu, 26 May 2022 11:19:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E785E4161A; Thu, 26 May 2022 11:19:51 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 89F4640DF7 for ; Thu, 26 May 2022 11:19:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653556788; x=1685092788; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I3VUe3XPJYaal9C0/esKea2GRxMuPeKP48TOAFQGg9s=; b=QKN+BdfZJ/2i5ND05n2VhNmx5hfDMk6f7o1CfvARA/jUSYSvLLIYPc7u BG7qo0Cd4lUVVM/zpvKIG/h5yrhc64j/LorBQ/hfr16eWZSP+IgFqbQ8u GL7zc3sM134Ce/KYsJizPUj1WacjiWluZjDnbWvDxpL7GIMlPlmSoglkO aKe/rJZ1RkqErblkP07jg+rNsni05apIKq3BKto5QTZP2dfy+INAWQ5Ng dmz5Pp4fJ+cOzBpqre4f9b8nEOtz0Otv2fehVAbPUrnITRVKFOtlYvQWc PXoBah5LBoV+agE5GTzsMZMcsHRoGe28blKicgV9XeHBwoXRrNEI87x/8 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10358"; a="360483838" X-IronPort-AV: E=Sophos;i="5.91,252,1647327600"; d="scan'208";a="360483838" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2022 02:19:48 -0700 X-IronPort-AV: E=Sophos;i="5.91,252,1647327600"; d="scan'208";a="677384566" Received: from shwdenpg560.ccr.corp.intel.com ([10.253.109.11]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2022 02:19:46 -0700 From: "Li, WeiyuanX" To: dts@dpdk.org, lijuan.tu@intel.com Cc: Weiyuan Li Subject: [dts][PATCH V3 2/2] test_plans/meson_tests: add test plan for meson tests Date: Thu, 26 May 2022 17:19:36 +0800 Message-Id: <20220526091937.1276-2-weiyuanx.li@intel.com> X-Mailer: git-send-email 2.34.1.windows.1 In-Reply-To: <20220526091937.1276-1-weiyuanx.li@intel.com> References: <20220526091937.1276-1-weiyuanx.li@intel.com> MIME-Version: 1.0 X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org From: Weiyuan Li Add new test plan for meson tests. Signed-off-by: Weiyuan Li Tested-by: Lingli Chen Reviewed-by: Lijuan Tu --- v2: -fix format issue. v3: -add verification that the test result is empty test_plans/index.rst | 1 + test_plans/meson_tests_test_plan.rst | 185 +++++++++++++++++++++++++++ 2 files changed, 186 insertions(+) create mode 100644 test_plans/meson_tests_test_plan.rst diff --git a/test_plans/index.rst b/test_plans/index.rst index 6978a2ef..45a23e7e 100644 --- a/test_plans/index.rst +++ b/test_plans/index.rst @@ -281,6 +281,7 @@ The following are the test plans for the DPDK DTS automated test system. unit_tests_ringpmd_test_plan unit_tests_ring_test_plan unit_tests_timer_test_plan + meson_tests_test_plan cmdline_test_plan hello_world_test_plan diff --git a/test_plans/meson_tests_test_plan.rst b/test_plans/meson_tests_test_plan.rst new file mode 100644 index 00000000..e199ec81 --- /dev/null +++ b/test_plans/meson_tests_test_plan.rst @@ -0,0 +1,185 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) <2022>, Intel Corporation + +=========== +Meson tests +=========== + +Description +=========== +This test plan describes how to run unit tests with meson. +Test tests have been classified into five different groups. + +- Fast tests. +- Performance tests. +- Driver tests. +- Extra tests. +- Debug tests. + +For more details, please refer to `Running DPDK Unit Tests with Meson `_. + +Prerequisites +============= + +Required Libraries +------------------ +* crypto relative cases require dependencies `libIPSec_MB` and `libcrypto`:: + + cryptodev_aesni_mb_autotest + cryptodev_aesni_gcm_autotest + cryptodev_null_autotest + cryptodev_openssl_autotest + cryptodev_openssl_asym_autotest + cryptodev_qat_autotest + cryptodev_sw_kasumi_autotest + cryptodev_sw_snow3g_autotest + cryptodev_sw_zuc_autotest + cryptodev_scheduler_autotest + + .. note:: + + For more details, please refer to `Crypto Device Drivers `_ + +* The following cases require dependencies `libpcap`:: + + bpf_convert_autotest + eal_flags_mem_autotest + flow_classify_autotest + efd_autotest + efd_perf_autotest + +General set up +-------------- + +* Compile DPDK:: + + # CC=gcc meson --werror -Denable_kmods=True -Dlibdir=lib --default-library=static + # ninja -C -j 110 + +* Load rte_kni driver:: + + # lsmod | grep rte_kni + # rmmod rte_kni.ko + # insmod .//kmod/rte_kni.ko lo_mode=lo_mode_fifo + + .. note:: + + Test case fast-tests/kni_autotest requires to load kni.ko. + +* Get the pci device id and interface of DUT and tester. + For example, 0000:18:00.0 and 0000:18:00.1 is pci device id, + ens785f0 and ens785f1 is interface:: + + # ./usertools/dpdk-devbind.py -s + ... + 0000:18:00.0 'Device 159b' if=ens785f0 drv=ice unused=vfio-pci + 0000:18:00.1 'Device 159b' if=ens785f1 drv=ice unused=vfio-pci + +* Bind the DUT port to dpdk:: + + # ./usertools/dpdk-devbind.py -b vfio-pci 0000:18:00.0 0000:18:00.1 + +.. note:: + + kni_autotest requires at least one ethernet port, but other can be none. + +Test case +========= + +Test case 1: test_fasts +------------------------------- +This section explains how to run the meson tests for fast-tests. + +test steps +~~~~~~~~~~ +1. Run test on DUT:: + + meson test -C x86_64-native-linuxapp-gcc --suite DPDK:fast-tests -t 1 --test-args="-c 0xff" + + * `-t` is used to specify the timeout of test case, unit is second. + * `--test-args option` is used to specify test specific parameters, + distributor_autotest and distributor_perf_autotest require core number less than 64. + +2. Open and view the results on DUT:: + + # cat ./x86_64-native-linuxapp-gcc/meson-logs/testlog.txt + +3. Check the result output log. All case results are "OK", for example:: + + 1/99 DPDK:fast-tests / acl_autotest OK 3.43s + 2/99 DPDK:fast-tests / atomic_autotest OK 6.21s + 3/99 DPDK:fast-tests / bitmap_autotest OK 1.18s + 4/99 DPDK:fast-tests / bpf_autotest OK 1.17s + 5/99 DPDK:fast-tests / bpf_convert_autotest OK 1.19s + +Test case 2: test_driver +------------------------------- +This section explains how to run the meson tests for driver-tests. + +test steps +~~~~~~~~~~ +1. Run test on DUT:: + + meson test -C x86_64-native-linuxapp-gcc --suite DPDK:driver-tests -t 1 --test-args="-c 0xff" + +2. Following test case 1 step 2 and step 3. + +Test case 3: test_debug +------------------------------- +This section explains how to run the meson tests for debug-tests. + +test steps +~~~~~~~~~~ +1. Run test on DUT:: + + meson test -C x86_64-native-linuxapp-gcc --suite DPDK:debug-tests -t 1 --test-args="-c 0xff" + +2.Following test case 1 step 2 and step 3. + +.. warning:: + + `The bug _` impacts the following cases:: + + DPDK:debug-tests/dump_struct_sizes + DPDK:debug-tests/dump_mempool + DPDK:debug-tests/dump_malloc_stats + DPDK:debug-tests/dump_devargs + DPDK:debug-tests/dump_log_types + DPDK:debug-tests/dump_ring + DPDK:debug-tests/dump_physmem + DPDK:debug-tests/dump_memzone + +Test case 4: test_extra +------------------------------- +This section explains how to run the meson tests for extra-tests. + +test steps +~~~~~~~~~~ +1. Run test on DUT:: + + meson test -C x86_64-native-linuxapp-gcc --suite DPDK:extra-tests -t 1 --test-args="-c 0xff" + +2. Following test case 1 step 2 and step 3. + +.. warning:: + + Extra-tests are know issues which are recorded in app/test/meson.build:: + + #Tests known to have issues or which don't belong in other tests lists. + extra_test_names = [ + 'alarm_autotest', # ee00af60170b ("test: remove strict timing requirements some tests") + 'cycles_autotest', # ee00af60170b ("test: remove strict timing requirements some tests") + 'delay_us_sleep_autotest', # ee00af60170b ("test: remove strict timing requirements some tests") + 'red_autotest', # https://bugs.dpdk.org/show_bug.cgi?id=826] + +Test case 5: test_perf +------------------------------- +This section explains how to run the meson tests for perf-tests. + +test steps +~~~~~~~~~~ +1. Run test on DUT:: + + meson test -C x86_64-native-linuxapp-gcc --suite DPDK:perf-tests -t 1 --test-args="-c 0xff" + +2. Following test case 1 step 2 and step 3. \ No newline at end of file