From patchwork Mon Dec 4 10:24:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 134796 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 641EF4366A; Mon, 4 Dec 2023 11:25:33 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4AF41410F1; Mon, 4 Dec 2023 11:24:43 +0100 (CET) Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mails.dpdk.org (Postfix) with ESMTP id 7D430410D5 for ; Mon, 4 Dec 2023 11:24:39 +0100 (CET) Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-40b4746ae3bso38083475e9.0 for ; Mon, 04 Dec 2023 02:24:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1701685479; x=1702290279; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3J4tuvvC25Dr3XVRHIMGMh8HcTBlhxLQyKucmyiGS50=; b=qgytLXMe89qHjBhy+ZW9AEtdo0GbP5h7qlfBh/lvefymItO5iU7Yk84+ua7Xi9Svhz ncnWhTlO0RrygAe7kQWzrox4HEsxcdEuXayikOg3kpi4H39GP4BAY3S8559qyw0ARmQX hnYaasSxOtjcrialbRBDyuawfoL/blIFpDmB24t+I455mloB3SM7J4CC3rEuTkDxYjFO Al8Bl9Er+qUmpFMP4xpH+AHVaoHd6vmlhUDvtTSD7EYA9WHghLGjUUusugxjhePnTp9h tCoxZ10WtXjVapc8/l2kyKMRrkk4xDS0K9bWJHm2NAPF/97byz+jh5HzxfwPQW+A2nB6 TcBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701685479; x=1702290279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3J4tuvvC25Dr3XVRHIMGMh8HcTBlhxLQyKucmyiGS50=; b=G3Rg+ucnRVNeMa6LjYh1wSXB2Vs3nAfcmEvhPSHSGF63/fZZnn7IqlHZFLDlhUSPCS /awKA34OLR66qUicYVTT/tRNtWmUCIdEYaUNoxWwdCAcyGjrN6sK6J08SkCtYPhmZhW5 x1ublCCMCGoyutGFdsxYASW7E6ObpBnCOWbtZ+3lgswHXOhcfdJmpEQ0lxjKWlR1oxW3 kS2chvIuACU22ofyLKvTaYcLhEQEGUjhTzb/sd/vkRmwfNeBmszxFfv5WscbRm92tiG8 Zx9f+6xqpuJhAWq8XMFTSw+CjgoInqGNJsWW63ib9rebc3nXyjnXBGceLOcDDkAOukaA RvYg== X-Gm-Message-State: AOJu0YxjJr1GH2MHpFBbskDfbQefJZISSU73PUEu7JMzKAvbpqCiAhDN 0EnkuSN/nMwal/kPda7Ifvvh0A== X-Google-Smtp-Source: AGHT+IH+LVUCD2ax8txg+fztEj0rj5FVivs4Tse9mkAvZEB0gNNb7gpRd3EFjle7PP2CFvgQ4dK5Gw== X-Received: by 2002:a7b:cbc9:0:b0:40c:a19:8d9c with SMTP id n9-20020a7bcbc9000000b0040c0a198d9cmr1021409wmi.162.1701685479088; Mon, 04 Dec 2023 02:24:39 -0800 (PST) Received: from jlinkes-PT-Latitude-5530.pantheon.local ([81.89.53.154]) by smtp.gmail.com with ESMTPSA id m28-20020a05600c3b1c00b0040b2b38a1fasm14255415wms.4.2023.12.04.02.24.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 02:24:38 -0800 (PST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v9 08/21] dts: test suite docstring update Date: Mon, 4 Dec 2023 11:24:16 +0100 Message-Id: <20231204102429.106709-9-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231204102429.106709-1-juraj.linkes@pantheon.tech> References: <20231123151344.162812-1-juraj.linkes@pantheon.tech> <20231204102429.106709-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Format according to the Google format and PEP257, with slight deviations. Signed-off-by: Juraj Linkeš --- dts/framework/test_suite.py | 231 +++++++++++++++++++++++++++--------- 1 file changed, 175 insertions(+), 56 deletions(-) diff --git a/dts/framework/test_suite.py b/dts/framework/test_suite.py index f9e66e814a..dfb391ffbd 100644 --- a/dts/framework/test_suite.py +++ b/dts/framework/test_suite.py @@ -2,8 +2,19 @@ # Copyright(c) 2010-2014 Intel Corporation # Copyright(c) 2023 PANTHEON.tech s.r.o. -""" -Base class for creating DTS test cases. +"""Features common to all test suites. + +The module defines the :class:`TestSuite` class which doesn't contain any test cases, and as such +must be extended by subclasses which add test cases. The :class:`TestSuite` contains the basics +needed by subclasses: + + * Test suite and test case execution flow, + * Testbed (SUT, TG) configuration, + * Packet sending and verification, + * Test case verification. + +The module also defines a function, :func:`get_test_suites`, +for gathering test suites from a Python module. """ import importlib @@ -11,7 +22,7 @@ import re from ipaddress import IPv4Interface, IPv6Interface, ip_interface from types import MethodType -from typing import Any, Union +from typing import Any, ClassVar, Union from scapy.layers.inet import IP # type: ignore[import] from scapy.layers.l2 import Ether # type: ignore[import] @@ -31,25 +42,44 @@ class TestSuite(object): - """ - The base TestSuite class provides methods for handling basic flow of a test suite: - * test case filtering and collection - * test suite setup/cleanup - * test setup/cleanup - * test case execution - * error handling and results storage - Test cases are implemented by derived classes. Test cases are all methods - starting with test_, further divided into performance test cases - (starting with test_perf_) and functional test cases (all other test cases). - By default, all test cases will be executed. A list of testcase str names - may be specified in conf.yaml or on the command line - to filter which test cases to run. - The methods named [set_up|tear_down]_[suite|test_case] should be overridden - in derived classes if the appropriate suite/test case fixtures are needed. + """The base class with methods for handling the basic flow of a test suite. + + * Test case filtering and collection, + * Test suite setup/cleanup, + * Test setup/cleanup, + * Test case execution, + * Error handling and results storage. + + Test cases are implemented by subclasses. Test cases are all methods starting with ``test_``, + further divided into performance test cases (starting with ``test_perf_``) + and functional test cases (all other test cases). + + By default, all test cases will be executed. A list of testcase names may be specified + in the YAML test run configuration file and in the :option:`--test-cases` command line argument + or in the :envvar:`DTS_TESTCASES` environment variable to filter which test cases to run. + The union of both lists will be used. Any unknown test cases from the latter lists + will be silently ignored. + + If the :option:`--re-run` command line argument or the :envvar:`DTS_RERUN` environment variable + is set, in case of a test case failure, the test case will be executed again until it passes + or it fails that many times in addition of the first failure. + + The methods named ``[set_up|tear_down]_[suite|test_case]`` should be overridden in subclasses + if the appropriate test suite/test case fixtures are needed. + + The test suite is aware of the testbed (the SUT and TG) it's running on. From this, it can + properly choose the IP addresses and other configuration that must be tailored to the testbed. + + Attributes: + sut_node: The SUT node where the test suite is running. + tg_node: The TG node where the test suite is running. """ sut_node: SutNode - is_blocking = False + tg_node: TGNode + #: Whether the test suite is blocking. A failure of a blocking test suite + #: will block the execution of all subsequent test suites in the current build target. + is_blocking: ClassVar[bool] = False _logger: DTSLOG _test_cases_to_run: list[str] _func: bool @@ -72,6 +102,20 @@ def __init__( func: bool, build_target_result: BuildTargetResult, ): + """Initialize the test suite testbed information and basic configuration. + + Process what test cases to run, create the associated + :class:`~.test_result.TestSuiteResult`, find links between ports + and set up default IP addresses to be used when configuring them. + + Args: + sut_node: The SUT node where the test suite will run. + tg_node: The TG node where the test suite will run. + test_cases: The list of test cases to execute. + If empty, all test cases will be executed. + func: Whether to run functional tests. + build_target_result: The build target result this test suite is run in. + """ self.sut_node = sut_node self.tg_node = tg_node self._logger = getLogger(self.__class__.__name__) @@ -95,6 +139,7 @@ def __init__( self._tg_ip_address_ingress = ip_interface("192.168.101.3/24") def _process_links(self) -> None: + """Construct links between SUT and TG ports.""" for sut_port in self.sut_node.ports: for tg_port in self.tg_node.ports: if (sut_port.identifier, sut_port.peer) == ( @@ -104,27 +149,42 @@ def _process_links(self) -> None: self._port_links.append(PortLink(sut_port=sut_port, tg_port=tg_port)) def set_up_suite(self) -> None: - """ - Set up test fixtures common to all test cases; this is done before - any test case is run. + """Set up test fixtures common to all test cases. + + This is done before any test case has been run. """ def tear_down_suite(self) -> None: - """ - Tear down the previously created test fixtures common to all test cases. + """Tear down the previously created test fixtures common to all test cases. + + This is done after all test have been run. """ def set_up_test_case(self) -> None: - """ - Set up test fixtures before each test case. + """Set up test fixtures before each test case. + + This is done before *each* test case. """ def tear_down_test_case(self) -> None: - """ - Tear down the previously created test fixtures after each test case. + """Tear down the previously created test fixtures after each test case. + + This is done after *each* test case. """ def configure_testbed_ipv4(self, restore: bool = False) -> None: + """Configure IPv4 addresses on all testbed ports. + + The configured ports are: + + * SUT ingress port, + * SUT egress port, + * TG ingress port, + * TG egress port. + + Args: + restore: If :data:`True`, will remove the configuration instead. + """ delete = True if restore else False enable = False if restore else True self._configure_ipv4_forwarding(enable) @@ -149,11 +209,17 @@ def _configure_ipv4_forwarding(self, enable: bool) -> None: self.sut_node.configure_ipv4_forwarding(enable) def send_packet_and_capture(self, packet: Packet, duration: float = 1) -> list[Packet]: - """ - Send a packet through the appropriate interface and - receive on the appropriate interface. - Modify the packet with l3/l2 addresses corresponding - to the testbed and desired traffic. + """Send and receive `packet` using the associated TG. + + Send `packet` through the appropriate interface and receive on the appropriate interface. + Modify the packet with l3/l2 addresses corresponding to the testbed and desired traffic. + + Args: + packet: The packet to send. + duration: Capture traffic for this amount of time after sending `packet`. + + Returns: + A list of received packets. """ packet = self._adjust_addresses(packet) return self.tg_node.send_packet_and_capture( @@ -161,13 +227,26 @@ def send_packet_and_capture(self, packet: Packet, duration: float = 1) -> list[P ) def get_expected_packet(self, packet: Packet) -> Packet: + """Inject the proper L2/L3 addresses into `packet`. + + Args: + packet: The packet to modify. + + Returns: + `packet` with injected L2/L3 addresses. + """ return self._adjust_addresses(packet, expected=True) def _adjust_addresses(self, packet: Packet, expected: bool = False) -> Packet: - """ + """L2 and L3 address additions in both directions. + Assumptions: - Two links between SUT and TG, one link is TG -> SUT, - the other SUT -> TG. + Two links between SUT and TG, one link is TG -> SUT, the other SUT -> TG. + + Args: + packet: The packet to modify. + expected: If :data:`True`, the direction is SUT -> TG, + otherwise the direction is TG -> SUT. """ if expected: # The packet enters the TG from SUT @@ -193,6 +272,19 @@ def _adjust_addresses(self, packet: Packet, expected: bool = False) -> Packet: return Ether(packet.build()) def verify(self, condition: bool, failure_description: str) -> None: + """Verify `condition` and handle failures. + + When `condition` is :data:`False`, raise an exception and log the last 10 commands + executed on both the SUT and TG. + + Args: + condition: The condition to check. + failure_description: A short description of the failure + that will be stored in the raised exception. + + Raises: + TestCaseVerifyError: `condition` is :data:`False`. + """ if not condition: self._fail_test_case_verify(failure_description) @@ -206,6 +298,19 @@ def _fail_test_case_verify(self, failure_description: str) -> None: raise TestCaseVerifyError(failure_description) def verify_packets(self, expected_packet: Packet, received_packets: list[Packet]) -> None: + """Verify that `expected_packet` has been received. + + Go through `received_packets` and check that `expected_packet` is among them. + If not, raise an exception and log the last 10 commands + executed on both the SUT and TG. + + Args: + expected_packet: The packet we're expecting to receive. + received_packets: The packets where we're looking for `expected_packet`. + + Raises: + TestCaseVerifyError: `expected_packet` is not among `received_packets`. + """ for received_packet in received_packets: if self._compare_packets(expected_packet, received_packet): break @@ -280,10 +385,14 @@ def _verify_l3_packet(self, received_packet: IP, expected_packet: IP) -> bool: return True def run(self) -> None: - """ - Setup, execute and teardown the whole suite. - Suite execution consists of running all test cases scheduled to be executed. - A test cast run consists of setup, execution and teardown of said test case. + """Set up, execute and tear down the whole suite. + + Test suite execution consists of running all test cases scheduled to be executed. + A test case run consists of setup, execution and teardown of said test case. + + Record the setup and the teardown and handle failures. + + The list of scheduled test cases is constructed when creating the :class:`TestSuite` object. """ test_suite_name = self.__class__.__name__ @@ -315,9 +424,7 @@ def run(self) -> None: raise BlockingTestSuiteError(test_suite_name) def _execute_test_suite(self) -> None: - """ - Execute all test cases scheduled to be executed in this suite. - """ + """Execute all test cases scheduled to be executed in this suite.""" if self._func: for test_case_method in self._get_functional_test_cases(): test_case_name = test_case_method.__name__ @@ -334,14 +441,18 @@ def _execute_test_suite(self) -> None: self._run_test_case(test_case_method, test_case_result) def _get_functional_test_cases(self) -> list[MethodType]: - """ - Get all functional test cases. + """Get all functional test cases defined in this TestSuite. + + Returns: + The list of functional test cases of this TestSuite. """ return self._get_test_cases(r"test_(?!perf_)") def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: - """ - Return a list of test cases matching test_case_regex. + """Return a list of test cases matching test_case_regex. + + Returns: + The list of test cases matching test_case_regex of this TestSuite. """ self._logger.debug(f"Searching for test cases in {self.__class__.__name__}.") filtered_test_cases = [] @@ -353,9 +464,7 @@ def _get_test_cases(self, test_case_regex: str) -> list[MethodType]: return filtered_test_cases def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool: - """ - Check whether the test case should be executed. - """ + """Check whether the test case should be scheduled to be executed.""" match = bool(re.match(test_case_regex, test_case_name)) if self._test_cases_to_run: return match and test_case_name in self._test_cases_to_run @@ -365,9 +474,9 @@ def _should_be_executed(self, test_case_name: str, test_case_regex: str) -> bool def _run_test_case( self, test_case_method: MethodType, test_case_result: TestCaseResult ) -> None: - """ - Setup, execute and teardown a test case in this suite. - Exceptions are caught and recorded in logs and results. + """Setup, execute and teardown a test case in this suite. + + Record the result of the setup and the teardown and handle failures. """ test_case_name = test_case_method.__name__ @@ -402,9 +511,7 @@ def _run_test_case( def _execute_test_case( self, test_case_method: MethodType, test_case_result: TestCaseResult ) -> None: - """ - Execute one test case and handle failures. - """ + """Execute one test case, record the result and handle failures.""" test_case_name = test_case_method.__name__ try: self._logger.info(f"Starting test case execution: {test_case_name}") @@ -425,6 +532,18 @@ def _execute_test_case( def get_test_suites(testsuite_module_path: str) -> list[type[TestSuite]]: + r"""Find all :class:`TestSuite`\s in a Python module. + + Args: + testsuite_module_path: The path to the Python module. + + Returns: + The list of :class:`TestSuite`\s found within the Python module. + + Raises: + ConfigurationError: The test suite module was not found. + """ + def is_test_suite(object: Any) -> bool: try: if issubclass(object, TestSuite) and object is not TestSuite: