From patchwork Thu Nov 23 15:13:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 134573 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 78B07433AC; Thu, 23 Nov 2023 16:14:56 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1997F42FE3; Thu, 23 Nov 2023 16:14:05 +0100 (CET) Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) by mails.dpdk.org (Postfix) with ESMTP id 0B99F42FCD for ; Thu, 23 Nov 2023 16:13:56 +0100 (CET) Received: by mail-wr1-f45.google.com with SMTP id ffacd0b85a97d-332e3ad436cso380935f8f.3 for ; Thu, 23 Nov 2023 07:13:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1700752436; x=1701357236; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6i4bo/AJKOFxXNVyyjE//kxyMsAoBQPx3oSImSMe8yo=; b=s8svDtUZEV0ucTE+CbduWfPDaIaj15gilnOJEJyWJgmql50Z+rBCcSMoZJ2LAUarlU HxvZl/3rRUqhvquXkA+nUYWBOZHVdTwyKFwyzJllWNyK3MRJu0o6PA6xIXpkepKlU/bf SNTZNxXY1BdYE/4YhPlijNY4vtDmWhUw4MpoGtN6sxTItdmUKPEqNRHkbkj5Jvdn7zU8 yjqZUeCG+xfZObQs4aQ+kMOdykFaroqLoE2Alup6RDa6T+P7i3sUMbrf2awpBbbkrvWe jX4k+zkBf5UFG/5PLVJQnAhXIKBcE2AIq5/nN3nDFtM0O1MrjKKW+Rn8Znve5pr8Sgpq GT5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700752436; x=1701357236; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6i4bo/AJKOFxXNVyyjE//kxyMsAoBQPx3oSImSMe8yo=; b=F4lQjbRzsHbP0mbQhXOs2Rc8daB8c0eD2BWzusYQYtMhPcNWWQmFazABGl+1FiWyI+ WUJre0VGi3k4n7/DIch1awFKo+YBved/ZhBNl6WJQSg8Bk1qjYG1x2MADe7JNbnP56jZ AVQOiovmJQ3eShe3ggPtivdLJhl3JZu1A1c4fXsqfrFjVV/QAK+x7thcoYuPm9AndGiN xaqsm2Tgo66saG3QV0dsx6YBWL3aFw/f/jokwBDxqmAY7NNpmRqDizk5ETbM2u5TnTbn buAdoAsuWixqYK5PXG9Kz4diOvkJyn69vYwj3jbPk4npsZSw4eOfhDH/KUQ5kDSkuwEK gQpg== X-Gm-Message-State: AOJu0YzmH8fdFx2gpneRoC9eZ9MX9LS2eC2ZCmLlcb/Cy75V3jLiSdEj 6C0ZCDMcBWsQtYmtoPVfGJUZp4Zhk4t8XHLajh646g== X-Google-Smtp-Source: AGHT+IHKY56vFtNyn6yV0X9Q3jvJW6xVjA2rtYnCwlNbu3EaNpW2NsA+V02MzqlKxtvg/VAFJJkNjA== X-Received: by 2002:a05:6000:b8a:b0:332:e62e:f0ba with SMTP id dl10-20020a0560000b8a00b00332e62ef0bamr803746wrb.18.1700752435742; Thu, 23 Nov 2023 07:13:55 -0800 (PST) Received: from jlinkes-PT-Latitude-5530.. ([84.245.121.10]) by smtp.gmail.com with ESMTPSA id q4-20020adfea04000000b003296b488961sm1870143wrm.31.2023.11.23.07.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Nov 2023 07:13:55 -0800 (PST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v8 07/21] dts: dts runner and main docstring update Date: Thu, 23 Nov 2023 16:13:30 +0100 Message-Id: <20231123151344.162812-8-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231123151344.162812-1-juraj.linkes@pantheon.tech> References: <20231115130959.39420-1-juraj.linkes@pantheon.tech> <20231123151344.162812-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Format according to the Google format and PEP257, with slight deviations. Signed-off-by: Juraj Linkeš --- dts/framework/dts.py | 131 ++++++++++++++++++++++++++++++++++++------- dts/main.py | 10 ++-- 2 files changed, 116 insertions(+), 25 deletions(-) diff --git a/dts/framework/dts.py b/dts/framework/dts.py index 356368ef10..e16d4578a0 100644 --- a/dts/framework/dts.py +++ b/dts/framework/dts.py @@ -3,6 +3,33 @@ # Copyright(c) 2022-2023 PANTHEON.tech s.r.o. # Copyright(c) 2022-2023 University of New Hampshire +r"""Test suite runner module. + +A DTS run is split into stages: + + #. Execution stage, + #. Build target stage, + #. Test suite stage, + #. Test case stage. + +The module is responsible for running tests on testbeds defined in the test run configuration. +Each setup or teardown of each stage is recorded in a :class:`~.test_result.DTSResult` or +one of its subclasses. The test case results are also recorded. + +If an error occurs, the current stage is aborted, the error is recorded and the run continues in +the next iteration of the same stage. The return code is the highest `severity` of all +:class:`~.exception.DTSError`\s. + +Example: + An error occurs in a build target setup. The current build target is aborted and the run + continues with the next build target. If the errored build target was the last one in the given + execution, the next execution begins. + +Attributes: + dts_logger: The logger instance used in this module. + result: The top level result used in the module. +""" + import sys from .config import ( @@ -23,9 +50,38 @@ def run_all() -> None: - """ - The main process of DTS. Runs all build targets in all executions from the main - config file. + """Run all build targets in all executions from the test run configuration. + + Before running test suites, executions and build targets are first set up. + The executions and build targets defined in the test run configuration are iterated over. + The executions define which tests to run and where to run them and build targets define + the DPDK build setup. + + The tests suites are set up for each execution/build target tuple and each scheduled + test case within the test suite is set up, executed and torn down. After all test cases + have been executed, the test suite is torn down and the next build target will be tested. + + All the nested steps look like this: + + #. Execution setup + + #. Build target setup + + #. Test suite setup + + #. Test case setup + #. Test case logic + #. Test case teardown + + #. Test suite teardown + + #. Build target teardown + + #. Execution teardown + + The test cases are filtered according to the specification in the test run configuration and + the :option:`--test-cases` command line argument or + the :envvar:`DTS_TESTCASES` environment variable. """ global dts_logger global result @@ -87,6 +143,8 @@ def run_all() -> None: def _check_dts_python_version() -> None: + """Check the required Python version - v3.10.""" + def RED(text: str) -> str: return f"\u001B[31;1m{str(text)}\u001B[0m" @@ -109,9 +167,16 @@ def _run_execution( execution: ExecutionConfiguration, result: DTSResult, ) -> None: - """ - Run the given execution. This involves running the execution setup as well as - running all build targets in the given execution. + """Run the given execution. + + This involves running the execution setup as well as running all build targets + in the given execution. After that, execution teardown is run. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: An execution's test run configuration. + result: The top level result object. """ dts_logger.info(f"Running execution with SUT '{execution.system_under_test_node.name}'.") execution_result = result.add_execution(sut_node.config) @@ -144,8 +209,18 @@ def _run_build_target( execution: ExecutionConfiguration, execution_result: ExecutionResult, ) -> None: - """ - Run the given build target. + """Run the given build target. + + This involves running the build target setup as well as running all test suites + in the given execution the build target is defined in. + After that, build target teardown is run. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + build_target: A build target's test run configuration. + execution: The build target's execution's test run configuration. + execution_result: The execution level result object associated with the execution. """ dts_logger.info(f"Running build target '{build_target.name}'.") build_target_result = execution_result.add_build_target(build_target) @@ -177,10 +252,20 @@ def _run_all_suites( execution: ExecutionConfiguration, build_target_result: BuildTargetResult, ) -> None: - """ - Use the given build_target to run execution's test suites - with possibly only a subset of test cases. - If no subset is specified, run all test cases. + """Run the execution's (possibly a subset) test suites using the current build target. + + The function assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. + + If a blocking test suite (such as the smoke test suite) fails, the rest of the test suites + in the current build target won't be executed. + + Args: + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated with the current build target. + build_target_result: The build target level result object associated + with the current build target. """ end_build_target = False if not execution.skip_smoke_tests: @@ -206,16 +291,22 @@ def _run_single_suite( build_target_result: BuildTargetResult, test_suite_config: TestSuiteConfig, ) -> None: - """Runs a single test suite. + """Run all test suite in a single test suite module. + + The function assumes the build target we're testing has already been built on the SUT node. + The current build target thus corresponds to the current DPDK build present on the SUT node. Args: - sut_node: Node to run tests on. - execution: Execution the test case belongs to. - build_target_result: Build target configuration test case is run on - test_suite_config: Test suite configuration + sut_node: The execution's SUT node. + tg_node: The execution's TG node. + execution: The execution's test run configuration associated with the current build target. + build_target_result: The build target level result object associated + with the current build target. + test_suite_config: Test suite test run configuration specifying the test suite module + and possibly a subset of test cases of test suites in that module. Raises: - BlockingTestSuiteError: If a test suite that was marked as blocking fails. + BlockingTestSuiteError: If a blocking test suite fails. """ try: full_suite_path = f"tests.TestSuite_{test_suite_config.test_suite}" @@ -239,9 +330,7 @@ def _run_single_suite( def _exit_dts() -> None: - """ - Process all errors and exit with the proper exit code. - """ + """Process all errors and exit with the proper exit code.""" result.process() if dts_logger: diff --git a/dts/main.py b/dts/main.py index 5d4714b0c3..b856ba86be 100755 --- a/dts/main.py +++ b/dts/main.py @@ -1,12 +1,10 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2010-2014 Intel Corporation -# Copyright(c) 2022 PANTHEON.tech s.r.o. +# Copyright(c) 2022-2023 PANTHEON.tech s.r.o. # Copyright(c) 2022 University of New Hampshire -""" -A test framework for testing DPDK. -""" +"""The DTS executable.""" import logging @@ -17,6 +15,10 @@ def main() -> None: """Set DTS settings, then run DTS. The DTS settings are taken from the command line arguments and the environment variables. + The settings object is stored in the module-level variable settings.SETTINGS which the entire + framework uses. After importing the module (or the variable), any changes to the variable are + not going to be reflected without a re-import. This means that the SETTINGS variable must + be modified before the settings module is imported anywhere else in the framework. """ settings.SETTINGS = settings.get_settings() from framework import dts