From patchwork Wed Oct 13 14:19:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Owen Hilyard X-Patchwork-Id: 101354 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DF49FA0C55; Wed, 13 Oct 2021 16:19:57 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D9646410DA; Wed, 13 Oct 2021 16:19:57 +0200 (CEST) Received: from mail-ot1-f100.google.com (mail-ot1-f100.google.com [209.85.210.100]) by mails.dpdk.org (Postfix) with ESMTP id 44D4F40E64 for ; Wed, 13 Oct 2021 16:19:56 +0200 (CEST) Received: by mail-ot1-f100.google.com with SMTP id v2-20020a05683018c200b0054e3acddd91so3848598ote.8 for ; Wed, 13 Oct 2021 07:19:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Z59Eohc2hTR857b1j/dzo6GpELCp+M+NPQ0SZTLDQeI=; b=C8YFMGo25+yCkgbJm98TbcCcPjzmkTWncbKQJGkczBz8JQFwZYgH892YfUCf4TvHZ4 I8fgPQ6peYCgRBqciASaNqHYAHsHf+jhWemQwhnMG2Mbqo9w/zr7Tp4crEV8GvkYIE0c hb6QW/4p8F1PTcwVlJuN0d1kfj5jOD+Qwvzyg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Z59Eohc2hTR857b1j/dzo6GpELCp+M+NPQ0SZTLDQeI=; b=jlqXu36lw8GC9XZELWCFg4nFIKotdqsGCPW7Rpsa071I2EmeZwnAR6YeJYeL6oxwO8 3oGkow2JrFzckkhZyQnL7EKk10p9ZHs3V7a+22UcKhpuGzY5CQFwO6z5wOdWbTkS9IcU SFgoBNolRombs5cibWzqkmei7bU84KNDp0mAsPbO3lyHdlgS5PgvUt9xrMuBJOtv6nif aNO0d8pV+OvCZNZdN2Gnh5i4PT5RJyv5STPK3A9fW8S+9P2cL+B9RaICTlykq/fT2aYI i9q8UYAPA5xFhKM/BP84DydzIgaIZF8/zlWiZ4gPpYBcXlR1eTw8ix0Q3sUu/yqwuXIa TnqQ== X-Gm-Message-State: AOAM533OjkqDkiLdMibhFccrOtCl00hhAjtSAdexWmKHd1+SSV2iNYuz R9c011y5G/Zn3E5k+QkYkVu9e/loPbNUu0KUzWuqhjsqJN5l7EVk4BPDeI45dzWDGgJ5iVhUp9D 6IPAxbhC76ZKtHZKUvzAH99KRaht/HZtj9vNR30w5KDlQqtI3wHtXpbEnB1mHo//q2UumE/qh+G gC+POUjp5LT5CSWw== X-Google-Smtp-Source: ABdhPJzEbDRORPEvp2oxMxiN/JgoaggbuYwR3+FvG7QjESoZ72gxrfuI1ZImGiU8Lw+azrtne6RWadzPYfU/ X-Received: by 2002:a05:6830:3093:: with SMTP id f19mr18681220ots.97.1634134795289; Wed, 13 Oct 2021 07:19:55 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [2606:4100:3880:1234::84]) by smtp-relay.gmail.com with ESMTPS id l20sm805420otk.11.2021.10.13.07.19.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Oct 2021 07:19:55 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::105d]) by postal.iol.unh.edu (Postfix) with ESMTP id C012D605247D; Wed, 13 Oct 2021 10:19:54 -0400 (EDT) From: ohilyard@iol.unh.edu To: dts@dpdk.org Cc: juraj.linkes@pantheon.tech, lijuan.tu@intel.com, Owen Hilyard Date: Wed, 13 Oct 2021 10:19:53 -0400 Message-Id: <20211013141953.28687-1-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Subject: [dts] [PATCH v2] ci/initial: Added script to get the tests for a patchset X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" From: Owen Hilyard Added a script to ci to allow getting the changed tests for a patchset to DTS. Instructions for use are in the module doc comment of the script. Added a dockerfile that should be updated over time to allow it to run all DTS CI scripts. Signed-off-by: Owen Hilyard --- ci/Dockerfile | 50 ++++++++ ci/README.txt | 38 ++++++ ci/dts_requirements.txt | 43 +++++++ ci/get_tests_for_patchset.py | 218 +++++++++++++++++++++++++++++++++++ ci/requirements.txt | 33 ++++++ 5 files changed, 382 insertions(+) create mode 100644 ci/Dockerfile create mode 100644 ci/README.txt create mode 100644 ci/dts_requirements.txt create mode 100644 ci/get_tests_for_patchset.py create mode 100644 ci/requirements.txt diff --git a/ci/Dockerfile b/ci/Dockerfile new file mode 100644 index 00000000..8ee53786 --- /dev/null +++ b/ci/Dockerfile @@ -0,0 +1,50 @@ +# BSD LICENSE +# +# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved. +# Copyright(c) 2021 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +# This container should be build in the ci directory, and then the +# DTS directory should be mounted as a volume at /dts/ +FROM python:3.9-slim-buster + +ENV DEBIAN_FRONTEND=noninteractive + +COPY requirements.txt . +COPY dts_requirements.txt dts_requirements.txt + +RUN apt-get update && apt-get install --no-install-recommends -y \ + # Add a C compiler for all of the c modules in DTS + build-essential make gcc git libpcap-dev\ + python3-pip + +RUN pip3 install -r requirements.txt +RUN pip3 install -r dts_requirements.txt +# install formatter +RUN pip3 install black \ No newline at end of file diff --git a/ci/README.txt b/ci/README.txt new file mode 100644 index 00000000..a833e255 --- /dev/null +++ b/ci/README.txt @@ -0,0 +1,38 @@ +# BSD LICENSE +# +# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved. +# Copyright(c) 2021 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +This directory contains scripts used for ci in DTS. Nothing in this directory +should be modified during the course of submitting a patch to DPDK. + +Additional python requirements only needed for running DTS in ci are present in the +requirements.txt file in this directory. + diff --git a/ci/dts_requirements.txt b/ci/dts_requirements.txt new file mode 100644 index 00000000..d88ccd26 --- /dev/null +++ b/ci/dts_requirements.txt @@ -0,0 +1,43 @@ +# BSD LICENSE +# +# Copyright(c) 2019 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +xlwt==1.3.0 +pexpect==4.7.0 +numpy==1.18.5 +docutils +pcapy +xlrd +scapy==2.4.4 +threadpool +matplotlib +gitpython +pydes \ No newline at end of file diff --git a/ci/get_tests_for_patchset.py b/ci/get_tests_for_patchset.py new file mode 100644 index 00000000..80f99045 --- /dev/null +++ b/ci/get_tests_for_patchset.py @@ -0,0 +1,218 @@ +# BSD LICENSE +# +# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved. +# Copyright(c) 2021 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +""" +This script should be run after a patchset has been applied to DTS. +It will iterate through the import tree in python, looking for any +test that could possibly have had behavior change due to the patchset. +It will then output all of those tests to standard out in a line-delimited +list. + +The script will also issue warning to standard error if any of the +"protected paths" are changed. The warning will be in the format: +"WARNING: {file_name} is protected" + +The script will also issue a warning if a config file is changed. This +warning is also sent to standard error and takes the form of: +"WARNING: {file_name} is a config file and was changed" + +This script can either be run natively or with the provided dockerfile. +""" + + +import argparse +import os +import pkgutil +import re +import subprocess +import sys +from pkgutil import walk_packages +from types import ModuleType +from typing import List, Iterable + +DTS_MAIN_BRANCH_REF: str = "origin/master" + +DTS_TEST_MODULE_PATH: str = "tests" +# Will be unnecessary once DTS moves to a normal module structure +DTS_MODULE_PATHS: List[str] = ["framework", "nic", "dep", DTS_TEST_MODULE_PATH] + +# This is primarily intended for folders which contain CI scripts, as these +# should not be modified in the course of normal CI. A file in any of these +# folders being modified will cause a warning to be emitted to standard error +DTS_PROTECTED_PATHS: List[str] = [ + "ci", +] + +# This is intended to help detect when a config files have been changed. +# The intention behind this detection is to enable additional regression +# tests related to ensuring stable config file formats +DTS_CONFIG_PATHS: List[str] = [ + "conf", + "execution.cfg", +] + + +def get_args() -> str: + parser: argparse.ArgumentParser = argparse.ArgumentParser( + description="After a patchset is applied, run this script" + "It will then output a list " + "of applicable test suites to standard output, with 1 " + "test suite per line. All other output, such as warnings," + " errors or informational messages will be sent to " + "standard error. This script may produce no output at all," + "in which case it should be assumed that there are no " + "applicable test suites.\n\n " + "Exit Codes:\n" + "value | meaning\n" + "----- | -------\n" + " 0 | OK \n" + " 1 | Error, check standard error", + ) + + dts_directory: str = os.path.dirname(os.path.dirname(os.path.join(os.getcwd(), __file__))) + parser.add_argument("-d", "--dts-directory", type=str, default=dts_directory, required=False) + args = parser.parse_args() + if not os.path.isdir(args.dts_directory): + print(f"{args.dts_directory} is not a directory.", file=sys.stderr) + exit(1) + + return args.dts_directory + + +def get_modified_files() -> List[str]: + cmd = ['git', 'diff', '--name-only', DTS_MAIN_BRANCH_REF, 'HEAD'] + process: subprocess.CompletedProcess = subprocess.run(cmd, capture_output=True) + if process.returncode != 0: + print(f"{' '.join(cmd)} returned {process.returncode}") + exit(1) + + # This explicit conversion to utf8 will catch encoding errors + stdout: bytes = process.stdout + stdout_str: str = stdout.decode("utf-8") + + return stdout_str.splitlines() + + +def get_names_of_modified_python_files(files: List[str]) -> List[str]: + return list( + map( + lambda f: str(re.sub("\\.py", "", f)), + map( + lambda f: os.path.basename(f), + filter( + lambda f: f.endswith(".py"), files + ) + ) + ) + ) + + +def get_modules() -> List[ModuleType]: + return list(map(lambda m: pkgutil.resolve_name(m.name), walk_packages(DTS_MODULE_PATHS))) + + +def get_module_imports(mod: ModuleType) -> Iterable[ModuleType]: + imports = [] + for attribute_label in dir(mod): + try: + attribute = getattr(mod, attribute_label) + except ModuleNotFoundError as _: # some standard library modules don't like being directly imported + continue + if isinstance(attribute, type(mod)): + imports.append(attribute) + return imports + + +def get_modules_in_tree(mod: ModuleType) -> Iterable[ModuleType]: + yield from get_module_imports(mod) + + +def get_only_test_suites(modules: Iterable[ModuleType]) -> Iterable[ModuleType]: + test_package_path = os.path.join(os.getcwd(), DTS_TEST_MODULE_PATH) + mod: ModuleType + return filter(lambda mod: mod.__name__.startswith("TestSuite_"), + filter(lambda mod: mod.__file__.startswith(test_package_path), + filter(lambda mod: "__file__" in dir(mod), modules))) + + +def get_test_suite_names(modules: Iterable[ModuleType]) -> Iterable[str]: + # Moving this into a set is there to ensure that there are no duplicates + return set(list(map(lambda mod: re.sub("TestSuite_", "", mod.__name__), modules))) + + +def get_tests_to_run() -> List[str]: + dts_directory: str + dts_directory = get_args() + + # This all needs to be done at the top level, so I have to do it here. + + # chdir to the DTS directory, since we want that to be + # the context for any commands that are run. + os.chdir(dts_directory) + + for path in DTS_MODULE_PATHS: + sys.path.append(os.path.join(dts_directory, path)) + + files: List[str] = get_modified_files() + changed_module_name = get_names_of_modified_python_files(files) + + for protected_folder in DTS_PROTECTED_PATHS: + for file_name in files: + if file_name.startswith(protected_folder): + print(f"WARNING: {file_name} is protected", file=sys.stderr) + + for config_file_path in DTS_CONFIG_PATHS: + for file_name in files: + if file_name.startswith(config_file_path): + print(f"WARNING: {file_name} is a config file and was changed", file=sys.stderr) + + # Each index is 1 level of the tree + module_list: List[ModuleType] = [pkgutil.resolve_name(name) for name in changed_module_name] + current_index: int = 0 + while current_index < len(module_list) and len(module_list) > 0: + mod = module_list[current_index] + if module_list.count(mod) == 1: + module_list = module_list + list(get_modules_in_tree(mod)) + current_index += 1 + + test_suites_to_run: List[str] = list(get_test_suite_names(get_only_test_suites(module_list))) + test_suites_to_run.sort() + return test_suites_to_run + + +def main(): + test_suites_to_run = get_tests_to_run() + print("\n".join(test_suites_to_run)) + + +if __name__ == "__main__": + main() diff --git a/ci/requirements.txt b/ci/requirements.txt new file mode 100644 index 00000000..74d51fc3 --- /dev/null +++ b/ci/requirements.txt @@ -0,0 +1,33 @@ +# BSD LICENSE +# +# Copyright(c) 2021 University of New Hampshire Interoperability Laboratory. All rights reserved. +# Copyright(c) 2021 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +argparse==1.4.0 \ No newline at end of file