From patchwork Mon Dec 4 10:24:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 134804 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C05C54366A; Mon, 4 Dec 2023 11:26:32 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7C10842D45; Mon, 4 Dec 2023 11:24:54 +0100 (CET) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mails.dpdk.org (Postfix) with ESMTP id 67FB4427D7 for ; Mon, 4 Dec 2023 11:24:47 +0100 (CET) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-40c08af319cso13806195e9.2 for ; Mon, 04 Dec 2023 02:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1701685487; x=1702290287; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t6KeZaLHiysMT+K7nNxDXG0i4w3eJCHNkMFr/vP4ebc=; b=JHoIVy3zSNrmgL0WPgDXktHzW7gUlkjwiGapVv1gKmFAPV7cfOxeSxThndAIImdVJT 2um280KDNEz6NoeRwYEEgx9m/j0CiztLJ96pqHObIr3+K51FI8dkAiBuH6wcbLqUHmHw bq6qcYOAswWB23o/IGxBFSxxArXpXRc86s0M+mK9/QS1c7EExqWOBqPfAudCMnjEnlYB 1scBA9lPljbrUlV2WXzt5SEHvRxaCLCBdFdlZJvd+buVLmaiXGAPYDKIIQKzh8a/PpWG xiHlAKdhhFzQvNmTIDlCYEBr+5ESbG8gWcLwe2m0tbqegw9YlJiYSyeLa+GvUW4kZWkW nbTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701685487; x=1702290287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t6KeZaLHiysMT+K7nNxDXG0i4w3eJCHNkMFr/vP4ebc=; b=wCXHUEnzfs4lRKUJW31O/lyD8fVmlf4LBHEHJ5K/YA4MWiYTdgkMrZ9lUo/FE5byB2 H9F5OTy9wkPD37G2s6BR4thcKgBJxEf+wN+YS3yhfeNzsRCAmAptZtMJBprW81uHYblb 3/x69DCKImMKG8vuU5jDEYLS9t8OcaJ76smOuG/UP9XccJVGPFQxxBDPnv9H58pLXh6w DRrzDVbOXtFRcoBwfMCDc0Qqk83pLJMQ9IQcNV/zI//pFWq3NxxF/Pl1/vnIT4S2Ys1S bsi9XFYg+rocZbV7hsK/sy/T6f0SnXKM8wCWB/bk5X49vsA70mNwtbp/Y22tSKoQk9Jw Ei7w== X-Gm-Message-State: AOJu0YwYMSiGOiWGZ5TtfVe8/huHym6Q+nBQWhgkOX6KaygT6PjBLlx5 0JtjrrFwu2thurFm7FlzinbZDA== X-Google-Smtp-Source: AGHT+IEwUS3wQjyd8qDni0zOHLt/lR+xVNcnMUWUwQXc4s5tgL2D/asIJ21xv2rlxu92Fwkn7m6FlA== X-Received: by 2002:a05:600c:4eca:b0:40a:6235:e82d with SMTP id g10-20020a05600c4eca00b0040a6235e82dmr2316431wmq.15.1701685487096; Mon, 04 Dec 2023 02:24:47 -0800 (PST) Received: from jlinkes-PT-Latitude-5530.pantheon.local ([81.89.53.154]) by smtp.gmail.com with ESMTPSA id m28-20020a05600c3b1c00b0040b2b38a1fasm14255415wms.4.2023.12.04.02.24.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 02:24:46 -0800 (PST) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, jspewock@iol.unh.edu, probb@iol.unh.edu, paul.szczepanek@arm.com, yoan.picchi@foss.arm.com, Luca.Vizzarro@arm.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v9 16/21] dts: posix and linux sessions docstring update Date: Mon, 4 Dec 2023 11:24:24 +0100 Message-Id: <20231204102429.106709-17-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231204102429.106709-1-juraj.linkes@pantheon.tech> References: <20231123151344.162812-1-juraj.linkes@pantheon.tech> <20231204102429.106709-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Format according to the Google format and PEP257, with slight deviations. Signed-off-by: Juraj Linkeš --- dts/framework/testbed_model/linux_session.py | 64 +++++++++++----- dts/framework/testbed_model/posix_session.py | 81 +++++++++++++++++--- 2 files changed, 114 insertions(+), 31 deletions(-) diff --git a/dts/framework/testbed_model/linux_session.py b/dts/framework/testbed_model/linux_session.py index 055765ba2d..0ab59cef85 100644 --- a/dts/framework/testbed_model/linux_session.py +++ b/dts/framework/testbed_model/linux_session.py @@ -2,6 +2,13 @@ # Copyright(c) 2023 PANTHEON.tech s.r.o. # Copyright(c) 2023 University of New Hampshire +"""Linux OS translator. + +Translate OS-unaware calls into Linux calls/utilities. Most of Linux distributions are mostly +compliant with POSIX standards, so this module only implements the parts that aren't. +This intermediate module implements the common parts of mostly POSIX compliant distributions. +""" + import json from ipaddress import IPv4Interface, IPv6Interface from typing import TypedDict, Union @@ -17,43 +24,52 @@ class LshwConfigurationOutput(TypedDict): + """The relevant parts of ``lshw``'s ``configuration`` section.""" + + #: link: str class LshwOutput(TypedDict): - """ - A model of the relevant information from json lshw output, e.g.: - { - ... - "businfo" : "pci@0000:08:00.0", - "logicalname" : "enp8s0", - "version" : "00", - "serial" : "52:54:00:59:e1:ac", - ... - "configuration" : { - ... - "link" : "yes", - ... - }, - ... + """A model of the relevant information from ``lshw``'s json output. + + Example: + :: + + { + ... + "businfo" : "pci@0000:08:00.0", + "logicalname" : "enp8s0", + "version" : "00", + "serial" : "52:54:00:59:e1:ac", + ... + "configuration" : { + ... + "link" : "yes", + ... + }, + ... """ + #: businfo: str + #: logicalname: NotRequired[str] + #: serial: NotRequired[str] + #: configuration: LshwConfigurationOutput class LinuxSession(PosixSession): - """ - The implementation of non-Posix compliant parts of Linux remote sessions. - """ + """The implementation of non-Posix compliant parts of Linux.""" @staticmethod def _get_privileged_command(command: str) -> str: return f"sudo -- sh -c '{command}'" def get_remote_cpus(self, use_first_core: bool) -> list[LogicalCore]: + """Overrides :meth:`~.os_session.OSSession.get_remote_cpus`.""" cpu_info = self.send_command("lscpu -p=CPU,CORE,SOCKET,NODE|grep -v \\#").stdout lcores = [] for cpu_line in cpu_info.splitlines(): @@ -65,18 +81,20 @@ def get_remote_cpus(self, use_first_core: bool) -> list[LogicalCore]: return lcores def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str: + """Overrides :meth:`~.os_session.OSSession.get_dpdk_file_prefix`.""" return dpdk_prefix - def setup_hugepages(self, hugepage_amount: int, force_first_numa: bool) -> None: + def setup_hugepages(self, hugepage_count: int, force_first_numa: bool) -> None: + """Overrides :meth:`~.os_session.OSSession.setup_hugepages`.""" self._logger.info("Getting Hugepage information.") hugepage_size = self._get_hugepage_size() hugepages_total = self._get_hugepages_total() self._numa_nodes = self._get_numa_nodes() - if force_first_numa or hugepages_total != hugepage_amount: + if force_first_numa or hugepages_total != hugepage_count: # when forcing numa, we need to clear existing hugepages regardless # of size, so they can be moved to the first numa node - self._configure_huge_pages(hugepage_amount, hugepage_size, force_first_numa) + self._configure_huge_pages(hugepage_count, hugepage_size, force_first_numa) else: self._logger.info("Hugepages already configured.") self._mount_huge_pages() @@ -132,6 +150,7 @@ def _configure_huge_pages(self, amount: int, size: int, force_first_numa: bool) self.send_command(f"echo {amount} | tee {hugepage_config_path}", privileged=True) def update_ports(self, ports: list[Port]) -> None: + """Overrides :meth:`~.os_session.OSSession.update_ports`.""" self._logger.debug("Gathering port info.") for port in ports: assert port.node == self.name, "Attempted to gather port info on the wrong node" @@ -161,6 +180,7 @@ def _update_port_attr(self, port: Port, attr_value: str | None, attr_name: str) ) def configure_port_state(self, port: Port, enable: bool) -> None: + """Overrides :meth:`~.os_session.OSSession.configure_port_state`.""" state = "up" if enable else "down" self.send_command(f"ip link set dev {port.logical_name} {state}", privileged=True) @@ -170,6 +190,7 @@ def configure_port_ip_address( port: Port, delete: bool, ) -> None: + """Overrides :meth:`~.os_session.OSSession.configure_port_ip_address`.""" command = "del" if delete else "add" self.send_command( f"ip address {command} {address} dev {port.logical_name}", @@ -178,5 +199,6 @@ def configure_port_ip_address( ) def configure_ipv4_forwarding(self, enable: bool) -> None: + """Overrides :meth:`~.os_session.OSSession.configure_ipv4_forwarding`.""" state = 1 if enable else 0 self.send_command(f"sysctl -w net.ipv4.ip_forward={state}", privileged=True) diff --git a/dts/framework/testbed_model/posix_session.py b/dts/framework/testbed_model/posix_session.py index 5657cc0bc9..d279bb8b53 100644 --- a/dts/framework/testbed_model/posix_session.py +++ b/dts/framework/testbed_model/posix_session.py @@ -2,6 +2,15 @@ # Copyright(c) 2023 PANTHEON.tech s.r.o. # Copyright(c) 2023 University of New Hampshire +"""POSIX compliant OS translator. + +Translates OS-unaware calls into POSIX compliant calls/utilities. POSIX is a set of standards +for portability between Unix operating systems which not all Linux distributions +(or the tools most frequently bundled with said distributions) adhere to. Most of Linux +distributions are mostly compliant though. +This intermediate module implements the common parts of mostly POSIX compliant distributions. +""" + import re from collections.abc import Iterable from pathlib import PurePath, PurePosixPath @@ -15,13 +24,21 @@ class PosixSession(OSSession): - """ - An intermediary class implementing the Posix compliant parts of - Linux and other OS remote sessions. - """ + """An intermediary class implementing the POSIX standard.""" @staticmethod def combine_short_options(**opts: bool) -> str: + """Combine shell options into one argument. + + These are options such as ``-x``, ``-v``, ``-f`` which are combined into ``-xvf``. + + Args: + opts: The keys are option names (usually one letter) and the bool values indicate + whether to include the option in the resulting argument. + + Returns: + The options combined into one argument. + """ ret_opts = "" for opt, include in opts.items(): if include: @@ -33,17 +50,19 @@ def combine_short_options(**opts: bool) -> str: return ret_opts def guess_dpdk_remote_dir(self, remote_dir: str | PurePath) -> PurePosixPath: + """Overrides :meth:`~.os_session.OSSession.guess_dpdk_remote_dir`.""" remote_guess = self.join_remote_path(remote_dir, "dpdk-*") result = self.send_command(f"ls -d {remote_guess} | tail -1") return PurePosixPath(result.stdout) def get_remote_tmp_dir(self) -> PurePosixPath: + """Overrides :meth:`~.os_session.OSSession.get_remote_tmp_dir`.""" return PurePosixPath("/tmp") def get_dpdk_build_env_vars(self, arch: Architecture) -> dict: - """ - Create extra environment variables needed for i686 arch build. Get information - from the node if needed. + """Overrides :meth:`~.os_session.OSSession.get_dpdk_build_env_vars`. + + Supported architecture: ``i686``. """ env_vars = {} if arch == Architecture.i686: @@ -63,6 +82,7 @@ def get_dpdk_build_env_vars(self, arch: Architecture) -> dict: return env_vars def join_remote_path(self, *args: str | PurePath) -> PurePosixPath: + """Overrides :meth:`~.os_session.OSSession.join_remote_path`.""" return PurePosixPath(*args) def copy_from( @@ -70,6 +90,7 @@ def copy_from( source_file: str | PurePath, destination_file: str | PurePath, ) -> None: + """Overrides :meth:`~.os_session.OSSession.copy_from`.""" self.remote_session.copy_from(source_file, destination_file) def copy_to( @@ -77,6 +98,7 @@ def copy_to( source_file: str | PurePath, destination_file: str | PurePath, ) -> None: + """Overrides :meth:`~.os_session.OSSession.copy_to`.""" self.remote_session.copy_to(source_file, destination_file) def remove_remote_dir( @@ -85,6 +107,7 @@ def remove_remote_dir( recursive: bool = True, force: bool = True, ) -> None: + """Overrides :meth:`~.os_session.OSSession.remove_remote_dir`.""" opts = PosixSession.combine_short_options(r=recursive, f=force) self.send_command(f"rm{opts} {remote_dir_path}") @@ -93,6 +116,7 @@ def extract_remote_tarball( remote_tarball_path: str | PurePath, expected_dir: str | PurePath | None = None, ) -> None: + """Overrides :meth:`~.os_session.OSSession.extract_remote_tarball`.""" self.send_command( f"tar xfm {remote_tarball_path} -C {PurePosixPath(remote_tarball_path).parent}", 60, @@ -109,6 +133,7 @@ def build_dpdk( rebuild: bool = False, timeout: float = SETTINGS.compile_timeout, ) -> None: + """Overrides :meth:`~.os_session.OSSession.build_dpdk`.""" try: if rebuild: # reconfigure, then build @@ -138,10 +163,12 @@ def build_dpdk( raise DPDKBuildError(f"DPDK build failed when doing '{e.command}'.") def get_dpdk_version(self, build_dir: str | PurePath) -> str: + """Overrides :meth:`~.os_session.OSSession.get_dpdk_version`.""" out = self.send_command(f"cat {self.join_remote_path(build_dir, 'VERSION')}", verify=True) return out.stdout def kill_cleanup_dpdk_apps(self, dpdk_prefix_list: Iterable[str]) -> None: + """Overrides :meth:`~.os_session.OSSession.kill_cleanup_dpdk_apps`.""" self._logger.info("Cleaning up DPDK apps.") dpdk_runtime_dirs = self._get_dpdk_runtime_dirs(dpdk_prefix_list) if dpdk_runtime_dirs: @@ -153,6 +180,14 @@ def kill_cleanup_dpdk_apps(self, dpdk_prefix_list: Iterable[str]) -> None: self._remove_dpdk_runtime_dirs(dpdk_runtime_dirs) def _get_dpdk_runtime_dirs(self, dpdk_prefix_list: Iterable[str]) -> list[PurePosixPath]: + """Find runtime directories DPDK apps are currently using. + + Args: + dpdk_prefix_list: The prefixes DPDK apps were started with. + + Returns: + The paths of DPDK apps' runtime dirs. + """ prefix = PurePosixPath("/var", "run", "dpdk") if not dpdk_prefix_list: remote_prefixes = self._list_remote_dirs(prefix) @@ -164,9 +199,13 @@ def _get_dpdk_runtime_dirs(self, dpdk_prefix_list: Iterable[str]) -> list[PurePo return [PurePosixPath(prefix, dpdk_prefix) for dpdk_prefix in dpdk_prefix_list] def _list_remote_dirs(self, remote_path: str | PurePath) -> list[str] | None: - """ - Return a list of directories of the remote_dir. - If remote_path doesn't exist, return None. + """Contents of remote_path. + + Args: + remote_path: List the contents of this path. + + Returns: + The contents of remote_path. If remote_path doesn't exist, return None. """ out = self.send_command(f"ls -l {remote_path} | awk '/^d/ {{print $NF}}'").stdout if "No such file or directory" in out: @@ -175,6 +214,17 @@ def _list_remote_dirs(self, remote_path: str | PurePath) -> list[str] | None: return out.splitlines() def _get_dpdk_pids(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> list[int]: + """Find PIDs of running DPDK apps. + + Look at each "config" file found in dpdk_runtime_dirs and find the PIDs of processes + that opened those file. + + Args: + dpdk_runtime_dirs: The paths of DPDK apps' runtime dirs. + + Returns: + The PIDs of running DPDK apps. + """ pids = [] pid_regex = r"p(\d+)" for dpdk_runtime_dir in dpdk_runtime_dirs: @@ -193,6 +243,14 @@ def _remote_files_exists(self, remote_path: PurePath) -> bool: return not result.return_code def _check_dpdk_hugepages(self, dpdk_runtime_dirs: Iterable[str | PurePath]) -> None: + """Check there aren't any leftover hugepages. + + If any hugepages are found, emit a warning. The hugepages are investigated in the + "hugepage_info" file of dpdk_runtime_dirs. + + Args: + dpdk_runtime_dirs: The paths of DPDK apps' runtime dirs. + """ for dpdk_runtime_dir in dpdk_runtime_dirs: hugepage_info = PurePosixPath(dpdk_runtime_dir, "hugepage_info") if self._remote_files_exists(hugepage_info): @@ -208,9 +266,11 @@ def _remove_dpdk_runtime_dirs(self, dpdk_runtime_dirs: Iterable[str | PurePath]) self.remove_remote_dir(dpdk_runtime_dir) def get_dpdk_file_prefix(self, dpdk_prefix: str) -> str: + """Overrides :meth:`~.os_session.OSSession.get_dpdk_file_prefix`.""" return "" def get_compiler_version(self, compiler_name: str) -> str: + """Overrides :meth:`~.os_session.OSSession.get_compiler_version`.""" match compiler_name: case "gcc": return self.send_command( @@ -228,6 +288,7 @@ def get_compiler_version(self, compiler_name: str) -> str: raise ValueError(f"Unknown compiler {compiler_name}") def get_node_info(self) -> NodeInfo: + """Overrides :meth:`~.os_session.OSSession.get_node_info`.""" os_release_info = self.send_command( "awk -F= '$1 ~ /^NAME$|^VERSION$/ {print $2}' /etc/os-release", SETTINGS.timeout,