From patchwork Mon Jul 17 11:07:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Juraj_Linke=C5=A1?= X-Patchwork-Id: 129576 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CF5B942E9B; Mon, 17 Jul 2023 13:07:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E8E1D42D43; Mon, 17 Jul 2023 13:07:27 +0200 (CEST) Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by mails.dpdk.org (Postfix) with ESMTP id A450040698 for ; Mon, 17 Jul 2023 13:07:25 +0200 (CEST) Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-992af8b3b1bso617254766b.1 for ; Mon, 17 Jul 2023 04:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pantheon.tech; s=google; t=1689592045; x=1692184045; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zi96yCSIZ5WtpQpQtuNhnvhJw5DyhS9QIGoChHCG2Q4=; b=LfETNG/PlUiof6K2FB8eKBTwo1W0J0rkfiamp9u3qIIkz+ZzjR8vKOZ/MrdFpNbB80 3i/NERHDTZIl1WUG0MGqakdl+Lt6HIleiKsg+6MzxJiwdgT15eKkbD6cdN+n5bAi/SQM mUMJ0VgoURzvsVnNi+upZQ6wVfUVlBdOkYEoxmD5ncMiXE2k2R597JO5fARx1ZoRtawq 55lk+1kVIBItPhhn3+LzTkljjSZ6RkPCty2jcdMLK9VG75ysCAt3NSA4PMZrkbAHu082 HVRmZxwd0IMKoU7DSTDTx1YDnpqbRHb9kGiBzFBgFrpkgKCKFFeTrL+Inc1HgkY3nVNK lZ7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689592045; x=1692184045; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zi96yCSIZ5WtpQpQtuNhnvhJw5DyhS9QIGoChHCG2Q4=; b=ZIp3KcnUWC5dbGq+rUWfyv2/+NCc4kmvXiRyZq/OKY87GrE1RAtCKK14cUsHaoBDuR xBHfiuEK7l5sXflxJLgkOHwgGkzyYUBK7fCOB12OlQ1Jo8QtUzD8E2FLgJbB/xGIvQ00 NTUHRLdmADKcm6X6s2Y/nPBK6g7GO2DO3Q9gyAuaJ+wb1FVn/oan0pvdYReOgTXpvQ// sV11xGQXrKD5oXT4uMw64pAs7l8zX5sGnZDWrnhqjgSTFesR0ADlZVzKU/upR3pFS56y oQ3CDB0t1bJUMS6bGBF7oIyVrwJ7MQVVFXbkdTPWG3wiMmWRe/f3HaumvfBGtz8unUbN EVaA== X-Gm-Message-State: ABy/qLYUGBKEP7ZbU+eM3dIeArf7b1+kaFRRbCKXDNmNjNSdeitH+znC KYPz6G1figdJXLWJSt0wohx3pw== X-Google-Smtp-Source: APBJJlG9T+AkvF2RukeVfft9y7z0PowOuq+PhP0VJc1vH6ZDfLL2VlqOS6LmNt3V3TP7PNmCIhA2tA== X-Received: by 2002:a17:906:28d9:b0:991:f383:d5c3 with SMTP id p25-20020a17090628d900b00991f383d5c3mr9961280ejd.74.1689592045293; Mon, 17 Jul 2023 04:07:25 -0700 (PDT) Received: from jlinkes-PT-Latitude-5530.. (ip-46.34.239.87.o2inet.sk. [46.34.239.87]) by smtp.gmail.com with ESMTPSA id s21-20020a170906355500b0098de7d28c34sm8995051eja.193.2023.07.17.04.07.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jul 2023 04:07:25 -0700 (PDT) From: =?utf-8?q?Juraj_Linke=C5=A1?= To: thomas@monjalon.net, Honnappa.Nagarahalli@arm.com, lijuan.tu@intel.com, jspewock@iol.unh.edu, probb@iol.unh.edu Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= Subject: [PATCH v2 5/6] dts: scapy traffic generator implementation Date: Mon, 17 Jul 2023 13:07:08 +0200 Message-Id: <20230717110709.39220-6-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230717110709.39220-1-juraj.linkes@pantheon.tech> References: <20230420093109.594704-1-juraj.linkes@pantheon.tech> <20230717110709.39220-1-juraj.linkes@pantheon.tech> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Scapy is a traffic generator capable of sending and receiving traffic. Since it's a software traffic generator, it's not suitable for performance testing, but it is suitable for functional testing. Signed-off-by: Juraj Linkeš --- dts/framework/remote_session/__init__.py | 1 + .../remote_session/remote/__init__.py | 1 + dts/framework/testbed_model/scapy.py | 224 +++++++++++++++++- 3 files changed, 222 insertions(+), 4 deletions(-) diff --git a/dts/framework/remote_session/__init__.py b/dts/framework/remote_session/__init__.py index 1155dd8318..00b6d1f03a 100644 --- a/dts/framework/remote_session/__init__.py +++ b/dts/framework/remote_session/__init__.py @@ -22,6 +22,7 @@ CommandResult, InteractiveRemoteSession, InteractiveShell, + PythonShell, RemoteSession, SSHSession, TestPmdDevice, diff --git a/dts/framework/remote_session/remote/__init__.py b/dts/framework/remote_session/remote/__init__.py index 03fd309f2b..075f52b646 100644 --- a/dts/framework/remote_session/remote/__init__.py +++ b/dts/framework/remote_session/remote/__init__.py @@ -9,6 +9,7 @@ from .interactive_remote_session import InteractiveRemoteSession from .interactive_shell import InteractiveShell +from .python_shell import PythonShell from .remote_session import CommandResult, RemoteSession from .ssh_session import SSHSession from .testpmd_shell import TestPmdDevice, TestPmdShell diff --git a/dts/framework/testbed_model/scapy.py b/dts/framework/testbed_model/scapy.py index 1a23dc9fa3..af0d4dbb25 100644 --- a/dts/framework/testbed_model/scapy.py +++ b/dts/framework/testbed_model/scapy.py @@ -12,10 +12,21 @@ a local server proxy. """ +import inspect +import marshal +import time +import types +import xmlrpc.client +from xmlrpc.server import SimpleXMLRPCServer + +import scapy.all # type: ignore[import] +from scapy.layers.l2 import Ether # type: ignore[import] from scapy.packet import Packet # type: ignore[import] from framework.config import OS, ScapyTrafficGeneratorConfig -from framework.logger import getLogger +from framework.logger import DTSLOG, getLogger +from framework.remote_session import PythonShell +from framework.settings import SETTINGS from .capturing_traffic_generator import ( CapturingTrafficGenerator, @@ -24,6 +35,134 @@ from .hw.port import Port from .tg_node import TGNode +""" +========= BEGIN RPC FUNCTIONS ========= + +All of the functions in this section are intended to be exported to a python +shell which runs a scapy RPC server. These functions are made available via that +RPC server to the packet generator. To add a new function to the RPC server, +first write the function in this section. Then, if you need any imports, make sure to +add them to SCAPY_RPC_SERVER_IMPORTS as well. After that, add the function to the list +in EXPORTED_FUNCTIONS. Note that kwargs (keyword arguments) do not work via xmlrpc, +so you may need to construct wrapper functions around many scapy types. +""" + +""" +Add the line needed to import something in a normal python environment +as an entry to this array. It will be imported before any functions are +sent to the server. +""" +SCAPY_RPC_SERVER_IMPORTS = [ + "from scapy.all import *", + "import xmlrpc", + "import sys", + "from xmlrpc.server import SimpleXMLRPCServer", + "import marshal", + "import pickle", + "import types", + "import time", +] + + +def scapy_send_packets_and_capture( + xmlrpc_packets: list[xmlrpc.client.Binary], + send_iface: str, + recv_iface: str, + duration: float, +) -> list[bytes]: + """RPC function to send and capture packets. + + The function is meant to be executed on the remote TG node. + + Args: + xmlrpc_packets: The packets to send. These need to be converted to + xmlrpc.client.Binary before sending to the remote server. + send_iface: The logical name of the egress interface. + recv_iface: The logical name of the ingress interface. + duration: Capture for this amount of time, in seconds. + + Returns: + A list of bytes. Each item in the list represents one packet, which needs + to be converted back upon transfer from the remote node. + """ + scapy_packets = [scapy.all.Packet(packet.data) for packet in xmlrpc_packets] + sniffer = scapy.all.AsyncSniffer( + iface=recv_iface, + store=True, + started_callback=lambda *args: scapy.all.sendp(scapy_packets, iface=send_iface), + ) + sniffer.start() + time.sleep(duration) + return [scapy_packet.build() for scapy_packet in sniffer.stop(join=True)] + + +def scapy_send_packets( + xmlrpc_packets: list[xmlrpc.client.Binary], send_iface: str +) -> None: + """RPC function to send packets. + + The function is meant to be executed on the remote TG node. + It doesn't return anything, only sends packets. + + Args: + xmlrpc_packets: The packets to send. These need to be converted to + xmlrpc.client.Binary before sending to the remote server. + send_iface: The logical name of the egress interface. + + Returns: + A list of bytes. Each item in the list represents one packet, which needs + to be converted back upon transfer from the remote node. + """ + scapy_packets = [scapy.all.Packet(packet.data) for packet in xmlrpc_packets] + scapy.all.sendp(scapy_packets, iface=send_iface, realtime=True, verbose=True) + + +""" +Functions to be exposed by the scapy RPC server. +""" +RPC_FUNCTIONS = [ + scapy_send_packets, + scapy_send_packets_and_capture, +] + +""" +========= END RPC FUNCTIONS ========= +""" + + +class QuittableXMLRPCServer(SimpleXMLRPCServer): + """Basic XML-RPC server that may be extended + by functions serializable by the marshal module. + """ + + def __init__(self, *args, **kwargs): + kwargs["allow_none"] = True + super().__init__(*args, **kwargs) + self.register_introspection_functions() + self.register_function(self.quit) + self.register_function(self.add_rpc_function) + + def quit(self) -> None: + self._BaseServer__shutdown_request = True + return None + + def add_rpc_function(self, name: str, function_bytes: xmlrpc.client.Binary): + """Add a function to the server. + + This is meant to be executed remotely. + + Args: + name: The name of the function. + function_bytes: The code of the function. + """ + function_code = marshal.loads(function_bytes.data) + function = types.FunctionType(function_code, globals(), name) + self.register_function(function) + + def serve_forever(self, poll_interval: float = 0.5) -> None: + print("XMLRPC OK") + super().serve_forever(poll_interval) + class ScapyTrafficGenerator(CapturingTrafficGenerator): """Provides access to scapy functions via an RPC interface. @@ -41,10 +180,19 @@ class ScapyTrafficGenerator(CapturingTrafficGenerator): Arguments: tg_node: The node where the traffic generator resides. config: The user configuration of the traffic generator. + + Attributes: + session: The exclusive interactive remote session created by the Scapy + traffic generator where the XML-RPC server runs. + rpc_server_proxy: The object used by clients to execute functions + on the XML-RPC server. """ + session: PythonShell + rpc_server_proxy: xmlrpc.client.ServerProxy _config: ScapyTrafficGeneratorConfig _tg_node: TGNode + _logger: DTSLOG def __init__(self, tg_node: TGNode, config: ScapyTrafficGeneratorConfig): self._config = config @@ -57,8 +205,58 @@ def __init__(self, tg_node: TGNode, config: ScapyTrafficGeneratorConfig): self._tg_node.config.os == OS.linux ), "Linux is the only supported OS for scapy traffic generation" + self.session = self._tg_node.create_interactive_shell( + PythonShell, timeout=5, privileged=True + ) + + # import libs in remote python console + for import_statement in SCAPY_RPC_SERVER_IMPORTS: + self.session.send_command(import_statement) + + # start the server + xmlrpc_server_listen_port = 8000 + self._start_xmlrpc_server_in_remote_python(xmlrpc_server_listen_port) + + # connect to the server + server_url = ( + f"http://{self._tg_node.config.hostname}:{xmlrpc_server_listen_port}" + ) + self.rpc_server_proxy = xmlrpc.client.ServerProxy( + server_url, allow_none=True, verbose=SETTINGS.verbose + ) + + # add functions to the server + for function in RPC_FUNCTIONS: + # A slightly hacky way to move a function to the remote server. + # It is constructed from the name and code on the other side. + # Pickle cannot handle functions, nor can any of the other serialization + # frameworks aside from the libraries used to generate pyc files, which + # are even more messy to work with. + function_bytes = marshal.dumps(function.__code__) + self.rpc_server_proxy.add_rpc_function(function.__name__, function_bytes) + + def _start_xmlrpc_server_in_remote_python(self, listen_port: int): + # load the source of the function + src = inspect.getsource(QuittableXMLRPCServer) + # Lines with only whitespace break the repl if in the middle of a function + # or class, so strip all lines containing only whitespace + src = "\n".join( + [line for line in src.splitlines() if not line.isspace() and line != ""] + ) + + spacing = "\n" * 4 + + # execute it in the python terminal + self.session.send_command(spacing + src + spacing) + self.session.send_command( + f"server = QuittableXMLRPCServer(('0.0.0.0', {listen_port}));" + f"server.serve_forever()", + "XMLRPC OK", + ) + def _send_packets(self, packets: list[Packet], port: Port) -> None: - raise NotImplementedError() + packets = [packet.build() for packet in packets] + self.rpc_server_proxy.scapy_send_packets(packets, port.logical_name) def _send_packets_and_capture( self, @@ -68,7 +266,25 @@ def _send_packets_and_capture( duration: float, capture_name: str = _get_default_capture_name(), ) -> list[Packet]: - raise NotImplementedError() + binary_packets = [packet.build() for packet in packets] + + xmlrpc_packets: list[ + xmlrpc.client.Binary + ] = self.rpc_server_proxy.scapy_send_packets_and_capture( + binary_packets, + send_port.logical_name, + receive_port.logical_name, + duration, + ) # type: ignore[assignment] + + scapy_packets = [Ether(packet.data) for packet in xmlrpc_packets] + return scapy_packets def close(self): - pass + try: + self.rpc_server_proxy.quit() + except ConnectionRefusedError: + # Because the python instance closes, we get no RPC response. + # Thus, this error is expected + pass + self.session.close()