[v5,0/4] net/mlx5: introduce Tx datapath tracing
Message ID | 20230705153125.4657-1-viacheslavo@nvidia.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2EECD42DD9; Wed, 5 Jul 2023 17:32:15 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4051442D40; Wed, 5 Jul 2023 17:32:04 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2071.outbound.protection.outlook.com [40.107.96.71]) by mails.dpdk.org (Postfix) with ESMTP id F2D3542D2D for <dev@dpdk.org>; Wed, 5 Jul 2023 17:32:00 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EbFI1g1DZzDZvDN41jNpfaSSHw3mtZgMx4jEEa56P7txO5J0W8bJbaY3vUdcNKfMK9wKa14izDGrI1aoMGfeHiGudCTNbt/fWwS7sswOtApgyO8LxhKHxFgcWZfOY9gj5d85vMQ5iBJqfXxXZE5pTl01PNZ58UIp8PrQXNJOl16xTDC8VvSt/5OY2bgzPKUskaQGBTeVvufxfS6JW0jRTsUwim7r6TcTlXY2FGhX67IOjRkOUJ5yI3wndoGsMwLzk47thekZKc3XY4a5gFkLZCKqXIqA5Suv/gmjI9OSOwrncqTvxphfdHgM1xe3otInkE0ZOzmF9A3HHzZPwTFfsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=m05DbSzIFm5dwts1qVhOvtIQkwEx1IQzcy4Q6h8BbTc=; b=dNqAYc7x3BH49qULSB2eClbuRQOXdz+sTf0y+EWw1CxBTyrCSVMb3Iq6snhrWhg01H4ipvtk+IJErTniDbrAW2DXL5sIbrv8YCg+UzSzqRRmaIVJ/kIQwlaB3fi4NgY/TB/3eOPOUjxPCAuMm/X/Q+XXJOyvgCe9gkp/HtGV+1Z3mtFmESeSRrwzOTB8IMipe7GHCJSPTuGFEU5IDcj6AS8gXoVfK4Lxxl4huzD5g9gMvQe49C2matIvMLd+ArB2SUy0SewLgzEqvaht8joXHuwjlTwwJtOdVcrTuBGJ/Gi8oqXRoZXk4CdDw5HlgmvIjHyYP+FcOJYuxGAow7r4EA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m05DbSzIFm5dwts1qVhOvtIQkwEx1IQzcy4Q6h8BbTc=; b=U55WQVXQoLnPQgX8HB9rXLAY64NkoIFF2H2w7a59LM27jzf0ojOeAPvOz/dxWPQy4t+8kMwKpBqb2rhD+7XisP5sC87fvoB94rCH3zbEe/bNT7PRhpIndSyHYjgjScpqdxRcJh512xgtX0kLGUlrgKz1Dj8qyxVPSeMqWYnY2BlK64jll5f5gHQimOVtBTyf4PFHlTPjbty5v9zTdIpHcZml2aMs45l/i3Tg9Lfe/BVJaiR5JlEj5Q6cDBOBuDJDOTBDepMg8WOx9LinqT8nLl8doM1KikJcMLJ7mlt/yu3IsRfyzHux1yaL0q4wYmm6U8xAu5xRIHtJnMo5UEnwZQ== Received: from DS7PR07CA0020.namprd07.prod.outlook.com (2603:10b6:5:3af::12) by DM8PR12MB5494.namprd12.prod.outlook.com (2603:10b6:8:24::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.17; Wed, 5 Jul 2023 15:31:56 +0000 Received: from DM6NAM11FT045.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3af:cafe::8d) by DS7PR07CA0020.outlook.office365.com (2603:10b6:5:3af::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.18 via Frontend Transport; Wed, 5 Jul 2023 15:31:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT045.mail.protection.outlook.com (10.13.173.123) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.45 via Frontend Transport; Wed, 5 Jul 2023 15:31:55 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 5 Jul 2023 08:31:39 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 5 Jul 2023 08:31:38 -0700 From: Viacheslav Ovsiienko <viacheslavo@nvidia.com> To: <dev@dpdk.org> CC: <jerinj@marvell.com>, <rasland@nvidia.com> Subject: [PATCH v5 0/4] net/mlx5: introduce Tx datapath tracing Date: Wed, 5 Jul 2023 18:31:21 +0300 Message-ID: <20230705153125.4657-1-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230420100803.494-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT045:EE_|DM8PR12MB5494:EE_ X-MS-Office365-Filtering-Correlation-Id: 9acd4b50-e39d-4da7-5759-08db7d6cf73d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: s5tOV+2drStTXnPOiYTFjeo5P1ITdOQhAE8N4uB6TgC/t2q8f0xL8+zqEkC1kM2MTrL8zKNLHdDFU5btKI3FGWrlWFukpg3Q7PU/+KSjvyz0H66pp2T8w+hj0TNsTs+KBos9w9u4SZfINEwZMMJiADCTs80SuxhgYu0ItFh2VZTzqQ8KH0TYtA/Cih2mKZdljenCppY87a0SvKFm33j5RD7klSEERqqpa2zW+eJgv6jcMxs2HvVTZeW+qQprDvFi/geKTlkr0gJgJRJk+mEUZ4uSTFaD71JXRY7T7ccuIzGGO/m6ZNwboYEbdNu+kgmw3rt2+BSNL9q7gypMd9XYx3MD07E0sChtB7iUgtPotGqNRtPaL287CNB8y8V7stBkvaFoZ6MC5IC83KwGYn4Cnc6rB49XSiRZfy4Wo16QfIlc/U3ZBuz75m3FTTp5FyD447LTetwfi/Rlg/Qf7pmJY1uz3a17w+7BqSywc9nxwYd01oG8W4Mo6CMd7fPTN4735n23SH7jLB2gZw573XbkMGq3fU02IkfiR2RZx4IxmhFUpr9MFXOkf2fXul2CqBQQsIRqYrrpQmRW5bbBQ/DcWWldjKyyBzy7u2R22tX7J7MQnBcEB27fztZ5R5MsoVEomGujFZx150rKLamdMq8nRt8+9+pcJx5MTuwtzaSmFZUNCHS2x263y/MA3kJp0W5lEkil3xFIvE+v7Un9wjKIOZvWJa0NmL+F92Od3hButGnWB5VEUS39Gci+SVdYXGR4n841j2H+NuLbNu0nSjKXcAERpVVblXqOBQzIQ9hRlag= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(136003)(396003)(346002)(39860400002)(376002)(451199021)(46966006)(36840700001)(40470700004)(40480700001)(55016003)(40460700003)(6666004)(7636003)(54906003)(356005)(82740400003)(7696005)(478600001)(41300700001)(8936002)(8676002)(70586007)(4326008)(70206006)(6916009)(2616005)(316002)(336012)(6286002)(16526019)(36860700001)(186003)(83380400001)(47076005)(966005)(107886003)(1076003)(26005)(426003)(82310400005)(86362001)(5660300002)(2906002)(36756003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2023 15:31:55.9270 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9acd4b50-e39d-4da7-5759-08db7d6cf73d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT045.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR12MB5494 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org |
Message
Slava Ovsiienko
July 5, 2023, 3:31 p.m. UTC
The mlx5 provides the send scheduling on specific moment of time,
and for the related kind of applications it would be extremely useful
to have extra debug information - when and how packets were scheduled
and when the actual sending was completed by the NIC hardware (it helps
application to track the internal delay issues).
Because the DPDK tx datapath API does not suppose getting any feedback
from the driver and the feature looks like to be mlx5 specific, it seems
to be reasonable to engage exisiting DPDK datapath tracing capability.
The work cycle is supposed to be:
- compile appplication with enabled tracing
- run application with EAL parameters configuring the tracing in mlx5
Tx datapath
- store the dump file with gathered tracing information
- run analyzing scrypt (in Python) to combine related events (packet
firing and completion) and see the data in human-readable view
Below is the detailed instruction "how to" with mlx5 NIC to gather
all the debug data including the full timings information.
1. Build DPDK application with enabled datapath tracing
The meson option should be specified:
--enable_trace_fp=true
The c_args shoudl be specified:
-DALLOW_EXPERIMENTAL_API
The DPDK configuration examples:
meson configure --buildtype=debug -Denable_trace_fp=true
-Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build
meson configure --buildtype=debug -Denable_trace_fp=true
-Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build
meson configure --buildtype=release -Denable_trace_fp=true
-Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build
meson configure --buildtype=release -Denable_trace_fp=true
-Dc_args='-DALLOW_EXPERIMENTAL_API' build
2. Configuring the NIC
If the sending completion timings are important the NIC should be configured
to provide realtime timestamps, the REAL_TIME_CLOCK_ENABLE NV settings parameter
should be configured to TRUE, for example with command (and with following
FW/driver reset):
sudo mlxconfig -d /dev/mst/mt4125_pciconf0 s REAL_TIME_CLOCK_ENABLE=1
3. Run DPDK application to gather the traces
EAL parameters controlling trace capability in runtime
--trace=pmd.net.mlx5.tx - the regular expression enabling the tracepoints
with matching names at least "pmd.net.mlx5.tx"
must be enabled to gather all events needed
to analyze mlx5 Tx datapath and its timings.
By default all tracepoints are disabled.
--trace-dir=/var/log - trace storing directory
--trace-bufsz=<val>B|<val>K|<val>M - optional, trace data buffer size
per thread. The default is 1MB.
--trace-mode=overwrite|discard - optional, selects trace data buffer mode.
4. Installing or Building Babeltrace2 Package
The gathered trace data can be analyzed with a developed Python script.
To parse the trace, the data script uses the Babeltrace2 library.
The package should be either installed or built from source code as
shown below:
git clone https://github.com/efficios/babeltrace.git
cd babeltrace
./bootstrap
./configure -help
./configure --disable-api-doc --disable-man-pages
--disable-python-bindings-doc --enbale-python-plugins
--enable-python-binding
5. Running the Analyzing Script
The analyzing script is located in the folder: ./drivers/net/mlx5/tools
It requires Python3.6, Babeltrace2 packages and it takes the only parameter
of trace data file. For example:
./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39
6. Interpreting the Script Output Data
All the timings are given in nanoseconds.
The list of Tx (and coming Rx) bursts per port/queue is presented in the output.
Each list element contains the list of built WQEs with specific opcodes, and
each WQE contains the list of the encompassed packets to send.
Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
--
v2: - comment addressed: "dump_trace" command is replaced with "save_trace"
- Windows build failure addressed, Windows does not support tracing
v3: - tracepoint routines are moved to the net folder, no need to export
- documentation added
- testpmd patches moved out from series to the dedicated patches
v4: - Python comments addressed
- codestyle issues fixed
v5: - traces are moved to the dedicated files, otherwise registration
header caused wrong code generation for 3rd party files/objects
and resulted in performance drop
Viacheslav Ovsiienko (4):
net/mlx5: introduce tracepoints for mlx5 drivers
net/mlx5: add comprehensive send completion trace
net/mlx5: add Tx datapath trace analyzing script
doc: add mlx5 datapath tracing feature description
doc/guides/nics/mlx5.rst | 78 +++++++
drivers/net/mlx5/linux/mlx5_verbs.c | 8 +-
drivers/net/mlx5/meson.build | 1 +
drivers/net/mlx5/mlx5_devx.c | 8 +-
drivers/net/mlx5/mlx5_rx.h | 19 --
drivers/net/mlx5/mlx5_rxtx.h | 19 ++
drivers/net/mlx5/mlx5_trace.c | 25 +++
drivers/net/mlx5/mlx5_trace.h | 73 +++++++
drivers/net/mlx5/mlx5_tx.c | 9 +
drivers/net/mlx5/mlx5_tx.h | 89 +++++++-
drivers/net/mlx5/tools/mlx5_trace.py | 307 +++++++++++++++++++++++++++
11 files changed, 607 insertions(+), 29 deletions(-)
create mode 100644 drivers/net/mlx5/mlx5_trace.c
create mode 100644 drivers/net/mlx5/mlx5_trace.h
create mode 100755 drivers/net/mlx5/tools/mlx5_trace.py