From patchwork Tue Jul 11 15:15:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129447 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8297742E47; Tue, 11 Jul 2023 17:15:56 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6306A40A7D; Tue, 11 Jul 2023 17:15:56 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2071.outbound.protection.outlook.com [40.107.96.71]) by mails.dpdk.org (Postfix) with ESMTP id D14EE4003C for ; Tue, 11 Jul 2023 17:15:54 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d/PxrK0cI8ojwTVSsN86FbehTttGKjXD/B159GJcPFbwOrme6mVSIXJKBsK4WGbiSEJbFxN9BvWn6N6VSuqmBx/bvIl6pO/mlfMdscK1OTt3Dqliqgw7cRzURybd0m/uHIkGahveIY9zMA8JIv59DWZJGVJ1pDXIAivrKTnYDEBPrdkDSun5zxRqSrYMsvvbNvI7GvPbwn8vpH/BgQAllASIUWJHMOJfXCTw0R5FFPin6dIsZTWof3wpNOV+WNO6BhVNqH5Kx50sFJdmzucoaXGLE2lnAOvWgMF5koZfTts7WG64mK2caRqpTgIkQISSgCrjc3SU5bphnE/AMHjDLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/DsapU1Z/lqzk/PAxwff7QOfmmrMm5Xv0ofwjQ6zr/I=; b=W+pR/rAbSZo2p416ZqtLg9uetvDgArIgw6413CJ2+IvN1BAzVojckGSks2J6EAHpd/p9pffTRz9BS42SIAf1DwqnB6UTg1pkU1cnSR3oWugAtuD4feaZ7cpU6kMbSv0WI+2ur2vIjC+WIkytJ4MOPCb6PJdAl3Groz1i3VVusOJP5IjUoFoZ9nUHiTCodg++yEN9xH9dNPMMW7i72xw9K0DoV+quTCiHZsgxYZ7XIt3cEotiC0j/hGONiaiYfG4uWFPvN64riGB+OzM/cQFDBkypb6VQSjTR+2GigF2NfWKfa4sOSbeolNfgKK8kmfFGZO20qVYMS01AKpjW5Raq4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/DsapU1Z/lqzk/PAxwff7QOfmmrMm5Xv0ofwjQ6zr/I=; b=L3DuvvXRsVDMd7Ru5aosfI4dU7wlhHyDcgFn2v3NTwmd5cHrzVqq0HfGdcT/hlmpYce1fOdldq0XbpEk1DfOkUaZfk2CEFKwCsJLm08DOlGdpEU5PNbkTDLiUZ16DC9bsWIDKCpZJRC+zTOjQuof5TQWS6A+LgtLjO57Buhy0aoCOAe4NYl7wrYxMV/FYxpJSKZIrtHqQoVo2bQ9aJPFC5FK70VgeC0vkoVR80q3QG3+uB4fzlSpDCB59PutYKQwgkqudQwdPoFeS+ac1kBtqvELrMs3ukilL8riPB6tf6vnClMukaiWSohGX1QD2nGvihNUs9hL0IE4ieYsjTBbKg== Received: from MW4PR04CA0320.namprd04.prod.outlook.com (2603:10b6:303:82::25) by CY8PR12MB7435.namprd12.prod.outlook.com (2603:10b6:930:51::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.31; Tue, 11 Jul 2023 15:15:50 +0000 Received: from CO1NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:303:82:cafe::ed) by MW4PR04CA0320.outlook.office365.com (2603:10b6:303:82::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.31 via Frontend Transport; Tue, 11 Jul 2023 15:15:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CO1NAM11FT065.mail.protection.outlook.com (10.13.174.62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.18 via Frontend Transport; Tue, 11 Jul 2023 15:15:49 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Tue, 11 Jul 2023 08:15:40 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Tue, 11 Jul 2023 08:15:38 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v6 1/2] net/mlx5: add Tx datapath trace analyzing script Date: Tue, 11 Jul 2023 18:15:20 +0300 Message-ID: <20230711151521.30175-2-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230711151521.30175-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230711151521.30175-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT065:EE_|CY8PR12MB7435:EE_ X-MS-Office365-Filtering-Correlation-Id: 5bb01694-e615-4071-5ddc-08db8221b5ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7dA8okw0UJYVPMe8YSK1l8CWst+Mcd4zpfaDcwCB0jfSDhdccvydX2LIbxV5F5OzvhJfE2dpEIDuwFt4/JSAYzBPOfRGBNQ159q7LwXpR6xqn41rmzPLjd4YbBRH83QpfegLrs2kwfhkPniiPM4V8K7yvnPRaz3zpZpjDERNb8QtG1iF+pDgKZa7lasIIlRx09XWVAZV06dFWeqGgKAvZeXyl+aDUwJT/niQngaAvY+1jCpV75WWOUR12DKcFU16E+SKg6JtYrZBLPhdOUnstXIxE0ZQIap6FRAKdrKvzIZyDLzncwZ77w5c9RZQOLRDdAG5cNmY0zafHan7R/YRhUDz7cMbPvLvZwczMgBzVXx84nxRuhlDx4JHApCIhWnzg9+UslYj3H+rg/Ckt5TI+VFLSyqM4x1MVcT4QexMsC7fMfGlidN/fCDqw5GVQknS5jIdyrXkZ8aIHGjKzrM5j5rZxpmMCm3O4oaoam8tPn+r8fI1xgzeuVdqyzYsA9GM2RXO6sTdmUs0lf3rJGXF21bQHFUQM5U4bSy21RN4WwDdtNfsqhdbCdLwQl1HztU2F5iYdQN6nKIOcko7EgLXDvwSRoqaYs75rYHtp00SHV9HvkW46ReRg40mNJl2Op+r0L9YZJ9biaDnbmPpnKMkMj8o4Llkd7F1uD6FZtH72RL96Mwrk/HF/vk8lideFd7SWLaKNSVLOFzi+rBAS3J9+SCcuM4kk2lmRGXiZ4OM2WWxlFJLCxLKE4muDf7ZljqP X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(376002)(136003)(396003)(346002)(451199021)(36840700001)(46966006)(40470700004)(82740400003)(82310400005)(86362001)(54906003)(7636003)(6916009)(2906002)(316002)(5660300002)(356005)(41300700001)(4326008)(47076005)(36860700001)(426003)(83380400001)(2616005)(36756003)(336012)(70586007)(70206006)(6286002)(26005)(186003)(16526019)(107886003)(478600001)(1076003)(40480700001)(55016003)(8676002)(7696005)(40460700003)(8936002)(6666004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2023 15:15:49.9414 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5bb01694-e615-4071-5ddc-08db8221b5ed X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT065.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7435 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Python script is intended to analyze mlx5 PMD datapath traces and report: - tx_burst routine timings - how packets are pushed to WQEs - how packet sending is completed with timings Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/tools/mlx5_trace.py | 307 +++++++++++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100755 drivers/net/mlx5/tools/mlx5_trace.py diff --git a/drivers/net/mlx5/tools/mlx5_trace.py b/drivers/net/mlx5/tools/mlx5_trace.py new file mode 100755 index 0000000000..8c1fd0a350 --- /dev/null +++ b/drivers/net/mlx5/tools/mlx5_trace.py @@ -0,0 +1,307 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2023 NVIDIA Corporation & Affiliates + +""" +Analyzing the mlx5 PMD datapath tracings +""" +import sys +import argparse +import bt2 + +PFX_TX = "pmd.net.mlx5.tx." +PFX_TX_LEN = len(PFX_TX) + + +class MlxQueue: + """Queue container object""" + + def __init__(self): + self.done_burst = [] # completed bursts + self.wait_burst = [] # waiting for completion + self.pq_id = 0 + + def log(self): + """Log all queue bursts""" + for txb in self.done_burst: + txb.log() + + +class MlxMbuf: + """Packet mbufs container object""" + + def __init__(self): + self.wqe = 0 # wqe id + self.ptr = None # first packet mbuf pointer + self.len = 0 # packet data length + self.nseg = 0 # number of segments + + def log(self): + """Log mbuf""" + out_txt = " %X: %u" % (self.ptr, self.len) + if self.nseg != 1: + out_txt += " (%d segs)" % self.nseg + print(out_txt) + + +class MlxWqe: + """WQE container object""" + + def __init__(self): + self.mbuf = [] # list of mbufs in WQE + self.wait_ts = 0 # preceding wait/push timestamp + self.comp_ts = 0 # send/recv completion timestamp + self.opcode = 0 + + def log(self): + """Log WQE""" + wqe_id = (self.opcode >> 8) & 0xFFFF + wqe_op = self.opcode & 0xFF + out_txt = " %04X: " % wqe_id + if wqe_op == 0xF: + out_txt += "WAIT" + elif wqe_op == 0x29: + out_txt += "EMPW" + elif wqe_op == 0xE: + out_txt += "TSO " + elif wqe_op == 0xA: + out_txt += "SEND" + else: + out_txt += "0x%02X" % wqe_op + if self.comp_ts != 0: + out_txt += " (%d, %d)" % (self.wait_ts, self.comp_ts - self.wait_ts) + else: + out_txt += " (%d)" % self.wait_ts + print(out_txt) + for mbuf in self.mbuf: + mbuf.log() + + def comp(self, wqe_id, wqe_ts): + """Return 0 if WQE in not completedLog WQE""" + if self.comp_ts != 0: + return 1 + cur_id = (self.opcode >> 8) & 0xFFFF + if cur_id > wqe_id: + cur_id -= wqe_id + if cur_id <= 0x8000: + return 0 + else: + cur_id = wqe_id - cur_id + if cur_id >= 0x8000: + return 0 + self.comp_ts = wqe_ts + return 1 + + +class MlxBurst: + """Packet burst container object""" + + def __init__(self): + self.wqes = [] # issued burst WQEs + self.done = 0 # number of sent/recv packets + self.req = 0 # requested number of packets + self.call_ts = 0 # burst routine invocation + self.done_ts = 0 # burst routine done + self.queue = None + + def log(self): + """Log burst""" + port = self.queue.pq_id >> 16 + queue = self.queue.pq_id & 0xFFFF + if self.req == 0: + print( + "%u: tx(p=%u, q=%u, %u/%u pkts (incomplete)" + % (self.call_ts, port, queue, self.done, self.req) + ) + else: + print( + "%u: tx(p=%u, q=%u, %u/%u pkts in %u" + % ( + self.call_ts, + port, + queue, + self.done, + self.req, + self.done_ts - self.call_ts, + ) + ) + for wqe in self.wqes: + wqe.log() + + def comp(self, wqe_id, wqe_ts): + """Return 0 if not all of WQEs in burst completed""" + wlen = len(self.wqes) + if wlen == 0: + return 0 + for wqe in self.wqes: + if wqe.comp(wqe_id, wqe_ts) == 0: + return 0 + return 1 + + +class MlxTrace: + """Trace representing object""" + + def __init__(self): + self.tx_blst = {} # current Tx bursts per CPU + self.tx_qlst = {} # active Tx queues per port/queue + self.tx_wlst = {} # wait timestamp list per CPU + + def run(self, msg_it): + """Run over gathered tracing data and build database""" + for msg in msg_it: + if not isinstance(msg, bt2._EventMessageConst): + continue + event = msg.event + if event.name.startswith(PFX_TX): + do_tx(msg, self) + # Handling of other log event cathegories can be added here + + def log(self): + """Log gathered trace database""" + for pq_id in self.tx_qlst: + queue = self.tx_qlst.get(pq_id) + queue.log() + + +def do_tx_entry(msg, trace): + """Handle entry Tx busrt""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is not None: + # continue existing burst after WAIT + return + # allocate the new burst and append to the queue + burst = MlxBurst() + burst.call_ts = msg.default_clock_snapshot.ns_from_origin + trace.tx_blst[cpu_id] = burst + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = trace.tx_qlst.get(pq_id) + if queue is None: + # queue does not exist - allocate the new one + queue = MlxQueue() + queue.pq_id = pq_id + trace.tx_qlst[pq_id] = queue + burst.queue = queue + queue.wait_burst.append(burst) + + +def do_tx_exit(msg, trace): + """Handle exit Tx busrt""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + burst.done_ts = msg.default_clock_snapshot.ns_from_origin + burst.req = event["nb_req"] + burst.done = event["nb_sent"] + trace.tx_blst.pop(cpu_id) + + +def do_tx_wqe(msg, trace): + """Handle WQE record""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + wqe = MlxWqe() + wqe.wait_ts = trace.tx_wlst.get(cpu_id) + if wqe.wait_ts is None: + wqe.wait_ts = msg.default_clock_snapshot.ns_from_origin + wqe.opcode = event["opcode"] + burst.wqes.append(wqe) + + +def do_tx_wait(msg, trace): + """Handle WAIT record""" + event = msg.event + cpu_id = event["cpu_id"] + trace.tx_wlst[cpu_id] = event["ts"] + + +def do_tx_push(msg, trace): + """Handle WQE push event""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + if not burst.wqes: + return + wqe = burst.wqes[-1] + mbuf = MlxMbuf() + mbuf.wqe = event["wqe_id"] + mbuf.ptr = event["mbuf"] + mbuf.len = event["mbuf_pkt_len"] + mbuf.nseg = event["mbuf_nb_segs"] + wqe.mbuf.append(mbuf) + + +def do_tx_complete(msg, trace): + """Handle send completion event""" + event = msg.event + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = trace.tx_qlst.get(pq_id) + if queue is None: + return + qlen = len(queue.wait_burst) + if qlen == 0: + return + wqe_id = event["wqe_id"] + wqe_ts = event["ts"] + rmv = 0 + while rmv < qlen: + burst = queue.wait_burst[rmv] + if burst.comp(wqe_id, wqe_ts) == 0: + break + rmv += 1 + # mode completed burst to done list + if rmv != 0: + idx = 0 + while idx < rmv: + queue.done_burst.append(burst) + idx += 1 + del queue.wait_burst[0:rmv] + + +def do_tx(msg, trace): + """Handle Tx related records""" + name = msg.event.name[PFX_TX_LEN:] + if name == "entry": + do_tx_entry(msg, trace) + elif name == "exit": + do_tx_exit(msg, trace) + elif name == "wqe": + do_tx_wqe(msg, trace) + elif name == "wait": + do_tx_wait(msg, trace) + elif name == "push": + do_tx_push(msg, trace) + elif name == "complete": + do_tx_complete(msg, trace) + else: + print("Error: unrecognized Tx event name: %s" % msg.event.name, file=sys.stderr) + raise ValueError() + + +def main() -> int: + """Script entry point""" + try: + parser = argparse.ArgumentParser() + parser.add_argument("path", nargs=1, type=str, help="input trace folder") + args = parser.parse_args() + + mlx_tr = MlxTrace() + msg_it = bt2.TraceCollectionMessageIterator(args.path) + mlx_tr.run(msg_it) + mlx_tr.log() + return 0 + except ValueError: + return -1 + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Tue Jul 11 15:15:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129449 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 75D7D42E47; Tue, 11 Jul 2023 17:16:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7B39642B8E; Tue, 11 Jul 2023 17:16:00 +0200 (CEST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2062.outbound.protection.outlook.com [40.107.212.62]) by mails.dpdk.org (Postfix) with ESMTP id 7778F42B8C for ; Tue, 11 Jul 2023 17:15:59 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M6pMtnR+JpWBwHLy21iuO31BITFX0Issgh5C76bjL1aSCnc7MJ0QOahq/URu9UiSm2+kBpZooJJEHmS1tTI6y1LhdoRJK8yw6zRSgB+Hh8Z9i4etbVf1UJraQb+HaRk60qJhSCMLivSv5d51YDKxW+jE0bqih/qawQsoqUF+n5Hz5BahgIQPaCV+GwguzuDiMa7wPFXzIIrlJNa1fjh4koBWSec9YBoxdUMjZLSaG4cIuSqv1it/DN2rs8z05xyZTcnXs33XCom818+v49mAGOXBZUA8hm+WKTmNiF9QmdEpowTEUp2WZcNdjsrqbCw8vkPIlOQoOe2t08nU6cAc8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ekiuEmhkhIXAob/bCijX7JcQnFsnmwJcWNJJdYyszIY=; b=H3BPeROVFet62t/ICn4UmAmikoTBUO73VbzVe9YsLquLb+BSeXy/nEXl+QfxeDUuUTMZWjq0weP0vixyl8ofwzi/5/RYrSiHE8IGJbMez7ZsqawITY0vEFIvT+jo4tJYo/nXiOpnHdFwwzzPRxLrA/lzRLJ2/ZCbrZ9mG1ecgviLasal/JmzWlnveAuUSZ82RRDATwyXqS9fotIuDGH7rYWyxu8Xb7hwS9s0z9IvKu3Yoz+RZ4Sgs+KG7rOICZDHfu+Th0PfrEzUZgL59iPvLCRoX2tL74tMtyHS9RJEKQ96QvG3H82d106mVx+Q+NLClw2hJynglJ9SH3EL1Kh1pQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ekiuEmhkhIXAob/bCijX7JcQnFsnmwJcWNJJdYyszIY=; b=UuqWrmouiNOd8MVQvAjTWu2vRvJYVDFrVqQ0zyJv8JQBob0tVwhKzf8biy41dpzrIscKm7jQnnrlIlc6qbE6pCr3hP0wusKFW7dWHbCu73Syh3RIseSt10MkZDlKlwQiismt4wRoE/XjWGT2DW//pfH1Ggb+KVPHBOBiJQR3rZaPsmp+M0lco1Pg1f032/EnX22jlJLoohpcyxpcf6E0JqA5iyfRJydC7iDvE2D0jtKkWjHl9S8ZyYUt7iFKZcG3c5u5cqZmsIPaewW9Hk2BLXfzarq0xoEwLaWV/4o0e4OpRAXjYAFSSTdYua/yKhU9frFEkiaL5IeRmHxeqE83dA== Received: from SJ2PR07CA0016.namprd07.prod.outlook.com (2603:10b6:a03:505::16) by CY8PR12MB8299.namprd12.prod.outlook.com (2603:10b6:930:6c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.31; Tue, 11 Jul 2023 15:15:55 +0000 Received: from DM6NAM11FT015.eop-nam11.prod.protection.outlook.com (2603:10b6:a03:505:cafe::9c) by SJ2PR07CA0016.outlook.office365.com (2603:10b6:a03:505::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.20 via Frontend Transport; Tue, 11 Jul 2023 15:15:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT015.mail.protection.outlook.com (10.13.172.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6588.20 via Frontend Transport; Tue, 11 Jul 2023 15:15:55 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Tue, 11 Jul 2023 08:15:41 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Tue, 11 Jul 2023 08:15:40 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v6 2/2] doc: add mlx5 datapath tracing feature description Date: Tue, 11 Jul 2023 18:15:21 +0300 Message-ID: <20230711151521.30175-3-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230711151521.30175-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230711151521.30175-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT015:EE_|CY8PR12MB8299:EE_ X-MS-Office365-Filtering-Correlation-Id: 2d56252e-ba09-44fd-628c-08db8221b91c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: x9HA23Ajce1ZLjB4Uq+WEOE4mb8WXU45gGfXuaVT4UdgZExxE+Ba3RnXZKIg3MZcmCz0HwclhRBLPRWGY1xVa+1Ku4dwSwjuFnJCqYxfGVcyrnqg7W6ECJtaosWDaLiBRdpIktt73S6uDU8TAuD/vQE04oNV87pTEATabhKTZ5U3EsJMNXNVzX0E4hqYZ7Aoab/CZ0uVACxMY/4M9a2CFCWJCWr7RPGJvnKNUKEzBOx/yxe9Wx9nA3aPA8RAOrZDxidC66DAT7Zf6rHgZJ2VZhGEVq/GLxNggoGTPJls7+DmfqjYFRcnZB8M+ExNYVBdW8hWk54vYmC0abSwYLFRbv5C5I/Y1a7u2B/+WNeg3LO2RNjwvSRpwzIx63UnM9Kt37A573fsY1NdgCRNVNBzWo+7ZdQuAFDh8vnuB2N3Ki2lxQzRKdeuuB8SvXG1+HT0NBHvzmtMyKsrbAlfYUqFyMzRnSyctjligRsxpFTLBxNuaWjMAheom0PPP9jOrxjh/1UD3A9Fycui3O/0ThDWZqi5pv5H/ejGz7fgLgjKlXZKSYdPeGlW3KUnW/nOGv8aEFLJW+2OWmadyR2LBZp/TWbVccvlu3shEhjWmYpIgPHiSTBGdotGgFzzNmLSbGB03Kbuw29G4VbNAfpVEKMYFn0/BJfp1DY2lKmr+8qJVsQejU2gDBbGUcYmBtjumEE11gDqo3H4XelKveQX+1th7VjnSP93BVtdU5pV5ZTb32pcsVtBe+mY+NtR4hq+pTDS8XbCzMUoJrJb720Qg1PAUvMoLDSZH0tcjQS12akyAZE= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(376002)(39860400002)(136003)(396003)(451199021)(40470700004)(46966006)(36840700001)(86362001)(5660300002)(8936002)(8676002)(41300700001)(316002)(82310400005)(2906002)(70206006)(70586007)(6916009)(4326008)(356005)(82740400003)(7636003)(54906003)(83380400001)(6666004)(7696005)(478600001)(36860700001)(26005)(16526019)(6286002)(186003)(2616005)(1076003)(47076005)(107886003)(966005)(40460700003)(40480700001)(36756003)(336012)(426003)(55016003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jul 2023 15:15:55.2671 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2d56252e-ba09-44fd-628c-08db8221b91c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT015.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8299 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The mlx5 provides the send scheduling on specific moment of time, and for the related kind of applications it would be extremely useful to have extra debug information - when and how packets were scheduled and when the actual sending was completed by the NIC hardware (it helps application to track the internal delay issues). The patch adds the documentation for feature usage. Signed-off-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 74 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 505873ecfd..a407920555 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1923,6 +1923,80 @@ The procedure below is an example of using a ConnectX-5 adapter card (pf0) with $ echo "0000:82:00.2" >> /sys/bus/pci/drivers/mlx5_core/bind +How to trace Tx datapath +------------------------ + +The mlx5 PMD provides Tx datapath tracing capability with extra debug information: +when and how packets were scheduled +and when the actual sending was completed by the NIC hardware. + +Steps to enable Tx datapath tracing: + +#. Build DPDK application with enabled datapath tracing + + The Meson option ``--enable_trace_fp=true`` and + the C flag ``ALLOW_EXPERIMENTAL_API`` should be specified. + + .. code-block:: console + + meson configure --buildtype=debug -Denable_trace_fp=true + -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build + +#. Configure the NIC + + If the sending completion timings are important, + the NIC should be configured to provide realtime timestamps. + The non-volatile settings parameter ``REAL_TIME_CLOCK_ENABLE`` should be configured as one. + The ``mlxconfig`` utility is part of the MFT package. + + .. code-block:: console + + mlxconfig -d /dev/mst/mt4125_pciconf0 s REAL_TIME_CLOCK_ENABLE=1 + +#. Run application with EAL parameter enabling the tracing in mlx5 Tx datapath + + By default all tracepoints are disabled. + To analyze Tx datapath and its timings: ``--trace=pmd.net.mlx5.tx``. + +#. Commit the tracing data to the storage (with ``rte_trace_save()`` API call). + +#. Install or build the ``babeltrace2`` package + + The Python script analyzing gathered trace data uses the ``babeltrace2`` library. + The package should be either installed or built from source as shown below. + + .. code-block:: console + + git clone https://github.com/efficios/babeltrace.git + cd babeltrace + ./bootstrap + ./configure -help + ./configure --disable-api-doc --disable-man-pages + --disable-python-bindings-doc --enable-python-plugins + --enable-python-binding + +#. Run analyzing script + + ``mlx5_trace.py`` is used to combine related events (packet firing and completion) + and to show the results in human-readable view. + + The analyzing script is located in the DPDK source tree: ``drivers/net/mlx5/tools``. + + It requires Python 3.6, ``babeltrace2`` package. + + The parameter of the script is the trace data folder. + + .. code-block:: console + + mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 + +#. Interpreting the script output data + + All the timings are given in nanoseconds. + The list of Tx bursts per port/queue is presented in the output. + Each list element contains the list of built WQEs with specific opcodes. + Each WQE contains the list of the encompassed packets to send. + Host shaper -----------