From patchwork Wed Jun 28 11:09:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129047 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A8A7142D7F; Wed, 28 Jun 2023 13:10:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9584A410F6; Wed, 28 Jun 2023 13:10:33 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by mails.dpdk.org (Postfix) with ESMTP id 5D04B410F6 for ; Wed, 28 Jun 2023 13:10:32 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ey4pOlWRXOrXBhj5D4TJ9cI5HRovfdZNBXFYEq0BTmsHXTQBxZLhEKNq6aOnCj9T26WvgMR6SElDwuawKn8yuBVlXANRaOvvOYSegEtUkFKtdKhulxN+ZaSEptw98HU/6Rrm6XY1uwvSRcsQrRAJdeuyJq0WKT781BNCZC+Fvp7ZwndIRvYrya4QnC8LEHc8Zlkq6a4hv+glT6oQkaaWq4nEdktNCfMteTIG0vv1vA+X048fPpqY3ELpzDRLq1GEP1aUmtaoh8+39Xh+67tyQMIocjpZ++q8Mh4BA0kububBVQClGYQkQH4s02C+9ks4mRMhb45vyCtRUiN8kXfwgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fxb3TQjm4GVID4nsJniGX7xjurzAlgh2LYU/QSDzdUc=; b=kRT3wLIzNChJIbFhHrcUkdVK21ZWo9YzojcoUthn0NgDSjaqK+z+nvEb1j5YGG1KBzTtw9Ks++UPsuVjZZmgZ5LnRXjllroCNF5PplP1Lb7phEKMarP1d8iMLWqwwMcZRYhOXCFvOhHedW0EV58ibuaPdxJM1LyorSJvvKP4wv5y01Yo9cOCS78Z7DOzCDl87+kxJYoVppZNbALYfvza5gYf6sBgDUGfHdNkJWMZoWVFjLpLZQE5/YQj38iugcHVCIzRq/TvO/bjr6vYY5adFmry8dTpQv2vRC7v9EincNPkre8CCK9gMQlUNH2yO0P6oUHyWLUto+8Jl1boGpACcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fxb3TQjm4GVID4nsJniGX7xjurzAlgh2LYU/QSDzdUc=; b=fGu8fmnjj4wZCQAR3WBNiWrFteybUdkFoMb98YM/aB2j+twsB406sFEWudXhQ/sDMRHjXWT9pO6v5cmxTQ23JErpUMQVm2lRaArssQs2iqyw5WbPPe69pdFQR0NxgVQCBi8p6PowLawjBL8j6Cg+ARuhVg9MQtX8Xg+uB+ibBrvja92l1tP9nSJivwS+M3sTiTKd8aub9MYIotiL3Kl8L3KulmXxKPWUqp7Aexpricowqerf1lxl2lWd2HP45SfMIwD8DkJQ+hAnrjEj3y62vh/qs8+r0xdA66fqW/yfvh0X0aS4X/Fa9ONx440pRl2eDNYcIYNg40fYZgq8jdzC7Q== Received: from BN8PR03CA0019.namprd03.prod.outlook.com (2603:10b6:408:94::32) by MN0PR12MB5860.namprd12.prod.outlook.com (2603:10b6:208:37b::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.26; Wed, 28 Jun 2023 11:10:30 +0000 Received: from BN8NAM11FT093.eop-nam11.prod.protection.outlook.com (2603:10b6:408:94:cafe::59) by BN8PR03CA0019.outlook.office365.com (2603:10b6:408:94::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.19 via Frontend Transport; Wed, 28 Jun 2023 11:10:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT093.mail.protection.outlook.com (10.13.177.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.48 via Frontend Transport; Wed, 28 Jun 2023 11:10:29 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 28 Jun 2023 04:10:16 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 28 Jun 2023 04:10:15 -0700 From: Viacheslav Ovsiienko To: Subject: [PATCH v3 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Date: Wed, 28 Jun 2023 14:09:55 +0300 Message-ID: <20230628110958.1403-2-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230628110958.1403-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230628110958.1403-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT093:EE_|MN0PR12MB5860:EE_ X-MS-Office365-Filtering-Correlation-Id: 8cbf8b41-a1fd-4737-8bee-08db77c8489c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RYIGB7KmdJV9T9LnmLnosvEBZVyQIaUMAbpwVd6GVdiplRUqQu/zxO08OKQ10X1QWmX4dbC89T4QPKbY0RnBQ3Ed1a0sG3JScI3R/37bHZt4cNK2AazdDB46A1jYOH95YpOVI402lTnVHC2ydujN4+YnrNrHqFwurjJ+lYNMx5WZG2+RBktT9BrT03ssF1dkSGGcizCU88g9+M5IyG4VlWihERQjQynCf5ud1Yo3MzgRV8XtxSf2o4sqynFprpE6z9cDtaJZtd+dkli/p+7lGJDnAtYzbeyv7qrxZHVMOydPCkmJf6Lg89IWHcF8oY83mga1cDOcdcUWY67PYDbO3SflgQJd4X5SiESDF+4v2TklrDYDiGmWaQtY/4YCsCgbJGit8akQ2ZEc9UbHgaommjbuzuv7asG4htlbzptkK4CGQmOnsK4HQv8aMMOK2kPhv/DgKoPbeQY05jVASGoqmnruHK7lPpQY/m2Payh8wscPgr1mhvxrseTfRV2pCEquMnyzVDwsrABwqG+wzsb/RllbzPNr1bpBvffvRpNCkc4EEq+0FK9pAVlwYuET1j6ZK0PPW1b6MApQP669nMaY+08FazX290KA2jEn9GqMvj0Ij5YhoYmrSV5YZyo8+40qfT1/5DG1057uVv4UfcdVXAen5a3qQXzaDEtUTWH9e2iUj2Z5FgsZkDS1Rie4p4kUIe4Q6+PPFKdHTNOE5JxQD3Cy0LADUtXPYbK/3eyPJbLFgVszwDL5RtFrGQXDD3x5 X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(376002)(346002)(136003)(39860400002)(451199021)(46966006)(40470700004)(36840700001)(82310400005)(1076003)(36860700001)(70206006)(40460700003)(36756003)(30864003)(5660300002)(356005)(86362001)(8936002)(8676002)(6916009)(41300700001)(316002)(70586007)(40480700001)(7636003)(82740400003)(55016003)(47076005)(2906002)(26005)(6286002)(478600001)(186003)(16526019)(2616005)(7696005)(426003)(83380400001)(336012)(6666004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2023 11:10:29.6409 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8cbf8b41-a1fd-4737-8bee-08db77c8489c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT093.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5860 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is an intention to engage DPDK tracing capabilities for mlx5 PMDs monitoring and profiling in various modes. The patch introduces tracepoints for the Tx datapath in the ethernet device driver. To engage this tracing capability the following steps should be taken: - meson option -Denable_trace_fp=true - meson option -Dc_args='-DALLOW_EXPERIMENTAL_API' - EAL command line parameter --trace=pmd.net.mlx5.tx.* The Tx datapath tracing allows to get information how packets are pushed into hardware descriptors, time stamping for scheduled wait and send completions, etc. To provide the human readable form of trace results the dedicated post-processing script is presumed. Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5_rx.h | 19 ---------- drivers/net/mlx5/mlx5_rxtx.h | 19 ++++++++++ drivers/net/mlx5/mlx5_tx.c | 29 +++++++++++++++ drivers/net/mlx5/mlx5_tx.h | 72 +++++++++++++++++++++++++++++++++++- 4 files changed, 118 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index 3514edd84e..f42607dce4 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -377,25 +377,6 @@ mlx5_rx_mb2mr(struct mlx5_rxq_data *rxq, struct rte_mbuf *mb) return mlx5_mr_mempool2mr_bh(mr_ctrl, mb->pool, addr); } -/** - * Convert timestamp from HW format to linear counter - * from Packet Pacing Clock Queue CQE timestamp format. - * - * @param sh - * Pointer to the device shared context. Might be needed - * to convert according current device configuration. - * @param ts - * Timestamp from CQE to convert. - * @return - * UTC in nanoseconds - */ -static __rte_always_inline uint64_t -mlx5_txpp_convert_rx_ts(struct mlx5_dev_ctx_shared *sh, uint64_t ts) -{ - RTE_SET_USED(sh); - return (ts & UINT32_MAX) + (ts >> 32) * NS_PER_S; -} - /** * Set timestamp in mbuf dynamic field. * diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 876aa14ae6..b109d50758 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -43,4 +43,23 @@ int mlx5_queue_state_modify_primary(struct rte_eth_dev *dev, int mlx5_queue_state_modify(struct rte_eth_dev *dev, struct mlx5_mp_arg_queue_state_modify *sm); +/** + * Convert timestamp from HW format to linear counter + * from Packet Pacing Clock Queue CQE timestamp format. + * + * @param sh + * Pointer to the device shared context. Might be needed + * to convert according current device configuration. + * @param ts + * Timestamp from CQE to convert. + * @return + * UTC in nanoseconds + */ +static __rte_always_inline uint64_t +mlx5_txpp_convert_rx_ts(struct mlx5_dev_ctx_shared *sh, uint64_t ts) +{ + RTE_SET_USED(sh); + return (ts & UINT32_MAX) + (ts >> 32) * NS_PER_S; +} + #endif /* RTE_PMD_MLX5_RXTX_H_ */ diff --git a/drivers/net/mlx5/mlx5_tx.c b/drivers/net/mlx5/mlx5_tx.c index 14e1487e59..13e2d90e03 100644 --- a/drivers/net/mlx5/mlx5_tx.c +++ b/drivers/net/mlx5/mlx5_tx.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -232,6 +233,15 @@ mlx5_tx_handle_completion(struct mlx5_txq_data *__rte_restrict txq, MLX5_ASSERT((txq->fcqs[txq->cq_ci & txq->cqe_m] >> 16) == cqe->wqe_counter); #endif + if (__rte_trace_point_fp_is_enabled()) { + uint64_t ts = rte_be_to_cpu_64(cqe->timestamp); + uint16_t wqe_id = rte_be_to_cpu_16(cqe->wqe_counter); + + if (txq->rt_timestamp) + ts = mlx5_txpp_convert_rx_ts(NULL, ts); + rte_pmd_mlx5_trace_tx_complete(txq->port_id, txq->idx, + wqe_id, ts); + } ring_doorbell = true; ++txq->cq_ci; last_cqe = cqe; @@ -752,3 +762,22 @@ mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, } return -EINVAL; } + +/* TX burst subroutines trace points. */ +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_entry, + pmd.net.mlx5.tx.entry) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_exit, + pmd.net.mlx5.tx.exit) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_wqe, + pmd.net.mlx5.tx.wqe) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_wait, + pmd.net.mlx5.tx.wait) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_push, + pmd.net.mlx5.tx.push) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_complete, + pmd.net.mlx5.tx.complete) diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h index cc8f7e98aa..b90cdf1fcc 100644 --- a/drivers/net/mlx5/mlx5_tx.h +++ b/drivers/net/mlx5/mlx5_tx.h @@ -13,12 +13,61 @@ #include #include #include +#include #include #include #include "mlx5.h" #include "mlx5_autoconf.h" +#include "mlx5_rxtx.h" + +/* TX burst subroutines trace points. */ +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_entry, + RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t queue_id), + rte_trace_point_emit_u16(port_id); + rte_trace_point_emit_u16(queue_id); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_exit, + RTE_TRACE_POINT_ARGS(uint16_t nb_sent, uint16_t nb_req), + rte_trace_point_emit_u16(nb_sent); + rte_trace_point_emit_u16(nb_req); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_wqe, + RTE_TRACE_POINT_ARGS(uint32_t opcode), + rte_trace_point_emit_u32(opcode); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_wait, + RTE_TRACE_POINT_ARGS(uint64_t ts), + rte_trace_point_emit_u64(ts); +) + + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_push, + RTE_TRACE_POINT_ARGS(const struct rte_mbuf *mbuf, uint16_t wqe_id), + rte_trace_point_emit_ptr(mbuf); + rte_trace_point_emit_u32(mbuf->pkt_len); + rte_trace_point_emit_u16(mbuf->nb_segs); + rte_trace_point_emit_u16(wqe_id); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_complete, + RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t queue_id, + uint16_t wqe_id, uint64_t ts), + rte_trace_point_emit_u16(port_id); + rte_trace_point_emit_u16(queue_id); + rte_trace_point_emit_u64(ts); + rte_trace_point_emit_u16(wqe_id); +) /* TX burst subroutines return codes. */ enum mlx5_txcmp_code { @@ -764,6 +813,9 @@ mlx5_tx_cseg_init(struct mlx5_txq_data *__rte_restrict txq, cs->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); + if (__rte_trace_point_fp_is_enabled() && !loc->pkts_sent) + rte_pmd_mlx5_trace_tx_entry(txq->port_id, txq->idx); + rte_pmd_mlx5_trace_tx_wqe((txq->wqe_ci << 8) | opcode); } /** @@ -1692,6 +1744,7 @@ mlx5_tx_schedule_send(struct mlx5_txq_data *restrict txq, if (txq->wait_on_time) { /* The wait on time capability should be used. */ ts -= sh->txpp.skew; + rte_pmd_mlx5_trace_tx_wait(ts); mlx5_tx_cseg_init(txq, loc, wqe, 1 + sizeof(struct mlx5_wqe_wseg) / MLX5_WSEG_SIZE, @@ -1706,6 +1759,7 @@ mlx5_tx_schedule_send(struct mlx5_txq_data *restrict txq, if (unlikely(wci < 0)) return MLX5_TXCMP_CODE_SINGLE; /* Build the WAIT WQE with specified completion. */ + rte_pmd_mlx5_trace_tx_wait(ts - sh->txpp.skew); mlx5_tx_cseg_init(txq, loc, wqe, 1 + sizeof(struct mlx5_wqe_qseg) / MLX5_WSEG_SIZE, @@ -1810,6 +1864,7 @@ mlx5_tx_packet_multi_tso(struct mlx5_txq_data *__rte_restrict txq, wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 0, MLX5_OPCODE_TSO, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); ds = mlx5_tx_mseg_build(txq, loc, wqe, vlan, inlen, 1, olx); wqe->cseg.sq_ds = rte_cpu_to_be_32(txq->qp_num_8s | ds); txq->wqe_ci += (ds + 3) / 4; @@ -1892,6 +1947,7 @@ mlx5_tx_packet_multi_send(struct mlx5_txq_data *__rte_restrict txq, wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, ds, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_none(txq, loc, wqe, olx); dseg = &wqe->dseg[0]; do { @@ -2115,6 +2171,7 @@ mlx5_tx_packet_multi_inline(struct mlx5_txq_data *__rte_restrict txq, wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 0, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); ds = mlx5_tx_mseg_build(txq, loc, wqe, vlan, inlen, 0, olx); wqe->cseg.sq_ds = rte_cpu_to_be_32(txq->qp_num_8s | ds); txq->wqe_ci += (ds + 3) / 4; @@ -2318,8 +2375,8 @@ mlx5_tx_burst_tso(struct mlx5_txq_data *__rte_restrict txq, */ wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; - mlx5_tx_cseg_init(txq, loc, wqe, ds, - MLX5_OPCODE_TSO, olx); + mlx5_tx_cseg_init(txq, loc, wqe, ds, MLX5_OPCODE_TSO, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_eseg_data(txq, loc, wqe, vlan, hlen, 1, olx); dptr = rte_pktmbuf_mtod(loc->mbuf, uint8_t *) + hlen - vlan; dlen -= hlen - vlan; @@ -2688,6 +2745,7 @@ mlx5_tx_burst_empw_simple(struct mlx5_txq_data *__rte_restrict txq, /* Update sent data bytes counter. */ slen += dlen; #endif + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_dseg_ptr (txq, loc, dseg, rte_pktmbuf_mtod(loc->mbuf, uint8_t *), @@ -2926,6 +2984,7 @@ mlx5_tx_burst_empw_inline(struct mlx5_txq_data *__rte_restrict txq, tlen += sizeof(struct rte_vlan_hdr); if (room < tlen) break; + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_dseg_vlan(txq, loc, dseg, dptr, dlen, olx); #ifdef MLX5_PMD_SOFT_COUNTERS @@ -2935,6 +2994,7 @@ mlx5_tx_burst_empw_inline(struct mlx5_txq_data *__rte_restrict txq, } else { if (room < tlen) break; + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_dseg_empw(txq, loc, dseg, dptr, dlen, olx); } @@ -2980,6 +3040,7 @@ mlx5_tx_burst_empw_inline(struct mlx5_txq_data *__rte_restrict txq, if (MLX5_TXOFF_CONFIG(VLAN)) MLX5_ASSERT(!(loc->mbuf->ol_flags & RTE_MBUF_F_TX_VLAN)); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_dseg_ptr(txq, loc, dseg, dptr, dlen, olx); /* We have to store mbuf in elts.*/ txq->elts[txq->elts_head++ & txq->elts_m] = loc->mbuf; @@ -3194,6 +3255,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, seg_n, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_data(txq, loc, wqe, vlan, inlen, 0, olx); txq->wqe_ci += wqe_n; @@ -3256,6 +3318,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, ds, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_eseg_data(txq, loc, wqe, vlan, txq->inlen_mode, 0, olx); @@ -3297,6 +3360,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 4, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_dmin(txq, loc, wqe, vlan, olx); dptr = rte_pktmbuf_mtod(loc->mbuf, uint8_t *) + MLX5_ESEG_MIN_INLINE_SIZE - vlan; @@ -3338,6 +3402,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 3, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_none(txq, loc, wqe, olx); mlx5_tx_dseg_ptr (txq, loc, &wqe->dseg[0], @@ -3707,6 +3772,9 @@ mlx5_tx_burst_tmpl(struct mlx5_txq_data *__rte_restrict txq, #endif if (MLX5_TXOFF_CONFIG(INLINE) && loc.mbuf_free) __mlx5_tx_free_mbuf(txq, pkts, loc.mbuf_free, olx); + /* Trace productive bursts only. */ + if (__rte_trace_point_fp_is_enabled() && loc.pkts_sent) + rte_pmd_mlx5_trace_tx_exit(loc.pkts_sent, pkts_n); return loc.pkts_sent; } From patchwork Wed Jun 28 11:09:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129049 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 46EF642D7F; Wed, 28 Jun 2023 13:10:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2B57F42BDA; Wed, 28 Jun 2023 13:10:39 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2069.outbound.protection.outlook.com [40.107.220.69]) by mails.dpdk.org (Postfix) with ESMTP id DB6D442D0E for ; Wed, 28 Jun 2023 13:10:35 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gTtc7b0PhYPB+ky1SOT0umiT/+Ksu0D9JynjMcYYkMt5GmFhy89ao7wi815mY5bhcylfH2uJWEcb8VRLYrJZdlL9k25BT5rsDBt2qvdjQR7Gt4qZXdIzT6AAyM5Zyg9WrcaurYstuKxJZOZWoKG8KP4LeFuqA8vzsqa4SDUy+VXJaILYEqrvAuAYMb2yvIEblqZFCG7UazSkuYWZr7y8Gno3C8jvLqL9xFoXh0asWQU8Ju4L+B3Qwkh5BOnk0n/vKDWxfe6VjmyiSb5S2aIvwJzeoFI1+gzxtLcNyJn9bNHIazz/gABY7qzfYfLk7iRhPMlvO1IJmGqv46HK4/6BMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kCoGTrc5xMwnxWWKK9I96Mqre17KCB1xK7f1vmMc6b8=; b=YdjxquqSKv5KjQhOz0Bu2UH11gBD68FN5Ca5wtfSKZnKgAUPpNHB/ij56U2mJad8BpOZwF3FawMISqgqy1KvqkVfA9Vybzfw+qhrLYE2OOGSgfxiNdURwuX6Vu0PaaH0Yw48LQssitaJ79bAfcgnBsAj2fFGbwU5+eW4up+CeVmIEQ/kTAD+wCnjOFwqs+nqRFV68yqfvirZSjm62fNh0bI5luaXfNUYTv2rfst1/NVvy6g9iea8jQJFId2PCskDJmgtlZbNwZ67cCx+ErR0H12xicLjYhIB+PiVei9WXFzYlRnTmSU204MsZaukrpM4VzUTvaWU7Rxya5mo+Szy6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kCoGTrc5xMwnxWWKK9I96Mqre17KCB1xK7f1vmMc6b8=; b=RLOKoRaQYz1Gx9L3VQjgWEglDGVVnJn2yv0NuCFy+ae7bAGc6I4Nprg5UdcpMUv7nnnOBNDDfd9rAaCDRnl8bf6D08ZqMzUmxOAak26U719+vdTfe4QOZKmbDiyerK13b17a9fZe/HhBZLgvV8MZsApco1loIOB9sZR8e8fbVk2xWTIADih7nvj+tFLI+LV9KjA/RGhD5G3Lc84JAgk/G71MtYvVqWC5F9qbj/Q+OJG8IPdr9Jfe2dXsz6qbEHcbkpGk5WAIqL4Xyz0qA7bUqpZr/JhglvRMMNXCGoHjq6MniwIEk6oBOYtEwV6TpaPriIhI5OXcwHsbMhb+DhkQWw== Received: from DM6PR13CA0015.namprd13.prod.outlook.com (2603:10b6:5:bc::28) by BY5PR12MB4305.namprd12.prod.outlook.com (2603:10b6:a03:213::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.26; Wed, 28 Jun 2023 11:10:32 +0000 Received: from DM6NAM11FT082.eop-nam11.prod.protection.outlook.com (2603:10b6:5:bc:cafe::ed) by DM6PR13CA0015.outlook.office365.com (2603:10b6:5:bc::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.19 via Frontend Transport; Wed, 28 Jun 2023 11:10:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DM6NAM11FT082.mail.protection.outlook.com (10.13.173.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.48 via Frontend Transport; Wed, 28 Jun 2023 11:10:32 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 28 Jun 2023 04:10:19 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 28 Jun 2023 04:10:18 -0700 From: Viacheslav Ovsiienko To: Subject: [PATCH v3 2/4] net/mlx5: add comprehensive send completion trace Date: Wed, 28 Jun 2023 14:09:56 +0300 Message-ID: <20230628110958.1403-3-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230628110958.1403-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230628110958.1403-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT082:EE_|BY5PR12MB4305:EE_ X-MS-Office365-Filtering-Correlation-Id: 233942fd-e900-4638-8bbc-08db77c84a0b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: f93pryPKeoBJIh9WF0pStsaI76Q2Sm95TmrQmGxjM0FdADuACoOVpBuU5NPE1JHKZ9eNWrzqFHsrIlNsFDjdAMqyUa23bFuhdOMzQaVZJyf7P5Av8h7QBoANtRHuj86KKz4SqJ0dv64vW/ZwxPhREBt+I/VXbTIXIymqLYuYb2n6FcGOkDZBSGbvybBjcoS1D+aMzrZWybttaf3Uei5krcd1EH+/TPt8io9P61pbrQAqld6SHf+78XgT62yQrb1AVDZuKAbw7UFCxonOKQWWFnc+bn2HJFdUZOLS8v0JgQynHGWQEEF/mH5crA9PWhaTtKTI91oohE7SBKH6qtEedXN36BV5FxM2+kGMgSxzVnSdFyUjBWG0cSuiRHWzE0Io1HFwdk/gb+S9JTQPFCUPpM/UkFZBU/Oc/A7crfSKKzkqdH9eoPr/rYmiWAp2Ep2GOyzpySLKDxUkHO6Ixj4jCv87JJ13mbXmfxauvQ+Vje8nkrqqCI9QHJUtNt39KCVGzb6JNUG00YTj95H5NOzOOJKcq/uEukAHJFSOzCjUHgCcB+Nw+HQEJKS+WxaDVZVS5hu9YGp7Y9Niw0DH25+GJkdjOuwu3xDIHskchBj438qHibuKBidWxexLyqx9eYEi9+s1yOnwmUIlDNoINHH9GxlgrzMM0kPPN9QszAMJeM41Rt4dcLaUj7AI2JnqDs3ZCgsUQUQyMmCu0cr4sHgB6k4SaYqIVwhl40Qik4102Av27YC3P9py6sOyZiWrg6il X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(376002)(396003)(39860400002)(136003)(346002)(451199021)(36840700001)(46966006)(40470700004)(83380400001)(336012)(426003)(47076005)(8936002)(6666004)(316002)(7696005)(8676002)(41300700001)(40460700003)(36860700001)(36756003)(40480700001)(55016003)(82740400003)(86362001)(7636003)(356005)(82310400005)(2616005)(5660300002)(6916009)(478600001)(70206006)(2906002)(6286002)(186003)(16526019)(70586007)(26005)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2023 11:10:32.1086 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 233942fd-e900-4638-8bbc-08db77c84a0b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT082.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4305 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is the demand to trace the send completions of every WQE if time scheduling is enabled. The patch extends the size of completion queue and requests completion on every issued WQE in the send queue. As the result hardware provides CQE on each completed WQE and driver is able to fetch completion timestamp for dedicated operation. The add code is under conditional compilation RTE_ENABLE_TRACE_FP flag and does not impact the release code. Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/linux/mlx5_verbs.c | 8 +++- drivers/net/mlx5/mlx5_devx.c | 8 +++- drivers/net/mlx5/mlx5_tx.h | 63 +++++++++++++++++++++++++++-- 3 files changed, 71 insertions(+), 8 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 7233c2c7fa..b54f3ccd9a 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -968,8 +968,12 @@ mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx) rte_errno = EINVAL; return -rte_errno; } - cqe_n = desc / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + if (__rte_trace_point_fp_is_enabled() && + txq_data->offloads & RTE_ETH_TX_OFFLOAD_SEND_ON_TIMESTAMP) + cqe_n = UINT16_MAX / 2 - 1; + else + cqe_n = desc / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; txq_obj->cq = mlx5_glue->create_cq(priv->sh->cdev->ctx, cqe_n, NULL, NULL, 0); if (txq_obj->cq == NULL) { diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 4369d2557e..5082a7e178 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1465,8 +1465,12 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) MLX5_ASSERT(ppriv); txq_obj->txq_ctrl = txq_ctrl; txq_obj->dev = dev; - cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + if (__rte_trace_point_fp_is_enabled() && + txq_data->offloads & RTE_ETH_TX_OFFLOAD_SEND_ON_TIMESTAMP) + cqe_n = UINT16_MAX / 2 - 1; + else + cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; log_desc_n = log2above(cqe_n); cqe_n = 1UL << log_desc_n; if (cqe_n > UINT16_MAX) { diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h index b90cdf1fcc..47ee8bca4f 100644 --- a/drivers/net/mlx5/mlx5_tx.h +++ b/drivers/net/mlx5/mlx5_tx.h @@ -775,6 +775,54 @@ mlx5_tx_request_completion(struct mlx5_txq_data *__rte_restrict txq, } } +/** + * Set completion request flag for all issued WQEs. + * This routine is intended to be used with enabled fast path tracing + * and send scheduling on time to provide the detailed report in trace + * for send completions on every WQE. + * + * @param txq + * Pointer to TX queue structure. + * @param loc + * Pointer to burst routine local context. + * @param olx + * Configured Tx offloads mask. It is fully defined at + * compile time and may be used for optimization. + */ +static __rte_always_inline void +mlx5_tx_request_completion_trace(struct mlx5_txq_data *__rte_restrict txq, + struct mlx5_txq_local *__rte_restrict loc, + unsigned int olx) +{ + uint16_t head = txq->elts_comp; + + while (txq->wqe_comp != txq->wqe_ci) { + volatile struct mlx5_wqe *wqe; + uint32_t wqe_n; + + MLX5_ASSERT(loc->wqe_last); + wqe = txq->wqes + (txq->wqe_comp & txq->wqe_m); + if (wqe == loc->wqe_last) { + head = txq->elts_head; + head += MLX5_TXOFF_CONFIG(INLINE) ? + 0 : loc->pkts_sent - loc->pkts_copy; + txq->elts_comp = head; + } + /* Completion request flag was set on cseg constructing. */ +#ifdef RTE_LIBRTE_MLX5_DEBUG + txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head | + (wqe->cseg.opcode >> 8) << 16; +#else + txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head; +#endif + /* A CQE slot must always be available. */ + MLX5_ASSERT((txq->cq_pi - txq->cq_ci) <= txq->cqe_s); + /* Advance to the next WQE in the queue. */ + wqe_n = rte_be_to_cpu_32(wqe->cseg.sq_ds) & 0x3F; + txq->wqe_comp += RTE_ALIGN(wqe_n, 4) / 4; + } +} + /** * Build the Control Segment with specified opcode: * - MLX5_OPCODE_SEND @@ -801,7 +849,7 @@ mlx5_tx_cseg_init(struct mlx5_txq_data *__rte_restrict txq, struct mlx5_wqe *__rte_restrict wqe, unsigned int ds, unsigned int opcode, - unsigned int olx __rte_unused) + unsigned int olx) { struct mlx5_wqe_cseg *__rte_restrict cs = &wqe->cseg; @@ -810,8 +858,12 @@ mlx5_tx_cseg_init(struct mlx5_txq_data *__rte_restrict txq, opcode = MLX5_OPCODE_TSO | MLX5_OPC_MOD_MPW << 24; cs->opcode = rte_cpu_to_be_32((txq->wqe_ci << 8) | opcode); cs->sq_ds = rte_cpu_to_be_32(txq->qp_num_8s | ds); - cs->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << - MLX5_COMP_MODE_OFFSET); + if (MLX5_TXOFF_CONFIG(TXPP) && __rte_trace_point_fp_is_enabled()) + cs->flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); + else + cs->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << + MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); if (__rte_trace_point_fp_is_enabled() && !loc->pkts_sent) rte_pmd_mlx5_trace_tx_entry(txq->port_id, txq->idx); @@ -3709,7 +3761,10 @@ mlx5_tx_burst_tmpl(struct mlx5_txq_data *__rte_restrict txq, if (unlikely(loc.pkts_sent == loc.pkts_loop)) goto burst_exit; /* Request CQE generation if limits are reached. */ - mlx5_tx_request_completion(txq, &loc, olx); + if (MLX5_TXOFF_CONFIG(TXPP) && __rte_trace_point_fp_is_enabled()) + mlx5_tx_request_completion_trace(txq, &loc, olx); + else + mlx5_tx_request_completion(txq, &loc, olx); /* * Ring QP doorbell immediately after WQE building completion * to improve latencies. The pure software related data treatment From patchwork Wed Jun 28 11:09:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129048 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 40CE442D7F; Wed, 28 Jun 2023 13:10:45 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 08E3842BFE; Wed, 28 Jun 2023 13:10:37 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2079.outbound.protection.outlook.com [40.107.237.79]) by mails.dpdk.org (Postfix) with ESMTP id AB89642D0D for ; Wed, 28 Jun 2023 13:10:35 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C2hpnoRAC8nSz/dzzM2sbUa6d7lyi2qTjgQWtqh/JR+zwpcjE9HoJtSsYNS/1d5txbayABC23B5+iWnr0M6Yms++8hVYWK0TctyUlbncPSF+vabEiXcJMv8cggFZvfFralWzbyqmo9NjqxrcVD2IJgnFlhxmZ+CV2/oMBr7LxOos7xuGhYrLaSGVZpvCHaaxL2jF0+8TnjklkNcaFt6vjffA6jHuLuXENz9I1FrAaCli6cNidq8C1dFQFkF+el1buHeLBj9oncSkcr5bre97ADtvJvlNkOXU5GSVLJapHNF97X+y/RHIqYNinKhWVyCNELggJP79S/WaRqtTOCoORg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MYiacIuwMsvJ4xeh+TMKQLFoa38kzzHwJ+42WXNm0ZU=; b=BuzgUDoBxEdhdxtBUD9B/bp6c9EnPi9rabFLZTCgYsBCY3OkYIFcXLuRFQ/30eO9g7fPoqZ+z8TRicwHt7GpkJ4XELB2gUTP5nJNHfB78QHZldzOB80MFumllcOY0hgvpKFkzdYJWmYRVvdhPuGz41jralfmz8QP3PkCdSH0zMUE2xkg+Nds6K6BPN9cWJ68+MNQoc39+CpiemKM8TDeFNXk4afYu9IkxZKuCUf6sucKQwW80FZuzn5twgHfP3WZASghp1qYSqhc7VcZVAGsjISeRL85sjY3aKyu0oIAeOT+fEUNTV21dRhF90KxL6cv3u911YIpV8VeaqAgFy34nw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MYiacIuwMsvJ4xeh+TMKQLFoa38kzzHwJ+42WXNm0ZU=; b=HeTcOgSj930xyaaBe30LATj0RISwFaH2Wtl1VhrQxdJCyGTt3zI5ICJ4Mp18LKAHIUgUvPCvIUNgoiPCQT/wbk0LU+Ghq7S3LUIMshVwUiODpcueDvLiSIlMi3ZIiOsT7yjFw1VTB9+r+oRoYluG3OiKM5IcRPzRV7VQiCmDr06wPp4otd/P8jlIcIspb0vYejvmy1WVyDMDnxX8DWMWtIDwE+Pfw+pDWMQLvYxY3JRgi3/1yvRis0DYFRagfcyJHhBfrziNAdCOcfpAnPCGj2ereITQv/QASDxcpjZnI2Zdv1UVS6NtYqkFhKNFF/Rg5ILfDxJL/ADFAevKwGIWew== Received: from BN8PR04CA0024.namprd04.prod.outlook.com (2603:10b6:408:70::37) by SN7PR12MB6861.namprd12.prod.outlook.com (2603:10b6:806:266::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.24; Wed, 28 Jun 2023 11:10:34 +0000 Received: from BN8NAM11FT108.eop-nam11.prod.protection.outlook.com (2603:10b6:408:70:cafe::7b) by BN8PR04CA0024.outlook.office365.com (2603:10b6:408:70::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.34 via Frontend Transport; Wed, 28 Jun 2023 11:10:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT108.mail.protection.outlook.com (10.13.176.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.47 via Frontend Transport; Wed, 28 Jun 2023 11:10:33 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 28 Jun 2023 04:10:20 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 28 Jun 2023 04:10:19 -0700 From: Viacheslav Ovsiienko To: Subject: [PATCH v3 3/4] net/mlx5: add Tx datapath trace analyzing script Date: Wed, 28 Jun 2023 14:09:57 +0300 Message-ID: <20230628110958.1403-4-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230628110958.1403-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230628110958.1403-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT108:EE_|SN7PR12MB6861:EE_ X-MS-Office365-Filtering-Correlation-Id: e0bb871c-4d8c-4369-00e2-08db77c84afe X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m9dA1zbFUkqzsv8qZgGVEPNev+OpPEBJPQUmNrRYxZiqmICy5a8E3YqZv1aeg7wJog5ZMginDvaSiwL+ufMms8pD081t0YJEjLC8gTO10wtAhnfpZvGU0hFOG2h4e1uvODTYkxXZceSZZRrAHRoQBQZdQnDYdrcANskwPkbiZV+J9wZpkUncyisBIN0DkA5T93TpfCrpcpYhV5EYoU54WrYSgnF/Hhe4lb2vR47vX3vwqf+ulAr+6cAbZsU9mwWHJopid7pbc36FnKoCvH+MLKMb57Q8dE0XcjC7DXS3o3G0EuPSdJfK9nPpTbbIOdxTBzaV/xr4zSxkJfgeAM0W9HsdcpOEvpOf779ewAzSrstzSQEHy6JP3xsxVUwMAWD20HyjYUWceEgihzA+z5A8aL0xx5l5pLp1UdoaQHj2LnD/3yzuFff0oUOnSEEIJsDU5wbAQLwiz5sA/DMDV0oGaep1VM48ZqZsxxVq0dRTKFQY4F05q9tDDBEUrkov+IIkErN+k56ALSGcAuo4rmJESMaIBjLoUC5CeWdJI1sSrxyotS8FzqlKGe8TcbVM9aOpb81pooGBz6WvikwvmQ4c/SRbgqQw0H8W9+8SlcFUYBbPwvryzgMzrhOtP1IvaKpSOsB8NDyFqLM//Dqpd5dgkAEtLoXO82LnVhVJST5EVtlOou+GhYeMv2LySsu5Uovh2izhvz5wetImbmPEj6VSe8ew8BgfPrnIfu1iorfj9xYlrO/dFlTrYxuQZVAzZEHV X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(136003)(376002)(346002)(451199021)(36840700001)(46966006)(40470700004)(5660300002)(6916009)(70206006)(478600001)(36756003)(316002)(70586007)(8936002)(8676002)(2906002)(40460700003)(55016003)(36860700001)(41300700001)(82310400005)(186003)(7696005)(40480700001)(336012)(86362001)(26005)(16526019)(47076005)(6666004)(426003)(6286002)(356005)(1076003)(2616005)(7636003)(82740400003)(83380400001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2023 11:10:33.6350 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e0bb871c-4d8c-4369-00e2-08db77c84afe X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT108.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6861 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Python script is intended to analyze mlx5 PMD datapath traces and report: - tx_burst routine timings - how packets are pushed to WQEs - how packet sending is completed with timings Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/tools/mlx5_trace.py | 271 +++++++++++++++++++++++++++ 1 file changed, 271 insertions(+) create mode 100755 drivers/net/mlx5/tools/mlx5_trace.py diff --git a/drivers/net/mlx5/tools/mlx5_trace.py b/drivers/net/mlx5/tools/mlx5_trace.py new file mode 100755 index 0000000000..c8fa63a7b9 --- /dev/null +++ b/drivers/net/mlx5/tools/mlx5_trace.py @@ -0,0 +1,271 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2023 NVIDIA Corporation & Affiliates + +''' +Analyzing the mlx5 PMD datapath tracings +''' +import sys +import argparse +import pathlib +import bt2 + +PFX_TX = "pmd.net.mlx5.tx." +PFX_TX_LEN = len(PFX_TX) + +tx_blst = {} # current Tx bursts per CPU +tx_qlst = {} # active Tx queues per port/queue +tx_wlst = {} # wait timestamp list per CPU + +class mlx5_queue(object): + def __init__(self): + self.done_burst = [] # completed bursts + self.wait_burst = [] # waiting for completion + self.pq_id = 0 + + def log(self): + for txb in self.done_burst: + txb.log() + + +class mlx5_mbuf(object): + def __init__(self): + self.wqe = 0 # wqe id + self.ptr = None # first packet mbuf pointer + self.len = 0 # packet data length + self.nseg = 0 # number of segments + + def log(self): + out = " %X: %u" % (self.ptr, self.len) + if self.nseg != 1: + out += " (%d segs)" % self.nseg + print(out) + + +class mlx5_wqe(object): + def __init__(self): + self.mbuf = [] # list of mbufs in WQE + self.wait_ts = 0 # preceding wait/push timestamp + self.comp_ts = 0 # send/recv completion timestamp + self.opcode = 0 + + def log(self): + id = (self.opcode >> 8) & 0xFFFF + op = self.opcode & 0xFF + fl = self.opcode >> 24 + out = " %04X: " % id + if op == 0xF: + out += "WAIT" + elif op == 0x29: + out += "EMPW" + elif op == 0xE: + out += "TSO " + elif op == 0xA: + out += "SEND" + else: + out += "0x%02X" % op + if self.comp_ts != 0: + out += " (%d, %d)" % (self.wait_ts, self.comp_ts - self.wait_ts) + else: + out += " (%d)" % self.wait_ts + print(out) + for mbuf in self.mbuf: + mbuf.log() + + # return 0 if WQE in not completed + def comp(self, wqe_id, ts): + if self.comp_ts != 0: + return 1 + id = (self.opcode >> 8) & 0xFFFF + if id > wqe_id: + id -= wqe_id + if id <= 0x8000: + return 0 + else: + id = wqe_id - id + if id >= 0x8000: + return 0 + self.comp_ts = ts + return 1 + + +class mlx5_burst(object): + def __init__(self): + self.wqes = [] # issued burst WQEs + self.done = 0 # number of sent/recv packets + self.req = 0 # requested number of packets + self.call_ts = 0 # burst routine invocation + self.done_ts = 0 # burst routine done + self.queue = None + + def log(self): + port = self.queue.pq_id >> 16 + queue = self.queue.pq_id & 0xFFFF + if self.req == 0: + print("%u: tx(p=%u, q=%u, %u/%u pkts (incomplete)" % + (self.call_ts, port, queue, self.done, self.req)) + else: + print("%u: tx(p=%u, q=%u, %u/%u pkts in %u" % + (self.call_ts, port, queue, self.done, self.req, + self.done_ts - self.call_ts)) + for wqe in self.wqes: + wqe.log() + + # return 0 if not all of WQEs in burst completed + def comp(self, wqe_id, ts): + wlen = len(self.wqes) + if wlen == 0: + return 0 + for wqe in self.wqes: + if wqe.comp(wqe_id, ts) == 0: + return 0 + return 1 + + +def do_tx_entry(msg): + event = msg.event + cpu_id = event["cpu_id"] + burst = tx_blst.get(cpu_id) + if burst is not None: + # continue existing burst after WAIT + return + # allocate the new burst and append to the queue + burst = mlx5_burst() + burst.call_ts = msg.default_clock_snapshot.ns_from_origin + tx_blst[cpu_id] = burst + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = tx_qlst.get(pq_id) + if queue is None: + # queue does not exist - allocate the new one + queue = mlx5_queue(); + queue.pq_id = pq_id + tx_qlst[pq_id] = queue + burst.queue = queue + queue.wait_burst.append(burst) + + +def do_tx_exit(msg): + event = msg.event + cpu_id = event["cpu_id"] + burst = tx_blst.get(cpu_id) + if burst is None: + return + burst.done_ts = msg.default_clock_snapshot.ns_from_origin + burst.req = event["nb_req"] + burst.done = event["nb_sent"] + tx_blst.pop(cpu_id) + + +def do_tx_wqe(msg): + event = msg.event + cpu_id = event["cpu_id"] + burst = tx_blst.get(cpu_id) + if burst is None: + return + wqe = mlx5_wqe() + wqe.wait_ts = tx_wlst.get(cpu_id) + if wqe.wait_ts is None: + wqe.wait_ts = msg.default_clock_snapshot.ns_from_origin + wqe.opcode = event["opcode"] + burst.wqes.append(wqe) + + +def do_tx_wait(msg): + event = msg.event + cpu_id = event["cpu_id"] + tx_wlst[cpu_id] = event["ts"] + + +def do_tx_push(msg): + event = msg.event + cpu_id = event["cpu_id"] + burst = tx_blst.get(cpu_id) + if burst is None: + return + if not burst.wqes: + return + wqe = burst.wqes[-1] + mbuf = mlx5_mbuf() + mbuf.wqe = event["wqe_id"] + mbuf.ptr = event["mbuf"] + mbuf.len = event["mbuf_pkt_len"] + mbuf.nseg = event["mbuf_nb_segs"] + wqe.mbuf.append(mbuf) + + +def do_tx_complete(msg): + event = msg.event + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = tx_qlst.get(pq_id) + if queue is None: + return + qlen = len(queue.wait_burst) + if qlen == 0: + return + wqe_id = event["wqe_id"] + ts = event["ts"] + rmv = 0 + while rmv < qlen: + burst = queue.wait_burst[rmv] + if burst.comp(wqe_id, ts) == 0: + break + rmv += 1 + # mode completed burst to done list + if rmv != 0: + idx = 0 + while idx < rmv: + queue.done_burst.append(burst) + idx += 1 + del queue.wait_burst[0:rmv] + + +def do_tx(msg): + name = msg.event.name[PFX_TX_LEN:] + if name == "entry": + do_tx_entry(msg) + elif name == "exit": + do_tx_exit(msg) + elif name == "wqe": + do_tx_wqe(msg) + elif name == "wait": + do_tx_wait(msg) + elif name == "push": + do_tx_push(msg) + elif name == "complete": + do_tx_complete(msg) + else: + print("Error: unrecognized Tx event name: %s" % msg.event.name) + sys.exit(1) + + +def do_log(msg_it): + for msg in msg_it: + if type(msg) is not bt2._EventMessageConst: + continue + event = msg.event + if event.name.startswith(PFX_TX): + do_tx(msg) + # Handling of other log event cathegories can be added here + + +def do_print(): + for pq_id in tx_qlst: + queue = tx_qlst.get(pq_id) + queue.log() + + +def main(args): + parser = argparse.ArgumentParser() + parser.add_argument("path", + nargs = 1, + type = str, + help = "input trace folder") + args = parser.parse_args() + + msg_it = bt2.TraceCollectionMessageIterator(args.path) + do_log(msg_it) + do_print() + exit(0) + +if __name__ == "__main__": + main(sys.argv) From patchwork Wed Jun 28 11:09:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129050 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E8C7642D7F; Wed, 28 Jun 2023 13:10:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 540FB42D2C; Wed, 28 Jun 2023 13:10:42 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2063.outbound.protection.outlook.com [40.107.243.63]) by mails.dpdk.org (Postfix) with ESMTP id BFF5142BDA for ; Wed, 28 Jun 2023 13:10:38 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YkzVVMVDTMqAB/n3R9m8eTIqKAyhTCBsMZ1OmJUpjTkHeIeG9ptktTvZ6e6mM976C6rCrG+aOtNuUQrD7/ZPvs5hXT3M6uh65g9EUFN+8cbzRsy99Zj0Uy2uY0ri81e52/ShW9l4RwPOGn/KBEYhp1f2UWEzYDBCJjjHNc7DCJCCw8CNwD6gLKApSzgRMwYjdIkqKn9NQ0TOXSUvEt0Mx+tMrqeAWK/0c8hMsa3p0/mREUVHkydaWWvMte0GQCbrqdhS9bJZtZqqxAZ0uGtNazbsHuSsg0LabNvFMLZrcfq1lIHwfuARz60EKdHCfdcq0gWcWSrCYYL0fw+Xk8PW3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Mnfie/REdgpuW91CbhKBwim+GyE4PDrDM5wQP+fa24Y=; b=bx/zcf4eNq6BQZcPsVr3Ly+FLQrKieQQR+X3IIw6OgMX8PMDnvjIp+qgU8ctsWRBsaXOCEQJUL3zoY5XVQ6tj1fdah0bAEcdUWt1i/ZQn7thjXJ1NLoKPxWH3T5MWuQnJ5/B+zTNu0jB24m4HEBohsZ5RN0kO1MNEYuyDBevvd5RiV66qhhlj66wdzp8JSx7oA6MUst23gCELjZCPLbE0RwTQAQq1ON5N2WMSy2NyhurzmjBZfBFzubfvv5s1J6os2OxAtFntweWoxphoN2dKV4z5IZYq/CJV9BXLUNoXAeFbWxjvkWbIw1eipTVf1hS4PTz7BeH3cKvxRZmfZdRqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Mnfie/REdgpuW91CbhKBwim+GyE4PDrDM5wQP+fa24Y=; b=Sz3R2onR4EYEkfL4k1KCcQOLiuCF0XH/wmw09cuGa4VnNXbc7DPikGSGk1tN28ga1abLeWPV9kIyFJn1BDYbLEvp5wZEW5J5DOUSW++YFL7ZCP8RGzn33u50xiNrpAtrHgD804g6mx2p94GmsSfv/i3qOm3JASchl59i+GB9w+rtJDvX5eV+emhvhSdBfekdEqtbTIlxJ+GP/d6ZYVMhVzPsDZSaEJp5gprsbdDf/Av2wi/PgjAcpscmd2Id0xmoad7lPyWU2r2tIPU7AITXDWJ3AhZG5TtZ434RkFKO9rtwmYsfthStjZEZaL17Lok6jY8w5SEUeFRdMZZFfTHjBQ== Received: from BN8PR03CA0028.namprd03.prod.outlook.com (2603:10b6:408:94::41) by DM6PR12MB4880.namprd12.prod.outlook.com (2603:10b6:5:1bc::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.24; Wed, 28 Jun 2023 11:10:37 +0000 Received: from BN8NAM11FT093.eop-nam11.prod.protection.outlook.com (2603:10b6:408:94:cafe::8e) by BN8PR03CA0028.outlook.office365.com (2603:10b6:408:94::41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.19 via Frontend Transport; Wed, 28 Jun 2023 11:10:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT093.mail.protection.outlook.com (10.13.177.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.48 via Frontend Transport; Wed, 28 Jun 2023 11:10:36 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 28 Jun 2023 04:10:23 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 28 Jun 2023 04:10:22 -0700 From: Viacheslav Ovsiienko To: Subject: [PATCH v3 4/4] doc: add mlx5 datapath tracing feature description Date: Wed, 28 Jun 2023 14:09:58 +0300 Message-ID: <20230628110958.1403-5-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230628110958.1403-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230628110958.1403-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT093:EE_|DM6PR12MB4880:EE_ X-MS-Office365-Filtering-Correlation-Id: 57457118-5185-4f96-6204-08db77c84ccd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DARtcftQxj2D9ug6/szMCexJBLNUWcKulcpuJUIh2wbPJ6m1ccfPupcRGtD/sSh9mr6w9Miy+kPtkCxGH/T1b6svyUbGPkUJhZ4YjXrmDByADlIOLZhOhkRzXMOmcREmKrphL3F0MwaR/OpCUBofmwyakCEWVYm7HbEPqKuxyQqo1u/S85F21yV4jsSAOmxNzwwtBkKlkK8fcoNqWeWow4G2tIrWDhmkErtCWmnCL6W/Bec/YINd7tf0O3RONFlrL4Rlb0Q5AZ+auLbBaO4z4VkGhyN2ovhbE2ikIuY2cLs9aP4IqfAlLHUjjPcylcXVlRqHAGbLIOEhwQXkG990LqqZJiwi1VBGpk38zSTi5E3Rwy/qWUQLZUyfpwIHn7v4qbwUnD8VqnuOwlA5tjbFObrxuZIeQY19sqnZCnIEMjX4mLtNZ5SFs5QWWlINMaNANXdUTsx3JR2IC9d6zH745u9lOdSwGd3sqfzJoofOHHCnYFRDa3VRbubmk+PJk+kTQF8T02De9191XDFPXggCfnPrDLHu+Y+maSh0/iR151tXaO5aAXOAH2e0M80MSp6CbmdmKtFS/alDd01ah/JyEBWMsESNdT7wiKNjxYr1KRR/5p3PHx/Enu+WApwGDwmgOKYBF0Niuyeo1ZMkoi73iurz2u5zgs5UgLDvB9Jlge17uQ7/gVCD5QYog747iEAz1PryZc0x6HO7XNDBi+/ZrYMYF5nzUMacC+uKXRZbxCwEM+B9WXkHUDj/n2I2TFcOrxLbHJoUCxn5Rw+4YhottreZ1m08CLJsAOz9A8+dDXg= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(346002)(376002)(136003)(396003)(39860400002)(451199021)(36840700001)(46966006)(40470700004)(82310400005)(41300700001)(36860700001)(70206006)(966005)(47076005)(26005)(6666004)(83380400001)(7696005)(426003)(2616005)(336012)(478600001)(186003)(16526019)(6286002)(2906002)(1076003)(356005)(36756003)(40460700003)(5660300002)(70586007)(55016003)(82740400003)(7636003)(40480700001)(316002)(6916009)(8676002)(8936002)(86362001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2023 11:10:36.6718 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 57457118-5185-4f96-6204-08db77c84ccd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT093.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4880 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The mlx5 provides the send scheduling on specific moment of time, and for the related kind of applications it would be extremely useful to have extra debug information - when and how packets were scheduled and when the actual sending was completed by the NIC hardware (it helps application to track the internal delay issues). The patch adds the documentation for feature usage. Signed-off-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 77 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 0ed5cb5bc3..555f02ad2a 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -2081,3 +2081,80 @@ Set the flow engine to active(0) or standby(1) mode with specific flags:: This command works for software steering only. Default FDB jump should be disabled if switchdev is enabled. The mode will propagate to all the probed ports. + +Tx datapath tracing +^^^^^^^^^^^^^^^^^^^ + +The mlx5 provides the Tx datapath tracing capability with extra debug +information - when and how packets were scheduled and when the actual +sending was completed by the NIC hardware. The feature engages the +exisiting DPDK datapath tracing capability. + +Usage of the mlx5 Tx datapath tracing: + +#. Build DPDK application with enabled datapath tracking + + * The meson option should be specified: ``--enable_trace_fp=true`` + * The c_args shoudl be specified: ``-DALLOW_EXPERIMENTAL_API`` + + .. code-block:: console + + meson configure --buildtype=debug -Denable_trace_fp=true + -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build + + meson configure --buildtype=release -Denable_trace_fp=true + -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build + +#. Configure the NIC + + If the sending completion timings are important the NIC should be configured + to provide realtime timestamps, the ``REAL_TIME_CLOCK_ENABLE`` NV settings + parameter should be configured to TRUE. + + .. code-block:: console + + mlxconfig -d /dev/mst/mt4125_pciconf0 s REAL_TIME_CLOCK_ENABLE=1 + +#. Run application with EAL parameters configuring the tracing in mlx5 Tx datapath + + * ``--trace=pmd.net.mlx5.tx`` - the regular expression enabling the tracepoints + with matching names at least "pmd.net.mlx5.tx" must be enabled to gather all + events needed to analyze mlx5 Tx datapath and its timings. By default all + tracepoints are disabled. + +#. Store the tracing data file with gathered tracing information + +#. Install or build the ``Babeltrace2`` Package + + The gathered trace data can be analyzed with a developed Python script. + To parse the trace, the data script uses the ``Babeltrace2`` library. + The package should be either installed or built from source code as + shown below. + + .. code-block:: console + + git clone https://github.com/efficios/babeltrace.git + cd babeltrace + ./bootstrap + ./configure -help + ./configure --disable-api-doc --disable-man-pages + --disable-python-bindings-doc --enbale-python-plugins + --enable-python-binding + +#. Run analyzing scrypt (in Python) to combine related events (packet firing and + completion) and see the output in human-readable view + + The analyzing script is located in the folder: ``./drivers/net/mlx5/tools`` + It requires Python3.6, ``Babeltrace2`` packages and it takes the only parameter + of trace data file. + + .. code-block:: console + + ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 + +#. Interpreting the Script Output Data + + All the timings are given in nanoseconds. + The list of Tx bursts per port/queue is presented in the output. + Each list element contains the list of built WQEs with specific opcodes, and + each WQE contains the list of the encompassed packets to send.