From patchwork Wed Jul 5 11:10:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129291 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 60CF042DD9; Wed, 5 Jul 2023 13:11:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9906142BDA; Wed, 5 Jul 2023 13:11:15 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2053.outbound.protection.outlook.com [40.107.237.53]) by mails.dpdk.org (Postfix) with ESMTP id 532F540ED8 for ; Wed, 5 Jul 2023 13:11:13 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q4ZponRiun3V8fxtkdlJxpAQuCBvtP8UnsZj84pPOpZw/LrWSdi8IlBnBJD42DcPnZjhysiZyy3Yj9L0n+5kLeOxWfZnNE3nOu9E1kIcGfgF3PKX4K9ORmDjaxZQZqbCIi+0x4dhp3JihOPlol/Bxntp53LFNCei7DvfwVh6Med26vRYgqFFxTPDG50oZwJs27UfXGDvWsN8ZhWeTV3g+IL2X6/hduM4AKodkd0ElVxqjh4ObWTQ1SjS91dfFw3XWP/KDS7FN0YHT3N67w4w8ZbtJ5XbiE3dFL0u5BnB55YJSpszBtJn5cHA27rWATMOG8iXw+509Mj8dJCUVnEUvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fxb3TQjm4GVID4nsJniGX7xjurzAlgh2LYU/QSDzdUc=; b=YSWE/JqEJwD7A1+dQByWc1sQklLVJwT/g4+21Xld5Xk6/oGaaQ5lJvgWfRugML6c2fFuU4kUlBu984te/6YEbCCrWi/9KZMv2BmdewN7OIezlIbTMLIeHfxiKdbXwE5aaPB7Mn0mUoK0/k1VFUrEAGRADm4G1hAn+AlzMcEsVPCSxrhIivpTCEMfZ0rqkqq8LTPznce1rA6XZ9JCNcejuvBNJuL4dTzTULBdL2DjnnQR1i+ENb2MdNw4u1P61DcaZKF/cxRNrYs4YZ9g1FmTb8PMoTF3jqsLJr4nZhWwADQgJPBTujct71QmElsivtYKoSCTImEdIzg41MdeXK8OpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fxb3TQjm4GVID4nsJniGX7xjurzAlgh2LYU/QSDzdUc=; b=GRGYz7GNSg90x0tRoMJQvAPjovRAdZX1FwhgryypmUOUxv1KZGx66itMZIMVJxi7PCGlD+/WXiC+MNS6IU2CbXOBeI/qndMMQb8iEEaTzPHoZmlsOp5MVFXMvlwTagnzrdBgsFVf9/h6H7l6Ng6DwHUzhL3bJ/8SiNaB+q7u3lgfjBX5e0lJxR2XZOGO7/Ss6x+pW/fUD5eCBf2nKEu648LxPNteom+7OBaj702mW43qcB+uiKjl5p3iCLzD9JIGQRlJse5zUEmXEY5uc9YgC4DJPZwZUbD6vjoUSEAirDuxxtDTG1XJoHCg7c/fyBRdToUGA10/08+TUjJ9A2/YRA== Received: from BN0PR04CA0013.namprd04.prod.outlook.com (2603:10b6:408:ee::18) by CH3PR12MB8935.namprd12.prod.outlook.com (2603:10b6:610:169::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.17; Wed, 5 Jul 2023 11:11:09 +0000 Received: from BN8NAM11FT033.eop-nam11.prod.protection.outlook.com (2603:10b6:408:ee:cafe::40) by BN0PR04CA0013.outlook.office365.com (2603:10b6:408:ee::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.18 via Frontend Transport; Wed, 5 Jul 2023 11:11:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT033.mail.protection.outlook.com (10.13.177.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.44 via Frontend Transport; Wed, 5 Jul 2023 11:11:08 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 5 Jul 2023 04:10:56 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 5 Jul 2023 04:10:55 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v4 1/4] net/mlx5: introduce tracepoints for mlx5 drivers Date: Wed, 5 Jul 2023 14:10:35 +0300 Message-ID: <20230705111038.9935-2-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230705111038.9935-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230705111038.9935-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT033:EE_|CH3PR12MB8935:EE_ X-MS-Office365-Filtering-Correlation-Id: 516fb499-43e6-41a6-a881-08db7d4888ef X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7dRXy2wmMittVAINbK+sknKbn7xzQ1Twjj10PTVKR5obAYdgpz4DUJzCoP8gFnGBWV7ahIamIjb9DST7ZdK+FoenrQOL8Wfi9zRrsyZAZNPZXRTVD3wB4HVxJf4E45jyQ4SYZthjTQ8uTHoZe5MePuD6d0OA5+mnylwFbMa6PlfVi8Y0Oxj56NR60KRWnltfB6ssBXsBRzaZpIe23iyVk8u9wcBKkvlu4vPXqmH+u5101jS0bAE8JlMXiWQKH7DGzOp9WZtaOKwdT1lJCPFtfZ8GRSCtqBFlONyoEXNLWRWnzZcMsewo03vfyso+8e3bREj/cgKqEHd6I95BvU8pHyTcSDsTFvZHtHv3iU8ve+akYG2VcmuDtJOcKi5s0LfLQyKPeJsb8GHnXsV8a2mXiyMlPZ+HUwFcfpwhZRG1XcAggF+PjCFYtJgxVF2GR1GdpvMM4NCawzj2lAgf0QR9efDxqOZl+jf9x1n4301GqXSBK9aEjZs8aKdJ/Ah3Lcj2+JiB3bOrdylBbHoCAIJMU9gKz+TgyngDmlfzKWY76BiT1MYHdZ6hPXbqBCTBGhhAqU4X3kNNS/04GYP/4LXK7EQ+KxiZoavxzsgS9ORLI6F3FvMRNUzq7kkQLbpt4zBRmlwkfkNklVgh6ipcHaG/cLnBIwxkTSMVtflOX+5kAHVXRm5p9b8B0gRjUD2xsFATPaV47rvEulP4oeofsXyGnby/LvfGQDvcMMIZ+6Wswp+isGyo6IkS+cP0XZrtElv0 X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(376002)(346002)(136003)(451199021)(36840700001)(46966006)(40470700004)(6666004)(7696005)(478600001)(70586007)(54906003)(36860700001)(2616005)(83380400001)(47076005)(426003)(336012)(36756003)(86362001)(55016003)(40480700001)(30864003)(2906002)(82310400005)(16526019)(6286002)(70206006)(186003)(1076003)(26005)(107886003)(7636003)(356005)(82740400003)(6916009)(4326008)(40460700003)(316002)(41300700001)(8936002)(8676002)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2023 11:11:08.9550 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 516fb499-43e6-41a6-a881-08db7d4888ef X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT033.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8935 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is an intention to engage DPDK tracing capabilities for mlx5 PMDs monitoring and profiling in various modes. The patch introduces tracepoints for the Tx datapath in the ethernet device driver. To engage this tracing capability the following steps should be taken: - meson option -Denable_trace_fp=true - meson option -Dc_args='-DALLOW_EXPERIMENTAL_API' - EAL command line parameter --trace=pmd.net.mlx5.tx.* The Tx datapath tracing allows to get information how packets are pushed into hardware descriptors, time stamping for scheduled wait and send completions, etc. To provide the human readable form of trace results the dedicated post-processing script is presumed. Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5_rx.h | 19 ---------- drivers/net/mlx5/mlx5_rxtx.h | 19 ++++++++++ drivers/net/mlx5/mlx5_tx.c | 29 +++++++++++++++ drivers/net/mlx5/mlx5_tx.h | 72 +++++++++++++++++++++++++++++++++++- 4 files changed, 118 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index 3514edd84e..f42607dce4 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -377,25 +377,6 @@ mlx5_rx_mb2mr(struct mlx5_rxq_data *rxq, struct rte_mbuf *mb) return mlx5_mr_mempool2mr_bh(mr_ctrl, mb->pool, addr); } -/** - * Convert timestamp from HW format to linear counter - * from Packet Pacing Clock Queue CQE timestamp format. - * - * @param sh - * Pointer to the device shared context. Might be needed - * to convert according current device configuration. - * @param ts - * Timestamp from CQE to convert. - * @return - * UTC in nanoseconds - */ -static __rte_always_inline uint64_t -mlx5_txpp_convert_rx_ts(struct mlx5_dev_ctx_shared *sh, uint64_t ts) -{ - RTE_SET_USED(sh); - return (ts & UINT32_MAX) + (ts >> 32) * NS_PER_S; -} - /** * Set timestamp in mbuf dynamic field. * diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 876aa14ae6..b109d50758 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -43,4 +43,23 @@ int mlx5_queue_state_modify_primary(struct rte_eth_dev *dev, int mlx5_queue_state_modify(struct rte_eth_dev *dev, struct mlx5_mp_arg_queue_state_modify *sm); +/** + * Convert timestamp from HW format to linear counter + * from Packet Pacing Clock Queue CQE timestamp format. + * + * @param sh + * Pointer to the device shared context. Might be needed + * to convert according current device configuration. + * @param ts + * Timestamp from CQE to convert. + * @return + * UTC in nanoseconds + */ +static __rte_always_inline uint64_t +mlx5_txpp_convert_rx_ts(struct mlx5_dev_ctx_shared *sh, uint64_t ts) +{ + RTE_SET_USED(sh); + return (ts & UINT32_MAX) + (ts >> 32) * NS_PER_S; +} + #endif /* RTE_PMD_MLX5_RXTX_H_ */ diff --git a/drivers/net/mlx5/mlx5_tx.c b/drivers/net/mlx5/mlx5_tx.c index 14e1487e59..13e2d90e03 100644 --- a/drivers/net/mlx5/mlx5_tx.c +++ b/drivers/net/mlx5/mlx5_tx.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -232,6 +233,15 @@ mlx5_tx_handle_completion(struct mlx5_txq_data *__rte_restrict txq, MLX5_ASSERT((txq->fcqs[txq->cq_ci & txq->cqe_m] >> 16) == cqe->wqe_counter); #endif + if (__rte_trace_point_fp_is_enabled()) { + uint64_t ts = rte_be_to_cpu_64(cqe->timestamp); + uint16_t wqe_id = rte_be_to_cpu_16(cqe->wqe_counter); + + if (txq->rt_timestamp) + ts = mlx5_txpp_convert_rx_ts(NULL, ts); + rte_pmd_mlx5_trace_tx_complete(txq->port_id, txq->idx, + wqe_id, ts); + } ring_doorbell = true; ++txq->cq_ci; last_cqe = cqe; @@ -752,3 +762,22 @@ mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, } return -EINVAL; } + +/* TX burst subroutines trace points. */ +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_entry, + pmd.net.mlx5.tx.entry) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_exit, + pmd.net.mlx5.tx.exit) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_wqe, + pmd.net.mlx5.tx.wqe) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_wait, + pmd.net.mlx5.tx.wait) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_push, + pmd.net.mlx5.tx.push) + +RTE_TRACE_POINT_REGISTER(rte_pmd_mlx5_trace_tx_complete, + pmd.net.mlx5.tx.complete) diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h index cc8f7e98aa..b90cdf1fcc 100644 --- a/drivers/net/mlx5/mlx5_tx.h +++ b/drivers/net/mlx5/mlx5_tx.h @@ -13,12 +13,61 @@ #include #include #include +#include #include #include #include "mlx5.h" #include "mlx5_autoconf.h" +#include "mlx5_rxtx.h" + +/* TX burst subroutines trace points. */ +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_entry, + RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t queue_id), + rte_trace_point_emit_u16(port_id); + rte_trace_point_emit_u16(queue_id); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_exit, + RTE_TRACE_POINT_ARGS(uint16_t nb_sent, uint16_t nb_req), + rte_trace_point_emit_u16(nb_sent); + rte_trace_point_emit_u16(nb_req); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_wqe, + RTE_TRACE_POINT_ARGS(uint32_t opcode), + rte_trace_point_emit_u32(opcode); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_wait, + RTE_TRACE_POINT_ARGS(uint64_t ts), + rte_trace_point_emit_u64(ts); +) + + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_push, + RTE_TRACE_POINT_ARGS(const struct rte_mbuf *mbuf, uint16_t wqe_id), + rte_trace_point_emit_ptr(mbuf); + rte_trace_point_emit_u32(mbuf->pkt_len); + rte_trace_point_emit_u16(mbuf->nb_segs); + rte_trace_point_emit_u16(wqe_id); +) + +RTE_TRACE_POINT_FP( + rte_pmd_mlx5_trace_tx_complete, + RTE_TRACE_POINT_ARGS(uint16_t port_id, uint16_t queue_id, + uint16_t wqe_id, uint64_t ts), + rte_trace_point_emit_u16(port_id); + rte_trace_point_emit_u16(queue_id); + rte_trace_point_emit_u64(ts); + rte_trace_point_emit_u16(wqe_id); +) /* TX burst subroutines return codes. */ enum mlx5_txcmp_code { @@ -764,6 +813,9 @@ mlx5_tx_cseg_init(struct mlx5_txq_data *__rte_restrict txq, cs->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); + if (__rte_trace_point_fp_is_enabled() && !loc->pkts_sent) + rte_pmd_mlx5_trace_tx_entry(txq->port_id, txq->idx); + rte_pmd_mlx5_trace_tx_wqe((txq->wqe_ci << 8) | opcode); } /** @@ -1692,6 +1744,7 @@ mlx5_tx_schedule_send(struct mlx5_txq_data *restrict txq, if (txq->wait_on_time) { /* The wait on time capability should be used. */ ts -= sh->txpp.skew; + rte_pmd_mlx5_trace_tx_wait(ts); mlx5_tx_cseg_init(txq, loc, wqe, 1 + sizeof(struct mlx5_wqe_wseg) / MLX5_WSEG_SIZE, @@ -1706,6 +1759,7 @@ mlx5_tx_schedule_send(struct mlx5_txq_data *restrict txq, if (unlikely(wci < 0)) return MLX5_TXCMP_CODE_SINGLE; /* Build the WAIT WQE with specified completion. */ + rte_pmd_mlx5_trace_tx_wait(ts - sh->txpp.skew); mlx5_tx_cseg_init(txq, loc, wqe, 1 + sizeof(struct mlx5_wqe_qseg) / MLX5_WSEG_SIZE, @@ -1810,6 +1864,7 @@ mlx5_tx_packet_multi_tso(struct mlx5_txq_data *__rte_restrict txq, wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 0, MLX5_OPCODE_TSO, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); ds = mlx5_tx_mseg_build(txq, loc, wqe, vlan, inlen, 1, olx); wqe->cseg.sq_ds = rte_cpu_to_be_32(txq->qp_num_8s | ds); txq->wqe_ci += (ds + 3) / 4; @@ -1892,6 +1947,7 @@ mlx5_tx_packet_multi_send(struct mlx5_txq_data *__rte_restrict txq, wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, ds, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_none(txq, loc, wqe, olx); dseg = &wqe->dseg[0]; do { @@ -2115,6 +2171,7 @@ mlx5_tx_packet_multi_inline(struct mlx5_txq_data *__rte_restrict txq, wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 0, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); ds = mlx5_tx_mseg_build(txq, loc, wqe, vlan, inlen, 0, olx); wqe->cseg.sq_ds = rte_cpu_to_be_32(txq->qp_num_8s | ds); txq->wqe_ci += (ds + 3) / 4; @@ -2318,8 +2375,8 @@ mlx5_tx_burst_tso(struct mlx5_txq_data *__rte_restrict txq, */ wqe = txq->wqes + (txq->wqe_ci & txq->wqe_m); loc->wqe_last = wqe; - mlx5_tx_cseg_init(txq, loc, wqe, ds, - MLX5_OPCODE_TSO, olx); + mlx5_tx_cseg_init(txq, loc, wqe, ds, MLX5_OPCODE_TSO, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_eseg_data(txq, loc, wqe, vlan, hlen, 1, olx); dptr = rte_pktmbuf_mtod(loc->mbuf, uint8_t *) + hlen - vlan; dlen -= hlen - vlan; @@ -2688,6 +2745,7 @@ mlx5_tx_burst_empw_simple(struct mlx5_txq_data *__rte_restrict txq, /* Update sent data bytes counter. */ slen += dlen; #endif + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_dseg_ptr (txq, loc, dseg, rte_pktmbuf_mtod(loc->mbuf, uint8_t *), @@ -2926,6 +2984,7 @@ mlx5_tx_burst_empw_inline(struct mlx5_txq_data *__rte_restrict txq, tlen += sizeof(struct rte_vlan_hdr); if (room < tlen) break; + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_dseg_vlan(txq, loc, dseg, dptr, dlen, olx); #ifdef MLX5_PMD_SOFT_COUNTERS @@ -2935,6 +2994,7 @@ mlx5_tx_burst_empw_inline(struct mlx5_txq_data *__rte_restrict txq, } else { if (room < tlen) break; + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_dseg_empw(txq, loc, dseg, dptr, dlen, olx); } @@ -2980,6 +3040,7 @@ mlx5_tx_burst_empw_inline(struct mlx5_txq_data *__rte_restrict txq, if (MLX5_TXOFF_CONFIG(VLAN)) MLX5_ASSERT(!(loc->mbuf->ol_flags & RTE_MBUF_F_TX_VLAN)); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_dseg_ptr(txq, loc, dseg, dptr, dlen, olx); /* We have to store mbuf in elts.*/ txq->elts[txq->elts_head++ & txq->elts_m] = loc->mbuf; @@ -3194,6 +3255,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, seg_n, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_data(txq, loc, wqe, vlan, inlen, 0, olx); txq->wqe_ci += wqe_n; @@ -3256,6 +3318,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, ds, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); dseg = mlx5_tx_eseg_data(txq, loc, wqe, vlan, txq->inlen_mode, 0, olx); @@ -3297,6 +3360,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 4, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_dmin(txq, loc, wqe, vlan, olx); dptr = rte_pktmbuf_mtod(loc->mbuf, uint8_t *) + MLX5_ESEG_MIN_INLINE_SIZE - vlan; @@ -3338,6 +3402,7 @@ mlx5_tx_burst_single_send(struct mlx5_txq_data *__rte_restrict txq, loc->wqe_last = wqe; mlx5_tx_cseg_init(txq, loc, wqe, 3, MLX5_OPCODE_SEND, olx); + rte_pmd_mlx5_trace_tx_push(loc->mbuf, txq->wqe_ci); mlx5_tx_eseg_none(txq, loc, wqe, olx); mlx5_tx_dseg_ptr (txq, loc, &wqe->dseg[0], @@ -3707,6 +3772,9 @@ mlx5_tx_burst_tmpl(struct mlx5_txq_data *__rte_restrict txq, #endif if (MLX5_TXOFF_CONFIG(INLINE) && loc.mbuf_free) __mlx5_tx_free_mbuf(txq, pkts, loc.mbuf_free, olx); + /* Trace productive bursts only. */ + if (__rte_trace_point_fp_is_enabled() && loc.pkts_sent) + rte_pmd_mlx5_trace_tx_exit(loc.pkts_sent, pkts_n); return loc.pkts_sent; } From patchwork Wed Jul 5 11:10:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129290 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0E2D542DD9; Wed, 5 Jul 2023 13:11:19 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6638C410F2; Wed, 5 Jul 2023 13:11:14 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2063.outbound.protection.outlook.com [40.107.237.63]) by mails.dpdk.org (Postfix) with ESMTP id 1AEA040150 for ; Wed, 5 Jul 2023 13:11:13 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XUQic0JCHABo1RRcU1fetkJP+KEGU5BVbuAi62nxJZWZPmU/3ipfvKtgx6LeauGQoglUARK8jD+LzHGEq+OecE91E054AsVUU1icq2Fa040zQkIi9iCedb/e8QDDwGxhe8eSZ21CiFIDr/QmRuYyqlTYMpk79NSjFCo6k0bpDgYZMPlgWO8qIB3ZWy1nByRHl4qhQmw2QGV6XTa1j4lmC4fTxsFj9ODI+0BGPD7fY4ohEu5rFM0kn1a9yTn24RntQt68rVqP9LD2bPnt3+sCCTivh2ULawLp6M5LAeu0rY2yl//7j6uIbK+xGiSJB/d2TNJhCvORc+l0hPk2fnecUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kCoGTrc5xMwnxWWKK9I96Mqre17KCB1xK7f1vmMc6b8=; b=IBByxRm9UqkeYlQQ+9NmFKCHh/djfrkWdrixU3s00B0cWx9TCLCiW9dFQE9UbWeLr6whgq8FFnbZoDDQJMkQUk+ypeWRX/pilNtYfxAUaeugW/prijYOMtp0NmroX6h0njLMt9JNSwVpeZLqRI4FzB+yD7SaSXEdeyZ54u/+k083u93awrPjUVZHS0sDGMaWd/v4eVdQEqsnhuITVD+W/O+QVNwwW5+W7v4NMiyxv/p4hj8HUf9yaln+WQGkyrVR/pp1rXoTqjgW8RnixATRkaX/FWyHeUwq9SLtH74/HRJ+OA212+Q7IgxAvFFQOhOjxMooud8ZRRF1uzVClvS3Uw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kCoGTrc5xMwnxWWKK9I96Mqre17KCB1xK7f1vmMc6b8=; b=GlJOVEZyJhffVAUf661NV67a52klvGi526VmeM00BfB+2OdRCFoJStJJsXiKiHoRvML5WWJ+1PibhpDUIzt2hg962gWf8MzEmD9IlzBj5aeLFmYvZzRRbsvpjW4m6TspB+T24UI+p3byEEOFke3QK5Pw5RvKqtC2iYt5IqpB/A/aA84ikuMmidz4RsRG//QffG/vOFgc34x4x6fw21pxJYuSdxOZH7KEuo8U8Nuv0oo8AytlwoiZ0EtMWOJzxuUDfzGgKkCD/dq9p9Yxt1rxFfSBZE8gPYdsCdZk/BMXPVrE190elDp1pJK1OULr4lh15KxJCCv8NY9UvaB8eNphzQ== Received: from BN9PR03CA0533.namprd03.prod.outlook.com (2603:10b6:408:131::28) by BN9PR12MB5034.namprd12.prod.outlook.com (2603:10b6:408:104::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.17; Wed, 5 Jul 2023 11:11:10 +0000 Received: from BN8NAM11FT075.eop-nam11.prod.protection.outlook.com (2603:10b6:408:131:cafe::4c) by BN9PR03CA0533.outlook.office365.com (2603:10b6:408:131::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.18 via Frontend Transport; Wed, 5 Jul 2023 11:11:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT075.mail.protection.outlook.com (10.13.176.208) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.45 via Frontend Transport; Wed, 5 Jul 2023 11:11:10 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 5 Jul 2023 04:10:58 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 5 Jul 2023 04:10:57 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v4 2/4] net/mlx5: add comprehensive send completion trace Date: Wed, 5 Jul 2023 14:10:36 +0300 Message-ID: <20230705111038.9935-3-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230705111038.9935-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230705111038.9935-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT075:EE_|BN9PR12MB5034:EE_ X-MS-Office365-Filtering-Correlation-Id: a16a0e5d-2d7d-481d-98c2-08db7d488996 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oLIjd+EQo5xpWAi6AuhSl2+MXdJBOevlMcL+GMmXlpRoU2kBQy6xuZvW90Mj5ZjtX/qV+Yiyp0ohNQHWeG41B7mbsNnfsRaFQTLVSWHg3stP/e0VyRl0ZOtkmyyVCDHqGi1UxaQ4Cs1kPQazmhri9SORRunGVVn6lzEmj75C01sFqTpOkAIiAVOlN8AWrf6G3Uh+NLjg/rb9hT7q83MmGbDUxTaFepFN+Tr6rXW74o8RZiBtuQGjmtTbSoRx/cgq3J4tSb22qrXLf/FB8ypGuAlLXY73bNnTr57c5HVsjYnVZuAm+YWsr7/o1VzSpZSVNPDxO7fM+rwprmlt7fRxXQ+qbWIy2pV6FduPeF7FW3BQ4vDegF/xW6XF2JvjbbRcDu8og3TZgVD3OPabwyhoIUn4gPD+HixpPn2pl6D9F0o8OyeJQOhrv9KIUvhoXLvI7lBqcq03xOP1UgueSAS/TertNga34imybq6Q8zSo4vYhiFqoyOeOeyu8jo7aPVxLeL2jlmE/BhUhonqX4e53+Knnfr2U6wayDL4WgnnwkxYhOqBuBXUx2XQLF/aAUlEUH3/OrzJQ5kWyJZ/BCgozb3xqA119nW2Bg4NURDbyvfwnsGvy+EBYFoXGq6exE2ojNeU6jTNx/utUZLu6SpQX0bP2LZTfNA4K19pSOGG24l1+PATKhfCEne6VjV0DsOZoelWR8mmlj4M5tUvK7/hV9yRPX9u7xDcXN7CI6G0HVv9ZrQ/6T7B6crZ5Eq2rsMVA X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(396003)(39860400002)(346002)(136003)(376002)(451199021)(36840700001)(46966006)(40470700004)(1076003)(70206006)(6916009)(4326008)(70586007)(7636003)(356005)(82740400003)(2616005)(55016003)(16526019)(6286002)(40480700001)(186003)(86362001)(6666004)(7696005)(36756003)(26005)(54906003)(478600001)(107886003)(40460700003)(82310400005)(36860700001)(8936002)(8676002)(5660300002)(2906002)(47076005)(41300700001)(316002)(83380400001)(336012)(426003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2023 11:11:10.0491 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a16a0e5d-2d7d-481d-98c2-08db7d488996 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT075.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5034 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There is the demand to trace the send completions of every WQE if time scheduling is enabled. The patch extends the size of completion queue and requests completion on every issued WQE in the send queue. As the result hardware provides CQE on each completed WQE and driver is able to fetch completion timestamp for dedicated operation. The add code is under conditional compilation RTE_ENABLE_TRACE_FP flag and does not impact the release code. Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/linux/mlx5_verbs.c | 8 +++- drivers/net/mlx5/mlx5_devx.c | 8 +++- drivers/net/mlx5/mlx5_tx.h | 63 +++++++++++++++++++++++++++-- 3 files changed, 71 insertions(+), 8 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 7233c2c7fa..b54f3ccd9a 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -968,8 +968,12 @@ mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx) rte_errno = EINVAL; return -rte_errno; } - cqe_n = desc / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + if (__rte_trace_point_fp_is_enabled() && + txq_data->offloads & RTE_ETH_TX_OFFLOAD_SEND_ON_TIMESTAMP) + cqe_n = UINT16_MAX / 2 - 1; + else + cqe_n = desc / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; txq_obj->cq = mlx5_glue->create_cq(priv->sh->cdev->ctx, cqe_n, NULL, NULL, 0); if (txq_obj->cq == NULL) { diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 4369d2557e..5082a7e178 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1465,8 +1465,12 @@ mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) MLX5_ASSERT(ppriv); txq_obj->txq_ctrl = txq_ctrl; txq_obj->dev = dev; - cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + if (__rte_trace_point_fp_is_enabled() && + txq_data->offloads & RTE_ETH_TX_OFFLOAD_SEND_ON_TIMESTAMP) + cqe_n = UINT16_MAX / 2 - 1; + else + cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; log_desc_n = log2above(cqe_n); cqe_n = 1UL << log_desc_n; if (cqe_n > UINT16_MAX) { diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h index b90cdf1fcc..47ee8bca4f 100644 --- a/drivers/net/mlx5/mlx5_tx.h +++ b/drivers/net/mlx5/mlx5_tx.h @@ -775,6 +775,54 @@ mlx5_tx_request_completion(struct mlx5_txq_data *__rte_restrict txq, } } +/** + * Set completion request flag for all issued WQEs. + * This routine is intended to be used with enabled fast path tracing + * and send scheduling on time to provide the detailed report in trace + * for send completions on every WQE. + * + * @param txq + * Pointer to TX queue structure. + * @param loc + * Pointer to burst routine local context. + * @param olx + * Configured Tx offloads mask. It is fully defined at + * compile time and may be used for optimization. + */ +static __rte_always_inline void +mlx5_tx_request_completion_trace(struct mlx5_txq_data *__rte_restrict txq, + struct mlx5_txq_local *__rte_restrict loc, + unsigned int olx) +{ + uint16_t head = txq->elts_comp; + + while (txq->wqe_comp != txq->wqe_ci) { + volatile struct mlx5_wqe *wqe; + uint32_t wqe_n; + + MLX5_ASSERT(loc->wqe_last); + wqe = txq->wqes + (txq->wqe_comp & txq->wqe_m); + if (wqe == loc->wqe_last) { + head = txq->elts_head; + head += MLX5_TXOFF_CONFIG(INLINE) ? + 0 : loc->pkts_sent - loc->pkts_copy; + txq->elts_comp = head; + } + /* Completion request flag was set on cseg constructing. */ +#ifdef RTE_LIBRTE_MLX5_DEBUG + txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head | + (wqe->cseg.opcode >> 8) << 16; +#else + txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head; +#endif + /* A CQE slot must always be available. */ + MLX5_ASSERT((txq->cq_pi - txq->cq_ci) <= txq->cqe_s); + /* Advance to the next WQE in the queue. */ + wqe_n = rte_be_to_cpu_32(wqe->cseg.sq_ds) & 0x3F; + txq->wqe_comp += RTE_ALIGN(wqe_n, 4) / 4; + } +} + /** * Build the Control Segment with specified opcode: * - MLX5_OPCODE_SEND @@ -801,7 +849,7 @@ mlx5_tx_cseg_init(struct mlx5_txq_data *__rte_restrict txq, struct mlx5_wqe *__rte_restrict wqe, unsigned int ds, unsigned int opcode, - unsigned int olx __rte_unused) + unsigned int olx) { struct mlx5_wqe_cseg *__rte_restrict cs = &wqe->cseg; @@ -810,8 +858,12 @@ mlx5_tx_cseg_init(struct mlx5_txq_data *__rte_restrict txq, opcode = MLX5_OPCODE_TSO | MLX5_OPC_MOD_MPW << 24; cs->opcode = rte_cpu_to_be_32((txq->wqe_ci << 8) | opcode); cs->sq_ds = rte_cpu_to_be_32(txq->qp_num_8s | ds); - cs->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << - MLX5_COMP_MODE_OFFSET); + if (MLX5_TXOFF_CONFIG(TXPP) && __rte_trace_point_fp_is_enabled()) + cs->flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); + else + cs->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << + MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); if (__rte_trace_point_fp_is_enabled() && !loc->pkts_sent) rte_pmd_mlx5_trace_tx_entry(txq->port_id, txq->idx); @@ -3709,7 +3761,10 @@ mlx5_tx_burst_tmpl(struct mlx5_txq_data *__rte_restrict txq, if (unlikely(loc.pkts_sent == loc.pkts_loop)) goto burst_exit; /* Request CQE generation if limits are reached. */ - mlx5_tx_request_completion(txq, &loc, olx); + if (MLX5_TXOFF_CONFIG(TXPP) && __rte_trace_point_fp_is_enabled()) + mlx5_tx_request_completion_trace(txq, &loc, olx); + else + mlx5_tx_request_completion(txq, &loc, olx); /* * Ring QP doorbell immediately after WQE building completion * to improve latencies. The pure software related data treatment From patchwork Wed Jul 5 11:10:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129292 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 22B9B42DD9; Wed, 5 Jul 2023 13:11:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A01C942D37; Wed, 5 Jul 2023 13:11:17 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2076.outbound.protection.outlook.com [40.107.243.76]) by mails.dpdk.org (Postfix) with ESMTP id 74E89410F6 for ; Wed, 5 Jul 2023 13:11:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WvVivryjPFpKBqRj9AyxKy0Xk+2a/Hy/q4irOotKJ+3zlZnikfOWK6yn4zI0lpSB1NfKHxmLwMDkOIB599n9o4Me6EvxyJCwGUO3uDWIMFw0iYBG59v8ndpN++fus8pVvKY0NkDB/GYjIfwACmBIhH3y9iI/WLaM+WZqa1/RHeXAIOjvRTMefNLkyzoO3mGl8IESnXfUman00w1LCMrAdndcarB+u2uTTmbqy90WE5XZq4Yw3kXO3O4fxeXxXC8DqdU3t/UZHMgeGIABynujVgKk+7BHC9qZSxdKzUkFygmcpgnQ91+r6/ICG6EEskLd7BusMWmsjNSmTpQ88zilOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/DsapU1Z/lqzk/PAxwff7QOfmmrMm5Xv0ofwjQ6zr/I=; b=Q/gvtkfGiWrcbTmHCl2NvzmI+UUOUFNTjeMJ/d0AIWKgoB0KFeZSBQz0E/PeKCQEI4Y1FJ3Kcoh/dgYhaMAfFBC4Lj3YgZqVgtYXymLJjUuzCTiWU4rxDp+nw+P4TO/wKAphsPmYlX30S0Iu8QDioLnh0MB8KHAo1hXP5nDp7V0SNsynODtVigJ1dWfgJgT2vHU7RMTAE6I4+EUc2079ts6MbTRHbG0OCdePADQikzd2+fsaCHkkO+iy1xZLOwbDxFspHamYayKBh5huQpywxUNfjvOwpWzdS1L1tGVPYauKKJ3S44bhUvegyzePufj57u4UJuDzKfDkMBGoBAf8Ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/DsapU1Z/lqzk/PAxwff7QOfmmrMm5Xv0ofwjQ6zr/I=; b=MELezuBYU7mgSIlCybWAh2Q2yf34pkaxGhEMXd9N99EAU37wGIQeydh3QeuwAUoXlmFUe4Q1jMzabsIJR8oBGH3K2TjVRcWEzPlKHr4j99rBr/yalyiLo2Tx+DNl7e88/LcYwVNzIn8LiL3nypaY2sTnF2J1qZA3HbHqEnmSIbjDpYEx5pcpbOgAskHixr86tPP7SDJtINTfBJp8zVqdXBkeGbsxy8rNi/zMrd7+sOqIPR3dB6IX/dr/uLmKx7dgHPBEyyDv8bosDuFDjnHGWga1vbGkprQ165s1LZV2UCjnYf1527HkTBeBZEM0oT9LEheNe9D/CWJg/rARiNTrkg== Received: from BN9PR03CA0536.namprd03.prod.outlook.com (2603:10b6:408:131::31) by SJ1PR12MB6265.namprd12.prod.outlook.com (2603:10b6:a03:458::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Wed, 5 Jul 2023 11:11:12 +0000 Received: from BN8NAM11FT075.eop-nam11.prod.protection.outlook.com (2603:10b6:408:131:cafe::2d) by BN9PR03CA0536.outlook.office365.com (2603:10b6:408:131::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.18 via Frontend Transport; Wed, 5 Jul 2023 11:11:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT075.mail.protection.outlook.com (10.13.176.208) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.45 via Frontend Transport; Wed, 5 Jul 2023 11:11:11 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 5 Jul 2023 04:11:00 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 5 Jul 2023 04:10:58 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v4 3/4] net/mlx5: add Tx datapath trace analyzing script Date: Wed, 5 Jul 2023 14:10:37 +0300 Message-ID: <20230705111038.9935-4-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230705111038.9935-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230705111038.9935-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT075:EE_|SJ1PR12MB6265:EE_ X-MS-Office365-Filtering-Correlation-Id: 71aa4bd7-c5e7-4526-dae6-08db7d488a5e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nPcYUmRfxOzlaeq0SKuoVn5XuS7ohYAz4UQgifKMsAUSp3Yh0HAd+Mt4L66PUeBKAJWxRxdDddByQibpCiK0wKkN3nno4KXDVOzHLa/FoNkAFmR3onkIsKAFMdmMPx/b/GwUm3PgoS7ZWNdlCDVsZjAKKPsvuZ0RrwPFoWdswBBLv5cPzQEWdp829nq/SowyDVfP+MICqyPiEITtfZhH9QbZVKOHxCj43aHMehBlPyZ8MGiQomIT26UPODXzrGDOvb4mi0TQP5botKJXx7swJFd0y5AAG2a/9BhXlFpwtjFiJF6V8ZJWO0xWt4E90lNHoA+7a1s/a3xW4FtFhrY9abnH6FuYdQG7bvJNkTpuO2PFmKBpvOhNfurqMp/z3QSIpr/36/8sbfkDCxSmgQkVC/aNoLK/SohByiKtFZzj7BOKsARec5gt/ziicoJYmEdS07beqcBaOHh7pVoKN6BO77wS/kMgZPYLm3UWYHJG4BW31vE3GuDKWXe/i/OJh9j4wOogk1sZLROdfaZDrL+kI+ak1Y6XaT09WqV1aUiIIfpJZLbgA/REodbYMs6E2vBRLLnL97kzddZS4m/47dTGvQ5NQFZdMj4QePomOIZ/jwdHilfHYUsLEEpWSNHI8LHsXdX3Kw0Kr9n3vcDqB1JOVH1RAQEweWrQr9XDaHcbFKEDlDM4At3goAdDoHfqBZ5ifHjORLNe5J1VO1AtFBodFG4s3Tv5qBLa0O2JAOTc4lRikOMojdatdiBDAgZT4NXt X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(346002)(136003)(376002)(396003)(451199021)(40470700004)(36840700001)(46966006)(336012)(40460700003)(41300700001)(86362001)(6666004)(82310400005)(7636003)(107886003)(47076005)(83380400001)(82740400003)(356005)(6286002)(1076003)(16526019)(2616005)(26005)(426003)(36860700001)(40480700001)(55016003)(7696005)(54906003)(478600001)(316002)(36756003)(4326008)(70206006)(70586007)(6916009)(8676002)(8936002)(186003)(2906002)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2023 11:11:11.3459 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 71aa4bd7-c5e7-4526-dae6-08db7d488a5e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT075.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6265 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The Python script is intended to analyze mlx5 PMD datapath traces and report: - tx_burst routine timings - how packets are pushed to WQEs - how packet sending is completed with timings Signed-off-by: Viacheslav Ovsiienko --- drivers/net/mlx5/tools/mlx5_trace.py | 307 +++++++++++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100755 drivers/net/mlx5/tools/mlx5_trace.py diff --git a/drivers/net/mlx5/tools/mlx5_trace.py b/drivers/net/mlx5/tools/mlx5_trace.py new file mode 100755 index 0000000000..8c1fd0a350 --- /dev/null +++ b/drivers/net/mlx5/tools/mlx5_trace.py @@ -0,0 +1,307 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2023 NVIDIA Corporation & Affiliates + +""" +Analyzing the mlx5 PMD datapath tracings +""" +import sys +import argparse +import bt2 + +PFX_TX = "pmd.net.mlx5.tx." +PFX_TX_LEN = len(PFX_TX) + + +class MlxQueue: + """Queue container object""" + + def __init__(self): + self.done_burst = [] # completed bursts + self.wait_burst = [] # waiting for completion + self.pq_id = 0 + + def log(self): + """Log all queue bursts""" + for txb in self.done_burst: + txb.log() + + +class MlxMbuf: + """Packet mbufs container object""" + + def __init__(self): + self.wqe = 0 # wqe id + self.ptr = None # first packet mbuf pointer + self.len = 0 # packet data length + self.nseg = 0 # number of segments + + def log(self): + """Log mbuf""" + out_txt = " %X: %u" % (self.ptr, self.len) + if self.nseg != 1: + out_txt += " (%d segs)" % self.nseg + print(out_txt) + + +class MlxWqe: + """WQE container object""" + + def __init__(self): + self.mbuf = [] # list of mbufs in WQE + self.wait_ts = 0 # preceding wait/push timestamp + self.comp_ts = 0 # send/recv completion timestamp + self.opcode = 0 + + def log(self): + """Log WQE""" + wqe_id = (self.opcode >> 8) & 0xFFFF + wqe_op = self.opcode & 0xFF + out_txt = " %04X: " % wqe_id + if wqe_op == 0xF: + out_txt += "WAIT" + elif wqe_op == 0x29: + out_txt += "EMPW" + elif wqe_op == 0xE: + out_txt += "TSO " + elif wqe_op == 0xA: + out_txt += "SEND" + else: + out_txt += "0x%02X" % wqe_op + if self.comp_ts != 0: + out_txt += " (%d, %d)" % (self.wait_ts, self.comp_ts - self.wait_ts) + else: + out_txt += " (%d)" % self.wait_ts + print(out_txt) + for mbuf in self.mbuf: + mbuf.log() + + def comp(self, wqe_id, wqe_ts): + """Return 0 if WQE in not completedLog WQE""" + if self.comp_ts != 0: + return 1 + cur_id = (self.opcode >> 8) & 0xFFFF + if cur_id > wqe_id: + cur_id -= wqe_id + if cur_id <= 0x8000: + return 0 + else: + cur_id = wqe_id - cur_id + if cur_id >= 0x8000: + return 0 + self.comp_ts = wqe_ts + return 1 + + +class MlxBurst: + """Packet burst container object""" + + def __init__(self): + self.wqes = [] # issued burst WQEs + self.done = 0 # number of sent/recv packets + self.req = 0 # requested number of packets + self.call_ts = 0 # burst routine invocation + self.done_ts = 0 # burst routine done + self.queue = None + + def log(self): + """Log burst""" + port = self.queue.pq_id >> 16 + queue = self.queue.pq_id & 0xFFFF + if self.req == 0: + print( + "%u: tx(p=%u, q=%u, %u/%u pkts (incomplete)" + % (self.call_ts, port, queue, self.done, self.req) + ) + else: + print( + "%u: tx(p=%u, q=%u, %u/%u pkts in %u" + % ( + self.call_ts, + port, + queue, + self.done, + self.req, + self.done_ts - self.call_ts, + ) + ) + for wqe in self.wqes: + wqe.log() + + def comp(self, wqe_id, wqe_ts): + """Return 0 if not all of WQEs in burst completed""" + wlen = len(self.wqes) + if wlen == 0: + return 0 + for wqe in self.wqes: + if wqe.comp(wqe_id, wqe_ts) == 0: + return 0 + return 1 + + +class MlxTrace: + """Trace representing object""" + + def __init__(self): + self.tx_blst = {} # current Tx bursts per CPU + self.tx_qlst = {} # active Tx queues per port/queue + self.tx_wlst = {} # wait timestamp list per CPU + + def run(self, msg_it): + """Run over gathered tracing data and build database""" + for msg in msg_it: + if not isinstance(msg, bt2._EventMessageConst): + continue + event = msg.event + if event.name.startswith(PFX_TX): + do_tx(msg, self) + # Handling of other log event cathegories can be added here + + def log(self): + """Log gathered trace database""" + for pq_id in self.tx_qlst: + queue = self.tx_qlst.get(pq_id) + queue.log() + + +def do_tx_entry(msg, trace): + """Handle entry Tx busrt""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is not None: + # continue existing burst after WAIT + return + # allocate the new burst and append to the queue + burst = MlxBurst() + burst.call_ts = msg.default_clock_snapshot.ns_from_origin + trace.tx_blst[cpu_id] = burst + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = trace.tx_qlst.get(pq_id) + if queue is None: + # queue does not exist - allocate the new one + queue = MlxQueue() + queue.pq_id = pq_id + trace.tx_qlst[pq_id] = queue + burst.queue = queue + queue.wait_burst.append(burst) + + +def do_tx_exit(msg, trace): + """Handle exit Tx busrt""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + burst.done_ts = msg.default_clock_snapshot.ns_from_origin + burst.req = event["nb_req"] + burst.done = event["nb_sent"] + trace.tx_blst.pop(cpu_id) + + +def do_tx_wqe(msg, trace): + """Handle WQE record""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + wqe = MlxWqe() + wqe.wait_ts = trace.tx_wlst.get(cpu_id) + if wqe.wait_ts is None: + wqe.wait_ts = msg.default_clock_snapshot.ns_from_origin + wqe.opcode = event["opcode"] + burst.wqes.append(wqe) + + +def do_tx_wait(msg, trace): + """Handle WAIT record""" + event = msg.event + cpu_id = event["cpu_id"] + trace.tx_wlst[cpu_id] = event["ts"] + + +def do_tx_push(msg, trace): + """Handle WQE push event""" + event = msg.event + cpu_id = event["cpu_id"] + burst = trace.tx_blst.get(cpu_id) + if burst is None: + return + if not burst.wqes: + return + wqe = burst.wqes[-1] + mbuf = MlxMbuf() + mbuf.wqe = event["wqe_id"] + mbuf.ptr = event["mbuf"] + mbuf.len = event["mbuf_pkt_len"] + mbuf.nseg = event["mbuf_nb_segs"] + wqe.mbuf.append(mbuf) + + +def do_tx_complete(msg, trace): + """Handle send completion event""" + event = msg.event + pq_id = event["port_id"] << 16 | event["queue_id"] + queue = trace.tx_qlst.get(pq_id) + if queue is None: + return + qlen = len(queue.wait_burst) + if qlen == 0: + return + wqe_id = event["wqe_id"] + wqe_ts = event["ts"] + rmv = 0 + while rmv < qlen: + burst = queue.wait_burst[rmv] + if burst.comp(wqe_id, wqe_ts) == 0: + break + rmv += 1 + # mode completed burst to done list + if rmv != 0: + idx = 0 + while idx < rmv: + queue.done_burst.append(burst) + idx += 1 + del queue.wait_burst[0:rmv] + + +def do_tx(msg, trace): + """Handle Tx related records""" + name = msg.event.name[PFX_TX_LEN:] + if name == "entry": + do_tx_entry(msg, trace) + elif name == "exit": + do_tx_exit(msg, trace) + elif name == "wqe": + do_tx_wqe(msg, trace) + elif name == "wait": + do_tx_wait(msg, trace) + elif name == "push": + do_tx_push(msg, trace) + elif name == "complete": + do_tx_complete(msg, trace) + else: + print("Error: unrecognized Tx event name: %s" % msg.event.name, file=sys.stderr) + raise ValueError() + + +def main() -> int: + """Script entry point""" + try: + parser = argparse.ArgumentParser() + parser.add_argument("path", nargs=1, type=str, help="input trace folder") + args = parser.parse_args() + + mlx_tr = MlxTrace() + msg_it = bt2.TraceCollectionMessageIterator(args.path) + mlx_tr.run(msg_it) + mlx_tr.log() + return 0 + except ValueError: + return -1 + + +if __name__ == "__main__": + sys.exit(main()) From patchwork Wed Jul 5 11:10:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Slava Ovsiienko X-Patchwork-Id: 129293 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0E2C442DD9; Wed, 5 Jul 2023 13:11:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B8EA442BC9; Wed, 5 Jul 2023 13:11:20 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by mails.dpdk.org (Postfix) with ESMTP id E3E7040FAE for ; Wed, 5 Jul 2023 13:11:19 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GNhNV6FsUKJVYs2cR/gR4yqTP9oXHl8uwTBaGC4ZFGPfhX8CzyF0bDQ86n7Bs3AeHKGNtO7cSboSgNbpznPR203NwGuYQ26ezvzeI5mux+SbRsMs4Jv2cl5bDPCSbmkabHkJXBaWgNyTnhM3kKvtC/mxVFuiq88TJ10JR5gUEwYrAqqIBWzCqu5jnAw8AktrJva8kuEyHRgyiMPFMJUwX8d82lLq+muIlz5gMtITuZ4Y9BviaIqW4I8lolPPnYOgyeJZQjx+0BsFQliqOO3aeSP+5qkylLXCHfDwkoyu1vU7XX7oZCaWtAWbh3RsPnAsiS9dva/wgF0H7dR1KfWAvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zrc1juTuIMGABBUREkW+A4JVfHtTYTJx09SyLHnMBe4=; b=TsRvzB7T7HeizMFUn/qK17kXkXmQz4fwy+p02N54oHLnJImvuCMsvXieXqoF8tzygXni7ZRnsdMkhNUyz8tWglX8t6TFAr+qYZy6ngJrDItsWAlaoH6qOoW6WCWInZs6Jkcw/fak6zP8xtB7nCPJXIbf92ZLJQpgAJ+1e2Sb5EfGjMBG7/EYNwONeCLTH5uTWNhuxhlbAimNlyqWKNrS9Yx4A9qfN8S0lu4wviMlrKirbUtBHiwrsfDGIjW5gXclAHvHmQi/faBXLJ9iK61pHC7ji48llMzcuCpd2zLG1AcegjyZLVdVGPDIEqaFx8aBGWd5qElW9+IGxRyVsO3uog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zrc1juTuIMGABBUREkW+A4JVfHtTYTJx09SyLHnMBe4=; b=T3uRGNnjDiMi09u/bB7sOoO5Ku/JL4wxKg6e1roJ45fZYpYU97t+zdD8NWW7yIiF2ubqYzNOPVXEAe1q5Iyf9dfFJQb30pc+lBTcK4vRrDE6pcKV4fO3tmuajQW0xCdoQLlzPtCPukbgMZdkobOtlCnCMLpcphHMaop4KG2vWdZLbvrRyQpMwcahVHLbJMzTjnEg2gMn3+77sJCCk0bJ6c967ofGaf4keBY4nPaFvERv1qKcLvyCr6gLAO0B6wRzciVpqd6BRWAZiKj2AI7CO+u2/nU3HW82yNtPtm0EGDg4AA5iiOZQAkZXQfI9rtnpweu15R5bSBhRttV2xWqt6g== Received: from BN9PR03CA0523.namprd03.prod.outlook.com (2603:10b6:408:131::18) by BL1PR12MB5352.namprd12.prod.outlook.com (2603:10b6:208:314::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.17; Wed, 5 Jul 2023 11:11:14 +0000 Received: from BN8NAM11FT075.eop-nam11.prod.protection.outlook.com (2603:10b6:408:131:cafe::13) by BN9PR03CA0523.outlook.office365.com (2603:10b6:408:131::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.23 via Frontend Transport; Wed, 5 Jul 2023 11:11:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BN8NAM11FT075.mail.protection.outlook.com (10.13.176.208) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.45 via Frontend Transport; Wed, 5 Jul 2023 11:11:13 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 5 Jul 2023 04:11:01 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 5 Jul 2023 04:11:00 -0700 From: Viacheslav Ovsiienko To: CC: , Subject: [PATCH v4 4/4] doc: add mlx5 datapath tracing feature description Date: Wed, 5 Jul 2023 14:10:38 +0300 Message-ID: <20230705111038.9935-5-viacheslavo@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20230705111038.9935-1-viacheslavo@nvidia.com> References: <20230420100803.494-1-viacheslavo@nvidia.com> <20230705111038.9935-1-viacheslavo@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT075:EE_|BL1PR12MB5352:EE_ X-MS-Office365-Filtering-Correlation-Id: c2b7d226-89ad-4708-fee4-08db7d488bcd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tNtOd6/kbNsvCQTZvX5jHlrKVjpUZBzNIWoNfwACjXdbajHifwKiD5e7G8CrHpOMCZWY9fmhCRu8L657WsUvRgbWCv5MFKRqdLssVQjDQWeTYO+649DD93bh+uhbOld9OjzqgiY6M21ZEONig/bcQfwx75uWxODfyDCNzko1nPDe/JdTP/0g8gxoLIh/CBHTa5oyfpRSr1/3NTflwtR+1e7Nimc8v6TJ+gT8399Ve6p9S/j5grTVasHGMfdov7+JbtXqIvp7Ur3qVq0d7ybtjM6xXpV2kn7OPQV+RdHw3TQNYsbyzF/mMcF/LXmil7Lta4s73/a5e/eUrQRGc0zxIisWl6clzMdt4sGzGM9BbnBP5/289XQdWe06uOaMvwJ0UewimQ5SYPLPOZgnBfy71fo7wSB1Mc1Q7Rg7C/6Mv32/alJpRpiuDQuYEQ0+YNk74u2gRJE9z7V5RgsWucOswpmQk8beTd6w7WRWlpxGi9zlUqROX50GAgiTQNyHf665q2gs2swUkM//IIe/B2BCBLucUJo90+8nEsjwALc/tBUFnAK14x8chlYDJWXx8RFvJ71c6DcAFpEwTCQmWOiI8LLSyHAHQipecbxlAvXpIz1DqaZS6+btYxw4WTYuvWUyHpGIWbQDn+xQhhYGbgawY6HF2y8LOEfm3P5kskWUmRm/EwVYmQlb0fwxPFc+bBdH+4oWM811V6/W01WcE4lHXo3ohc1/ZVckwtfzQ+gVoarvvYr5LBzHI+acjGJ7EaWkytMEs+HE61NCqs5sl10sH89xO9qoI6hSEufgu5b9t+k= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(136003)(346002)(376002)(396003)(451199021)(46966006)(40470700004)(36840700001)(41300700001)(54906003)(55016003)(40480700001)(7696005)(36756003)(5660300002)(316002)(2906002)(8936002)(8676002)(6916009)(70586007)(4326008)(70206006)(40460700003)(426003)(82740400003)(336012)(966005)(6666004)(478600001)(107886003)(82310400005)(186003)(86362001)(6286002)(2616005)(356005)(7636003)(26005)(83380400001)(47076005)(36860700001)(1076003)(16526019); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2023 11:11:13.7676 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c2b7d226-89ad-4708-fee4-08db7d488bcd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT075.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5352 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The mlx5 provides the send scheduling on specific moment of time, and for the related kind of applications it would be extremely useful to have extra debug information - when and how packets were scheduled and when the actual sending was completed by the NIC hardware (it helps application to track the internal delay issues). The patch adds the documentation for feature usage. Signed-off-by: Viacheslav Ovsiienko --- doc/guides/nics/mlx5.rst | 78 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index b9843edbd9..1c8fc6f6d4 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -2077,3 +2077,81 @@ where: * ``sw_queue_id``: queue index in range [64536, 65535]. This range is the highest 1000 numbers. * ``hw_queue_id``: queue index given by HW in queue creation. + + +Tx datapath tracing +^^^^^^^^^^^^^^^^^^^ + +The mlx5 provides the Tx datapath tracing capability with extra debug +information - when and how packets were scheduled and when the actual +sending was completed by the NIC hardware. The feature engages the +existing DPDK datapath tracing capability. + +Usage of the mlx5 Tx datapath tracing: + +#. Build DPDK application with enabled datapath tracking + + * The meson option should be specified: ``--enable_trace_fp=true`` + * The c_args should be specified: ``-DALLOW_EXPERIMENTAL_API`` + + .. code-block:: console + + meson configure --buildtype=debug -Denable_trace_fp=true + -Dc_args='-DRTE_LIBRTE_MLX5_DEBUG -DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build + + meson configure --buildtype=release -Denable_trace_fp=true + -Dc_args='-DRTE_ENABLE_ASSERT -DALLOW_EXPERIMENTAL_API' build + +#. Configure the NIC + + If the sending completion timings are important the NIC should be configured + to provide realtime timestamps, the ``REAL_TIME_CLOCK_ENABLE`` NV settings + parameter should be configured as TRUE. + + .. code-block:: console + + mlxconfig -d /dev/mst/mt4125_pciconf0 s REAL_TIME_CLOCK_ENABLE=1 + +#. Run application with EAL parameters configuring the tracing in mlx5 Tx datapath + + * ``--trace=pmd.net.mlx5.tx`` - the regular expression enabling the tracepoints + with matching names at least "pmd.net.mlx5.tx" must be enabled to gather all + events needed to analyze mlx5 Tx datapath and its timings. By default all + tracepoints are disabled. + +#. Store the file with gathered tracing information + +#. Install or build the ``babeltrace2`` package + + The gathered trace data can be analyzed with a developed Python script. + To parse the trace, the data script uses the ``babeltrace2`` library. + The package should be either installed or built from source code as + shown below. + + .. code-block:: console + + git clone https://github.com/efficios/babeltrace.git + cd babeltrace + ./bootstrap + ./configure -help + ./configure --disable-api-doc --disable-man-pages + --disable-python-bindings-doc --enable-python-plugins + --enable-python-binding + +#. Run analyzing scrypt (in Python) to combine related events (packet firing and + completion) and see the output in human-readable view + + The analyzing script is located in the folder: ``./drivers/net/mlx5/tools`` + It requires Python3.6, ``babeltrace2`` packages and it takes the only parameter + of trace data file. + + .. code-block:: console + + ./mlx5_trace.py /var/log/rte-2023-01-23-AM-11-52-39 + +#. Interpreting the Script Output Data + + All the timings are given in nanoseconds. + The list of Tx bursts per port/queue is presented in the output. + Each list element contains the list of built WQEs with specific opcodes, and + each WQE contains the list of the encompassed packets to send.