From patchwork Fri Dec 22 15:05:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rushil Gupta X-Patchwork-Id: 135504 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4EC0B43760; Fri, 22 Dec 2023 16:05:58 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C1B0B402E7; Fri, 22 Dec 2023 16:05:57 +0100 (CET) Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by mails.dpdk.org (Postfix) with ESMTP id 35A26402A6 for ; Fri, 22 Dec 2023 16:05:56 +0100 (CET) Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbdd300d01bso1419494276.3 for ; Fri, 22 Dec 2023 07:05:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1703257555; x=1703862355; darn=dpdk.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=TKeLKwT3emnoOhbOyX2smzBDKI6WhcqkJ6I6STIuUfE=; b=GaS2YkIkNjvBQ4qaB2DBJFG7QGal1FTiaVQfbFa1RGwINyv4GvAr2gCowa1Nkam2Ds 6VnkcnmWlq0V2IBdU2n7hk+3IT54Dg+SAZkRwJwd4zmC61IiKtDECG8Ua9p859bm9a8G P1PImkeAxqUwdWCQ/pss5Oop8afpvyXCBO+n8d7Oiry2TLvVl60d00RNZwRiOV9Lczmg Wv37d2TgP+wHvHWYi0kRV9eS6g6N1EetvjyP9DaMbwlJCDqiAK+ED1VdoJjWiFIkfYbL I7BdAx+zsRRhMra8uW5IqhDI/QcL1JleYUjMJh7uA/9/d1oRVaqd8+RhKnEa+0XmvSvI c1FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703257555; x=1703862355; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TKeLKwT3emnoOhbOyX2smzBDKI6WhcqkJ6I6STIuUfE=; b=faG5NdmNT+Xt5kxamkQVoxUiEr8ILXUmdBv2KgHY0OC8v+eW73zvFgXEhj3/1hUahQ Aj0aETabyxldG/7v/i20iQT3fhUUo1ltwjYcSqZmauK/D03yS4INbNmFvwQerzZMwl5t +XGJ1yMVaBSenLHMEHZn7H1TlFsAiiaa6liwohKmoIpQPUAWYZDIWNcIpifUScyTYEgV CWL3fyH9J6wtGLcvF/IvMlZ20iaIs3uxKtUaNaMeoMuS2OdRqZiwu3cKdSHgXovkqDp7 RGx+LMqQ0amUCPz9Y6m8scGlJYdg/DHCotdczGQmCyY5PU7HkolCQ9AkPQrbSNd/d6Cl JLsw== X-Gm-Message-State: AOJu0Yy5jFopta7IFJfXrow5c5xsnoumAeIUOuLknHVpyotC71wq39sn Av/RF97gRkBmnLiQ/N1uJkl1DF/2qnCjNiufQ3M= X-Google-Smtp-Source: AGHT+IHIm8LK3W3ZxQxNOiJFZoypdFJkexbQWUPKBv3yJBQCi1gmva6aqmKTGskOMt3bz5ytMpGwpU/aVdoD X-Received: from rushilg.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:f27]) (user=rushilg job=sendgmr) by 2002:a25:804d:0:b0:dbd:af15:eeb7 with SMTP id a13-20020a25804d000000b00dbdaf15eeb7mr28496ybn.13.1703257555547; Fri, 22 Dec 2023 07:05:55 -0800 (PST) Date: Fri, 22 Dec 2023 15:05:53 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231222150553.2695916-1-rushilg@google.com> Subject: [PATCH] net/gve: Enable stats reporting for GQ format From: Rushil Gupta To: qi.z.zhang@intel.com, ferruh.yigit@amd.com Cc: junfeng.guo@intel.com, dev@dpdk.org, Rushil Gupta , Joshua Washington X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Read from shared region to retrieve imissed statistics for GQ. Tested using `show port xstats ` in interactive mode. This metric can be triggered by using queues > cores. Signed-off-by: Rushil Gupta Reviewed-by: Joshua Washington --- drivers/net/gve/base/gve_adminq.h | 11 +++ drivers/net/gve/gve_ethdev.c | 142 ++++++++++++++++++++++++++++-- drivers/net/gve/gve_ethdev.h | 20 ++++- 3 files changed, 167 insertions(+), 6 deletions(-) diff --git a/drivers/net/gve/base/gve_adminq.h b/drivers/net/gve/base/gve_adminq.h index e30b184913..f05362f85f 100644 --- a/drivers/net/gve/base/gve_adminq.h +++ b/drivers/net/gve/base/gve_adminq.h @@ -314,6 +314,17 @@ struct gve_stats_report { GVE_CHECK_STRUCT_LEN(8, gve_stats_report); +/* Numbers of gve tx/rx stats in stats report. */ +#define GVE_TX_STATS_REPORT_NUM 6 +#define GVE_RX_STATS_REPORT_NUM 2 + +/* Interval to schedule a stats report update, 20000ms. */ +#define GVE_STATS_REPORT_TIMER_PERIOD 20000 + +/* Numbers of NIC tx/rx stats in stats report. */ +#define NIC_TX_STATS_REPORT_NUM 0 +#define NIC_RX_STATS_REPORT_NUM 4 + enum gve_stat_names { /* stats from gve */ TX_WAKE_CNT = 1, diff --git a/drivers/net/gve/gve_ethdev.c b/drivers/net/gve/gve_ethdev.c index ecd37ff37f..bb535a863f 100644 --- a/drivers/net/gve/gve_ethdev.c +++ b/drivers/net/gve/gve_ethdev.c @@ -125,6 +125,73 @@ gve_link_update(struct rte_eth_dev *dev, __rte_unused int wait_to_complete) return rte_eth_linkstatus_set(dev, &link); } +static int gve_alloc_stats_report(struct gve_priv *priv, + uint16_t nb_tx_queues, uint16_t nb_rx_queues) +{ + char z_name[RTE_MEMZONE_NAMESIZE]; + int tx_stats_cnt; + int rx_stats_cnt; + + tx_stats_cnt = (GVE_TX_STATS_REPORT_NUM + NIC_TX_STATS_REPORT_NUM) * + nb_tx_queues; + rx_stats_cnt = (GVE_RX_STATS_REPORT_NUM + NIC_RX_STATS_REPORT_NUM) * + nb_rx_queues; + priv->stats_report_len = sizeof(struct gve_stats_report) + + sizeof(struct stats) * (tx_stats_cnt + rx_stats_cnt); + + snprintf(z_name, sizeof(z_name), "stats_report_%s", priv->pci_dev->device.name); + priv->stats_report_mem = rte_memzone_reserve_aligned(z_name, + priv->stats_report_len, + rte_socket_id(), + RTE_MEMZONE_IOVA_CONTIG, PAGE_SIZE); + + if (!priv->stats_report_mem) + return -ENOMEM; + + /* offset by skipping stats written by gve. */ + priv->stats_start_idx = (GVE_TX_STATS_REPORT_NUM * nb_tx_queues) + + (GVE_RX_STATS_REPORT_NUM * nb_rx_queues); + priv->stats_end_idx = priv->stats_start_idx + + (NIC_TX_STATS_REPORT_NUM * nb_tx_queues) + + (NIC_RX_STATS_REPORT_NUM * nb_rx_queues) - 1; + + return 0; +} + +static void gve_free_stats_report(struct rte_eth_dev *dev) +{ + struct gve_priv *priv = dev->data->dev_private; + rte_memzone_free(priv->stats_report_mem); +} + +/* Read Rx NIC stats from shared region */ +static void gve_get_imissed_from_nic(struct rte_eth_dev *dev) +{ + struct gve_stats_report *stats_report; + struct gve_rx_queue *rxq; + struct gve_priv *priv; + struct stats stat; + int queue_id; + int stat_id; + int i; + + priv = dev->data->dev_private; + stats_report = (struct gve_stats_report *) + priv->stats_report_mem->addr; + + for (i = priv->stats_start_idx; i <= priv->stats_end_idx; i++) { + stat = stats_report->stats[i]; + queue_id = cpu_to_be32(stat.queue_id); + rxq = dev->data->rx_queues[queue_id]; + if (rxq == NULL) + continue; + stat_id = cpu_to_be32(stat.stat_name); + /* Update imissed. */ + if (stat_id == RX_NO_BUFFERS_POSTED) + rxq->stats.imissed = cpu_to_be64(stat.value); + } +} + static int gve_start_queues(struct rte_eth_dev *dev) { @@ -176,6 +243,7 @@ gve_start_queues(struct rte_eth_dev *dev) static int gve_dev_start(struct rte_eth_dev *dev) { + struct gve_priv *priv; int ret; ret = gve_start_queues(dev); @@ -187,6 +255,27 @@ gve_dev_start(struct rte_eth_dev *dev) dev->data->dev_started = 1; gve_link_update(dev, 0); + priv = dev->data->dev_private; + /* No stats available yet for Dqo. */ + if (gve_is_gqi(priv)) + { + ret = gve_alloc_stats_report(priv, + dev->data->nb_tx_queues, + dev->data->nb_rx_queues); + if (ret != 0) { + PMD_DRV_LOG(ERR, + "Failed to allocate region for stats reporting."); + return ret; + } + ret = gve_adminq_report_stats(priv, priv->stats_report_len, + priv->stats_report_mem->iova, + GVE_STATS_REPORT_TIMER_PERIOD); + if (ret != 0) { + PMD_DRV_LOG(ERR, "gve_adminq_report_stats command failed."); + return ret; + } + } + return 0; } @@ -200,6 +289,9 @@ gve_dev_stop(struct rte_eth_dev *dev) dev->data->dev_started = 0; + if (gve_is_gqi(dev->data->dev_private)) + gve_free_stats_report(dev); + return 0; } @@ -352,6 +444,8 @@ static int gve_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) { uint16_t i; + if (gve_is_gqi(dev->data->dev_private)) + gve_get_imissed_from_nic(dev); for (i = 0; i < dev->data->nb_tx_queues; i++) { struct gve_tx_queue *txq = dev->data->tx_queues[i]; @@ -372,6 +466,7 @@ gve_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) stats->ibytes += rxq->stats.bytes; stats->ierrors += rxq->stats.errors; stats->rx_nombuf += rxq->stats.no_mbufs; + stats->imissed += rxq->stats.imissed; } return 0; @@ -443,6 +538,7 @@ static const struct gve_xstats_name_offset rx_xstats_name_offset[] = { { "errors", RX_QUEUE_STATS_OFFSET(errors) }, { "mbuf_alloc_errors", RX_QUEUE_STATS_OFFSET(no_mbufs) }, { "mbuf_alloc_errors_bulk", RX_QUEUE_STATS_OFFSET(no_mbufs_bulk) }, + { "imissed", RX_QUEUE_STATS_OFFSET(imissed) }, }; static int @@ -614,17 +710,53 @@ gve_teardown_device_resources(struct gve_priv *priv) gve_clear_device_resources_ok(priv); } +static uint8_t +pci_dev_find_capability(struct rte_pci_device *pdev, int cap) +{ + uint8_t pos, id; + uint16_t ent; + int loops; + int ret; + + ret = rte_pci_read_config(pdev, &pos, sizeof(pos), PCI_CAPABILITY_LIST); + if (ret != sizeof(pos)) + return 0; + + loops = (PCI_CFG_SPACE_SIZE - PCI_STD_HEADER_SIZEOF) / PCI_CAP_SIZEOF; + + while (pos && loops--) { + ret = rte_pci_read_config(pdev, &ent, sizeof(ent), pos); + if (ret != sizeof(ent)) + return 0; + + id = ent & 0xff; + if (id == 0xff) + break; + + if (id == cap) + return pos; + + pos = (ent >> 8); + } + + return 0; +} + static int pci_dev_msix_vec_count(struct rte_pci_device *pdev) { - off_t msix_pos = rte_pci_find_capability(pdev, RTE_PCI_CAP_ID_MSIX); + uint8_t msix_cap = pci_dev_find_capability(pdev, PCI_CAP_ID_MSIX); uint16_t control; + int ret; - if (msix_pos > 0 && rte_pci_read_config(pdev, &control, sizeof(control), - msix_pos + RTE_PCI_MSIX_FLAGS) == sizeof(control)) - return (control & RTE_PCI_MSIX_FLAGS_QSIZE) + 1; + if (!msix_cap) + return 0; - return 0; + ret = rte_pci_read_config(pdev, &control, sizeof(control), msix_cap + PCI_MSIX_FLAGS); + if (ret != sizeof(control)) + return 0; + + return (control & PCI_MSIX_FLAGS_QSIZE) + 1; } static int diff --git a/drivers/net/gve/gve_ethdev.h b/drivers/net/gve/gve_ethdev.h index 58d8943e71..0e1bfec871 100644 --- a/drivers/net/gve/gve_ethdev.h +++ b/drivers/net/gve/gve_ethdev.h @@ -8,13 +8,25 @@ #include #include #include -#include #include "base/gve.h" /* TODO: this is a workaround to ensure that Tx complq is enough */ #define DQO_TX_MULTIPLIER 4 +/* + * Following macros are derived from linux/pci_regs.h, however, + * we can't simply include that header here, as there is no such + * file for non-Linux platform. + */ +#define PCI_CFG_SPACE_SIZE 256 +#define PCI_CAPABILITY_LIST 0x34 /* Offset of first capability list entry */ +#define PCI_STD_HEADER_SIZEOF 64 +#define PCI_CAP_SIZEOF 4 +#define PCI_CAP_ID_MSIX 0x11 /* MSI-X */ +#define PCI_MSIX_FLAGS 2 /* Message Control */ +#define PCI_MSIX_FLAGS_QSIZE 0x07FF /* Table size */ + #define GVE_DEFAULT_RX_FREE_THRESH 64 #define GVE_DEFAULT_TX_FREE_THRESH 32 #define GVE_DEFAULT_TX_RS_THRESH 32 @@ -85,6 +97,7 @@ struct gve_rx_stats { uint64_t errors; uint64_t no_mbufs; uint64_t no_mbufs_bulk; + uint64_t imissed; }; struct gve_xstats_name_offset { @@ -272,6 +285,11 @@ struct gve_priv { struct gve_tx_queue **txqs; struct gve_rx_queue **rxqs; + + uint32_t stats_report_len; + const struct rte_memzone *stats_report_mem; + uint16_t stats_start_idx; /* start index of array of stats written by NIC */ + uint16_t stats_end_idx; /* end index of array of stats written by NIC */ }; static inline bool