[RFC] net/af_packet: make stats reset reliable
Checks
Commit Message
For stats reset, use an offset instead of zeroing out actual stats values,
get_stats() displays diff between stats and offset.
This way stats only updated in datapath and offset only updated in stats
reset function. This makes stats reset function more reliable.
As stats only written by single thread, we can remove 'volatile' qualifier
which should improve the performance in datapath.
Signed-off-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
Cc: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Morten Brørup <mb@smartsharesystems.com>
This update triggered by mail list discussion [1].
[1]
https://inbox.dpdk.org/dev/3b2cf48e-2293-4226-b6cd-5f4dd3969f99@lysator.liu.se/
---
drivers/net/af_packet/rte_eth_af_packet.c | 69 +++++++++++++++--------
1 file changed, 44 insertions(+), 25 deletions(-)
Comments
> +static uint64_t
> +stats_get_diff(uint64_t stats, uint64_t offset)
> +{
> + if (stats >= offset)
> + return stats - offset;
> + /* unlikely wraparound case */
> + return UINT64_MAX + stats - offset;
The numbers are unsigned, so wrapping comes for free.
Remove the comparison and always return stats - offset.
Using uint8_t for easier explanation, if offset is 255 and stats is 0, then the diff should be 1.
Returning stats - offset:
stats - offset = 0 - 255 = 0 - 0xFF = 1.
Returning UINT8_MAX + stats - offset is wrong:
UINT8_MAX + stats - offset = 255 - 0 - 255 = 0.
Besides that, it looks good to me.
While reviewing, I came across the rx_mbuf_alloc_failed counter in the rte_eth_dev_data structure:
https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/rte_ethdev.c#L3145
https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/ethdev_driver.h#L127
Doesn't it have the same problem?
BTW, the af_packet PMD doesn't increase the rx_mbuf_alloc_failed counter on mbuf allocation failures. But that's a separate bug.
On 4/26/2024 12:33 PM, Morten Brørup wrote:
>> +static uint64_t
>> +stats_get_diff(uint64_t stats, uint64_t offset)
>> +{
>> + if (stats >= offset)
>> + return stats - offset;
>> + /* unlikely wraparound case */
>> + return UINT64_MAX + stats - offset;
>
> The numbers are unsigned, so wrapping comes for free.
>
> Remove the comparison and always return stats - offset.
>
> Using uint8_t for easier explanation, if offset is 255 and stats is 0, then the diff should be 1.
> Returning stats - offset:
> stats - offset = 0 - 255 = 0 - 0xFF = 1.
>
> Returning UINT8_MAX + stats - offset is wrong:
> UINT8_MAX + stats - offset = 255 - 0 - 255 = 0.
>
> Besides that, it looks good to me.
>
Yes, it is wrong, and thanks for removing comparison tip.
But thinking twice, taking wrapping into account for a uint64_t variable
can be being too cautious anyway. I will remove it completely.
>
> While reviewing, I came across the rx_mbuf_alloc_failed counter in the rte_eth_dev_data structure:
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/rte_ethdev.c#L3145
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/ethdev_driver.h#L127
>
> Doesn't it have the same problem?
>
stats reset problem? af_packet is not collecting 'rx_mbuf_alloc_failed',
so nothing to do there for af_packet.
>
> BTW, the af_packet PMD doesn't increase the rx_mbuf_alloc_failed counter on mbuf allocation failures. But that's a separate bug.
>
Yes it is missing, but I wouldn't call it a bug, just one of the stats
is missing. And yes this can be handled separately if required.
> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> Sent: Friday, 26 April 2024 15.38
>
> On 4/26/2024 12:33 PM, Morten Brørup wrote:
[...]
> > While reviewing, I came across the rx_mbuf_alloc_failed counter in the
> rte_eth_dev_data structure:
> > https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/rte_ethdev.c#L3145
> >
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/ethdev_driver.h#L127
> >
> > Doesn't it have the same problem?
> >
>
> stats reset problem? af_packet is not collecting 'rx_mbuf_alloc_failed',
> so nothing to do there for af_packet.
Agreed, not related to af_packet or this patch.
I'm just wondering if a similar or other patch should be applied to rx_mbuf_alloc_failed in the ethdev layer.
rx_mbuf_alloc_failed is shared by lcores, so perhaps it should be atomic, or atomically incremented by drivers using it.
>
> >
> > BTW, the af_packet PMD doesn't increase the rx_mbuf_alloc_failed counter on
> mbuf allocation failures. But that's a separate bug.
> >
>
> Yes it is missing, but I wouldn't call it a bug, just one of the stats
> is missing. And yes this can be handled separately if required.
OK, then just some stats missing, not a bug.
Quite useful stats for debugging a production system, if mbuf allocations fail.
But still, not related to this patch.
Recheck-request: iol-compile-amd64-testing
The DPDK Community Lab updated to the latest Alpine image yesterday, which
resulted in all Alpine builds failing. The failure is unrelated to your
patch, and this recheck should remove the fail on Patchwork, as we have
disabled Alpine testing for now.
On 2024-04-26 13:33, Morten Brørup wrote:
>> +static uint64_t
>> +stats_get_diff(uint64_t stats, uint64_t offset)
>> +{
>> + if (stats >= offset)
>> + return stats - offset;
>> + /* unlikely wraparound case */
>> + return UINT64_MAX + stats - offset;
>
> The numbers are unsigned, so wrapping comes for free.
>
With 64-bit counters, will they ever wrap? If you constantly run 100
Gbps it'll take > 1000 years before the byte counter wrap.
> Remove the comparison and always return stats - offset.
>
> Using uint8_t for easier explanation, if offset is 255 and stats is 0, then the diff should be 1.
> Returning stats - offset:
> stats - offset = 0 - 255 = 0 - 0xFF = 1.
>
> Returning UINT8_MAX + stats - offset is wrong:
> UINT8_MAX + stats - offset = 255 - 0 - 255 = 0.
>
> Besides that, it looks good to me.
>
>
> While reviewing, I came across the rx_mbuf_alloc_failed counter in the rte_eth_dev_data structure:
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/rte_ethdev.c#L3145
> https://elixir.bootlin.com/dpdk/v24.03/source/lib/ethdev/ethdev_driver.h#L127
>
> Doesn't it have the same problem?
>
>
> BTW, the af_packet PMD doesn't increase the rx_mbuf_alloc_failed counter on mbuf allocation failures. But that's a separate bug.
>
@@ -51,8 +51,10 @@ struct pkt_rx_queue {
uint16_t in_port;
uint8_t vlan_strip;
- volatile unsigned long rx_pkts;
- volatile unsigned long rx_bytes;
+ uint64_t rx_pkts;
+ uint64_t rx_bytes;
+ uint64_t rx_pkts_offset;
+ uint64_t rx_bytes_offset;
};
struct pkt_tx_queue {
@@ -64,9 +66,12 @@ struct pkt_tx_queue {
unsigned int framecount;
unsigned int framenum;
- volatile unsigned long tx_pkts;
- volatile unsigned long err_pkts;
- volatile unsigned long tx_bytes;
+ uint64_t tx_pkts;
+ uint64_t err_pkts;
+ uint64_t tx_bytes;
+ uint64_t tx_pkts_offset;
+ uint64_t err_pkts_offset;
+ uint64_t tx_bytes_offset;
};
struct pmd_internals {
@@ -385,8 +390,18 @@ eth_dev_info(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
return 0;
}
+
+static uint64_t
+stats_get_diff(uint64_t stats, uint64_t offset)
+{
+ if (stats >= offset)
+ return stats - offset;
+ /* unlikely wraparound case */
+ return UINT64_MAX + stats - offset;
+}
+
static int
-eth_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *igb_stats)
+eth_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
{
unsigned i, imax;
unsigned long rx_total = 0, tx_total = 0, tx_err_total = 0;
@@ -396,27 +411,29 @@ eth_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *igb_stats)
imax = (internal->nb_queues < RTE_ETHDEV_QUEUE_STAT_CNTRS ?
internal->nb_queues : RTE_ETHDEV_QUEUE_STAT_CNTRS);
for (i = 0; i < imax; i++) {
- igb_stats->q_ipackets[i] = internal->rx_queue[i].rx_pkts;
- igb_stats->q_ibytes[i] = internal->rx_queue[i].rx_bytes;
- rx_total += igb_stats->q_ipackets[i];
- rx_bytes_total += igb_stats->q_ibytes[i];
+ struct pkt_rx_queue *rxq = &internal->rx_queue[i];
+ stats->q_ipackets[i] = stats_get_diff(rxq->rx_pkts, rxq->rx_pkts_offset);
+ stats->q_ibytes[i] = stats_get_diff(rxq->rx_bytes, rxq->rx_bytes_offset);
+ rx_total += stats->q_ipackets[i];
+ rx_bytes_total += stats->q_ibytes[i];
}
imax = (internal->nb_queues < RTE_ETHDEV_QUEUE_STAT_CNTRS ?
internal->nb_queues : RTE_ETHDEV_QUEUE_STAT_CNTRS);
for (i = 0; i < imax; i++) {
- igb_stats->q_opackets[i] = internal->tx_queue[i].tx_pkts;
- igb_stats->q_obytes[i] = internal->tx_queue[i].tx_bytes;
- tx_total += igb_stats->q_opackets[i];
- tx_err_total += internal->tx_queue[i].err_pkts;
- tx_bytes_total += igb_stats->q_obytes[i];
+ struct pkt_tx_queue *txq = &internal->tx_queue[i];
+ stats->q_opackets[i] = stats_get_diff(txq->tx_pkts, txq->tx_pkts_offset);
+ stats->q_obytes[i] = stats_get_diff(txq->tx_bytes, txq->tx_bytes_offset);
+ tx_total += stats->q_opackets[i];
+ tx_err_total += stats_get_diff(txq->err_pkts, txq->err_pkts_offset);
+ tx_bytes_total += stats->q_obytes[i];
}
- igb_stats->ipackets = rx_total;
- igb_stats->ibytes = rx_bytes_total;
- igb_stats->opackets = tx_total;
- igb_stats->oerrors = tx_err_total;
- igb_stats->obytes = tx_bytes_total;
+ stats->ipackets = rx_total;
+ stats->ibytes = rx_bytes_total;
+ stats->opackets = tx_total;
+ stats->oerrors = tx_err_total;
+ stats->obytes = tx_bytes_total;
return 0;
}
@@ -427,14 +444,16 @@ eth_stats_reset(struct rte_eth_dev *dev)
struct pmd_internals *internal = dev->data->dev_private;
for (i = 0; i < internal->nb_queues; i++) {
- internal->rx_queue[i].rx_pkts = 0;
- internal->rx_queue[i].rx_bytes = 0;
+ struct pkt_rx_queue *rxq = &internal->rx_queue[i];
+ rxq->rx_pkts_offset = rxq->rx_pkts;
+ rxq->rx_bytes_offset = rxq->rx_bytes;
}
for (i = 0; i < internal->nb_queues; i++) {
- internal->tx_queue[i].tx_pkts = 0;
- internal->tx_queue[i].err_pkts = 0;
- internal->tx_queue[i].tx_bytes = 0;
+ struct pkt_tx_queue *txq = &internal->tx_queue[i];
+ txq->tx_pkts_offset = txq->tx_pkts;
+ txq->err_pkts_offset = txq->err_pkts;
+ txq->tx_bytes_offset = txq->tx_bytes;
}
return 0;