ethdev: document that stats reset APIs are not thread-safe
Checks
Commit Message
Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
safe has performance impact on datapath.
Instead document APIs as not thread safe and add condition for reliable
stats reset functionality, forwarding should be stopped.
Cc: stable@dpdk.org
Signed-off-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
Cc: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Morten Brørup <mb@smartsharesystems.com>
This update triggered by mail list discussion [1].
[1]
https://inbox.dpdk.org/dev/3b2cf48e-2293-4226-b6cd-5f4dd3969f99@lysator.liu.se/
---
lib/ethdev/rte_ethdev.h | 6 ++++++
1 file changed, 6 insertions(+)
Comments
> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> Sent: Thursday, 25 April 2024 18.53
>
> Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
> safe has performance impact on datapath.
>
> Instead document APIs as not thread safe and add condition for reliable
> stats reset functionality, forwarding should be stopped.
I'm not sure stopping forwarding suffices.
NIC hardware counters will keep progressing unless RX and TX is stopped at NIC level.
I don't have any suggestions for a better wording, though. :-(
Anyway, better with the patch than without...
Acked-by: Morten Brørup <mb@smartsharesystems.com>
On Fri, 26 Apr 2024 14:20:01 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:
> > From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> > Sent: Thursday, 25 April 2024 18.53
> >
> > Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
> > safe has performance impact on datapath.
> >
> > Instead document APIs as not thread safe and add condition for reliable
> > stats reset functionality, forwarding should be stopped.
>
> I'm not sure stopping forwarding suffices.
> NIC hardware counters will keep progressing unless RX and TX is stopped at NIC level.
>
> I don't have any suggestions for a better wording, though. :-(
>
> Anyway, better with the patch than without...
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>
The safest option would be:
rte_eth_dev_stop
rte_eth_stats_reset
rte_eth_dev_start
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, 26 April 2024 17.14
>
> On Fri, 26 Apr 2024 14:20:01 +0200
> Morten Brørup <mb@smartsharesystems.com> wrote:
>
> > > From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> > > Sent: Thursday, 25 April 2024 18.53
> > >
> > > Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
> > > safe has performance impact on datapath.
> > >
> > > Instead document APIs as not thread safe and add condition for reliable
> > > stats reset functionality, forwarding should be stopped.
> >
> > I'm not sure stopping forwarding suffices.
> > NIC hardware counters will keep progressing unless RX and TX is stopped at
> NIC level.
> >
> > I don't have any suggestions for a better wording, though. :-(
> >
> > Anyway, better with the patch than without...
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> >
>
> The safest option would be:
> rte_eth_dev_stop
> rte_eth_stats_reset
> rte_eth_dev_start
Yes, but this will cause packet loss.
Network admins should be able to clear the counters without causing packet loss.
Recheck-request: iol-compile-amd64-testing
The DPDK Community Lab updated to the latest Alpine image yesterday, which
resulted in all Alpine builds failing. The failure is unrelated to your
patch, and this recheck should remove the fail on Patchwork, as we have
disabled Alpine testing for now.
On Fri, 26 Apr 2024 17:17:25 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:
> > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > Sent: Friday, 26 April 2024 17.14
> >
> > On Fri, 26 Apr 2024 14:20:01 +0200
> > Morten Brørup <mb@smartsharesystems.com> wrote:
> >
> > > > From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> > > > Sent: Thursday, 25 April 2024 18.53
> > > >
> > > > Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
> > > > safe has performance impact on datapath.
> > > >
> > > > Instead document APIs as not thread safe and add condition for reliable
> > > > stats reset functionality, forwarding should be stopped.
> > >
> > > I'm not sure stopping forwarding suffices.
> > > NIC hardware counters will keep progressing unless RX and TX is stopped at
> > NIC level.
> > >
> > > I don't have any suggestions for a better wording, though. :-(
> > >
> > > Anyway, better with the patch than without...
> > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > >
> >
> > The safest option would be:
> > rte_eth_dev_stop
> > rte_eth_stats_reset
> > rte_eth_dev_start
>
> Yes, but this will cause packet loss.
> Network admins should be able to clear the counters without causing packet loss.
>
Another alternative would be to a "reset zero" approach.
When reset is done, capture the current state of all the relevant counters.
Record that, and subtract those on next request. Never modify the
underlying counters.
To make it MT safe, use lock on reset and something super lightweight
like seqlock on the stats request side. Might be hard with the size and variable
length nature of xstats thoug.
On 2024-04-26 14:20, Morten Brørup wrote:
>> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
>> Sent: Thursday, 25 April 2024 18.53
>>
>> Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
>> safe has performance impact on datapath.
>>
>> Instead document APIs as not thread safe and add condition for reliable
>> stats reset functionality, forwarding should be stopped.
>
What does "forwarding" mean here? Sounds like something from testpmd.
Isn't what needs to stop here other threads calling into the ethdev API
for this particular device?
> I'm not sure stopping forwarding suffices.
> NIC hardware counters will keep progressing unless RX and TX is stopped at NIC level.
>
Why would that be an issue?
> I don't have any suggestions for a better wording, though. :-(
>
> Anyway, better with the patch than without...
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>
On 2024-04-26 17:17, Morten Brørup wrote:
>> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
>> Sent: Friday, 26 April 2024 17.14
>>
>> On Fri, 26 Apr 2024 14:20:01 +0200
>> Morten Brørup <mb@smartsharesystems.com> wrote:
>>
>>>> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
>>>> Sent: Thursday, 25 April 2024 18.53
>>>>
>>>> Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
>>>> safe has performance impact on datapath.
>>>>
>>>> Instead document APIs as not thread safe and add condition for reliable
>>>> stats reset functionality, forwarding should be stopped.
>>>
>>> I'm not sure stopping forwarding suffices.
>>> NIC hardware counters will keep progressing unless RX and TX is stopped at
>> NIC level.
>>>
>>> I don't have any suggestions for a better wording, though. :-(
>>>
>>> Anyway, better with the patch than without...
>>> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>>>
>>
>> The safest option would be:
>> rte_eth_dev_stop
>> rte_eth_stats_reset
>> rte_eth_dev_start
>
> Yes, but this will cause packet loss.
> Network admins should be able to clear the counters without causing packet loss.
>
For sure, and they would do it via some O&M interface, which would in
turn talk the control plane, which in turn would talk to the data plane
(including <rte_ethdev.h>).
Either the control plane or the O&M layer could keep track of a reset
offset. Doesn't have to be on the PMD level.
> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Sunday, 28 April 2024 17.53
>
> On 2024-04-26 17:17, Morten Brørup wrote:
> >> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> >> Sent: Friday, 26 April 2024 17.14
> >>
> >> On Fri, 26 Apr 2024 14:20:01 +0200
> >> Morten Brørup <mb@smartsharesystems.com> wrote:
> >>
> >>>> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> >>>> Sent: Thursday, 25 April 2024 18.53
> >>>>
> >>>> Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
> >>>> safe has performance impact on datapath.
> >>>>
> >>>> Instead document APIs as not thread safe and add condition for reliable
> >>>> stats reset functionality, forwarding should be stopped.
> >>>
> >>> I'm not sure stopping forwarding suffices.
> >>> NIC hardware counters will keep progressing unless RX and TX is stopped at
> >> NIC level.
> >>>
> >>> I don't have any suggestions for a better wording, though. :-(
> >>>
> >>> Anyway, better with the patch than without...
> >>> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> >>>
> >>
> >> The safest option would be:
> >> rte_eth_dev_stop
> >> rte_eth_stats_reset
> >> rte_eth_dev_start
> >
> > Yes, but this will cause packet loss.
> > Network admins should be able to clear the counters without causing packet
> loss.
> >
>
> For sure, and they would do it via some O&M interface, which would in
> turn talk the control plane, which in turn would talk to the data plane
> (including <rte_ethdev.h>).
>
> Either the control plane or the O&M layer could keep track of a reset
> offset. Doesn't have to be on the PMD level.
Exactly.
Snapshotting the "reset" offset in the control plane has many advantages.
Let's promote that design pattern, rather than trying to work around different quirks required by different drivers.
> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Sunday, 28 April 2024 17.49
>
> On 2024-04-26 14:20, Morten Brørup wrote:
> >> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
> >> Sent: Thursday, 25 April 2024 18.53
> >>
> >> Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
> >> safe has performance impact on datapath.
> >>
> >> Instead document APIs as not thread safe and add condition for reliable
> >> stats reset functionality, forwarding should be stopped.
> >
>
> What does "forwarding" mean here? Sounds like something from testpmd.
>
> Isn't what needs to stop here other threads calling into the ethdev API
> for this particular device?
Doesn't suffice.
The NIC hardware counters increment if the NIC's DMA engine receives or transmits packets.
E.g. in the RX direction, the NIC hardware counters increment according to what the NIC detects happening on the wire. If some packets are DMA'ed into memory and the associated RX descriptors are filled, the NIC's hardware counters progress. It doesn't matter if the software has looked at those RX descriptors or not.
>
> > I'm not sure stopping forwarding suffices.
> > NIC hardware counters will keep progressing unless RX and TX is stopped at
> NIC level.
> >
>
> Why would that be an issue?
My point is:
Unless you stop everything, including the NIC hardware RX & TX DMA, the counters might not be zero when returning from the stats_reset() functions.
>
> > I don't have any suggestions for a better wording, though. :-(
I think it is fine providing a notice about MT-unsafety in the API documentation.
But the only way to prevent it is useless in real applications, as it would cause packet loss. So it's not helpful describing how to do that.
> >
> > Anyway, better with the patch than without...
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> >
On 2024-04-29 09:57, Morten Brørup wrote:
>> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
>> Sent: Sunday, 28 April 2024 17.49
>>
>> On 2024-04-26 14:20, Morten Brørup wrote:
>>>> From: Ferruh Yigit [mailto:ferruh.yigit@amd.com]
>>>> Sent: Thursday, 25 April 2024 18.53
>>>>
>>>> Making 'rte_eth_stats_reset()' and 'rte_eth_xstats_reset()' APIs thread
>>>> safe has performance impact on datapath.
>>>>
>>>> Instead document APIs as not thread safe and add condition for reliable
>>>> stats reset functionality, forwarding should be stopped.
>>>
>>
>> What does "forwarding" mean here? Sounds like something from testpmd.
>>
>> Isn't what needs to stop here other threads calling into the ethdev API
>> for this particular device?
>
> Doesn't suffice.
> The NIC hardware counters increment if the NIC's DMA engine receives or transmits packets.
> E.g. in the RX direction, the NIC hardware counters increment according to what the NIC detects happening on the wire. If some packets are DMA'ed into memory and the associated RX descriptors are filled, the NIC's hardware counters progress. It doesn't matter if the software has looked at those RX descriptors or not.
>
>>
>>> I'm not sure stopping forwarding suffices.
>>> NIC hardware counters will keep progressing unless RX and TX is stopped at
>> NIC level.
>>>
>>
>> Why would that be an issue?
>
> My point is:
> Unless you stop everything, including the NIC hardware RX & TX DMA, the counters might not be zero when returning from the stats_reset() functions.
>
The reset can still be MT safe, even though there is no way to retrieve
the exact counter state prior to the reset and also is no guarantee that
the counters are zero immediately after the function returns (whatever
that means).
If you do want to be able to do that, the means to that end is to teach
the reset function to return the prior counter state as a part of the
reset call, not to force the user to have the system grind to a halt.
>>
>>> I don't have any suggestions for a better wording, though. :-(
>
> I think it is fine providing a notice about MT-unsafety in the API documentation.
> But the only way to prevent it is useless in real applications, as it would cause packet loss. So it's not helpful describing how to do that.
>
>>>
>>> Anyway, better with the patch than without...
>>> Acked-by: Morten Brørup <mb@smartsharesystems.com>
>>>
On Mon, 29 Apr 2024 08:20:13 +0200
Morten Brørup <mb@smartsharesystems.com> wrote:
> > turn talk the control plane, which in turn would talk to the data plane
> > (including <rte_ethdev.h>).
> >
> > Either the control plane or the O&M layer could keep track of a reset
> > offset. Doesn't have to be on the PMD level.
>
> Exactly.
>
> Snapshotting the "reset" offset in the control plane has many advantages.
>
> Let's promote that design pattern, rather than trying to work around different quirks required by different drivers.
>
The other thing that can help in software drivers is keeping per-queue stats.
@@ -3136,6 +3136,9 @@ int rte_eth_stats_get(uint16_t port_id, struct rte_eth_stats *stats);
/**
* Reset the general I/O statistics of an Ethernet device.
*
+ * API is not multi-thread safe.
+ * Application should stop forwarding before calling this API.
+ *
* @param port_id
* The port identifier of the Ethernet device.
* @return
@@ -3296,6 +3299,9 @@ int rte_eth_xstats_get_id_by_name(uint16_t port_id, const char *xstat_name,
/**
* Reset extended statistics of an Ethernet device.
*
+ * API is not multi-thread safe.
+ * Application should stop forwarding before calling this API.
+ *
* @param port_id
* The port identifier of the Ethernet device.
* @return