doc/eal: add caveat about spinlocks from non-pinned threads
Checks
Commit Message
Need to warn users of DPDK spinlocks from non-pinned threads.
This is similar wording to Linux documentation in pthread_spin_init.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
1 file changed, 10 insertions(+)
Comments
On Fri, Jun 10, 2022 at 08:28:19AM -0700, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> ---
> doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>
> The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>
> ++ locking
> +
> + If a pthread, that is not pinned to an lcore acquires a lock such as a
> + DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> + then there is a possibility of large application delays.
> + The problem is that if a thread is scheduled off the CPU while it holds
> + a lock, then other threads will waste time spinning on the lock until
> + the lock holder is once more rescheduled and releases the lock.
> +
> +
> cgroup control
> ~~~~~~~~~~~~~~
>
> --
> 2.35.1
On Fri, Jun 10, 2022 at 08:28:19AM -0700, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>
> The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>
> ++ locking
> +
> + If a pthread, that is not pinned to an lcore acquires a lock such as a
nit: suggest not using term pthread but instead just say thread as not
to imply a specific platform/implementation.
> + DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> + then there is a possibility of large application delays.
> + The problem is that if a thread is scheduled off the CPU while it holds
> + a lock, then other threads will waste time spinning on the lock until
'until the lock holder' -> 'until the thread holding the lock'
but i'm not really fussed, just a suggestion.
> + the lock holder is once more rescheduled and releases the lock.
> +
> +
> cgroup control
> ~~~~~~~~~~~~~~
>
> --
> 2.35.1
On Fri, 10 Jun 2022 16:48:15 -0700
Tyler Retzlaff <roretzla@linux.microsoft.com> wrote:
> >
> > ++ locking
> > +
> > + If a pthread, that is not pinned to an lcore acquires a lock such as a
>
> nit: suggest not using term pthread but instead just say thread as not
> to imply a specific platform/implementation.
>
> > + DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> > + then there is a possibility of large application delays.
> > + The problem is that if a thread is scheduled off the CPU while it holds
> > + a lock, then other threads will waste time spinning on the lock until
>
> 'until the lock holder' -> 'until the thread holding the lock'
>
> but i'm not really fussed, just a suggestion.
Sure, that wording was from existing pthread_spin_init() man page
On 2022/6/10 23:28, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>
> The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>
> ++ locking
> +
> + If a pthread, that is not pinned to an lcore acquires a lock such as a
> + DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
Some APIs inherently use rte_spinlock, just like rte_malloc/rte_eal_alarm_set,
Because DPDK API mainly use rte_spinlock to support thread-safty.
Suggest declare DPDK API mainly use rte_spinlock to support thread-safty, so
if the caller thread is not pinned to an lcore may encount a possibility of
large application delays.
> + then there is a possibility of large application delays.
> + The problem is that if a thread is scheduled off the CPU while it holds
> + a lock, then other threads will waste time spinning on the lock until
> + the lock holder is once more rescheduled and releases the lock.
> +
> +
> cgroup control
> ~~~~~~~~~~~~~~
>
>
On Sat, 11 Jun 2022 09:55:00 +0800
fengchengwen <fengchengwen@huawei.com> wrote:
> On 2022/6/10 23:28, Stephen Hemminger wrote:
> > Need to warn users of DPDK spinlocks from non-pinned threads.
> > This is similar wording to Linux documentation in pthread_spin_init.
> >
> > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > ---
> > doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
> > 1 file changed, 10 insertions(+)
> >
> > diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> > index 5f0748fba1c0..45d3de8d84f6 100644
> > --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> > +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> > @@ -797,6 +797,16 @@ Known Issues
> >
> > The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
> >
> > ++ locking
> > +
> > + If a pthread, that is not pinned to an lcore acquires a lock such as a
> > + DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
>
> Some APIs inherently use rte_spinlock, just like rte_malloc/rte_eal_alarm_set,
> Because DPDK API mainly use rte_spinlock to support thread-safty.
>
> Suggest declare DPDK API mainly use rte_spinlock to support thread-safty, so
> if the caller thread is not pinned to an lcore may encount a possibility of
> large application delays.
I copied text from pthread_spinlock man page. The same caveats apply to
pthread_spinlocks as DPDK; therefore using same wording seemed appropriate.
But it is worth mentioning that other API's may depend on spinlocks internally.
On 2022-06-10 17:28, Stephen Hemminger wrote:
> Need to warn users of DPDK spinlocks from non-pinned threads.
> This is similar wording to Linux documentation in pthread_spin_init.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 5f0748fba1c0..45d3de8d84f6 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -797,6 +797,16 @@ Known Issues
>
> The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
>
> ++ locking
> +
Isn't this problem more general than locks? The use of any
non-preemption safe data structures potentially causes such delays.
Regular DPDK rings for sure. The lock-less stack? The hash library?
Both actual and open-coded spinlocks internal to the APIs are also very
common.
> + If a pthread, that is not pinned to an lcore acquires a lock such as a
> + DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
> + then there is a possibility of large application delays.
Pinning or not doesn't matter. What matters is if the thread is
preempted and thus is prevented from making progress for a long time.
> + The problem is that if a thread is scheduled off the CPU while it holds
> + a lock, then other threads will waste time spinning on the lock until
> + the lock holder is once more rescheduled and releases the lock.
> +
> +
> cgroup control
> ~~~~~~~~~~~~~~
>
@@ -797,6 +797,16 @@ Known Issues
The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
++ locking
+
+ If a pthread, that is not pinned to an lcore acquires a lock such as a
+ DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+ then there is a possibility of large application delays.
+ The problem is that if a thread is scheduled off the CPU while it holds
+ a lock, then other threads will waste time spinning on the lock until
+ the lock holder is once more rescheduled and releases the lock.
+
+
cgroup control
~~~~~~~~~~~~~~