[2/3] net/ionic: remove duplicate barriers

  These barriers are duplicated by the barriers inside
rte_write64(). Remove them to improve performance.

Signed-off-by: Neel Patel <neel.patel@amd.com>
Signed-off-by: Andrew Boyer <andrew.boyer@amd.com>
---
 drivers/net/ionic/ionic_main.c        | 1 -
 drivers/net/ionic/ionic_rxtx_sg.c     | 1 -
 drivers/net/ionic/ionic_rxtx_simple.c | 1 -
 3 files changed, 3 deletions(-)
  

Hi Andrew,

I think that this barrier may have been added to ensure any writes to q->hw_index
and q->head_idx complete before ionic_q_flush computes val. Dependency chains
can also prevent reordering if that's the case this barrier is not required.
However, I have the following concern.

> diff --git a/drivers/net/ionic/ionic_main.c b/drivers/net/ionic/ionic_main.c
> index 1f24f64a33..814bb3b8f4 100644
> --- a/drivers/net/ionic/ionic_main.c
> +++ b/drivers/net/ionic/ionic_main.c
> @@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct
> ionic_admin_ctx *ctx)
>  	q->head_idx = Q_NEXT_TO_POST(q, 1);
> 
>  	/* Ring doorbell */
> -	rte_wmb();
>  	ionic_q_flush(q);
> 
>  err_out:

Ionic_q_flush(q) uses q->hw_index and q->head_idx to compute the value of val which
it writes to the address stored in q->db. I can see that q->head_idx is updated before
the removed rte_wmb(), therefore it's safe to assume that 
" q->head_idx = Q_NEXT_TO_POST(q, 1)" and 
"val = IONIC_DBELL_QID(q->hw_index) | q->head_idx" will maintain the program order
due to that dependency. But I don't know if there exists a dependency chain 
over q->hw_index. If not, then that may have been the motive behind this barrier.

> diff --git a/drivers/net/ionic/ionic_rxtx_sg.c
> b/drivers/net/ionic/ionic_rxtx_sg.c
> index e8dec99c04..3820fd850f 100644
> --- a/drivers/net/ionic/ionic_rxtx_sg.c
> +++ b/drivers/net/ionic/ionic_rxtx_sg.c
> @@ -218,7 +218,6 @@ ionic_xmit_pkts_sg(void *tx_queue, struct rte_mbuf
> **tx_pkts,
>  	}
> 
>  	if (nb_tx > 0) {
> -		rte_wmb();
>  		ionic_txq_flush(q);
> 
>  		txq->last_wdog_cycles = rte_get_timer_cycles(); diff --git
> a/drivers/net/ionic/ionic_rxtx_simple.c
> b/drivers/net/ionic/ionic_rxtx_simple.c
> index 9674b4d1df..4c9b9415ad 100644
> --- a/drivers/net/ionic/ionic_rxtx_simple.c
> +++ b/drivers/net/ionic/ionic_rxtx_simple.c
> @@ -191,7 +191,6 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf
> **tx_pkts,
>  	}
> 
>  	if (nb_tx > 0) {
> -		rte_wmb();
>  		ionic_txq_flush(q);
> 
>  		txq->last_wdog_cycles = rte_get_timer_cycles();
> --
> 2.17.1
  

> -----Original Message-----
> From: Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com>
> Sent: Monday, February 19, 2024 4:17 PM
> To: Andrew Boyer <andrew.boyer@amd.com>; dev@dpdk.org
> Cc: Neel Patel <neel.patel@amd.com>; nd <nd@arm.com>; Honnappa
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>
> Subject: RE: [PATCH 2/3] net/ionic: remove duplicate barriers
> 
> Hi Andrew,
> 
> I think that this barrier may have been added to ensure any writes to q-
> >hw_index and q->head_idx complete before ionic_q_flush computes val.
> Dependency chains can also prevent reordering if that's the case this barrier is
> not required.
> However, I have the following concern.
> 
> > diff --git a/drivers/net/ionic/ionic_main.c
> > b/drivers/net/ionic/ionic_main.c index 1f24f64a33..814bb3b8f4 100644
> > --- a/drivers/net/ionic/ionic_main.c
> > +++ b/drivers/net/ionic/ionic_main.c
> > @@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct
> > ionic_admin_ctx *ctx)
> >  	q->head_idx = Q_NEXT_TO_POST(q, 1);
> >
> >  	/* Ring doorbell */
> > -	rte_wmb();
> >  	ionic_q_flush(q);
> >
> >  err_out:
> 
> Ionic_q_flush(q) uses q->hw_index and q->head_idx to compute the value of
> val which it writes to the address stored in q->db. I can see that q->head_idx is
> updated before the removed rte_wmb(), therefore it's safe to assume that " q-
> >head_idx = Q_NEXT_TO_POST(q, 1)" and "val = IONIC_DBELL_QID(q-
> >hw_index) | q->head_idx" will maintain the program order due to that
> dependency. But I don't know if there exists a dependency chain over q-
> >hw_index. If not, then that may have been the motive behind this barrier.
> 

Since q->hw_index is also updated in the same CPU ionic_q_flush will
always see the correct value, consequently val should be always correct. 
It's safe to remove this barrier.

> > diff --git a/drivers/net/ionic/ionic_rxtx_sg.c
> > b/drivers/net/ionic/ionic_rxtx_sg.c
> > index e8dec99c04..3820fd850f 100644
> > --- a/drivers/net/ionic/ionic_rxtx_sg.c
> > +++ b/drivers/net/ionic/ionic_rxtx_sg.c
> > @@ -218,7 +218,6 @@ ionic_xmit_pkts_sg(void *tx_queue, struct
> rte_mbuf
> > **tx_pkts,
> >  	}
> >
> >  	if (nb_tx > 0) {
> > -		rte_wmb();
> >  		ionic_txq_flush(q);
> >
> >  		txq->last_wdog_cycles = rte_get_timer_cycles(); diff --git
> > a/drivers/net/ionic/ionic_rxtx_simple.c
> > b/drivers/net/ionic/ionic_rxtx_simple.c
> > index 9674b4d1df..4c9b9415ad 100644
> > --- a/drivers/net/ionic/ionic_rxtx_simple.c
> > +++ b/drivers/net/ionic/ionic_rxtx_simple.c
> > @@ -191,7 +191,6 @@ ionic_xmit_pkts(void *tx_queue, struct rte_mbuf
> > **tx_pkts,
> >  	}
> >
> >  	if (nb_tx > 0) {
> > -		rte_wmb();
> >  		ionic_txq_flush(q);
> >
> >  		txq->last_wdog_cycles = rte_get_timer_cycles();
> > --
> > 2.17.1
  

On Feb 19, 2024, at 9:02 PM, Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com> wrote:

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.


-----Original Message-----
From: Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com<mailto:wathsala.vithanage@arm.com>>
Sent: Monday, February 19, 2024 4:17 PM
To: Andrew Boyer <andrew.boyer@amd.com<mailto:andrew.boyer@amd.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Cc: Neel Patel <neel.patel@amd.com<mailto:neel.patel@amd.com>>; nd <nd@arm.com<mailto:nd@arm.com>>; Honnappa
Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>; nd <nd@arm.com<mailto:nd@arm.com>>
Subject: RE: [PATCH 2/3] net/ionic: remove duplicate barriers

Hi Andrew,

I think that this barrier may have been added to ensure any writes to q-
hw_index and q->head_idx complete before ionic_q_flush computes val.
Dependency chains can also prevent reordering if that's the case this barrier is
not required.
However, I have the following concern.

diff --git a/drivers/net/ionic/ionic_main.c
b/drivers/net/ionic/ionic_main.c index 1f24f64a33..814bb3b8f4 100644
--- a/drivers/net/ionic/ionic_main.c
+++ b/drivers/net/ionic/ionic_main.c
@@ -223,7 +223,6 @@ ionic_adminq_post(struct ionic_lif *lif, struct
ionic_admin_ctx *ctx)
   q->head_idx = Q_NEXT_TO_POST(q, 1);

   /* Ring doorbell */
-   rte_wmb();
   ionic_q_flush(q);

err_out:

Ionic_q_flush(q) uses q->hw_index and q->head_idx to compute the value of
val which it writes to the address stored in q->db. I can see that q->head_idx is
updated before the removed rte_wmb(), therefore it's safe to assume that " q-
head_idx = Q_NEXT_TO_POST(q, 1)" and "val = IONIC_DBELL_QID(q-
hw_index) | q->head_idx" will maintain the program order due to that
dependency. But I don't know if there exists a dependency chain over q-
hw_index. If not, then that may have been the motive behind this barrier.


Since q->hw_index is also updated in the same CPU ionic_q_flush will
always see the correct value, consequently val should be always correct.
It's safe to remove this barrier.


Thank you for the careful review. We agree with this analysis.

-Andrew