diff mbox series

[09/22] net/bnxt: use dedicated cpr for async events

Message ID 20190718033616.37605-10-ajit.khaparde@broadcom.com (mailing list archive)
State Changes Requested, archived
Delegated to: Ferruh Yigit
Headers
Series bnxt patchset |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation fail Compilation issues

Commit Message

Ajit Khaparde July 18, 2019, 3:36 a.m. UTC 
  From: Lance Richardson <lance.richardson@broadcom.com>

This commit enables the creation of a dedicated completion
ring for asynchronous event handling instead of handling these
events on a receive completion ring.

For the stingray platform and other platforms needing tighter
control of resource utilization, we retain the ability to
process async events on a receive completion ring. This behavior
is controlled by a compile-time configuration variable.

For Thor-based adapters, we use a dedicated NQ (notification
queue) ring for async events (async events can't currently
be received on a completion ring due to a firmware limitation).

Rename "def_cp_ring" to "async_cp_ring" to better reflect its
purpose (async event notifications) and to avoid confusion with
VNIC default receive completion rings.

Signed-off-by: Lance Richardson <lance.richardson@broadcom.com>
Reviewed-by: Somnath Kotur <somnath.kotur@broadcom.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
---
 config/common_base                           |   1 +
 config/defconfig_arm64-stingray-linuxapp-gcc |   3 +
 drivers/net/bnxt/bnxt.h                      |  10 +-
 drivers/net/bnxt/bnxt_ethdev.c               |  13 +-
 drivers/net/bnxt/bnxt_hwrm.c                 |  16 +-
 drivers/net/bnxt/bnxt_hwrm.h                 |   2 +
 drivers/net/bnxt/bnxt_irq.c                  |  47 +++---
 drivers/net/bnxt/bnxt_ring.c                 | 145 ++++++++++++++++---
 drivers/net/bnxt/bnxt_ring.h                 |   3 +
 drivers/net/bnxt/bnxt_rxr.c                  |   2 +-
 drivers/net/bnxt/bnxt_rxtx_vec_sse.c         |   2 +-
 11 files changed, 195 insertions(+), 49 deletions(-)

Comments

Ferruh Yigit July 22, 2019, 2:57 p.m. UTC | #1 
On 7/18/2019 4:36 AM, Ajit Khaparde wrote:
> From: Lance Richardson <lance.richardson@broadcom.com>
> 
> This commit enables the creation of a dedicated completion
> ring for asynchronous event handling instead of handling these
> events on a receive completion ring.
> 
> For the stingray platform and other platforms needing tighter
> control of resource utilization, we retain the ability to
> process async events on a receive completion ring. This behavior
> is controlled by a compile-time configuration variable.
> 
> For Thor-based adapters, we use a dedicated NQ (notification
> queue) ring for async events (async events can't currently
> be received on a completion ring due to a firmware limitation).
> 
> Rename "def_cp_ring" to "async_cp_ring" to better reflect its
> purpose (async event notifications) and to avoid confusion with
> VNIC default receive completion rings.
> 
> Signed-off-by: Lance Richardson <lance.richardson@broadcom.com>
> Reviewed-by: Somnath Kotur <somnath.kotur@broadcom.com>
> Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> ---
>  config/common_base                           |   1 +
>  config/defconfig_arm64-stingray-linuxapp-gcc |   3 +
>  drivers/net/bnxt/bnxt.h                      |  10 +-
>  drivers/net/bnxt/bnxt_ethdev.c               |  13 +-
>  drivers/net/bnxt/bnxt_hwrm.c                 |  16 +-
>  drivers/net/bnxt/bnxt_hwrm.h                 |   2 +
>  drivers/net/bnxt/bnxt_irq.c                  |  47 +++---
>  drivers/net/bnxt/bnxt_ring.c                 | 145 ++++++++++++++++---
>  drivers/net/bnxt/bnxt_ring.h                 |   3 +
>  drivers/net/bnxt/bnxt_rxr.c                  |   2 +-
>  drivers/net/bnxt/bnxt_rxtx_vec_sse.c         |   2 +-
>  11 files changed, 195 insertions(+), 49 deletions(-)
> 
> diff --git a/config/common_base b/config/common_base
> index 8ef75c203..487a9b811 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -212,6 +212,7 @@ CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
>  # Compile burst-oriented Broadcom BNXT PMD driver
>  #
>  CONFIG_RTE_LIBRTE_BNXT_PMD=y
> +CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n

Normally we don't want to introduce new config time options as much as possible.

This is what happens when patches slip in the last minute, please Ajit try to
send patches before proposal next time.

And back to the config option, is it something have to be a compile time flag
(if so why?), can it be replaced by a devarg?
Thomas Monjalon July 22, 2019, 3:06 p.m. UTC | #2 
22/07/2019 16:57, Ferruh Yigit:
> On 7/18/2019 4:36 AM, Ajit Khaparde wrote:
> > From: Lance Richardson <lance.richardson@broadcom.com>
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -212,6 +212,7 @@ CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
> >  # Compile burst-oriented Broadcom BNXT PMD driver
> >  #
> >  CONFIG_RTE_LIBRTE_BNXT_PMD=y
> > +CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n
> 
> Normally we don't want to introduce new config time options as much as possible.
> 
> This is what happens when patches slip in the last minute, please Ajit try to
> send patches before proposal next time.
> 
> And back to the config option, is it something have to be a compile time flag
> (if so why?), can it be replaced by a devarg?

I confirm it is nack.
I don't see any reason not to replace it with a runtime devargs.
Lance Richardson July 22, 2019, 5:57 p.m. UTC | #3 
On Mon, Jul 22, 2019 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 22/07/2019 16:57, Ferruh Yigit:
> > On 7/18/2019 4:36 AM, Ajit Khaparde wrote:
> > > From: Lance Richardson <lance.richardson@broadcom.com>
> > > --- a/config/common_base
> > > +++ b/config/common_base
> > > @@ -212,6 +212,7 @@ CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
> > >  # Compile burst-oriented Broadcom BNXT PMD driver
> > >  #
> > >  CONFIG_RTE_LIBRTE_BNXT_PMD=y
> > > +CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n
> >
> > Normally we don't want to introduce new config time options as much as possible.
> >
> > This is what happens when patches slip in the last minute, please Ajit try to
> > send patches before proposal next time.
> >
> > And back to the config option, is it something have to be a compile time flag
> > (if so why?), can it be replaced by a devarg?
>
> I confirm it is nack.
> I don't see any reason not to replace it with a runtime devargs.
>
>

This build-time configuration option was introduced because the
"shared async completion
ring" configuration is needed for one specific platform,
arm64-stingray-linux-gcc. This
configuration has a number of drawbacks and should be avoided
otherwise. Making it
a run-time option will add complexity and introduce the possibility
that existing stingray
users will not know that they need to specify this option as of 19.08,
so it would be good
if some other way could be found to handle this.

Other than perhaps using RTE_ARCH_ARM64 to select the shared completion ring
configuration (which would obviously affect all ARM64 users), are
there any other options
that might be acceptable?

It's maybe worthwhile to note that the last several DPDK releases use
the "shared completion
ring" configuration.

Thanks and regards,

   Lance
Ferruh Yigit July 22, 2019, 6:34 p.m. UTC | #4 
On 7/22/2019 6:57 PM, Lance Richardson wrote:
> On Mon, Jul 22, 2019 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 22/07/2019 16:57, Ferruh Yigit:
>>> On 7/18/2019 4:36 AM, Ajit Khaparde wrote:
>>>> From: Lance Richardson <lance.richardson@broadcom.com>
>>>> --- a/config/common_base
>>>> +++ b/config/common_base
>>>> @@ -212,6 +212,7 @@ CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
>>>>  # Compile burst-oriented Broadcom BNXT PMD driver
>>>>  #
>>>>  CONFIG_RTE_LIBRTE_BNXT_PMD=y
>>>> +CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n
>>>
>>> Normally we don't want to introduce new config time options as much as possible.
>>>
>>> This is what happens when patches slip in the last minute, please Ajit try to
>>> send patches before proposal next time.
>>>
>>> And back to the config option, is it something have to be a compile time flag
>>> (if so why?), can it be replaced by a devarg?
>>
>> I confirm it is nack.
>> I don't see any reason not to replace it with a runtime devargs.
>>
>>
> 
> This build-time configuration option was introduced because the
> "shared async completion
> ring" configuration is needed for one specific platform,
> arm64-stingray-linux-gcc. This
> configuration has a number of drawbacks and should be avoided
> otherwise. Making it
> a run-time option will add complexity and introduce the possibility
> that existing stingray
> users will not know that they need to specify this option as of 19.08,
> so it would be good
> if some other way could be found to handle this.

I see this provides a configuration transparent to user, but won't this kill the
binary portability? If a distro packages an 'armv8a' target and distributes it,
this won't be usable in your 'stingray' platform for the driver.

But if this is added as a devargs, it can be usable even with distributed
binaries, by providing proper devargs to binary for a specific platform. And
documenting this devargs in NIC guide can help users to figure it out.

> 
> Other than perhaps using RTE_ARCH_ARM64 to select the shared completion ring
> configuration (which would obviously affect all ARM64 users), are
> there any other options
> that might be acceptable?
> 
> It's maybe worthwhile to note that the last several DPDK releases use
> the "shared completion
> ring" configuration.
> 

I dropped this patch from next-net since discussion is going on, I did my best
to resolve the conflicts but please confirm the final result in next-net.
Thomas Monjalon July 23, 2019, 8:04 a.m. UTC | #5 
22/07/2019 20:34, Ferruh Yigit:
> On 7/22/2019 6:57 PM, Lance Richardson wrote:
> > On Mon, Jul 22, 2019 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >> 22/07/2019 16:57, Ferruh Yigit:
> >>> On 7/18/2019 4:36 AM, Ajit Khaparde wrote:
> >>>> From: Lance Richardson <lance.richardson@broadcom.com>
> >>>> --- a/config/common_base
> >>>> +++ b/config/common_base
> >>>> @@ -212,6 +212,7 @@ CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
> >>>>  # Compile burst-oriented Broadcom BNXT PMD driver
> >>>>  #
> >>>>  CONFIG_RTE_LIBRTE_BNXT_PMD=y
> >>>> +CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n
> >>>
> >>> Normally we don't want to introduce new config time options as much as possible.
> >>>
> >>> This is what happens when patches slip in the last minute, please Ajit try to
> >>> send patches before proposal next time.
> >>>
> >>> And back to the config option, is it something have to be a compile time flag
> >>> (if so why?), can it be replaced by a devarg?
> >>
> >> I confirm it is nack.
> >> I don't see any reason not to replace it with a runtime devargs.
> > 
> > This build-time configuration option was introduced because the
> > "shared async completion
> > ring" configuration is needed for one specific platform,
> > arm64-stingray-linux-gcc. This
> > configuration has a number of drawbacks and should be avoided
> > otherwise. Making it
> > a run-time option will add complexity and introduce the possibility
> > that existing stingray
> > users will not know that they need to specify this option as of 19.08,
> > so it would be good
> > if some other way could be found to handle this.
> 
> I see this provides a configuration transparent to user, but won't this kill the
> binary portability? If a distro packages an 'armv8a' target and distributes it,
> this won't be usable in your 'stingray' platform for the driver.
> 
> But if this is added as a devargs, it can be usable even with distributed
> binaries, by providing proper devargs to binary for a specific platform. And
> documenting this devargs in NIC guide can help users to figure it out.
> 
> > Other than perhaps using RTE_ARCH_ARM64 to select the shared completion ring
> > configuration (which would obviously affect all ARM64 users), are
> > there any other options
> > that might be acceptable?

Can you detect the platform in the PMD?
Lance Richardson July 23, 2019, 10:53 a.m. UTC | #6 
Yes, the platform can be detected at runtime
from the PCI device ID. This would be a better
approach than adding a configuration parameter,
I think. I'll take a look.

Thanks,

   Lance

On Tue, Jul 23, 2019 at 4:04 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 22/07/2019 20:34, Ferruh Yigit:
> > On 7/22/2019 6:57 PM, Lance Richardson wrote:
> > > On Mon, Jul 22, 2019 at 11:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >> 22/07/2019 16:57, Ferruh Yigit:
> > >>> On 7/18/2019 4:36 AM, Ajit Khaparde wrote:
> > >>>> From: Lance Richardson <lance.richardson@broadcom.com>
> > >>>> --- a/config/common_base
> > >>>> +++ b/config/common_base
> > >>>> @@ -212,6 +212,7 @@ CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
> > >>>>  # Compile burst-oriented Broadcom BNXT PMD driver
> > >>>>  #
> > >>>>  CONFIG_RTE_LIBRTE_BNXT_PMD=y
> > >>>> +CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n
> > >>>
> > >>> Normally we don't want to introduce new config time options as much as possible.
> > >>>
> > >>> This is what happens when patches slip in the last minute, please Ajit try to
> > >>> send patches before proposal next time.
> > >>>
> > >>> And back to the config option, is it something have to be a compile time flag
> > >>> (if so why?), can it be replaced by a devarg?
> > >>
> > >> I confirm it is nack.
> > >> I don't see any reason not to replace it with a runtime devargs.
> > >
> > > This build-time configuration option was introduced because the
> > > "shared async completion
> > > ring" configuration is needed for one specific platform,
> > > arm64-stingray-linux-gcc. This
> > > configuration has a number of drawbacks and should be avoided
> > > otherwise. Making it
> > > a run-time option will add complexity and introduce the possibility
> > > that existing stingray
> > > users will not know that they need to specify this option as of 19.08,
> > > so it would be good
> > > if some other way could be found to handle this.
> >
> > I see this provides a configuration transparent to user, but won't this kill the
> > binary portability? If a distro packages an 'armv8a' target and distributes it,
> > this won't be usable in your 'stingray' platform for the driver.
> >
> > But if this is added as a devargs, it can be usable even with distributed
> > binaries, by providing proper devargs to binary for a specific platform. And
> > documenting this devargs in NIC guide can help users to figure it out.
> >
> > > Other than perhaps using RTE_ARCH_ARM64 to select the shared completion ring
> > > configuration (which would obviously affect all ARM64 users), are
> > > there any other options
> > > that might be acceptable?
>
> Can you detect the platform in the PMD?
>
>
Lance Richardson July 23, 2019, 9:27 p.m. UTC | #7 
On Mon, Jul 22, 2019 at 2:34 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
<snip>
> I dropped this patch from next-net since discussion is going on, I did my best
> to resolve the conflicts but please confirm the final result in next-net.
>

I did some basic testing of the current head of dpdk-next-net and diffed
against the original version, everything looks OK.

I will follow up with another patch that has the build-time configuration stuff
removed.

Thanks,

    Lance
diff mbox series

Patch

diff --git a/config/common_base b/config/common_base
index 8ef75c203..487a9b811 100644
--- a/config/common_base
+++ b/config/common_base
@@ -212,6 +212,7 @@  CONFIG_RTE_LIBRTE_BNX2X_DEBUG_PERIODIC=n
 # Compile burst-oriented Broadcom BNXT PMD driver
 #
 CONFIG_RTE_LIBRTE_BNXT_PMD=y
+CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=n
 
 #
 # Compile burst-oriented Chelsio Terminator (CXGBE) PMD
diff --git a/config/defconfig_arm64-stingray-linuxapp-gcc b/config/defconfig_arm64-stingray-linuxapp-gcc
index 7b33aa7af..acfb1c207 100644
--- a/config/defconfig_arm64-stingray-linuxapp-gcc
+++ b/config/defconfig_arm64-stingray-linuxapp-gcc
@@ -12,5 +12,8 @@  CONFIG_RTE_ARCH_ARM_TUNE="cortex-a72"
 CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
 CONFIG_RTE_LIBRTE_VHOST_NUMA=n
 
+# Conserve cpr resources by using rx cpr for async events.
+CONFIG_RTE_LIBRTE_BNXT_SHARED_ASYNC_RING=y
+
 CONFIG_RTE_EAL_IGB_UIO=y
 CONFIG_RTE_KNI_KMOD=n
diff --git a/drivers/net/bnxt/bnxt.h b/drivers/net/bnxt/bnxt.h
index 3ccf784e5..8bd8f536c 100644
--- a/drivers/net/bnxt/bnxt.h
+++ b/drivers/net/bnxt/bnxt.h
@@ -33,6 +33,14 @@ 
 #define BNXT_MAX_RX_RING_DESC	8192
 #define BNXT_DB_SIZE		0x80
 
+#ifdef RTE_LIBRTE_BNXT_SHARED_ASYNC_RING
+/* Async events are handled on rx queue 0 completion ring. */
+#define BNXT_NUM_ASYNC_CPR	0
+#else
+/* Async events are handled on a dedicated completion ring. */
+#define BNXT_NUM_ASYNC_CPR	1
+#endif
+
 /* Chimp Communication Channel */
 #define GRCPF_REG_CHIMP_CHANNEL_OFFSET		0x0
 #define GRCPF_REG_CHIMP_COMM_TRIGGER		0x100
@@ -387,7 +395,7 @@  struct bnxt {
 	uint16_t		fw_tx_port_stats_ext_size;
 
 	/* Default completion ring */
-	struct bnxt_cp_ring_info	*def_cp_ring;
+	struct bnxt_cp_ring_info	*async_cp_ring;
 	uint32_t		max_ring_grps;
 	struct bnxt_ring_grp_info	*grp_info;
 
diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index 429ebe555..fe7837df2 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -198,12 +198,17 @@  static void bnxt_free_mem(struct bnxt *bp)
 	bnxt_free_stats(bp);
 	bnxt_free_tx_rings(bp);
 	bnxt_free_rx_rings(bp);
+	bnxt_free_async_cp_ring(bp);
 }
 
 static int bnxt_alloc_mem(struct bnxt *bp)
 {
 	int rc;
 
+	rc = bnxt_alloc_async_ring_struct(bp);
+	if (rc)
+		goto alloc_mem_err;
+
 	rc = bnxt_alloc_vnic_mem(bp);
 	if (rc)
 		goto alloc_mem_err;
@@ -216,6 +221,10 @@  static int bnxt_alloc_mem(struct bnxt *bp)
 	if (rc)
 		goto alloc_mem_err;
 
+	rc = bnxt_alloc_async_cp_ring(bp);
+	if (rc)
+		goto alloc_mem_err;
+
 	return 0;
 
 alloc_mem_err:
@@ -617,8 +626,8 @@  static int bnxt_dev_configure_op(struct rte_eth_dev *eth_dev)
 	/* Inherit new configurations */
 	if (eth_dev->data->nb_rx_queues > bp->max_rx_rings ||
 	    eth_dev->data->nb_tx_queues > bp->max_tx_rings ||
-	    eth_dev->data->nb_rx_queues + eth_dev->data->nb_tx_queues >
-	    bp->max_cp_rings ||
+	    eth_dev->data->nb_rx_queues + eth_dev->data->nb_tx_queues
+		+ BNXT_NUM_ASYNC_CPR > bp->max_cp_rings ||
 	    eth_dev->data->nb_rx_queues + eth_dev->data->nb_tx_queues >
 	    bp->max_stat_ctx)
 		goto resource_error;
diff --git a/drivers/net/bnxt/bnxt_hwrm.c b/drivers/net/bnxt/bnxt_hwrm.c
index bd4250a3a..52b2119a5 100644
--- a/drivers/net/bnxt/bnxt_hwrm.c
+++ b/drivers/net/bnxt/bnxt_hwrm.c
@@ -740,9 +740,12 @@  int bnxt_hwrm_func_reserve_vf_resc(struct bnxt *bp, bool test)
 	req.num_tx_rings = rte_cpu_to_le_16(bp->tx_nr_rings);
 	req.num_rx_rings = rte_cpu_to_le_16(bp->rx_nr_rings *
 					    AGG_RING_MULTIPLIER);
-	req.num_stat_ctxs = rte_cpu_to_le_16(bp->rx_nr_rings + bp->tx_nr_rings);
+	req.num_stat_ctxs = rte_cpu_to_le_16(bp->rx_nr_rings +
+					     bp->tx_nr_rings +
+					     BNXT_NUM_ASYNC_CPR);
 	req.num_cmpl_rings = rte_cpu_to_le_16(bp->rx_nr_rings +
-					      bp->tx_nr_rings);
+					      bp->tx_nr_rings +
+					      BNXT_NUM_ASYNC_CPR);
 	req.num_vnics = rte_cpu_to_le_16(bp->rx_nr_rings);
 	if (bp->vf_resv_strategy ==
 	    HWRM_FUNC_RESOURCE_QCAPS_OUTPUT_VF_RESV_STRATEGY_MINIMAL_STATIC) {
@@ -2079,7 +2082,7 @@  int bnxt_free_all_hwrm_ring_grps(struct bnxt *bp)
 	return rc;
 }
 
-static void bnxt_free_nq_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr)
+void bnxt_free_nq_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr)
 {
 	struct bnxt_ring *cp_ring = cpr->cp_ring_struct;
 
@@ -2089,9 +2092,10 @@  static void bnxt_free_nq_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr)
 	memset(cpr->cp_desc_ring, 0, cpr->cp_ring_struct->ring_size *
 				     sizeof(*cpr->cp_desc_ring));
 	cpr->cp_raw_cons = 0;
+	cpr->valid = 0;
 }
 
-static void bnxt_free_cp_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr)
+void bnxt_free_cp_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr)
 {
 	struct bnxt_ring *cp_ring = cpr->cp_ring_struct;
 
@@ -3225,7 +3229,7 @@  int bnxt_hwrm_func_cfg_def_cp(struct bnxt *bp)
 	req.enables = rte_cpu_to_le_32(
 			HWRM_FUNC_CFG_INPUT_ENABLES_ASYNC_EVENT_CR);
 	req.async_event_cr = rte_cpu_to_le_16(
-			bp->def_cp_ring->cp_ring_struct->fw_ring_id);
+			bp->async_cp_ring->cp_ring_struct->fw_ring_id);
 	rc = bnxt_hwrm_send_message(bp, &req, sizeof(req), BNXT_USE_CHIMP_MB);
 
 	HWRM_CHECK_RESULT();
@@ -3245,7 +3249,7 @@  int bnxt_hwrm_vf_func_cfg_def_cp(struct bnxt *bp)
 	req.enables = rte_cpu_to_le_32(
 			HWRM_FUNC_VF_CFG_INPUT_ENABLES_ASYNC_EVENT_CR);
 	req.async_event_cr = rte_cpu_to_le_16(
-			bp->def_cp_ring->cp_ring_struct->fw_ring_id);
+			bp->async_cp_ring->cp_ring_struct->fw_ring_id);
 	rc = bnxt_hwrm_send_message(bp, &req, sizeof(req), BNXT_USE_CHIMP_MB);
 
 	HWRM_CHECK_RESULT();
diff --git a/drivers/net/bnxt/bnxt_hwrm.h b/drivers/net/bnxt/bnxt_hwrm.h
index 37aaa1a9e..c882fc2a1 100644
--- a/drivers/net/bnxt/bnxt_hwrm.h
+++ b/drivers/net/bnxt/bnxt_hwrm.h
@@ -119,6 +119,8 @@  int bnxt_free_all_hwrm_stat_ctxs(struct bnxt *bp);
 int bnxt_free_all_hwrm_rings(struct bnxt *bp);
 int bnxt_free_all_hwrm_ring_grps(struct bnxt *bp);
 int bnxt_alloc_all_hwrm_ring_grps(struct bnxt *bp);
+void bnxt_free_cp_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr);
+void bnxt_free_nq_ring(struct bnxt *bp, struct bnxt_cp_ring_info *cpr);
 int bnxt_set_hwrm_vnic_filters(struct bnxt *bp, struct bnxt_vnic_info *vnic);
 int bnxt_clear_hwrm_vnic_filters(struct bnxt *bp, struct bnxt_vnic_info *vnic);
 void bnxt_free_all_hwrm_resources(struct bnxt *bp);
diff --git a/drivers/net/bnxt/bnxt_irq.c b/drivers/net/bnxt/bnxt_irq.c
index 6c4dce401..9ff16ddd8 100644
--- a/drivers/net/bnxt/bnxt_irq.c
+++ b/drivers/net/bnxt/bnxt_irq.c
@@ -21,7 +21,7 @@  static void bnxt_int_handler(void *param)
 {
 	struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)param;
 	struct bnxt *bp = eth_dev->data->dev_private;
-	struct bnxt_cp_ring_info *cpr = bp->def_cp_ring;
+	struct bnxt_cp_ring_info *cpr = bp->async_cp_ring;
 	struct cmpl_base *cmp;
 	uint32_t raw_cons;
 	uint32_t cons;
@@ -42,19 +42,19 @@  static void bnxt_int_handler(void *param)
 
 		bnxt_event_hwrm_resp_handler(bp, cmp);
 		raw_cons = NEXT_RAW_CMP(raw_cons);
-	};
+	}
 
 	cpr->cp_raw_cons = raw_cons;
-	B_CP_DB_REARM(cpr, cpr->cp_raw_cons);
+	if (BNXT_HAS_NQ(bp))
+		bnxt_db_nq_arm(cpr);
+	else
+		B_CP_DB_REARM(cpr, cpr->cp_raw_cons);
 }
 
 void bnxt_free_int(struct bnxt *bp)
 {
 	struct bnxt_irq *irq;
 
-	if (bp->irq_tbl == NULL)
-		return;
-
 	irq = bp->irq_tbl;
 	if (irq) {
 		if (irq->requested) {
@@ -70,19 +70,35 @@  void bnxt_free_int(struct bnxt *bp)
 
 void bnxt_disable_int(struct bnxt *bp)
 {
-	struct bnxt_cp_ring_info *cpr = bp->def_cp_ring;
+	struct bnxt_cp_ring_info *cpr = bp->async_cp_ring;
+
+	if (BNXT_NUM_ASYNC_CPR == 0)
+		return;
+
+	if (!cpr || !cpr->cp_db.doorbell)
+		return;
 
 	/* Only the default completion ring */
-	if (cpr != NULL && cpr->cp_db.doorbell != NULL)
+	if (BNXT_HAS_NQ(bp))
+		bnxt_db_nq(cpr);
+	else
 		B_CP_DB_DISARM(cpr);
 }
 
 void bnxt_enable_int(struct bnxt *bp)
 {
-	struct bnxt_cp_ring_info *cpr = bp->def_cp_ring;
+	struct bnxt_cp_ring_info *cpr = bp->async_cp_ring;
+
+	if (BNXT_NUM_ASYNC_CPR == 0)
+		return;
+
+	if (!cpr || !cpr->cp_db.doorbell)
+		return;
 
 	/* Only the default completion ring */
-	if (cpr != NULL && cpr->cp_db.doorbell != NULL)
+	if (BNXT_HAS_NQ(bp))
+		bnxt_db_nq_arm(cpr);
+	else
 		B_CP_DB_ARM(cpr);
 }
 
@@ -90,7 +106,7 @@  int bnxt_setup_int(struct bnxt *bp)
 {
 	uint16_t total_vecs;
 	const int len = sizeof(bp->irq_tbl[0].name);
-	int i, rc = 0;
+	int i;
 
 	/* DPDK host only supports 1 MSI-X vector */
 	total_vecs = 1;
@@ -104,14 +120,11 @@  int bnxt_setup_int(struct bnxt *bp)
 			bp->irq_tbl[i].handler = bnxt_int_handler;
 		}
 	} else {
-		rc = -ENOMEM;
-		goto setup_exit;
+		PMD_DRV_LOG(ERR, "bnxt_irq_tbl setup failed\n");
+		return -ENOMEM;
 	}
-	return 0;
 
-setup_exit:
-	PMD_DRV_LOG(ERR, "bnxt_irq_tbl setup failed\n");
-	return rc;
+	return 0;
 }
 
 int bnxt_request_int(struct bnxt *bp)
diff --git a/drivers/net/bnxt/bnxt_ring.c b/drivers/net/bnxt/bnxt_ring.c
index a9952e02c..05a9a200c 100644
--- a/drivers/net/bnxt/bnxt_ring.c
+++ b/drivers/net/bnxt/bnxt_ring.c
@@ -5,6 +5,7 @@ 
 
 #include <rte_bitmap.h>
 #include <rte_memzone.h>
+#include <rte_malloc.h>
 #include <unistd.h>
 
 #include "bnxt.h"
@@ -369,6 +370,7 @@  static int bnxt_alloc_cmpl_ring(struct bnxt *bp, int queue_index,
 {
 	struct bnxt_ring *cp_ring = cpr->cp_ring_struct;
 	uint32_t nq_ring_id = HWRM_NA_SIGNATURE;
+	int cp_ring_index = queue_index + BNXT_NUM_ASYNC_CPR;
 	uint8_t ring_type;
 	int rc = 0;
 
@@ -383,13 +385,13 @@  static int bnxt_alloc_cmpl_ring(struct bnxt *bp, int queue_index,
 		}
 	}
 
-	rc = bnxt_hwrm_ring_alloc(bp, cp_ring, ring_type, queue_index,
+	rc = bnxt_hwrm_ring_alloc(bp, cp_ring, ring_type, cp_ring_index,
 				  HWRM_NA_SIGNATURE, nq_ring_id);
 	if (rc)
 		return rc;
 
 	cpr->cp_cons = 0;
-	bnxt_set_db(bp, &cpr->cp_db, ring_type, queue_index,
+	bnxt_set_db(bp, &cpr->cp_db, ring_type, cp_ring_index,
 		    cp_ring->fw_ring_id);
 	bnxt_db_cq(cpr);
 
@@ -400,6 +402,7 @@  static int bnxt_alloc_nq_ring(struct bnxt *bp, int queue_index,
 			      struct bnxt_cp_ring_info *nqr)
 {
 	struct bnxt_ring *nq_ring = nqr->cp_ring_struct;
+	int nq_ring_index = queue_index + BNXT_NUM_ASYNC_CPR;
 	uint8_t ring_type;
 	int rc = 0;
 
@@ -408,12 +411,12 @@  static int bnxt_alloc_nq_ring(struct bnxt *bp, int queue_index,
 
 	ring_type = HWRM_RING_ALLOC_INPUT_RING_TYPE_NQ;
 
-	rc = bnxt_hwrm_ring_alloc(bp, nq_ring, ring_type, queue_index,
+	rc = bnxt_hwrm_ring_alloc(bp, nq_ring, ring_type, nq_ring_index,
 				  HWRM_NA_SIGNATURE, HWRM_NA_SIGNATURE);
 	if (rc)
 		return rc;
 
-	bnxt_set_db(bp, &nqr->cp_db, ring_type, queue_index,
+	bnxt_set_db(bp, &nqr->cp_db, ring_type, nq_ring_index,
 		    nq_ring->fw_ring_id);
 	bnxt_db_nq(nqr);
 
@@ -490,14 +493,16 @@  int bnxt_alloc_hwrm_rx_ring(struct bnxt *bp, int queue_index)
 	struct bnxt_ring *cp_ring = cpr->cp_ring_struct;
 	struct bnxt_cp_ring_info *nqr = rxq->nq_ring;
 	struct bnxt_rx_ring_info *rxr = rxq->rx_ring;
-	int rc = 0;
+	int rc;
 
 	if (BNXT_HAS_NQ(bp)) {
-		if (bnxt_alloc_nq_ring(bp, queue_index, nqr))
+		rc = bnxt_alloc_nq_ring(bp, queue_index, nqr);
+		if (rc)
 			goto err_out;
 	}
 
-	if (bnxt_alloc_cmpl_ring(bp, queue_index, cpr, nqr))
+	rc = bnxt_alloc_cmpl_ring(bp, queue_index, cpr, nqr);
+	if (rc)
 		goto err_out;
 
 	if (BNXT_HAS_RING_GRPS(bp)) {
@@ -505,22 +510,24 @@  int bnxt_alloc_hwrm_rx_ring(struct bnxt *bp, int queue_index)
 		bp->grp_info[queue_index].cp_fw_ring_id = cp_ring->fw_ring_id;
 	}
 
-	if (!queue_index) {
+	if (!BNXT_NUM_ASYNC_CPR && !queue_index) {
 		/*
-		 * In order to save completion resources, use the first
-		 * completion ring from PF or VF as the default completion ring
-		 * for async event and HWRM forward response handling.
+		 * If a dedicated async event completion ring is not enabled,
+		 * use the first completion ring from PF or VF as the default
+		 * completion ring for async event handling.
 		 */
-		bp->def_cp_ring = cpr;
+		bp->async_cp_ring = cpr;
 		rc = bnxt_hwrm_set_async_event_cr(bp);
 		if (rc)
 			goto err_out;
 	}
 
-	if (bnxt_alloc_rx_ring(bp, queue_index))
+	rc = bnxt_alloc_rx_ring(bp, queue_index);
+	if (rc)
 		goto err_out;
 
-	if (bnxt_alloc_rx_agg_ring(bp, queue_index))
+	rc = bnxt_alloc_rx_agg_ring(bp, queue_index);
+	if (rc)
 		goto err_out;
 
 	rxq->rx_buf_use_size = BNXT_MAX_MTU + RTE_ETHER_HDR_LEN +
@@ -545,6 +552,9 @@  int bnxt_alloc_hwrm_rx_ring(struct bnxt *bp, int queue_index)
 		    bp->eth_dev->data->rx_queue_state[queue_index]);
 
 err_out:
+	PMD_DRV_LOG(ERR,
+		    "Failed to allocate receive queue %d, rc %d.\n",
+		    queue_index, rc);
 	return rc;
 }
 
@@ -583,15 +593,13 @@  int bnxt_alloc_hwrm_rings(struct bnxt *bp)
 		}
 
 		bnxt_hwrm_set_ring_coal(bp, &coal, cp_ring->fw_ring_id);
-
-		if (!i) {
+		if (!BNXT_NUM_ASYNC_CPR && !i) {
 			/*
-			 * In order to save completion resource, use the first
-			 * completion ring from PF or VF as the default
-			 * completion ring for async event & HWRM
-			 * forward response handling.
+			 * If a dedicated async event completion ring is not
+			 * enabled, use the first completion ring as the default
+			 * completion ring for async event handling.
 			 */
-			bp->def_cp_ring = cpr;
+			bp->async_cp_ring = cpr;
 			rc = bnxt_hwrm_set_async_event_cr(bp);
 			if (rc)
 				goto err_out;
@@ -652,3 +660,98 @@  int bnxt_alloc_hwrm_rings(struct bnxt *bp)
 err_out:
 	return rc;
 }
+
+/* Allocate dedicated async completion ring. */
+int bnxt_alloc_async_cp_ring(struct bnxt *bp)
+{
+	struct bnxt_cp_ring_info *cpr = bp->async_cp_ring;
+	struct bnxt_ring *cp_ring = cpr->cp_ring_struct;
+	uint8_t ring_type;
+	int rc;
+
+	if (BNXT_NUM_ASYNC_CPR == 0)
+		return 0;
+
+	if (BNXT_HAS_NQ(bp))
+		ring_type = HWRM_RING_ALLOC_INPUT_RING_TYPE_NQ;
+	else
+		ring_type = HWRM_RING_ALLOC_INPUT_RING_TYPE_L2_CMPL;
+
+	rc = bnxt_hwrm_ring_alloc(bp, cp_ring, ring_type, 0,
+				  HWRM_NA_SIGNATURE, HWRM_NA_SIGNATURE);
+
+	if (rc)
+		return rc;
+
+	cpr->cp_cons = 0;
+	cpr->valid = 0;
+	bnxt_set_db(bp, &cpr->cp_db, ring_type, 0,
+		    cp_ring->fw_ring_id);
+
+	if (BNXT_HAS_NQ(bp))
+		bnxt_db_nq(cpr);
+	else
+		bnxt_db_cq(cpr);
+
+	return bnxt_hwrm_set_async_event_cr(bp);
+}
+
+/* Free dedicated async completion ring. */
+void bnxt_free_async_cp_ring(struct bnxt *bp)
+{
+	struct bnxt_cp_ring_info *cpr = bp->async_cp_ring;
+
+	if (BNXT_NUM_ASYNC_CPR == 0 || cpr == NULL)
+		return;
+
+	if (BNXT_HAS_NQ(bp))
+		bnxt_free_nq_ring(bp, cpr);
+	else
+		bnxt_free_cp_ring(bp, cpr);
+
+	bnxt_free_ring(cpr->cp_ring_struct);
+	rte_free(cpr->cp_ring_struct);
+	cpr->cp_ring_struct = NULL;
+	rte_free(cpr);
+	bp->async_cp_ring = NULL;
+}
+
+int bnxt_alloc_async_ring_struct(struct bnxt *bp)
+{
+	struct bnxt_cp_ring_info *cpr = NULL;
+	struct bnxt_ring *ring = NULL;
+	unsigned int socket_id;
+
+	if (BNXT_NUM_ASYNC_CPR == 0)
+		return 0;
+
+	socket_id = rte_lcore_to_socket_id(rte_get_master_lcore());
+
+	cpr = rte_zmalloc_socket("cpr",
+				 sizeof(struct bnxt_cp_ring_info),
+				 RTE_CACHE_LINE_SIZE, socket_id);
+	if (cpr == NULL)
+		return -ENOMEM;
+
+	ring = rte_zmalloc_socket("bnxt_cp_ring_struct",
+				  sizeof(struct bnxt_ring),
+				  RTE_CACHE_LINE_SIZE, socket_id);
+	if (ring == NULL) {
+		rte_free(cpr);
+		return -ENOMEM;
+	}
+
+	ring->bd = (void *)cpr->cp_desc_ring;
+	ring->bd_dma = cpr->cp_desc_mapping;
+	ring->ring_size = rte_align32pow2(DEFAULT_CP_RING_SIZE);
+	ring->ring_mask = ring->ring_size - 1;
+	ring->vmem_size = 0;
+	ring->vmem = NULL;
+
+	bp->async_cp_ring = cpr;
+	cpr->cp_ring_struct = ring;
+
+	return bnxt_alloc_rings(bp, 0, NULL, NULL,
+				bp->async_cp_ring, NULL,
+				"def_cp");
+}
diff --git a/drivers/net/bnxt/bnxt_ring.h b/drivers/net/bnxt/bnxt_ring.h
index e5cef3a1d..04c7b04b8 100644
--- a/drivers/net/bnxt/bnxt_ring.h
+++ b/drivers/net/bnxt/bnxt_ring.h
@@ -75,6 +75,9 @@  int bnxt_alloc_rings(struct bnxt *bp, uint16_t qidx,
 			    const char *suffix);
 int bnxt_alloc_hwrm_rx_ring(struct bnxt *bp, int queue_index);
 int bnxt_alloc_hwrm_rings(struct bnxt *bp);
+int bnxt_alloc_async_cp_ring(struct bnxt *bp);
+void bnxt_free_async_cp_ring(struct bnxt *bp);
+int bnxt_alloc_async_ring_struct(struct bnxt *bp);
 
 static inline void bnxt_db_write(struct bnxt_db_info *db, uint32_t idx)
 {
diff --git a/drivers/net/bnxt/bnxt_rxr.c b/drivers/net/bnxt/bnxt_rxr.c
index 54a2cf5fd..1e068f817 100644
--- a/drivers/net/bnxt/bnxt_rxr.c
+++ b/drivers/net/bnxt/bnxt_rxr.c
@@ -564,7 +564,7 @@  uint16_t bnxt_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 				nb_rx_pkts++;
 			if (rc == -EBUSY)	/* partial completion */
 				break;
-		} else {
+		} else if (!BNXT_NUM_ASYNC_CPR) {
 			evt =
 			bnxt_event_hwrm_resp_handler(rxq->bp,
 						     (struct cmpl_base *)rxcmp);
diff --git a/drivers/net/bnxt/bnxt_rxtx_vec_sse.c b/drivers/net/bnxt/bnxt_rxtx_vec_sse.c
index c358506f8..3ef016073 100644
--- a/drivers/net/bnxt/bnxt_rxtx_vec_sse.c
+++ b/drivers/net/bnxt/bnxt_rxtx_vec_sse.c
@@ -257,7 +257,7 @@  bnxt_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
 			mbuf->packet_type = bnxt_parse_pkt_type(rxcmp, rxcmp1);
 
 			rx_pkts[nb_rx_pkts++] = mbuf;
-		} else {
+		} else if (!BNXT_NUM_ASYNC_CPR) {
 			evt =
 			bnxt_event_hwrm_resp_handler(rxq->bp,
 						     (struct cmpl_base *)rxcmp);