From patchwork Thu Jun 16 09:41:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 112886 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A2118A00BE; Thu, 16 Jun 2022 11:48:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E82E242C26; Thu, 16 Jun 2022 11:47:56 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 5FA8042BF3 for ; Thu, 16 Jun 2022 11:47:52 +0200 (CEST) Received: from dggpeml500024.china.huawei.com (unknown [172.30.72.57]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4LNy1q1h9BzSgxv; Thu, 16 Jun 2022 17:44:31 +0800 (CST) Received: from localhost.localdomain (10.67.165.24) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 16 Jun 2022 17:47:50 +0800 From: Chengwen Feng To: , CC: , , , , , Subject: [PATCH v8 1/4] ethdev: support device error recovery notification Date: Thu, 16 Jun 2022 17:41:19 +0800 Message-ID: <20220616094122.1909-2-fengchengwen@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20220616094122.1909-1-fengchengwen@huawei.com> References: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> <20220616094122.1909-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.165.24] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP Some PMDs (e.g. hns3) could detect hardware or firmware errors, and try to recover from the errors. In this process, the PMD sets the data path pointers to dummy functions (which will prevent the crash), and also make sure the control path operations failed with retcode -EBUSY. Also in this process, from the perspective of application, services are affected. For example, the Rx/Tx bust APIs cannot receive and send packets, and the control plane API return failure. In some service scenarios, application needs to be aware of the event to determine whether to migrate services. So three events were introduced: 1. RTE_ETH_EVENT_ERR_RECOVERING: the PMD must trigger this event to notify the application that it detected a hardware or firmware error and tries to recover. 2. RTE_ETH_EVENT_RECOVER_SUCCESS: the PMD must trigger this event to notify the application that it has recovered from the error. And PMD already re-configures the port to the state prior to the error. 3. RTE_ETH_EVENT_RECOVER_FAILED: the PMD must trigger this event to notify the application that it has failed to recover from the error. The port may not be usable anymore. Note: the error recovery of these events is mainly performed by the PMD. Unlike the RTE_ETH_EVENT_INTR_RESET which the error recovery is performed by the application. The PMD must ensure that the above two error handling methods cannot be used at the same time. Signed-off-by: Kalesh AP Signed-off-by: Somnath Kotur Signed-off-by: Chengwen Feng Reviewed-by: Ajit Khaparde --- doc/guides/prog_guide/poll_mode_drv.rst | 32 +++++++++++++++++++++++++ doc/guides/rel_notes/release_22_07.rst | 11 +++++++++ lib/ethdev/rte_ethdev.h | 6 +++++ 3 files changed, 49 insertions(+) diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index 9d081b1cba..6398917485 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -627,3 +627,35 @@ by application. The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger the application to handle reset event. It is duty of application to handle all synchronization before it calls rte_eth_dev_reset(). + +Error Recovery Notification +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some PMDs (e.g. hns3) could detect hardware or firmware errors, and try to +recover from the errors. In this process, the PMD sets the data path pointers +to dummy functions (which will prevent the crash), and also make sure the +control path operations failed with retcode -EBUSY. + +Also in this process, from the perspective of application, services are +affected. For example, the Rx/Tx bust APIs cannot receive and send packets, +and the control plane API return failure. + +In some service scenarios, application needs to be aware of the event to +determine whether to migrate services. So three events was introduced. + +The PMD must trigger RTE_ETH_EVENT_ERR_RECOVERING event to notify the +application that it detected a hardware or firmware error and tries to recover. + +The PMD must trigger RTE_ETH_EVENT_RECOVER_SUCCESS event to notify the +application that it has recovered from the error. And PMD already re-configures +the port to the state prior to the error. + +The PMD must trigger RTE_ETH_EVENT_RECOVER_FAILED event to notify the +application that it has failed to recover from the error. The port may not be +usable anymore. + +.. note:: + The error recovery of these events is mainly performed by the PMD. + Unlike the RTE_ETH_EVENT_INTR_RESET which the error recovery is + performed by the application. The PMD must ensure that the above two + error handling methods cannot be used at the same time. diff --git a/doc/guides/rel_notes/release_22_07.rst b/doc/guides/rel_notes/release_22_07.rst index 6fc044edaa..b237bd3303 100644 --- a/doc/guides/rel_notes/release_22_07.rst +++ b/doc/guides/rel_notes/release_22_07.rst @@ -108,6 +108,17 @@ New Features Added an API which can get the device type of vDPA device. +* **Added error recover notification.** + + Added error recover notification to application including: + + * Added new event: ``RTE_ETH_EVENT_ERR_RECOVERING`` for the PMD to report + that the port is recovering from an error. + * Added new event: ``RTE_ETH_EVENT_RECOVER_SUCCESS`` for the PMD to report + that the port recover successful from an error. + * Added new event: ``RTE_ETH_EVENT_RECOVER_FAILED`` for the PMD to report + that the prot recover failed from an error. + * **Updated Amazon ena driver.** The new driver version (v2.7.0) includes: diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 045ee64747..6998f6f0be 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -3928,6 +3928,12 @@ enum rte_eth_event_type { * @see rte_eth_rx_avail_thresh_set() */ RTE_ETH_EVENT_RX_AVAIL_THRESH, + /** Port recovering from a hardware or firmware error */ + RTE_ETH_EVENT_ERR_RECOVERING, + /** Port recovers successful from the error */ + RTE_ETH_EVENT_RECOVER_SUCCESS, + /** Port recovers failed from the error */ + RTE_ETH_EVENT_RECOVER_FAILED, RTE_ETH_EVENT_MAX /**< max value of this enum */ }; From patchwork Thu Jun 16 09:41:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 112887 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7AB06A00BE; Thu, 16 Jun 2022 11:48:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E16C042C2A; Thu, 16 Jun 2022 11:47:57 +0200 (CEST) Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by mails.dpdk.org (Postfix) with ESMTP id 7D0E042C15 for ; Thu, 16 Jun 2022 11:47:52 +0200 (CEST) Received: from dggpeml500024.china.huawei.com (unknown [172.30.72.55]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4LNy3M266hz1K9vB; Thu, 16 Jun 2022 17:45:51 +0800 (CST) Received: from localhost.localdomain (10.67.165.24) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 16 Jun 2022 17:47:50 +0800 From: Chengwen Feng To: , CC: , , , , , Subject: [PATCH v8 2/4] app/testpmd: handle error recovery notification event Date: Thu, 16 Jun 2022 17:41:20 +0800 Message-ID: <20220616094122.1909-3-fengchengwen@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20220616094122.1909-1-fengchengwen@huawei.com> References: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> <20220616094122.1909-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.165.24] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP This patch handles error recovery notification event. Signed-off-by: Kalesh AP Signed-off-by: Somnath Kotur Signed-off-by: Chengwen Feng Reviewed-by: Ajit Khaparde --- app/test-pmd/parameters.c | 10 ++++++++-- app/test-pmd/testpmd.c | 8 +++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index 641c9c767e..779a699371 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -175,9 +175,9 @@ usage(char* progname) printf(" --no-rmv-interrupt: disable device removal interrupt.\n"); printf(" --bitrate-stats=N: set the logical core N to perform " "bit-rate calculation.\n"); - printf(" --print-event : " + printf(" --print-event : " "enable print of designated event or all of them.\n"); - printf(" --mask-event : " + printf(" --mask-event : " "disable print of designated event or all of them.\n"); printf(" --flow-isolate-all: " "requests flow API isolated mode on all ports at initialization time.\n"); @@ -461,6 +461,12 @@ parse_event_printing_config(const char *optarg, int enable) mask = UINT32_C(1) << RTE_ETH_EVENT_DESTROY; else if (!strcmp(optarg, "flow_aged")) mask = UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED; + else if (!strcmp(optarg, "err_recovering")) + mask = UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING; + else if (!strcmp(optarg, "recover_success")) + mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVER_SUCCESS; + else if (!strcmp(optarg, "recover_failed")) + mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVER_FAILED; else if (!strcmp(optarg, "all")) mask = ~UINT32_C(0); else { diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 04c39adc21..96f5c8a63b 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -421,6 +421,9 @@ static const char * const eth_event_desc[] = { [RTE_ETH_EVENT_DESTROY] = "device released", [RTE_ETH_EVENT_FLOW_AGED] = "flow aged", [RTE_ETH_EVENT_RX_AVAIL_THRESH] = "RxQ available descriptors threshold reached", + [RTE_ETH_EVENT_ERR_RECOVERING] = "error recovering", + [RTE_ETH_EVENT_RECOVER_SUCCESS] = "error recover successful", + [RTE_ETH_EVENT_RECOVER_FAILED] = "error recover failed", [RTE_ETH_EVENT_MAX] = NULL, }; @@ -435,7 +438,10 @@ uint32_t event_print_mask = (UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN) | (UINT32_C(1) << RTE_ETH_EVENT_IPSEC) | (UINT32_C(1) << RTE_ETH_EVENT_MACSEC) | (UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV) | - (UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED); + (UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED) | + (UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING) | + (UINT32_C(1) << RTE_ETH_EVENT_RECOVER_SUCCESS) | + (UINT32_C(1) << RTE_ETH_EVENT_RECOVER_FAILED); /* * Decide if all memory are locked for performance. */ From patchwork Thu Jun 16 09:41:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 112884 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1BB8EA00BE; Thu, 16 Jun 2022 11:47:58 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E938B42C1A; Thu, 16 Jun 2022 11:47:54 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 6298242C10 for ; Thu, 16 Jun 2022 11:47:52 +0200 (CEST) Received: from dggpeml500024.china.huawei.com (unknown [172.30.72.54]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4LNy3t3QS2zjXfD; Thu, 16 Jun 2022 17:46:18 +0800 (CST) Received: from localhost.localdomain (10.67.165.24) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 16 Jun 2022 17:47:50 +0800 From: Chengwen Feng To: , CC: , , , , , Subject: [PATCH v8 3/4] net/hns3: support error recovery notification event Date: Thu, 16 Jun 2022 17:41:21 +0800 Message-ID: <20220616094122.1909-4-fengchengwen@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20220616094122.1909-1-fengchengwen@huawei.com> References: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> <20220616094122.1909-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.165.24] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch supports error recovery notification event: 1. RTE_ETH_EVENT_ERR_RECOVERING: the PMD report this event to nofify the application that it detected a hardware or firmware error and tries to recover. 2. RTE_ETH_EVENT_RECOVER_SUCCESS: the PMD report this event to nofify the application that it has recovered from the error. And PMD already re-configures the port to the state prior to the error. 3. RTE_ETH_EVENT_RECOVER_FAILED: the PMD report this event to notify the application that it has failed to recover from the error. The port may not be usable anymore. Signed-off-by: Chengwen Feng --- drivers/net/hns3/hns3_intr.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/net/hns3/hns3_intr.c b/drivers/net/hns3/hns3_intr.c index 3ca2e1e338..d6d2209f73 100644 --- a/drivers/net/hns3/hns3_intr.c +++ b/drivers/net/hns3/hns3_intr.c @@ -1486,6 +1486,27 @@ static const struct hns3_hw_err_type hns3_hw_error_type[] = { } }; +static void +hns3_report_reset_begin(struct hns3_hw *hw) +{ + struct rte_eth_dev *dev = &rte_eth_devices[hw->data->port_id]; + rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_ERR_RECOVERING, NULL); +} + +static void +hns3_report_reset_success(struct hns3_hw *hw) +{ + struct rte_eth_dev *dev = &rte_eth_devices[hw->data->port_id]; + rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_RECOVER_SUCCESS, NULL); +} + +static void +hns3_report_reset_failed(struct hns3_hw *hw) +{ + struct rte_eth_dev *dev = &rte_eth_devices[hw->data->port_id]; + rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_RECOVER_FAILED, NULL); +} + static int hns3_config_ncsi_hw_err_int(struct hns3_adapter *hns, bool en) { @@ -2645,6 +2666,7 @@ hns3_reset_pre(struct hns3_adapter *hns) if (hw->reset.stage == RESET_STAGE_NONE) { __atomic_store_n(&hns->hw.reset.resetting, 1, __ATOMIC_RELAXED); hw->reset.stage = RESET_STAGE_DOWN; + hns3_report_reset_begin(hw); ret = hw->reset.ops->stop_service(hns); hns3_clock_gettime(&tv); if (ret) { @@ -2754,6 +2776,7 @@ hns3_reset_post(struct hns3_adapter *hns) hns3_clock_calctime_ms(&tv_delta), tv.tv_sec, tv.tv_usec); hw->reset.level = HNS3_NONE_RESET; + hns3_report_reset_success(hw); } return 0; @@ -2799,6 +2822,7 @@ hns3_reset_fail_handle(struct hns3_adapter *hns) hns3_clock_calctime_ms(&tv_delta), tv.tv_sec, tv.tv_usec); hw->reset.level = HNS3_NONE_RESET; + hns3_report_reset_failed(hw); } /* From patchwork Thu Jun 16 09:41:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 112885 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2F87EA00BE; Thu, 16 Jun 2022 11:48:03 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E6D9F42C1E; Thu, 16 Jun 2022 11:47:55 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 7746342C13 for ; Thu, 16 Jun 2022 11:47:52 +0200 (CEST) Received: from dggpeml500024.china.huawei.com (unknown [172.30.72.57]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4LNy3t4bqdzjYC2; Thu, 16 Jun 2022 17:46:18 +0800 (CST) Received: from localhost.localdomain (10.67.165.24) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Thu, 16 Jun 2022 17:47:50 +0800 From: Chengwen Feng To: , CC: , , , , , Subject: [PATCH v8 4/4] net/bnxt: notify applications about device reset/recovery Date: Thu, 16 Jun 2022 17:41:22 +0800 Message-ID: <20220616094122.1909-5-fengchengwen@huawei.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20220616094122.1909-1-fengchengwen@huawei.com> References: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> <20220616094122.1909-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.67.165.24] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP When the driver receives RESET_NOTIFY async event from FW or detects an error condition, it should update the application that FW is going to reset. Once the driver recovers from the reset, update the reset recovery status to application as well. Signed-off-by: Kalesh AP Signed-off-by: Somnath Kotur Reviewed-by: Ajit Khaparde Reviewed-by: Chengwen Feng --- drivers/net/bnxt/bnxt_cpr.c | 4 ++++ drivers/net/bnxt/bnxt_ethdev.c | 11 ++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c index 99af0f9e87..5bb376d4d5 100644 --- a/drivers/net/bnxt/bnxt_cpr.c +++ b/drivers/net/bnxt/bnxt_cpr.c @@ -180,6 +180,10 @@ void bnxt_handle_async_event(struct bnxt *bp, return; } + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_ERR_RECOVERING, + NULL); + pthread_mutex_lock(&bp->err_recovery_lock); event_data = data1; /* timestamp_lo/hi values are in units of 100ms */ diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 0f0f40b95b..cc02951d95 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -4397,13 +4397,18 @@ static void bnxt_dev_recover(void *arg) PMD_DRV_LOG(INFO, "Port: %u Recovered from FW reset\n", bp->eth_dev->data->port_id); pthread_mutex_unlock(&bp->err_recovery_lock); - + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_RECOVER_SUCCESS, + NULL); return; err_start: bnxt_dev_stop(bp->eth_dev); err: bp->flags |= BNXT_FLAG_FATAL_ERROR; bnxt_uninit_resources(bp, false); + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_RECOVER_FAILED, + NULL); if (bp->eth_dev->data->dev_conf.intr_conf.rmv) rte_eth_dev_callback_process(bp->eth_dev, RTE_ETH_EVENT_INTR_RMV, @@ -4575,6 +4580,10 @@ static void bnxt_check_fw_health(void *arg) PMD_DRV_LOG(ERR, "Detected FW dead condition\n"); + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_ERR_RECOVERING, + NULL); + if (bnxt_is_primary_func(bp)) wait_msec = info->primary_func_wait_period; else