From patchwork Fri Jan 28 12:48:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh A P X-Patchwork-Id: 106661 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 38853A034D; Fri, 28 Jan 2022 13:21:17 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EFBBD42896; Fri, 28 Jan 2022 13:21:12 +0100 (CET) Received: from relay.smtp-ext.broadcom.com (lpdvsmtp10.broadcom.com [192.19.11.229]) by mails.dpdk.org (Postfix) with ESMTP id 0F1ED40141 for ; Fri, 28 Jan 2022 13:21:11 +0100 (CET) Received: from dhcp-10-123-153-22.dhcp.broadcom.net (bgccx-dev-host-lnx2.bec.broadcom.net [10.123.153.22]) by relay.smtp-ext.broadcom.com (Postfix) with ESMTP id 4FCEBC0000D9; Fri, 28 Jan 2022 04:21:09 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 relay.smtp-ext.broadcom.com 4FCEBC0000D9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=broadcom.com; s=dkimrelay; t=1643372470; bh=BvT9gpvfUp7iQuFXPjlk2+nRbQ1PBQf9K0dSi0QStH4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pFqhq04U2wgSQBkQ/sVPHuW5SxYhbZUzIZPmP1tLAv+EUY5IsKDFaGCn7bRX/aA0L 6vJDI/5nX4dPhqUnbdzs8v0LauKx2hqv5ySTPkcnIQY7bXp3j3llRKXJ/aYzX2Ql9M rBaBO2hz/2Wte5tVTo1VBztvfSm5w56YNsTc+/0o= From: Kalesh A P To: dev@dpdk.org Cc: ferruh.yigit@intel.com, ajit.khaparde@broadcom.com, asafp@nvidia.com Subject: [dpdk-dev] [PATCH v7 1/4] ethdev: support device reset and recovery events Date: Fri, 28 Jan 2022 18:18:27 +0530 Message-Id: <20220128124830.427-2-kalesh-anakkur.purayil@broadcom.com> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> References: <20201009034832.10302-1-kalesh-anakkur.purayil@broadcom.com> <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP Adding support for the device reset and recovery events in the rte_eth_event framework. FW error and FW reset conditions would be managed internally by the PMD without needing application intervention. In such cases, PMD would need reset/recovery events to notify application that PMD is undergoing a reset. While most of the recovery process is transparent to the application since most of the driver ensures recovery from FW reset or FW error conditions, the application will have to reprogram any flows which were offloaded to the underlying hardware. Signed-off-by: Kalesh AP Signed-off-by: Somnath Kotur Reviewed-by: Ajit Khaparde --- doc/guides/prog_guide/poll_mode_drv.rst | 24 ++++++++++++++++++++++++ lib/ethdev/rte_ethdev.h | 18 ++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index 6831289..9ecc0e4 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -623,3 +623,27 @@ by application. The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger the application to handle reset event. It is duty of application to handle all synchronization before it calls rte_eth_dev_reset(). + +Error recovery support +~~~~~~~~~~~~~~~~~~~~~~ + +When the PMD detects a FW reset or error condition, it may try to recover +from the error without needing the application intervention. In such cases, +PMD would need events to notify the application that it is undergoing +an error recovery. + +The PMD should trigger RTE_ETH_EVENT_ERR_RECOVERING event to notify the +application that PMD detected a FW reset or FW error condition. PMD may +try to recover from the error by itself. Data path may be quiesced and +control path operations may fail during the recovery period. The application +should stop polling till it receives RTE_ETH_EVENT_RECOVERED event from the PMD. + +The PMD should trigger RTE_ETH_EVENT_RECOVERED event to notify the application +that the it has recovered from the error condition. PMD re-configures the port +to the state prior to the error condition. Control path and data path are up now. +Since the device has undergone a reset, flow rules offloaded prior to reset +may be lost and the application should recreate the rules again. + +The PMD should trigger RTE_ETH_EVENT_INTR_RMV event to notify the application +that it has failed to recover from the error condition. The device may not be +usable anymore. diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 147cc1c..a46819f 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -3818,6 +3818,24 @@ enum rte_eth_event_type { RTE_ETH_EVENT_DESTROY, /**< port is released */ RTE_ETH_EVENT_IPSEC, /**< IPsec offload related event */ RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected */ + RTE_ETH_EVENT_ERR_RECOVERING, + /**< port recovering from an error + * + * PMD detected a FW reset or error condition. + * PMD will try to recover from the error. + * Data path may be quiesced and Control path operations + * may fail at this time. + */ + RTE_ETH_EVENT_RECOVERED, + /**< port recovered from an error + * + * PMD has recovered from the error condition. + * Control path and Data path are up now. + * PMD re-configures the port to the state prior to the error. + * Since the device has undergone a reset, flow rules + * offloaded prior to reset may be lost and + * the application should recreate the rules again. + */ RTE_ETH_EVENT_MAX /**< max value of this enum */ }; From patchwork Fri Jan 28 12:48:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh A P X-Patchwork-Id: 106662 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BF184A034D; Fri, 28 Jan 2022 13:21:22 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CBF714289F; Fri, 28 Jan 2022 13:21:13 +0100 (CET) Received: from relay.smtp-ext.broadcom.com (relay.smtp-ext.broadcom.com [192.19.11.229]) by mails.dpdk.org (Postfix) with ESMTP id C223542890 for ; Fri, 28 Jan 2022 13:21:12 +0100 (CET) Received: from dhcp-10-123-153-22.dhcp.broadcom.net (bgccx-dev-host-lnx2.bec.broadcom.net [10.123.153.22]) by relay.smtp-ext.broadcom.com (Postfix) with ESMTP id 1EDC6C0000EC; Fri, 28 Jan 2022 04:21:10 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 relay.smtp-ext.broadcom.com 1EDC6C0000EC DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=broadcom.com; s=dkimrelay; t=1643372472; bh=cABqB8X2S2RGDVsg8wc56XfAIbyi35KkguJ99abzivQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YfcwHNVb9e0yssLexGrIeihkvMTskxvpmj1y4+GuFubfJGHWKN4yrzC2LUISt9xsk w445bQmBSflwTeJKvwT5envTI36tG709GIw0ya8/P8o450ecXVfTIrdOFoiXREtcd7 hOHg5HbQb3PBUzlxw8FsKLdf/EK1kURlUqVxUvho= From: Kalesh A P To: dev@dpdk.org Cc: ferruh.yigit@intel.com, ajit.khaparde@broadcom.com, asafp@nvidia.com Subject: [dpdk-dev] [PATCH v7 2/4] app/testpmd: handle device recovery event Date: Fri, 28 Jan 2022 18:18:28 +0530 Message-Id: <20220128124830.427-3-kalesh-anakkur.purayil@broadcom.com> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> References: <20201009034832.10302-1-kalesh-anakkur.purayil@broadcom.com> <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP Added code to handle error recovery events in testpmd. This is an indication from the PMD that it is undergoing an error recovery and recovered from the error condition. Signed-off-by: Kalesh AP Signed-off-by: Somnath Kotur Reviewed-by: Ajit Khaparde --- app/test-pmd/parameters.c | 8 ++++++-- app/test-pmd/testpmd.c | 10 +++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index daf6a31..deea29f 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -175,9 +175,9 @@ usage(char* progname) printf(" --no-rmv-interrupt: disable device removal interrupt.\n"); printf(" --bitrate-stats=N: set the logical core N to perform " "bit-rate calculation.\n"); - printf(" --print-event : " + printf(" --print-event : " "enable print of designated event or all of them.\n"); - printf(" --mask-event : " + printf(" --mask-event : " "disable print of designated event or all of them.\n"); printf(" --flow-isolate-all: " "requests flow API isolated mode on all ports at initialization time.\n"); @@ -461,6 +461,10 @@ parse_event_printing_config(const char *optarg, int enable) mask = UINT32_C(1) << RTE_ETH_EVENT_DESTROY; else if (!strcmp(optarg, "flow_aged")) mask = UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED; + else if (!strcmp(optarg, "err_recovering")) + mask = UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING; + else if (!strcmp(optarg, "recovered")) + mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERED; else if (!strcmp(optarg, "all")) mask = ~UINT32_C(0); else { diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index e1da961..7b64751 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -417,6 +417,8 @@ static const char * const eth_event_desc[] = { [RTE_ETH_EVENT_NEW] = "device probed", [RTE_ETH_EVENT_DESTROY] = "device released", [RTE_ETH_EVENT_FLOW_AGED] = "flow aged", + [RTE_ETH_EVENT_ERR_RECOVERING] = "device error, recovery in progress", + [RTE_ETH_EVENT_RECOVERED] = "device recovered", [RTE_ETH_EVENT_MAX] = NULL, }; @@ -431,7 +433,9 @@ uint32_t event_print_mask = (UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN) | (UINT32_C(1) << RTE_ETH_EVENT_IPSEC) | (UINT32_C(1) << RTE_ETH_EVENT_MACSEC) | (UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV) | - (UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED); + (UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED) | + (UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING) | + (UINT32_C(1) << RTE_ETH_EVENT_RECOVERED); /* * Decide if all memory are locked for performance. */ @@ -3564,6 +3568,10 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param, ports[port_id].port_status = RTE_PORT_CLOSED; printf("Port %u is closed\n", port_id); break; + case RTE_ETH_EVENT_RECOVERED: + /* for now, flush flows to avoid displaying stale entries */ + port_flow_flush(port_id); + break; default: break; } From patchwork Fri Jan 28 12:48:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh A P X-Patchwork-Id: 106663 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9182CA034D; Fri, 28 Jan 2022 13:21:29 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1FDE3428BC; Fri, 28 Jan 2022 13:21:16 +0100 (CET) Received: from relay.smtp-ext.broadcom.com (lpdvsmtp10.broadcom.com [192.19.11.229]) by mails.dpdk.org (Postfix) with ESMTP id 8D029428A6 for ; Fri, 28 Jan 2022 13:21:14 +0100 (CET) Received: from dhcp-10-123-153-22.dhcp.broadcom.net (bgccx-dev-host-lnx2.bec.broadcom.net [10.123.153.22]) by relay.smtp-ext.broadcom.com (Postfix) with ESMTP id E22E9C0000EA; Fri, 28 Jan 2022 04:21:12 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 relay.smtp-ext.broadcom.com E22E9C0000EA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=broadcom.com; s=dkimrelay; t=1643372474; bh=EIrM3VBGU9W9CnHvP4GFXpsrZJKjR6q3GRf9+F0djnk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZIIs/Icwnj9r+yEhAFiJ1UN9Bgz6Efu0jTkUGhMKAYFC/z29V7VKzqKPn+zyqyJxS O+5tbjSgk23CEdU1O/k5aE+MnBvO9r0JaVuz5ZkK9tJ0CCYMXT8NTHRTIn5IFAeAR5 NZn4/c24408SnW9W4c89SrSyDitFc1j6auY66HDg= From: Kalesh A P To: dev@dpdk.org Cc: ferruh.yigit@intel.com, ajit.khaparde@broadcom.com, asafp@nvidia.com Subject: [dpdk-dev] [PATCH v7 3/4] net/bnxt: notify applications about device reset/recovery Date: Fri, 28 Jan 2022 18:18:29 +0530 Message-Id: <20220128124830.427-4-kalesh-anakkur.purayil@broadcom.com> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> References: <20201009034832.10302-1-kalesh-anakkur.purayil@broadcom.com> <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP When the driver receives RESET_NOTIFY async event from FW or detects an error condition, it should update the application that FW is going to reset. Once the driver recovers from the reset, update the reset recovery status to application as well. Signed-off-by: Kalesh AP Signed-off-by: Somnath Kotur Reviewed-by: Ajit Khaparde --- drivers/net/bnxt/bnxt_cpr.c | 4 ++++ drivers/net/bnxt/bnxt_ethdev.c | 8 +++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c index 9b9285b..738eac7 100644 --- a/drivers/net/bnxt/bnxt_cpr.c +++ b/drivers/net/bnxt/bnxt_cpr.c @@ -180,6 +180,10 @@ void bnxt_handle_async_event(struct bnxt *bp, return; } + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_ERR_RECOVERING, + NULL); + pthread_mutex_lock(&bp->err_recovery_lock); event_data = data1; /* timestamp_lo/hi values are in units of 100ms */ diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index cc95211..579e571 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -4383,7 +4383,9 @@ static void bnxt_dev_recover(void *arg) PMD_DRV_LOG(INFO, "Port: %u Recovered from FW reset\n", bp->eth_dev->data->port_id); pthread_mutex_unlock(&bp->err_recovery_lock); - + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_RECOVERED, + NULL); return; err_start: bnxt_dev_stop(bp->eth_dev); @@ -4561,6 +4563,10 @@ static void bnxt_check_fw_health(void *arg) PMD_DRV_LOG(ERR, "Detected FW dead condition\n"); + rte_eth_dev_callback_process(bp->eth_dev, + RTE_ETH_EVENT_ERR_RECOVERING, + NULL); + if (bnxt_is_primary_func(bp)) wait_msec = info->primary_func_wait_period; else From patchwork Fri Jan 28 12:48:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kalesh A P X-Patchwork-Id: 106664 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D1ED4A034D; Fri, 28 Jan 2022 13:21:34 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 09F88428C8; Fri, 28 Jan 2022 13:21:17 +0100 (CET) Received: from relay.smtp-ext.broadcom.com (saphodev.broadcom.com [192.19.11.229]) by mails.dpdk.org (Postfix) with ESMTP id 69954428C3 for ; Fri, 28 Jan 2022 13:21:16 +0100 (CET) Received: from dhcp-10-123-153-22.dhcp.broadcom.net (bgccx-dev-host-lnx2.bec.broadcom.net [10.123.153.22]) by relay.smtp-ext.broadcom.com (Postfix) with ESMTP id B1728C0000D9; Fri, 28 Jan 2022 04:21:14 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 relay.smtp-ext.broadcom.com B1728C0000D9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=broadcom.com; s=dkimrelay; t=1643372476; bh=vfU32IRSqgGcQn2uU6dKFS4yzFRVgcDSeB+/Mu8SbKg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J+fiTomOp+VathPnBmN6Bjv3PeibOGDr65JaRCv4Q2szN+WZJWtAcenT/Pbr1Z/4S /1XZ0OcaUKcHRhD69Cvq24OFOzfRiV4WjRtBA0cSRqtD35QvIuNs8S0VVy3Nx4R967 YZsqPbMQqg4Dk14HpxRc0zBn+GXM3VVALCtCMCLQ= From: Kalesh A P To: dev@dpdk.org Cc: ferruh.yigit@intel.com, ajit.khaparde@broadcom.com, asafp@nvidia.com Subject: [dpdk-dev] [PATCH v7 4/4] doc: update release notes Date: Fri, 28 Jan 2022 18:18:30 +0530 Message-Id: <20220128124830.427-5-kalesh-anakkur.purayil@broadcom.com> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> References: <20201009034832.10302-1-kalesh-anakkur.purayil@broadcom.com> <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Kalesh AP Updated release notes with new error recovery event support. Signed-off-by: Kalesh AP --- doc/guides/rel_notes/release_22_03.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst index 33be324..88755ba 100644 --- a/doc/guides/rel_notes/release_22_03.rst +++ b/doc/guides/rel_notes/release_22_03.rst @@ -24,6 +24,21 @@ DPDK Release 22.03 New Features ------------ +* **Added error recovery support.** + + Added error recovery support to detect and recover from device errors including: + + * Added new event: ``RTE_ETH_EVENT_ERR_RECOVERING`` for the driver to report + that the port is recovering from an error. + * Added new event: ``RTE_ETH_EVENT_RECOVERED`` for the driver to report + that the port has recovered from an error. + +* **Updated Broadcom bnxt driver.** + + Updated the Broadcom bnxt driver with new features and improvements, including: + + * Added support to handle device recovery events. + .. This section should contain new features added in this release. Sample format: