From patchwork Tue Apr 11 09:11:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 125921 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 000224291B; Tue, 11 Apr 2023 11:13:48 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1E06241611; Tue, 11 Apr 2023 11:12:52 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id EC63F42BAC for ; Tue, 11 Apr 2023 11:12:49 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 33B8V8Co021687 for ; Tue, 11 Apr 2023 02:12:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=kRHigjH98bdhwoGGG41niOq+Tw9a6wNDtexyz4Y2ZTM=; b=U6DwpM7XbCNsNU9jRHEm5f1jLnwzqdhZnbnHguXHrTje7kkhnIAfA9eUUUIv+OvSzYXS MANRTE7AxI6dr/TR4GKgoDKjjtnKeo2wpSWXYXI52F4e4quL+60OY6HipVGwHkFnHCci SP7tEukZn6uzN51WZ62m09wIa0y/T2diLT5JdARSuyrgY6WvIkedzQPjkAhcjWGELj75 VryeLHwKnJUjS+LoXh5CRXTsxy5QIIfrQxhag3h0QwUmXJTelUY3Xithd5dr07KxWZBE TuCaauV1nCoAMuspSKAhxFOIlHHaKcGbpJlmw+FaaTcRBlaTNIHuHlaZMCZyxY8tECwY EQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3pvt73b1vf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Tue, 11 Apr 2023 02:12:48 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.48; Tue, 11 Apr 2023 02:12:47 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.48 via Frontend Transport; Tue, 11 Apr 2023 02:12:47 -0700 Received: from hyd1588t430.caveonetworks.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 060863F706A; Tue, 11 Apr 2023 02:12:44 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Kumar Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Harman Kalra Subject: [PATCH 16/21] common/cnxk: sync between mbox up and down messages Date: Tue, 11 Apr 2023 14:41:39 +0530 Message-ID: <20230411091144.1087887-16-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230411091144.1087887-1-ndabilpuram@marvell.com> References: <20230411091144.1087887-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: -G0CObFgzCp8dbPqw0k4M2vxphjLO4cG X-Proofpoint-ORIG-GUID: -G0CObFgzCp8dbPqw0k4M2vxphjLO4cG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-11_05,2023-04-06_03,2023-02-09_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Harman Kalra An issue is observed where if PF is with DPDK and VF as kernel netdev does not responds to link events. It was due to recent design change in kernel where sender checks whether previous interrupt is received before triggering current interrupt by waiting for mailbox data register to become zero. Signed-off-by: Harman Kalra --- drivers/common/cnxk/roc_dev.c | 20 ++++++++- drivers/common/cnxk/roc_mbox.c | 64 +++++++++++++++++++++-------- drivers/common/cnxk/roc_mbox.h | 15 +++++++ drivers/common/cnxk/roc_mbox_priv.h | 6 ++- 4 files changed, 84 insertions(+), 21 deletions(-) diff --git a/drivers/common/cnxk/roc_dev.c b/drivers/common/cnxk/roc_dev.c index 5e4e564ebe..e5a5cd7c10 100644 --- a/drivers/common/cnxk/roc_dev.c +++ b/drivers/common/cnxk/roc_dev.c @@ -195,7 +195,8 @@ af_pf_wait_msg(struct dev *dev, uint16_t vf, int num_msg) vf_msg->rc = msg->rc; vf_msg->pcifunc = msg->pcifunc; /* Send to VF */ - mbox_msg_send(&dev->mbox_vfpf_up, vf); + mbox_msg_send_up(&dev->mbox_vfpf_up, vf); + mbox_wait_for_zero(&dev->mbox_vfpf_up, vf); } } @@ -498,6 +499,7 @@ pf_vf_mbox_send_up_msg(struct dev *dev, void *rec_msg) /* Send to VF */ mbox_msg_send(vf_mbox, vf); + mbox_wait_for_zero(&dev->mbox_vfpf_up, vf); } } @@ -631,6 +633,7 @@ static void roc_pf_vf_mbox_irq(void *param) { struct dev *dev = param; + uint64_t mbox_data; uint64_t intr; intr = plt_read64(dev->bar2 + RVU_VF_INT); @@ -640,6 +643,13 @@ roc_pf_vf_mbox_irq(void *param) plt_write64(intr, dev->bar2 + RVU_VF_INT); plt_base_dbg("Irq 0x%" PRIx64 "(pf:%d,vf:%d)", intr, dev->pf, dev->vf); + /* Reading for UP/DOWN message, next message sending will be delayed + * by 1ms until this region is zeroed mbox_wait_for_zero() + */ + mbox_data = plt_read64(dev->bar2 + RVU_VF_VFPF_MBOX0); + if (mbox_data) + plt_write64(!mbox_data, dev->bar2 + RVU_VF_VFPF_MBOX0); + /* First process all configuration messages */ process_msgs(dev, dev->mbox); @@ -651,6 +661,7 @@ static void roc_af_pf_mbox_irq(void *param) { struct dev *dev = param; + uint64_t mbox_data; uint64_t intr; intr = plt_read64(dev->bar2 + RVU_PF_INT); @@ -660,6 +671,13 @@ roc_af_pf_mbox_irq(void *param) plt_write64(intr, dev->bar2 + RVU_PF_INT); plt_base_dbg("Irq 0x%" PRIx64 "(pf:%d,vf:%d)", intr, dev->pf, dev->vf); + /* Reading for UP/DOWN message, next message sending will be delayed + * by 1ms until this region is zeroed mbox_wait_for_zero() + */ + mbox_data = plt_read64(dev->bar2 + RVU_PF_PFAF_MBOX0); + if (mbox_data) + plt_write64(!mbox_data, dev->bar2 + RVU_PF_PFAF_MBOX0); + /* First process all configuration messages */ process_msgs(dev, dev->mbox); diff --git a/drivers/common/cnxk/roc_mbox.c b/drivers/common/cnxk/roc_mbox.c index 7dcd188ca7..5338a960d9 100644 --- a/drivers/common/cnxk/roc_mbox.c +++ b/drivers/common/cnxk/roc_mbox.c @@ -10,18 +10,6 @@ #include "roc_api.h" #include "roc_priv.h" -#define RVU_AF_AFPF_MBOX0 (0x02000) -#define RVU_AF_AFPF_MBOX1 (0x02008) - -#define RVU_PF_PFAF_MBOX0 (0xC00) -#define RVU_PF_PFAF_MBOX1 (0xC08) - -#define RVU_PF_VFX_PFVF_MBOX0 (0x0000) -#define RVU_PF_VFX_PFVF_MBOX1 (0x0008) - -#define RVU_VF_VFPF_MBOX0 (0x0000) -#define RVU_VF_VFPF_MBOX1 (0x0008) - /* RCLK, SCLK in MHz */ uint16_t dev_rclk_freq; uint16_t dev_sclk_freq; @@ -194,10 +182,31 @@ mbox_alloc_msg_rsp(struct mbox *mbox, int devid, int size, int size_rsp) /** * @internal - * Send a mailbox message + * Synchronization between UP and DOWN messages */ -void -mbox_msg_send(struct mbox *mbox, int devid) +bool +mbox_wait_for_zero(struct mbox *mbox, int devid) +{ + uint64_t data; + + data = plt_read64((volatile void *)(mbox->reg_base + + (mbox->trigger | (devid << mbox->tr_shift)))); + + /* If data is non-zero wait for ~1ms and return to caller + * whether data has changed to zero or not after the wait. + */ + if (data) + usleep(1000); + else + return true; + + data = plt_read64((volatile void *)(mbox->reg_base + + (mbox->trigger | (devid << mbox->tr_shift)))); + return data == 0; +} + +static void +mbox_msg_send_data(struct mbox *mbox, int devid, uint8_t data) { struct mbox_dev *mdev = &mbox->dev[devid]; struct mbox_hdr *tx_hdr = @@ -223,9 +232,28 @@ mbox_msg_send(struct mbox *mbox, int devid) /* The interrupt should be fired after num_msgs is written * to the shared memory */ - plt_write64(1, (volatile void *)(mbox->reg_base + - (mbox->trigger | - (devid << mbox->tr_shift)))); + plt_write64(data, (volatile void *)(mbox->reg_base + + (mbox->trigger | (devid << mbox->tr_shift)))); +} + +/** + * @internal + * Send a mailbox message + */ +void +mbox_msg_send(struct mbox *mbox, int devid) +{ + mbox_msg_send_data(mbox, devid, MBOX_DOWN_MSG); +} + +/** + * @internal + * Send an UP mailbox message + */ +void +mbox_msg_send_up(struct mbox *mbox, int devid) +{ + mbox_msg_send_data(mbox, devid, MBOX_UP_MSG); } /** diff --git a/drivers/common/cnxk/roc_mbox.h b/drivers/common/cnxk/roc_mbox.h index 3d5746b9b8..93c5451c0f 100644 --- a/drivers/common/cnxk/roc_mbox.h +++ b/drivers/common/cnxk/roc_mbox.h @@ -35,6 +35,21 @@ struct mbox_msghdr { int __io rc; /* Msg processed response code */ }; +#define RVU_AF_AFPF_MBOX0 (0x02000) +#define RVU_AF_AFPF_MBOX1 (0x02008) + +#define RVU_PF_PFAF_MBOX0 (0xC00) +#define RVU_PF_PFAF_MBOX1 (0xC08) + +#define RVU_PF_VFX_PFVF_MBOX0 (0x0000) +#define RVU_PF_VFX_PFVF_MBOX1 (0x0008) + +#define RVU_VF_VFPF_MBOX0 (0x0000) +#define RVU_VF_VFPF_MBOX1 (0x0008) + +#define MBOX_DOWN_MSG 1 +#define MBOX_UP_MSG 2 + /* Mailbox message types */ #define MBOX_MSG_MASK 0xFFFF #define MBOX_MSG_INVALID 0xFFFE diff --git a/drivers/common/cnxk/roc_mbox_priv.h b/drivers/common/cnxk/roc_mbox_priv.h index 4fafca6f72..354c8fa52a 100644 --- a/drivers/common/cnxk/roc_mbox_priv.h +++ b/drivers/common/cnxk/roc_mbox_priv.h @@ -71,10 +71,12 @@ struct mbox { const char *mbox_id2name(uint16_t id); int mbox_id2size(uint16_t id); void mbox_reset(struct mbox *mbox, int devid); -int mbox_init(struct mbox *mbox, uintptr_t hwbase, uintptr_t reg_base, - int direction, int ndevsi, uint64_t intr_offset); +int mbox_init(struct mbox *mbox, uintptr_t hwbase, uintptr_t reg_base, int direction, int ndevsi, + uint64_t intr_offset); void mbox_fini(struct mbox *mbox); void mbox_msg_send(struct mbox *mbox, int devid); +void mbox_msg_send_up(struct mbox *mbox, int devid); +bool mbox_wait_for_zero(struct mbox *mbox, int devid); int mbox_wait_for_rsp(struct mbox *mbox, int devid); int mbox_wait_for_rsp_tmo(struct mbox *mbox, int devid, uint32_t tmo); int mbox_get_rsp(struct mbox *mbox, int devid, void **msg);