From patchwork Tue Sep 19 01:29:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Mody, Rasesh" X-Patchwork-Id: 28893 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EC3521B21C; Tue, 19 Sep 2017 03:31:38 +0200 (CEST) Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0082.outbound.protection.outlook.com [104.47.37.82]) by dpdk.org (Postfix) with ESMTP id D12C51B19E for ; Tue, 19 Sep 2017 03:31:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=MIuFpU4OR66DRpPa0B5zkR/YqDXlTFggIpb5IDDkZyQ=; b=U4ZE4P36XX+zOd4UyVrbcbqv7E/IRvFekMP70anq4q8x9JGXDwDg652adznnbnSbRECzuyMhNqJ8xJ7HKrCtVwssCdXPQI9WKY4+/9QYOFStWFNPmDX8vLKkjg0AZMg3ditmPTL7z/l3PtPlfADX52BD23CSBG0Up/ughZAzvD4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Rasesh.Mody@cavium.com; Received: from cavium.com (198.186.0.2) by SN4PR0701MB3838.namprd07.prod.outlook.com (2603:10b6:803:4f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.56.11; Tue, 19 Sep 2017 01:31:18 +0000 From: Rasesh Mody To: dev@dpdk.org, ferruh.yigit@intel.com Cc: Rasesh Mody , Dept-EngDPDKDev@cavium.com Date: Mon, 18 Sep 2017 18:29:58 -0700 Message-Id: <1505784633-1171-19-git-send-email-rasesh.mody@cavium.com> X-Mailer: git-send-email 1.7.10.3 In-Reply-To: <1505784633-1171-1-git-send-email-rasesh.mody@cavium.com> References: <1505784633-1171-1-git-send-email-rasesh.mody@cavium.com> MIME-Version: 1.0 X-Originating-IP: [198.186.0.2] X-ClientProxiedBy: SN4PR0501CA0015.namprd05.prod.outlook.com (2603:10b6:803:40::28) To SN4PR0701MB3838.namprd07.prod.outlook.com (2603:10b6:803:4f::13) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 88f30299-9113-4c5a-1fb1-08d4fefe20db X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:SN4PR0701MB3838; X-Microsoft-Exchange-Diagnostics: 1; SN4PR0701MB3838; 3:fByGLNQPqfdJw3eFzihBAYiN15B9VsfuawNZWFRfdfU4vmZ/uK2gK8uWtrnL6tKMj6wJ654zfb8lJ9GShZtBG8ak6RVZdqgGiiZ26LgSAGqqCiWizfIZS/uS4o2Xy0S2h3ToVJfXtWfScINfWHEfx1akvOCm0irfr3Rs4kLyNOQoZw8HTJ2JTbyemfWE5DXad06u4oQx7i8KlG0UY9gzljeGn+WwTyHcKT6/tNnbrn+flk1pGIFEY31cmx2zpQcu; 25:kV/Mjgf/hX+aeXKsdmhsrw78H1S7rPfqA18R4u3FGNExZICw9CU+/ps4UO5QKxYaVJKf/Z0896oJZwEoMpoHVJqk5P5Bw2nZ6p9HFVUauEtc0ecUI1Hz1DmoBzQfHeYvu3FgU3a8l2mdBvd8nw6izt0Z4DVhk8zUOKxZkmxF/+jA35Wrrr8Ts3XBFhB17OFYYwgDN7FU4d4UM+ay3NFJIiNENC8cCyPoY/MVqfSKK/fJBMxpYoo9nFZ//mjhO+svUaSI+h4CsFNZ/q1uXRUv8+yPWJMbkQSen64/vbzxslUj+Xgo9zSr9VgSlFJET3DIyQxRMT78bPXcsywv9z9VsA==; 31:ypF0JuPpFYsMpCR1JEcXGA1jf6POIFKCdJah20Cemsc4Hk+k9sJRSFZ49f1u41tJrlBLFwIn2ozAOi/EWaB422hZ93gamV2hM0++I5U47ODYSmvMasAYb9W9KgRd1r4m4g58NiW7VoOThpIzKiDYknTX0QvoqV7CsnvT99piDlFnfc2NtaZU4AYeHl76ClvvjbtjM3OofdlS3qzY012JkTUmnEh4sthpZ9e66pSHLtc= X-MS-TrafficTypeDiagnostic: SN4PR0701MB3838: X-Microsoft-Exchange-Diagnostics: 1; SN4PR0701MB3838; 20:Lg8ByKmgM4TYdlzdi6lByAcaljs0eunfHLQghmiGo1wNvzlp28tNT0+RDse28mvd+FgMgl+MnRZ1EoJnBkV7R5vPog2HL69F1BbGYJMGbE6U+fZ+milB36GpmhkPCML5tdL7H0lVG7++qbb5SuhQ1dkMCnvQhj+4gAYFM+2QL85oCNQFaJqI8lHWGQBIuy9V1IY1JIS0jG/PCz2w+OMbhFFTDtvqUJVGJjwG9oR5k8Afo+EeW6VHWPBGT//9IBF9UIaTf9qZHiQHW5EtSGQeNZ0LAraIaiOzTBQTqnKF4LO9T/YLuCYLhpD+8nObxKL1pK73mod9mcgIJsfWxzMNpSJBOrsmlug8L0L6/6qbRP7dIk/QeQ2BsrXTLeLjHYceJ8Si6dxaAtjxys6/uxVoD3/PRDTYzQ34K14+E5ijHJkmYEZpAhJK8V9INtHScQYMQYomORg5dSZ3mdXQVHod1DmM0KP1OpPJoAlHB75zbYbfiW4LfL23CBKTDIzcdUHe; 4:ZjC1D6JtUOabMQ6G36Dp7dWRjyWWepMI8aZaCzzjhLAzZu9cw9iSbCEkom/8oB2hoU3E7ik0jlXwEbeTBgyBldCFGWbgdXkHwJi96zEs1XboHiSErIuaCYTRTPygykQASY3nAvHHaHnSTyUUI2NwCVXR7UiUf8+Hg4TK2uDPbAWshtH3f0NOjagGfNAUesHe+dyV+WeiU7XyvgQiFnljp4QGNHc+Xq7ZvpFV61+Mili60Uh2+qXQvpJKeqVZFhCgQIDxtrV66awLaNP4Z1rQWjUliunW1FMJzUU/WnEZQeI= X-Exchange-Antispam-Report-Test: UriScan:(17755550239193); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(10201501046)(93006095)(93001095)(3002001)(6041248)(20161123555025)(20161123562025)(20161123558100)(20161123560025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:SN4PR0701MB3838; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:SN4PR0701MB3838; X-Forefront-PRVS: 04359FAD81 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(376002)(346002)(199003)(189002)(48376002)(33646002)(68736007)(316002)(21086003)(16526017)(16586007)(72206003)(36756003)(478600001)(97736004)(86362001)(66066001)(25786009)(69596002)(47776003)(50986999)(81166006)(8936002)(50226002)(8676002)(5660300001)(7736002)(53936002)(305945005)(81156014)(55016002)(50466002)(76176999)(6116002)(3846002)(2950100002)(101416001)(4326008)(4720700003)(2906002)(107886003)(6666003)(189998001)(106356001)(5003940100001)(105586002); DIR:OUT; SFP:1101; SCL:1; SRVR:SN4PR0701MB3838; H:cavium.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN4PR0701MB3838; 23:/gIP/JwgA3elo9HRLQ3ti3UjyhPZq/qqj1Ftao9?= IA/MuCyLwrvB+WnnU/DdcfUq23Zlv2EMZK1j4qUnZ87AJYR7bMAD0bBzOCITZ5952US8nUGJNpW4yhVQt+uHrAkRoOLhAPu9vHG16C1ZuJrxlcFV65gcS4yWXod/YiETQxWGCuAxP+CYuwksbpykRaN6HQxwyKC7IVoakLCEm8WH3UdhIBNSYsTex+ItkDbdnOhCWtm5m6qvQ6A2fDZ2nTP5d+s5QbyeXPslMNMx2LreL0ZA+uk69jwKXJjqu69OD2fjWu5nkDjI2rMeO+kO1cLXrtNtaVlQXKlvv6CQyD4695XOyh68rg61nckq/YeRHlWIHx+fIPs9rLQQqCfJexpXFpTyFGarkVd4BOXZcTFsLOuQ0pdUMOlCYtyRZbIsXjNjCjOxzTuD6ITHmkgxv2xbTjQGssaA0MzR3ZVfkFz33i2k2inGGpa6K35WghzxF/HdzLv2EQiA8NphAK+4cTx4c8cVLM9RIhkjoazjEtkXpkc2/AMwMhwXAs+SlGohdufCGsxSPpjqgYY6IVGmz2sZGYyEPbJU8ERpr3MmPeP7sxkQf/HCQ6BK01B8Fmr9RrNbnojtnqGbpaFkB4j9jFnQhYQu4V+m6LY5adIkb6HyV9EbBU3M1I4E98WN0i1LAZHObIlBb7JTHMuamIwq9PfUVmSJ9n20z99W9g0Yji/DL+WqfNorstrUn0zVtsP4t6gPTsF7aoh6zLG3qWpe+4rvQRE0SKzkF+UlasunI1uIdBT4B2M6wss6QhKfCSvm4vcH7bzfkTWqbdlitaznN7KTI37S+pR5mu919hJF4VhOadQivz1HgtaZvLJEw57LZ7uzXqsmo4LH1TraiNX0bzTVTw4oLLT37LXmBXzZIzoqwqUNQi6D2PR1SVMFwUFbv2cJg0bHpTg1/fRZTSgySHeUG4Z4o2Mui+FiQe2i2ZtxBZKf7lA9thqaC2GtE9OrjRdz3scD0fRMJXFbZaG7bvNMm06hXKWJHjl2rOHNLPzd8TlqbYoLqsgNaLRu9kNZzEKVGAFmWhaacJEBnHhtkH6Z3Ia+ROwbj/2lItPGNJ8P2IE1b7u6OUx56NdlxCLlorR0= X-Microsoft-Exchange-Diagnostics: 1; SN4PR0701MB3838; 6:kqnOEjERq77GTmV8P/B8hsiUX8zkDnE8TOkACioANjAGEaLZFkkfKBfpdlQfrlygmOUwA1h4pTxGJialx+m5NX6c1/SJJI22ePVtMwgB2NAoru1BejGjU4JnEidCNGjnyK3hOJoWwZ4/zZS449c5260Hy42NAARLk97eMJo/BbY6p0ENvWSz/7+H6N/xe2d2CzpnwyRp/Y/BGauJsndRFasIrUyDJqXmCraTZrLyM3MyqdQ4U8us5dqfgwPX1vQNmYR8QNEF5W8QbZRkFVVSxuRnIt4pMmmo7JMNk1C0oZKLyjEAZaOYPel0gJqou0FRDiLx7voXSe+hsfoqzofyOg==; 5:aUbA5dwUDnOgLcJsvns79ll6T223XLV0RlquAiB3apEc4BAjYEFQx9E9W0qiN1lUOpN/ic9PS/cw+/+BCN+lWHWMLdCe5AwFfca8H/BAdFVXdGAfckoZbdag1dFr4d/T0Sie1/V7tRZqdfXL6TKk2Q==; 24:prfgbbC+ArjiEAUV1nAdePINqED6nsbk5OF/DcpL02QJ8qW1mZO0DRZk8KG58glxlkf8kyV+WhWtMf1U06OBn4f5pW42TEZ8DEWAFTRHNDA=; 7:YNOxgV68xsuUn6919xni8EdjRIgTidERpJv7pzcCdOHMjL2Ge4W5PoKcc+YYnK2JQkuEFXiB3+UT7gmIlV+KupNbRaEo1dHEJS2KZ670RMZaFetwDq4pEXeroPn6FO0MImYLXDv/hmIEIFVuGtNJ8KoGB0I9vKUyA/R+ynEt+gkMz/mwPlP6Qo1Fkugz0vO+w+RJKHjo+vsUbSTxAWDvD5mHiUYvGtxi4leLUOpPLic= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2017 01:31:18.9805 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR0701MB3838 Subject: [dpdk-dev] [PATCH 18/53] net/qede/base: avoid possible race condition X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" There's a possible race in multiple VF scenarios for base driver users that use the optional APIs ecore_iov_pf_get_and_clear_pending_events, ecore_iov_pf_add_pending_events. If the client doesn't synchronize the two calls, it's possible for the PF to clear a VF pending message indication without ever getting it [as 'get & clear' isn't atomic], leading to VF timeout on the command. The solution is to switch into a per-VF indication rather than having a bitfield for the various VFs with pending events. As part of the solution, the setting/clearing of the indications is done internally by base driver. As a result, ecore_iov_pf_add_pending_events is no longer needed and ecore_iov_pf_get_and_clear_pending_events loses the 'and_clear' from its name as its now a proper getter. A VF would be considered 'pending' [I.e., get_pending_events() should have '1' for it in its bitfield] beginning with the PF's base driver recognizing a message sent by that VF [in SP_DPC] and ending only when that VF message is processed. Signed-off-by: Rasesh Mody --- drivers/net/qede/base/ecore_iov_api.h | 16 +++-------- drivers/net/qede/base/ecore_sriov.c | 47 ++++++++++++++++++--------------- drivers/net/qede/base/ecore_sriov.h | 4 ++- 3 files changed, 33 insertions(+), 34 deletions(-) diff --git a/drivers/net/qede/base/ecore_iov_api.h b/drivers/net/qede/base/ecore_iov_api.h index 4fce6b6..4ec6217 100644 --- a/drivers/net/qede/base/ecore_iov_api.h +++ b/drivers/net/qede/base/ecore_iov_api.h @@ -345,21 +345,13 @@ struct ecore_public_vf_info* u16 vfid, bool b_enabled_only); /** - * @brief Set pending events bitmap for given @vfid - * - * @param p_hwfn - * @param vfid - */ -void ecore_iov_pf_add_pending_events(struct ecore_hwfn *p_hwfn, u8 vfid); - -/** - * @brief Copy pending events bitmap in @events and clear - * original copy of events + * @brief fills a bitmask of all VFs which have pending unhandled + * messages. * * @param p_hwfn */ -void ecore_iov_pf_get_and_clear_pending_events(struct ecore_hwfn *p_hwfn, - u64 *events); +void ecore_iov_pf_get_pending_events(struct ecore_hwfn *p_hwfn, + u64 *events); /** * @brief Copy VF's message to PF's buffer diff --git a/drivers/net/qede/base/ecore_sriov.c b/drivers/net/qede/base/ecore_sriov.c index b8e03d0..7ac533e 100644 --- a/drivers/net/qede/base/ecore_sriov.c +++ b/drivers/net/qede/base/ecore_sriov.c @@ -3755,8 +3755,7 @@ static enum _ecore_status_t ecore_iov_vf_flr_poll(struct ecore_hwfn *p_hwfn, ack_vfs[vfid / 32] |= (1 << (vfid % 32)); p_hwfn->pf_iov_info->pending_flr[rel_vf_id / 64] &= ~(1ULL << (rel_vf_id % 64)); - p_hwfn->pf_iov_info->pending_events[rel_vf_id / 64] &= - ~(1ULL << (rel_vf_id % 64)); + p_vf->vf_mbx.b_pending_msg = false; } return rc; @@ -3886,12 +3885,22 @@ void ecore_iov_process_mbx_req(struct ecore_hwfn *p_hwfn, mbx = &p_vf->vf_mbx; /* ecore_iov_process_mbx_request */ - DP_VERBOSE(p_hwfn, - ECORE_MSG_IOV, - "VF[%02x]: Processing mailbox message\n", p_vf->abs_vf_id); +#ifndef CONFIG_ECORE_SW_CHANNEL + if (!mbx->b_pending_msg) { + DP_NOTICE(p_hwfn, true, + "VF[%02x]: Trying to process mailbox message when none is pending\n", + p_vf->abs_vf_id); + return; + } + mbx->b_pending_msg = false; +#endif mbx->first_tlv = mbx->req_virt->first_tlv; + DP_VERBOSE(p_hwfn, ECORE_MSG_IOV, + "VF[%02x]: Processing mailbox message [type %04x]\n", + p_vf->abs_vf_id, mbx->first_tlv.tl.type); + OSAL_IOV_VF_MSG_TYPE(p_hwfn, p_vf->relative_vf_id, mbx->first_tlv.tl.type); @@ -4016,26 +4025,20 @@ void ecore_iov_process_mbx_req(struct ecore_hwfn *p_hwfn, #endif } -void ecore_iov_pf_add_pending_events(struct ecore_hwfn *p_hwfn, u8 vfid) +void ecore_iov_pf_get_pending_events(struct ecore_hwfn *p_hwfn, + u64 *events) { - u64 add_bit = 1ULL << (vfid % 64); + int i; - /* TODO - add locking mechanisms [no atomics in ecore, so we can't - * add the lock inside the ecore_pf_iov struct]. - */ - p_hwfn->pf_iov_info->pending_events[vfid / 64] |= add_bit; -} + OSAL_MEM_ZERO(events, sizeof(u64) * ECORE_VF_ARRAY_LENGTH); -void ecore_iov_pf_get_and_clear_pending_events(struct ecore_hwfn *p_hwfn, - u64 *events) -{ - u64 *p_pending_events = p_hwfn->pf_iov_info->pending_events; + ecore_for_each_vf(p_hwfn, i) { + struct ecore_vf_info *p_vf; - /* TODO - Take a lock */ - OSAL_MEMCPY(events, p_pending_events, - sizeof(u64) * ECORE_VF_ARRAY_LENGTH); - OSAL_MEMSET(p_pending_events, 0, - sizeof(u64) * ECORE_VF_ARRAY_LENGTH); + p_vf = &p_hwfn->pf_iov_info->vfs_array[i]; + if (p_vf->vf_mbx.b_pending_msg) + events[i / 64] |= 1ULL << (i % 64); + } } static struct ecore_vf_info * @@ -4069,6 +4072,8 @@ static enum _ecore_status_t ecore_sriov_vfpf_msg(struct ecore_hwfn *p_hwfn, */ p_vf->vf_mbx.pending_req = (((u64)vf_msg->hi) << 32) | vf_msg->lo; + p_vf->vf_mbx.b_pending_msg = true; + return OSAL_PF_VF_MSG(p_hwfn, p_vf->relative_vf_id); } diff --git a/drivers/net/qede/base/ecore_sriov.h b/drivers/net/qede/base/ecore_sriov.h index d190126..8923730 100644 --- a/drivers/net/qede/base/ecore_sriov.h +++ b/drivers/net/qede/base/ecore_sriov.h @@ -45,6 +45,9 @@ struct ecore_iov_vf_mbx { /* Address in VF where a pending message is located */ dma_addr_t pending_req; + /* Message from VF awaits handling */ + bool b_pending_msg; + u8 *offset; #ifdef CONFIG_ECORE_SW_CHANNEL @@ -168,7 +171,6 @@ struct ecore_vf_info { */ struct ecore_pf_iov { struct ecore_vf_info vfs_array[E4_MAX_NUM_VFS]; - u64 pending_events[ECORE_VF_ARRAY_LENGTH]; u64 pending_flr[ECORE_VF_ARRAY_LENGTH]; #ifndef REMOVE_DBG