From patchwork Fri Nov 3 11:20:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubham Rohila X-Patchwork-Id: 133834 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8C69B43279; Fri, 3 Nov 2023 12:20:35 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 58E194027F; Fri, 3 Nov 2023 12:20:35 +0100 (CET) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2072.outbound.protection.outlook.com [40.107.94.72]) by mails.dpdk.org (Postfix) with ESMTP id 68E8440273 for ; Fri, 3 Nov 2023 12:20:33 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PhgiAjbEdGD6vKOk7TLBJEETjCvSd5gNyCyllGkv+aADFVw3eR/WKdluOHwyjJwInn+SYSw6QcdJEA7EPloWox06WAs1Lq0f7qjuetPNAo34C4WkSi+NWotdrM3u/Rgl/mf3RnVn/Z8HAvGZ7z4pnMFytIy2KbvCNomhcDOu0F7MVixFAb5+2vzS3WuqivUnz7r/PaFfpOosBsBKmnfXm5KiBFwZsEMzIRWYpy+EkSohUXgXN0BCWs5p0pGFvne4waoqPK/BomL/6nnMtLH4EDp2M61kZgAvB8UQB2Wtxzqed2Ln2IVeVFhBuQCHGPxY5az3g1McrIgzieI0IONxAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nLFRa3rTyzSJRZpvWJP2bsc8PWTABxNWe5eJ+Ojby88=; b=XQSKViaFf4XlbbqYnDeWK/Uc4MF11/wJviZJ4vWWa0AuqtajbU2FZAgthMpVdhb54DJNjIAwhvqBQL2j/9l9YA+dLoEnTPl/fS1Vff2tGbY3m8U+bRtJ9UPBKEBqSxM2mW9yKInc1iSjsptC1cEC7IxW4EjxHer8tmGvcMowNVTW0fvSbekuOAQVGODFO+GndErMqnB4fJ6nJuEFMrTz/76LerQ3e3hXlPXU8JxIkgY/dMJJ7tzYQrkgh2mlxg37G+Q6D2FMonH+9JWI6kYb8WPcEemJ6aK9QNTBrcQfmFr66uCA48KVLnLAEXwAV3RcdnTxIS+1kBdyTeuEGRW/wg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nLFRa3rTyzSJRZpvWJP2bsc8PWTABxNWe5eJ+Ojby88=; b=lh7Mo+X9Ttx6x8QM346D9S0gKgak7Q9jXjHCdHZbWBLTixbizOBfqlhQ390qxG/fHAubZ4/gmn8KlbNBZg6RQ//pzwc1nEbxwecGcSMaueUOe3W5BFWTXfwSi07OERNbM4KaXEom9D6KJApNtmHfTSJCfRji8e6Wz/scQ6t8Nfg= Received: from MN2PR10CA0019.namprd10.prod.outlook.com (2603:10b6:208:120::32) by SJ0PR12MB7068.namprd12.prod.outlook.com (2603:10b6:a03:4ae::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.21; Fri, 3 Nov 2023 11:20:31 +0000 Received: from BL6PEPF0001AB51.namprd04.prod.outlook.com (2603:10b6:208:120:cafe::6c) by MN2PR10CA0019.outlook.office365.com (2603:10b6:208:120::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.22 via Frontend Transport; Fri, 3 Nov 2023 11:20:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF0001AB51.mail.protection.outlook.com (10.167.242.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6954.21 via Frontend Transport; Fri, 3 Nov 2023 11:20:30 +0000 Received: from SATLEXMB06.amd.com (10.181.40.147) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Fri, 3 Nov 2023 06:20:30 -0500 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB06.amd.com (10.181.40.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Fri, 3 Nov 2023 06:20:29 -0500 Received: from xhdipdslab61.xilinx.com (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.32 via Frontend Transport; Fri, 3 Nov 2023 06:20:27 -0500 From: Shubham Rohila To: , , , CC: , , , Shubham Rohila Subject: [PATCH v2 1/2] bus/cdx: add support for devices without MSI Date: Fri, 3 Nov 2023 16:50:15 +0530 Message-ID: <20231103112016.1945684-1-shubham.rohila@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231027162212.3353973-1-ferruh.yigit@amd.com> References: <20231027162212.3353973-1-ferruh.yigit@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB51:EE_|SJ0PR12MB7068:EE_ X-MS-Office365-Filtering-Correlation-Id: eff42f36-3dc4-400f-9382-08dbdc5ee3b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6aybAgqQi1QYUVZ6p1Y2M5BdsoEIH/m8u9jwiTIH0GxmmhKj/fRRyFbfhDfCy9xQ0bF87M5A+uRv1WVDLHAJ92k8eAc7bE9U5hQ6ZNuCo7NTns0RGvzZUhYp552cj5Wl7aoyjUtaOGtYXH9ZqCY42bRlqqyPdW6+/cPoEnJ/1U/6Ch2Phoa3BVexThrfLlAbpsKMDocAVGU4FZXi74R6CuHeT/nLksaFEpN3hcDdiypG2d07g2X+SrGZ5FLnMDzzsen89JKg+iiEkXcVFYOGblcNqdAR6Dj3ZJyjrP/vb4z8CnXwcXJhxOrDYdneyf/klJkbf64X7c8BD7H6vMeLJ6Da3zv8z0eF0zJv+NaramjiLo8k7vD0IgHJnikJ6sPxNbPuc97oRxvIoJAnhhUalYT2y9Cmdj3+09m1FbTdzhiyT9ZZ3QpTx9XQmzlZYJQUmsMFIm+EITIkb9VN8uJy94USE//zch+k7VA3dOL/ZYLP/O6ZRymwz/qh9tm/yLrchZpA+wBHiduwccdq3to37yuYYU5dPgIkSRSymDlaWwhihsnJE6UH1XB7vopJbRKkn8KAUpynO7b0BQkqQLeFSj4j3SqFawQL9XWWc1d7dwas5oANd7OuVSn1ld/NJSsww3BYW98NGj/eKJJzN/bZn71SMUCZpoQvEscvTOhPHuy6VLNO8OZuZOMOuLw+wek6P9QBP21J4ZKTJpzq6RQsK8mxzqKipRv1ZyQ9IOXl9hZDto/F1Q/Xl3k5wFF7RB1Su34px/eghHwMkvOvt3fltA== X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(4636009)(136003)(396003)(39860400002)(346002)(376002)(230922051799003)(451199024)(82310400011)(1800799009)(186009)(64100799003)(36840700001)(40470700004)(46966006)(40460700003)(40480700001)(2616005)(26005)(1076003)(36860700001)(478600001)(54906003)(83380400001)(426003)(2906002)(6666004)(336012)(110136005)(44832011)(41300700001)(70206006)(5660300002)(70586007)(8936002)(4326008)(47076005)(316002)(8676002)(36756003)(356005)(82740400003)(81166007)(86362001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2023 11:20:30.7952 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: eff42f36-3dc4-400f-9382-08dbdc5ee3b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB51.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB7068 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Nikhil Agarwal Update the cleanup routine for cdx device to support device without MSI. Also, set vfio_dev_fd for such devices This fd can be used for BME reload operations. Signed-off-by: Nikhil Agarwal Signed-off-by: Shubham Rohila Acked-by: Nipun Gupta --- v2 - New patch in the series drivers/bus/cdx/cdx.c | 2 +- drivers/bus/cdx/cdx_vfio.c | 19 +++++++++---------- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/drivers/bus/cdx/cdx.c b/drivers/bus/cdx/cdx.c index 541aae76c3..62b108e082 100644 --- a/drivers/bus/cdx/cdx.c +++ b/drivers/bus/cdx/cdx.c @@ -405,9 +405,9 @@ cdx_probe_one_driver(struct rte_cdx_driver *dr, return ret; error_probe: + cdx_vfio_unmap_resource(dev); rte_intr_instance_free(dev->intr_handle); dev->intr_handle = NULL; - cdx_vfio_unmap_resource(dev); error_map_device: return ret; } diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c index 8a3ac0b995..8cac79782e 100644 --- a/drivers/bus/cdx/cdx_vfio.c +++ b/drivers/bus/cdx/cdx_vfio.c @@ -101,13 +101,12 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev) struct mapped_cdx_res_list *vfio_res_list; int ret, vfio_dev_fd; - if (rte_intr_fd_get(dev->intr_handle) < 0) - return -1; - - if (close(rte_intr_fd_get(dev->intr_handle)) < 0) { - CDX_BUS_ERR("Error when closing eventfd file descriptor for %s", - dev->device.name); - return -1; + if (rte_intr_fd_get(dev->intr_handle) >= 0) { + if (close(rte_intr_fd_get(dev->intr_handle)) < 0) { + CDX_BUS_ERR("Error when closing eventfd file descriptor for %s", + dev->device.name); + return -1; + } } vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); @@ -185,6 +184,9 @@ cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd, { int i, ret; + if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd)) + return -1; + if (num_irqs == 0) return 0; @@ -227,9 +229,6 @@ cdx_vfio_setup_interrupts(struct rte_cdx_device *dev, int vfio_dev_fd, if (rte_intr_type_set(dev->intr_handle, RTE_INTR_HANDLE_VFIO_MSIX)) return -1; - if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd)) - return -1; - return 0; } From patchwork Fri Nov 3 11:20:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shubham Rohila X-Patchwork-Id: 133835 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8BB9443279; Fri, 3 Nov 2023 12:20:41 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 733C840691; Fri, 3 Nov 2023 12:20:40 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2075.outbound.protection.outlook.com [40.107.243.75]) by mails.dpdk.org (Postfix) with ESMTP id DF8574067D for ; Fri, 3 Nov 2023 12:20:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GVS96u8jUKbSbJkX5DhF5He6acjxA85qlDBELvC6PNORshSkFX1Jiac1yufCFHnncDkg/faYN01+AHByW0Zvv0/0QmDXUv8WQ4sXF+1zoA4WV+nwbM1jEMH4VyhK6lfzBAllE99Wn7eKbMS6vNJB6XJnxfiEexClkuUrPw803QPFJIkDM+EAEPR9+OV/60QBePeQI2iUpyXrU15hxO/w4EezGUZSdyGhKCTtWFmTjj2P5s1U5Vb/scGxDm6xHlY4KJndwPo4EFguvhegTU2+YyhHLXT5fkJ524IQOuFxPjYkT08aPE/xU9VlQp747d7ntSYtY5tIcU1BAd9NA7mbEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R6lmhbqBY/pzmgv1VGaKpQpMhXOUpjORDh1a5RMZqCE=; b=HeBlO5eUUDZy4aOKqfUAhmigGmgcDrQeJ2OoJa1zrTrXijWafyKS1Lsnr4eJQfxE///6HFEpbEZiTbFiaEsqPwnftwcWJl7H1V7x9vB+G4RvgY2szUhPHxbIbfbhepDYAnIYcUV4s/WU4fOzJ5LeB4lPeJ1EnDCLoSoLrylXjSzmCMDtVz1LYwEP1P1w2be4rdKHrNIOYzZC9yrVlgUKjEUUGKJbj9Q6EEolfUB/wlUwW350g/yH0ob3hWpNPQySDO4o9hHdzSBM3a71bBtmmDeRRi+BOGkDlV7X4rkPgW7BOMMrLA3zlSV9W7kaX8Dm0f+nez/wX1gvahDbmro6Jw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R6lmhbqBY/pzmgv1VGaKpQpMhXOUpjORDh1a5RMZqCE=; b=hHsiyD3JBQg9uAICoF1EkLuqsfbz8qhy18COpNf+uCa1U9pLj3efCO0aavQpNn3MLkQkrzoCD2Ht0pJbZtbfOzQ6geGJwpNqc5Wp7OXjfFkRPeqn1jbMXlUR5ZDv4EpwSVf7Afc3LKBfjoDljkJ763Y1EJnxYKVHlPrVsomsrbQ= Received: from MN2PR19CA0031.namprd19.prod.outlook.com (2603:10b6:208:178::44) by SJ2PR12MB7964.namprd12.prod.outlook.com (2603:10b6:a03:4cf::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.24; Fri, 3 Nov 2023 11:20:36 +0000 Received: from BL6PEPF0001AB4A.namprd04.prod.outlook.com (2603:10b6:208:178:cafe::a4) by MN2PR19CA0031.outlook.office365.com (2603:10b6:208:178::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.22 via Frontend Transport; Fri, 3 Nov 2023 11:20:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF0001AB4A.mail.protection.outlook.com (10.167.242.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6954.19 via Frontend Transport; Fri, 3 Nov 2023 11:20:35 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Fri, 3 Nov 2023 06:20:35 -0500 Received: from xhdipdslab61.xilinx.com (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.32 via Frontend Transport; Fri, 3 Nov 2023 06:20:33 -0500 From: Shubham Rohila To: , , , CC: , , , Shubham Rohila Subject: [PATCH v2 2/2] bus/cdx: add bus master enable/disable APIs Date: Fri, 3 Nov 2023 16:50:16 +0530 Message-ID: <20231103112016.1945684-2-shubham.rohila@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231103112016.1945684-1-shubham.rohila@amd.com> References: <20231027162212.3353973-1-ferruh.yigit@amd.com> <20231103112016.1945684-1-shubham.rohila@amd.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4A:EE_|SJ2PR12MB7964:EE_ X-MS-Office365-Filtering-Correlation-Id: 7b45d533-4ba6-491a-b894-08dbdc5ee6c7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: StfUmH/LqGaTvf8xHmK6V4bMu4Wkon03ODfdh7C27mw6fNYcooFQNyhzlFKCyc67RynJhzNKg3BeeKPuA2PcaOZxOqljT5j5kHWYDdzh3OFlI3oq2UEjEu5xYYxShArqqV4Iqg4K6Cj5eLl53AhMLRTin04H3GfAKNDgWK8k38CVYvja2NSL3BmNHWj2lF7Iqkx+UISoOeDpFC9moT983TmQQjal5BFUcTgkoGcUlleqh2nBLUK6Nr6EMisBLL1BRr/AZQk8z3Bc1DHbhedq44BTnDFXxPeFgvjFPCsigB67+mhxs78p48C/MOXAEKqrnHZh9+XWH5Yct4vFHwSyTr32xgsXSlEjvN91Hh4X0tq4i4CTKzm/pbTCPllzo2DUhIdV/hrBRJrLA0+j2GP70pY3fDqKzLtEsW+1aEBnIoZNe3nwlUEh0FjJM6ClocCL0zKMXHtJxt6uemkmMhjMme58/5BmuAk3mrsH5CyNQ9ArH339talaZ+2iWfL9QRdy/zLiJ9E41pm9QdOrwb3/t9TPxOUDoRrpf/MuWnfdCGoaDOenuz40StuN2vClxKb19qS6mXIuFzmuBUySraExHmrrcuSBN0EBpp5uixJBc02UKTE73USW8YJxLWzdaqMVQmjzcOWX5CMbYaQCvi8Do2AB3OpPsCiIqUFlAByJ48dhC6Lxp6RDtGrCh3Cueq7R0FiMvkPATjMqIcQ0qqs0ZloxbffI8VFGrDPZAdEDTQ0+FfYQpGDUX6cNQdSwY8WbTLhoCywK7zLFXuYZ5WjcmA== X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(4636009)(376002)(396003)(136003)(346002)(39860400002)(230922051799003)(82310400011)(186009)(451199024)(64100799003)(1800799009)(46966006)(36840700001)(40470700004)(47076005)(26005)(6666004)(478600001)(54906003)(2906002)(81166007)(40480700001)(70206006)(110136005)(70586007)(1076003)(82740400003)(2616005)(36756003)(336012)(426003)(40460700003)(41300700001)(44832011)(316002)(5660300002)(356005)(83380400001)(86362001)(36860700001)(4326008)(8676002)(8936002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2023 11:20:35.9473 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7b45d533-4ba6-491a-b894-08dbdc5ee6c7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7964 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Define rte_cdx_vfio_bm_enable and rte_cdx_vfio_bm_disable to enable or disable bus master functionality for cdx devices. Signed-off-by: Shubham Rohila Acked-by: Nipun Gupta --- v2 - Fix indentations and alphabetical orderings. - Wrap checks in parenthesis for readability - Assign argsz before using in rte_cdx_vfio_bm_disable - Remove unecessary variable initialisation drivers/bus/cdx/bus_cdx_driver.h | 25 ++++++++ drivers/bus/cdx/cdx_vfio.c | 103 +++++++++++++++++++++++++++++++ drivers/bus/cdx/version.map | 2 + lib/eal/include/rte_vfio.h | 27 ++++++++ 4 files changed, 157 insertions(+) diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h index 1c9a64c87a..211f8e406b 100644 --- a/drivers/bus/cdx/bus_cdx_driver.h +++ b/drivers/bus/cdx/bus_cdx_driver.h @@ -16,6 +16,7 @@ extern "C" { #include #include +#include #include #include @@ -178,6 +179,30 @@ int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle); __rte_internal int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle); +/** + * Enable Bus Mastering for CDX bus devices. + * + * @param dev + * Pointer to the cdx device. + * + * @return + * 0 on success, -1 on error. + */ +__rte_internal +int rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev); + +/** + * Disable Bus Mastering for CDX bus devices. + * + * @param dev + * Pointer to the cdx device. + * + * @return + * 0 on success, -1 on error. + */ +__rte_internal +int rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev); + /** * Unregister a CDX driver. * diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c index 8cac79782e..1481022f13 100644 --- a/drivers/bus/cdx/cdx_vfio.c +++ b/drivers/bus/cdx/cdx_vfio.c @@ -102,6 +102,10 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev) int ret, vfio_dev_fd; if (rte_intr_fd_get(dev->intr_handle) >= 0) { + if (rte_cdx_vfio_bm_disable(dev) < 0) + CDX_BUS_ERR("Error when disabling bus master for %s", + dev->device.name); + if (close(rte_intr_fd_get(dev->intr_handle)) < 0) { CDX_BUS_ERR("Error when closing eventfd file descriptor for %s", dev->device.name); @@ -255,6 +259,16 @@ cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd, return -1; } + /* + * Enable Bus mastering for the device. errno is set as ENOTTY if + * device does not support configuring bus master. + */ + if (rte_cdx_vfio_bm_enable(dev) && (errno != -ENOTTY)) { + CDX_BUS_ERR("Bus master enable failure! Error: %d (%s)", errno, + strerror(errno)); + return -1; + } + return 0; } @@ -595,3 +609,92 @@ rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle) return ret; } + +/* enable Bus Mastering */ +int +rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev) +{ + struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; + struct vfio_device_feature_bus_master *vfio_bm_feature; + struct vfio_device_feature *feature; + int vfio_dev_fd, ret; + size_t argsz; + + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + if (vfio_dev_fd < 0) + return -1; + + argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master); + + feature = (struct vfio_device_feature *)malloc(argsz); + if (!feature) + return -ENOMEM; + + vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data; + + feature->argsz = argsz; + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE; + feature->flags |= VFIO_DEVICE_FEATURE_SET; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret) { + CDX_BUS_ERR("Bus Master configuring not supported for device: %s, error: %d (%s)\n", + dev->name, errno, strerror(errno)); + free(feature); + return ret; + } + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET; + vfio_bm_feature->op = VFIO_DEVICE_FEATURE_SET_MASTER; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret < 0) + CDX_BUS_ERR("BM Enable Error for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + + free(feature); + return ret; +} + +/* Disable Bus Mastering */ +int +rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev) +{ + struct vfio_device_feature_bus_master *vfio_bm_feature; + struct vfio_device_feature *feature; + int vfio_dev_fd, ret; + size_t argsz; + + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + if (vfio_dev_fd < 0) + return -1; + + argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master); + + feature = (struct vfio_device_feature *)malloc(argsz); + if (!feature) + return -ENOMEM; + + vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data; + + feature->argsz = argsz; + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE; + feature->flags |= VFIO_DEVICE_FEATURE_SET; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret) { + CDX_BUS_ERR("Bus Master configuring not supported for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + free(feature); + return ret; + } + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET; + vfio_bm_feature->op = VFIO_DEVICE_FEATURE_CLEAR_MASTER; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret < 0) + CDX_BUS_ERR("BM Disable Error for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + + free(feature); + return ret; +} diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map index 0a15d39ae8..7a22cf5b4b 100644 --- a/drivers/bus/cdx/version.map +++ b/drivers/bus/cdx/version.map @@ -5,6 +5,8 @@ INTERNAL { rte_cdx_register; rte_cdx_unmap_device; rte_cdx_unregister; + rte_cdx_vfio_bm_disable; + rte_cdx_vfio_bm_enable; rte_cdx_vfio_intr_disable; rte_cdx_vfio_intr_enable; diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h index 3487c4f2a2..22832afd0f 100644 --- a/lib/eal/include/rte_vfio.h +++ b/lib/eal/include/rte_vfio.h @@ -73,6 +73,33 @@ struct vfio_info_cap_header { #define RTE_VFIO_CAP_MSIX_MAPPABLE 3 #endif +/* VFIO_DEVICE_FEATURE is defined for kernel version 5.7 and newer. */ +#ifdef VFIO_DEVICE_FEATURE +#define RTE_VFIO_DEVICE_FEATURE VFIO_DEVICE_FEATURE +#else +#define RTE_VFIO_DEVICE_FEATURE _IO(VFIO_TYPE, VFIO_BASE + 17) +struct vfio_device_feature { + __u32 argsz; + __u32 flags; +#define VFIO_DEVICE_FEATURE_MASK (0xffff) /* 16-bit feature index */ +#define VFIO_DEVICE_FEATURE_GET (1 << 16) /* Get feature into data[] */ +#define VFIO_DEVICE_FEATURE_SET (1 << 17) /* Set feature from data[] */ +#define VFIO_DEVICE_FEATURE_PROBE (1 << 18) /* Probe feature support */ + __u8 data[]; +}; +#endif + +#ifdef VFIO_DEVICE_FEATURE_BUS_MASTER +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER VFIO_DEVICE_FEATURE_BUS_MASTER +#else +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER 10 +struct vfio_device_feature_bus_master { + __u32 op; +#define VFIO_DEVICE_FEATURE_CLEAR_MASTER 0 /* Clear Bus Master */ +#define VFIO_DEVICE_FEATURE_SET_MASTER 1 /* Set Bus Master */ +}; +#endif + #else /* not VFIO_PRESENT */ /* we don't need an actual definition, only pointer is used */