From patchwork Fri Oct 27 16:22:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ferruh Yigit X-Patchwork-Id: 133513 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0F48543218; Fri, 27 Oct 2023 18:22:27 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B349A4029A; Fri, 27 Oct 2023 18:22:26 +0200 (CEST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2045.outbound.protection.outlook.com [40.107.212.45]) by mails.dpdk.org (Postfix) with ESMTP id 6AE0440263 for ; Fri, 27 Oct 2023 18:22:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mxlreYdZwBgJa2fUrwVQEJpo+Au5JhOwZ7gwofE77YwBlP4qe8KpVcEEqgcC+0ATpBR09Qd2Gb0kE7PLolt6+ssDiWxeVlOzDkJjv4qYO8ij/GV54mOPuuhkfZEQz10ug26qZT21gbGhoHpon7jarOIKyh7xiDXkve0O6LoC8XjaBYpa0kClx2GlVLFtgarVpAjU73I+QzlMXOOnQV0Bd5I0wBhZCShfq3ochEewggzbk2Unp5l9dkllrMGUrqGGTWAER6vgEt8+jtCBo6Hxj8ux9L3LLaPL02dGaIOWgYvcA6xzOAAi/C7egv9oFcM9mHtqgtGyCFd0q3mR53BhDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4ahzJ7CJCpg1RYAHSUX/+tQWSMC+nm2iGaIqKFtmcms=; b=GT8q5Y3CfHeeFsBz5tUbNPrjNeQSySv0konOaj+n80P4hzIDuNTumGoy9OnQSUciWE4UzhlNhMLz1tpLwO0BLSjFXzXA3cINfGSksnYE6dptn8dBm6FK8eVBdn4fp1ouOvlX7wgVOKaMLhjMDp1mu+csnVlyyqVlz+t/9s/T9T2fCTAusIV8YlGb118Gw3vtlO8/n2iRCepb/MR1W7bpCwzlv0DuhA6hEzZb9HpKF+UjYDLjKCWMMR8Imc4tbaTJHoyvA49r2CcILmc3U+KtSU66T4WpKWS5C5S771X8NggLNKInCRHpoMajPC0tEvdLPXx+Ib3WKp2wO4OBKmxSzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4ahzJ7CJCpg1RYAHSUX/+tQWSMC+nm2iGaIqKFtmcms=; b=G4XPxdg1tI5sNP9v5ib9g2IGaqz3eZFHp3LvO2dO/nUDcCJrMSmoXsdearnkVSUodj/SA0H++QB0kux24PdmBxfAuMBCPE6l7unEXsEK8mXHo/pin4w+bDP6yJg3/PGYWiqaWvsjQH0a9kCCaIZAt+MqI206VndNUWvXLW5uJC8= Received: from BL0PR01CA0022.prod.exchangelabs.com (2603:10b6:208:71::35) by CY8PR12MB7219.namprd12.prod.outlook.com (2603:10b6:930:59::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33; Fri, 27 Oct 2023 16:22:23 +0000 Received: from BL02EPF0001A102.namprd05.prod.outlook.com (2603:10b6:208:71:cafe::64) by BL0PR01CA0022.outlook.office365.com (2603:10b6:208:71::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19 via Frontend Transport; Fri, 27 Oct 2023 16:22:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL02EPF0001A102.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6933.15 via Frontend Transport; Fri, 27 Oct 2023 16:22:22 +0000 Received: from telcodpdk.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Fri, 27 Oct 2023 11:22:21 -0500 From: Ferruh Yigit To: , , , CC: , , , Shubham Rohila Subject: [PATCH] Add bus master enable/disable APIs for CDX devices Date: Fri, 27 Oct 2023 17:22:11 +0100 Message-ID: <20231027162212.3353973-1-ferruh.yigit@amd.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A102:EE_|CY8PR12MB7219:EE_ X-MS-Office365-Filtering-Correlation-Id: 4909987a-3306-425c-efe0-08dbd708e66d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TqefJw04tuWocmQyCotKzuRVGqQCD9HFbK9+F/jhR2uFXSO3wQUyIIW/S/BUa2m//Fr5clDXAAul2V386sp9Vvl1i1HyYk/BP2n9ZglcHDBW3KTvzCObaZwQhFrXlBTurauZastl6SYOaE8bTRN2qFF9oE++21HHW4NCMZRte/ut/xdhd/J143xRjy/uVrieIWak8nGr9ij0cnsdhq/VnXSzrDOu5k5c2pAt5GQPuS2LSutWRWKPX4U1mHo56j2vTLzjXAsvXQGuD2hT0lhVEdCqfCG/UW85MrwoH2t5Cg5DC5DDPGcJdxKqQxY3zS5bdGgxk6593haCVLVz1PgnBPiI1/j7g0zXKLv6Eb0D1Ip+gSfh7tkKQLYsfs05XiOt0W7+KnpgLuTvHs0YmKF6eIOrd4h6U2krtpyIrkTpx9Lj6rtu1cyF7/VKT3zPpfV75KI4Fs2rKoxaxmLDZo813xC1prF0AhvE09j20GlB1U5uT3QnV6mxfvedFDJZbq3pdiX0GC5idPOHEQ3O/v4d0EnVlGL9c/ZHgxcGO/S7bEROLQKPZyPZ1/gIPqo4oubFXTkH8mBCfy9+xrOpChuFD2hVn8iB8fPvGa403JYFNWD5Z1TXfeIvKvdVa5OK75xsfpMY0cLxQYcg/DNMblf5/OfVIdqkNjvi0HYZytvTrSsuZhXKiHACcwbL8Dsmu6UN0JJJQ1ERlp1b3o9/eZONEn6eXis5gJE/KcbzEiAMqVIyQmOFMXw0O6QFHBtaOx6QOStCFF7dSP7yNVzs1uWFoA== X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(4636009)(136003)(396003)(376002)(39860400002)(346002)(230922051799003)(64100799003)(186009)(82310400011)(451199024)(1800799009)(36840700001)(40470700004)(46966006)(16526019)(26005)(5660300002)(36756003)(2906002)(44832011)(41300700001)(36860700001)(86362001)(316002)(8936002)(4326008)(8676002)(7696005)(1076003)(478600001)(356005)(81166007)(110136005)(40460700003)(2616005)(54906003)(82740400003)(70586007)(426003)(83380400001)(336012)(70206006)(40480700001)(47076005)(6666004)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2023 16:22:22.8263 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4909987a-3306-425c-efe0-08dbd708e66d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A102.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7219 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Shubham Rohila Define rte_cdx_vfio_bm_enable and rte_cdx_vfio_bm_disable to enable or disable bus master functionality for cdx devices. Signed-off-by: Shubham Rohila --- drivers/bus/cdx/bus_cdx_driver.h | 25 ++++++++ drivers/bus/cdx/cdx_vfio.c | 97 ++++++++++++++++++++++++++++++++ drivers/bus/cdx/version.map | 2 + lib/eal/include/rte_vfio.h | 27 +++++++++ 4 files changed, 151 insertions(+) diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h index 1c9a64c87ac1..2f94a6c3afff 100644 --- a/drivers/bus/cdx/bus_cdx_driver.h +++ b/drivers/bus/cdx/bus_cdx_driver.h @@ -16,6 +16,7 @@ extern "C" { #include #include +#include #include #include @@ -178,6 +179,30 @@ int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle); __rte_internal int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle); +/** + * Enable Bus Mastering for CDX bus devices. + * + * @param dev + * Pointer to the cdx device. + * + * @return + * 0 on success, -1 on error. + */ +__rte_internal +int rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev); + +/** + * Disable Bus Mastering for CDX bus devices. + * + * @param dev + * Pointer to the cdx device. + * + * @return + * 0 on success, -1 on error. + */ +__rte_internal +int rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev); + /** * Unregister a CDX driver. * diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c index 8a3ac0b99537..a0abf71483aa 100644 --- a/drivers/bus/cdx/cdx_vfio.c +++ b/drivers/bus/cdx/cdx_vfio.c @@ -256,6 +256,16 @@ cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd, return -1; } + /* + * Enable Bus mastering for the device. errno is set as ENOTTY if + * device does not support configuring bus master. + */ + if (rte_cdx_vfio_bm_enable(dev) && errno != -ENOTTY) { + CDX_BUS_ERR("Bus master enable failure! Error: %d (%s)", errno, + strerror(errno)); + return -1; + } + return 0; } @@ -596,3 +606,90 @@ rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle) return ret; } + +/* enable Bus Mastering */ +int +rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev) +{ + struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; + struct vfio_device_feature_bus_master *vfio_bm_feature; + struct vfio_device_feature *feature; + int vfio_dev_fd, ret = 0; + size_t argsz = 0; + + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + if (vfio_dev_fd < 0) + return -1; + + argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master); + + feature = (struct vfio_device_feature *)malloc(argsz); + if (!feature) + return -ENOMEM; + + vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data; + + feature->argsz = argsz; + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE; + feature->flags |= VFIO_DEVICE_FEATURE_SET; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret) { + CDX_BUS_ERR("Bus Master configuring not supported for device: %s, error: %d (%s)\n", + dev->name, errno, strerror(errno)); + free(feature); + return ret; + } + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET; + vfio_bm_feature->op = VFIO_DEVICE_FEATURE_SET_MASTER; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret < 0) + CDX_BUS_ERR("BM Enable Error for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + + free(feature); + return ret; +} + +/* Disable Bus Mastering */ +int +rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev) +{ + struct vfio_device_feature_bus_master *vfio_bm_feature; + struct vfio_device_feature *feature; + int vfio_dev_fd, ret = 0; + size_t argsz = 0; + + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + if (vfio_dev_fd < 0) + return -1; + + feature = (struct vfio_device_feature *)malloc(argsz); + if (!feature) + return -ENOMEM; + + vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data; + + feature->argsz = argsz; + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE; + feature->flags |= VFIO_DEVICE_FEATURE_SET; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret) { + CDX_BUS_ERR("Bus Master configuring not supported for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + free(feature); + return ret; + } + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET; + vfio_bm_feature->op = VFIO_DEVICE_FEATURE_CLEAR_MASTER; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret < 0) + CDX_BUS_ERR("BM Disable Error for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + + free(feature); + return ret; +} diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map index 0a15d39ae82a..5438f76029e0 100644 --- a/drivers/bus/cdx/version.map +++ b/drivers/bus/cdx/version.map @@ -7,6 +7,8 @@ INTERNAL { rte_cdx_unregister; rte_cdx_vfio_intr_disable; rte_cdx_vfio_intr_enable; + rte_cdx_vfio_bm_enable; + rte_cdx_vfio_bm_disable; local: *; }; diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h index 3487c4f2a251..1c791b3636e9 100644 --- a/lib/eal/include/rte_vfio.h +++ b/lib/eal/include/rte_vfio.h @@ -73,6 +73,33 @@ struct vfio_info_cap_header { #define RTE_VFIO_CAP_MSIX_MAPPABLE 3 #endif +/* VFIO_DEVICE_FEATURE is defined for kernel version 5.7 and newer. */ +#ifdef VFIO_DEVICE_FEATURE +#define RTE_VFIO_DEVICE_FEATURE VFIO_DEVICE_FEATURE +#else +#define RTE_VFIO_DEVICE_FEATURE _IO(VFIO_TYPE, VFIO_BASE + 17) +struct vfio_device_feature { + __u32 argsz; + __u32 flags; +#define VFIO_DEVICE_FEATURE_MASK (0xffff) /* 16-bit feature index */ +#define VFIO_DEVICE_FEATURE_GET (1 << 16) /* Get feature into data[] */ +#define VFIO_DEVICE_FEATURE_SET (1 << 17) /* Set feature from data[] */ +#define VFIO_DEVICE_FEATURE_PROBE (1 << 18) /* Probe feature support */ + __u8 data[]; +}; +#endif + +#ifdef VFIO_DEVICE_FEATURE_BUS_MASTER +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER VFIO_DEVICE_FEATURE_BUS_MASTER +#else +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER 10 +struct vfio_device_feature_bus_master { + __u32 op; +#define VFIO_DEVICE_FEATURE_CLEAR_MASTER 0 /* Clear Bus Master */ +#define VFIO_DEVICE_FEATURE_SET_MASTER 1 /* Set Bus Master */ +}; +#endif + #else /* not VFIO_PRESENT */ /* we don't need an actual definition, only pointer is used */