From patchwork Mon Aug 14 16:10:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Shukla X-Patchwork-Id: 27585 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 260269121; Mon, 14 Aug 2017 18:12:42 +0200 (CEST) Received: from NAM03-DM3-obe.outbound.protection.outlook.com (mail-dm3nam03on0089.outbound.protection.outlook.com [104.47.41.89]) by dpdk.org (Postfix) with ESMTP id 3C8059111 for ; Mon, 14 Aug 2017 18:12:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=jmlnlUh48PLLT4XZ40rvNGobpyqEkXG+5f1RoGRmNjw=; b=SG/Kn/r271pIyQb+v6XIF3o3mbw2vo2FFETcSJXd2dKV966e/0/6EhnOITslxsl+KTAN9pfJ6EZT2OxdcX88oqvq3BbCnz/SOCEd8MaTc2aZxPtNi5310GchPU+tHYT3h9YV/di+7arPZ8iTJ45dRXTGoXcRbJ/TIYWjj5re3ME= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; Received: from localhost.localdomain (14.140.2.178) by CY4PR07MB3093.namprd07.prod.outlook.com (10.172.115.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1341.21; Mon, 14 Aug 2017 16:12:33 +0000 From: Santosh Shukla To: dev@dpdk.org Cc: olivier.matz@6wind.com, thomas@monjalon.net, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, maxime.coquelin@redhat.com, sergio.gonzalez.monroy@intel.com, bruce.richardson@intel.com, shreyansh.jain@nxp.com, gaetan.rivet@6wind.com, anatoly.burakov@intel.com, stephen@networkplumber.org, Santosh Shukla Date: Mon, 14 Aug 2017 21:40:52 +0530 Message-Id: <20170814161059.6684-6-santosh.shukla@caviumnetworks.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170814161059.6684-1-santosh.shukla@caviumnetworks.com> References: <20170724084004.25542-1-santosh.shukla@caviumnetworks.com> <20170814161059.6684-1-santosh.shukla@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: PN1PR01CA0087.INDPRD01.PROD.OUTLOOK.COM (10.174.144.155) To CY4PR07MB3093.namprd07.prod.outlook.com (10.172.115.7) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aa9ef86a-0aa3-4bc5-41bd-08d4e32f48af X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:CY4PR07MB3093; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 3:tqlkZbSKsdWsFPUFpxDfhm76EkhRTdy9z24M8OCvT9g6PkVyDBRQF90c6f0qo3b75prU7sapMTWkXOSI/Halh43dnXFZThPFnBwekxPT/WzK918Wlf4BKs3K1li/u2nAjXi6Y01yg7Y389ie5XOPcIUCqxSuRsKEjC3wD2nG5tqtY0J0xXCiVwKT7s1INiPRv81AOX1O3whYmCeO8FNSNOy2nceGyPC6HcZ0bQsJJ6pxbSMpbSupHXfTxG4/+gWX; 25:cLs24/zyEIdXIoOsjmAAe4fY3YTAl9JkaICrBG7Ug00eAjWraOgkFw6uS0kS2bROntxBIDAUZ/aU7DY9GcGwALkpswFpy5cJUjvGcmohTB1eETOECpGMutujbKcCit29uF86OsLt7XaDO2ujYAljoBPd+YA8Iwb5k2meGtbPniXmBvhomQf4di72iNUlLBVrmichnT+7xh+JuQprdNuGOAh1UuGOutQSYtOOKM2gEWlMk+5Gd90daZyR1+YC9n65pTsOqO/hfIZsiDEdODpy630/S1j0RWPabG5CCeEaQKFzBRBX7Qvmwbw7VEGsRZHQwlYFs4/jkqE4Xa3cNsHowA==; 31:5xX+kofutXzrB0atcWH5MO/e0UOeyS+8wvCCJCTG46y7+V/3UEzQrwGJoPSXO+e2wUFma97/kyqxKyT9vH3do3bo6wT1rCx6isquBFkvOc0lpoPteu+hI4W9FEzjXiSCAav1PZtbAPmrG0wnsUmfOSAFwustISykOK2Nwd+4dQlWp7hRDPNLeT9V+Vc361g7n6hDFeYEId1PHN17eSsAZ+DSyG2U4spDFmyQmK6No9I= X-MS-TrafficTypeDiagnostic: CY4PR07MB3093: X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 20:XbkTDc8ebwtG/GVEaVAZAYPvXUeP08GjiFTYCRO0NH/GP0gVbaHrha7YXFrhRik1x0JKPg/sgRIFefZZoYiERFfZ/r33LxMhWVJjtuvemPWR9hHWehuPNCnwOOSTg+gisl/4+YNlO5OeqKOvPfyN+YzvGeaCJhfMun/7iJdwl7oKHLBKfuvl6mRv4iXzWPEXSNOH2iY3nIjtOYReKmO7+OrfkbarM5adQ450fCfAWY3HNrerilbr6b1LgSIdsYiAg52SQ5jvuXHVb65Flr99E5SyP4kRursAhoIEZHDViLw6q+3zhRkD3+1zp0WO/aIkfwmdj+OB2yyNO3z7TM9xfH6+LzbmnRZBskRm4uRXEN++K15gZS3WTPkU6+FNktnz9H9cYinNTLp7KRwKOm3y4UJbFfweWUISAvL5EL3e95/oeH4XtBrTS92nexfTUMRUDSiJb1Wx7C1fH71BC04gow3pE6Vk79ojSitEYAVCqndlGr+jxLueNsDsnEN5pZ91+azceaL7HM3h6VJrTicpB3jyi56AwVYh/YPG/FPCwN6s9GJs/oNVAQbe7DLqw7tXPR7qg/21Etl4MU9wrugGyLwkIMjcU9Z2nax5k30zr0U=; 4:0JdxpKXSg1KdqUEXUuEMWPSoBVXnYz1TqmtbZxdnqH7uAQjppwRPDQu/oVy9oY9YaLAClx39Ew9lFq90RN7MrTLpXiwbi6LBhDHy5iA5IU3Yrk3I9wzmePDO9sOV3EbqMTk06K89hoGOtTxEhJFJrXb40n07x0O5bg2BmyRRsM9/qmQLGjZpdv2WXHuwkbyUier1LiX3pInJkHwfzIbWlEo3fSZe+xePp2479uN0ap+q7wZEvHd2dwZkLpPqhCyMh0pVt6mMORmGBu0ktFDo5ll6KNKcWzguxeVtNVcq4sU= X-Exchange-Antispam-Report-Test: UriScan:(185117386973197); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(100000703101)(100105400095)(3002001)(10201501046)(6041248)(20161123562025)(20161123560025)(20161123558100)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY4PR07MB3093; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY4PR07MB3093; X-Forefront-PRVS: 039975700A X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(7370300001)(4630300001)(6069001)(6009001)(189002)(199003)(76176999)(101416001)(1076002)(50986999)(8656003)(6116002)(66066001)(72206003)(7350300001)(5009440100003)(3846002)(107886003)(47776003)(110136004)(2361001)(8676002)(42186005)(6486002)(50466002)(48376002)(105586002)(106356001)(189998001)(4326008)(6506006)(5660300001)(2351001)(25786009)(6512007)(81156014)(478600001)(36756003)(33646002)(7416002)(5003940100001)(7736002)(305945005)(68736007)(97736004)(5890100001)(6666003)(50226002)(6916009)(42882006)(2906002)(2950100002)(81166006)(53936002)(110426004); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3093; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3093; 23:fLcmpqebsAF5PF83GwwnbJeG9oYhOeLkCZcvK4Tst?= y4sa+g+Id5FO0BUoN2pGiLNmTRNOAR0U+DhMsebQBmWryh+R0D+dvVTH8TbnRZUZkaDmNObDIAwmL89XbSh7eBynzvMUHNXavpR+LodMciy8WuYZsH7qaW3/1m5h6+Hoddb15JNleBcj85ta98sLRKUwtDqeYPpeqVXtK+8KCOGcrfEmeC4hk8c7uRDVTdkzy+8lHA0QcwP0cCPOxn0OLd3k8fP1BVBb+i7Gg9sHVdJ6sipYkmrWVqsd1h5uLUNsH81ca8E/5ghPEZYgLcj9ajlE+bBBNy5s2gJy6xPlD4hNu2zUfQMu4MGxMFPRSC6l8S8wc6G3xNmMdqsucqPoImsAfaG6DpKQY7enj86jOjd1pgVAtSFI5caTWX+TqLR7GDDqrsJkHjvjeanHhr9CpKWSbA/P0xlRCxxf/7jdcYihZNfWjcwRThCg6Jks+pyp6lfFJ8YQuq9n8iKTE9bqo5D2QbgQYOoMqOf++Ro5ifslkt1caLZ5ADdYrZykGIV1WetPb8r3KGENubbphG5199wOGZzQcfuy4p0JeP6sEVTpKGW7z0AjiBhoNfoweYr1vMVldS8nifXZaFwcWYSHHlzzVHGpp7L2P6wDIL3nj8s+Rf9/A/gaekiPjN15Rsp2uCxfntAPnnr853AiQ2+kOuRd85ML/rPOs1a0MPRjGN+g/OD+1PyDrfEHrWzv3E8k1lrWHnORQtD2vCoTIYyYSQ1HkwdNrM2Sq0IocbsUQA8bAbc/dGv5tPK8Dngc9P1sChpKnd6O3bXhDdjTLok5kzNoGc6veMPZOeVeN6CDmUxzn08ycqWkg32Js6byQQQGKnhkHWJ2NEHNq/bg5y9x/jBAmB1ZID6SsPVSt+cOONdPlSa0e69tQj+EGRzB36EXq9OB8LJ1LT+m+F2EJkXNVbidZWVfEa4H/M0NacuGEzQj0BLwwMyMn+FioKMdFS5lshHXSbM8WG3wjoz01tzp1NOx3exiqntfCfdvp2zCp2A9v/X+/R/jQ0uhgziTIRt5K6Z72MAkKswlfClE7QdJ1hybFxnLEQBANaAnFmFszRUyfBzHNnXtAsVBlD1QDxi8sDkiKxiDD1gX02/mrPnAA8fS+51NH97ziCFLQHEDkbqAi05ROlq/P53XEjluO0bNcnUeYejuarwKz+zax3cOemJEBG2gX+JE7cM5T03IjcTrTvZJeBfOAsX7dEb2FpKSYpobW4RwuM6Z0IKC5E4FRZbXw1Glg9+Z5AArSIPBF+KeYdFfVZ8gUdmApQ8Dvfon9I= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3093; 6:o93Dg85bPTZfwVuC4IwJwKdWnDvmBh2iDlynOLwhCERTCU3DlrCSGeKdHs+0oMtROfrAJ+sSI/Ni9l8Ia/R2/LeABMRwM4Zccj6SYdqggErR89SDUb8DAVGZ/9a8aJkqxX0K23lgY0aYsLlqrERowAcvwOCtecVN+CTsuc6Q3ztOAmVWsgG24Mclgu0+P9wvRW94jY637onw5S88MPZb/yyrMxneAwucJvNfq0SzFhoA8NWEQq82voisRCjHtCltwI4ctaHDynrkO/p+tMxjhReG+rE4HJBO5AAz0W5NSV2ZFV7CHTE0rcOk+oL1iIDK0gj+nMhBQaWqWeITz1Eshg==; 5:QZuzsJAiE9ndWiDbXASP/JNzBlcNMSmsKlpu5wSBhpbPoFer2265bXnRtM4tCU8cey7mCDEQ++tZzkE38GVGgmb7jS6J6B5bDsM5uVesP/nf4VSbCI+R6re1DSlwQ6O56ByWa9PvOvRA92TkPyYfXA==; 24:fk97V9F1xRIQrQe0l20879x2h7sp0y46Uo3ppHFyAF4tddzeLOkD54PLRGuFrH2OUqYJutiMkwqE+j6BOlcmKcJyuxn+eDkMacDzsuKeSOY=; 7:G71YmdLjiULhFVxNzHyMKTXgbcBxQK6n4qU4sP6pp3hH9TrXANc6A6RVf2GeyJN0/1Dm5wVi/zEZkQNDK/qOUWpvciH5BDH1Ap6AdnImLFGS1ReALEtwtDtTH37Kd04UeyxL2k6UqvxnMUzOpCw6EMjSP95FTxMNDFyUYw4DFcMIsSsEgeaR7kp5xX9yKmqZJQUfQ9AN2iLnwTgc3h66neAFB5ykuG7ntwuNR03/bbY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2017 16:12:33.8182 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3093 Subject: [dpdk-dev] [PATCH v6 05/12] linuxapp/eal_pci: get iommu class X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Get iommu class of PCI device on the bus and returns preferred iova mapping mode for that bus. Algorithm for iova scheme selection for PCI bus: 0. If no device bound then return with RTE_IOVA_DC mapping mode, else goto 1). 1. Look for device attached to vfio kdrv and has .drv_flag set to RTE_PCI_DRV_IOVA_AS_VA. 2. Look for any device attached to UIO class of driver. 3. Check for vfio-noiommu mode enabled. If 2) & 3) is false and 1) is true then select mapping scheme as RTE_IOVA_VA. Otherwise use default mapping scheme (RTE_IOVA_PA). Signed-off-by: Santosh Shukla Signed-off-by: Jerin Jacob Reviewed-by: Maxime Coquelin Acked-by: Hemant Agrawal --- lib/librte_eal/linuxapp/eal/eal_pci.c | 95 +++++++++++++++++++++++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 19 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.h | 4 ++ lib/librte_eal/linuxapp/eal/rte_eal_version.map | 1 + 4 files changed, 119 insertions(+) diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index 8951ce742..9725fd493 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -45,6 +45,7 @@ #include "eal_filesystem.h" #include "eal_private.h" #include "eal_pci_init.h" +#include "eal_vfio.h" /** * @file @@ -487,6 +488,100 @@ rte_pci_scan(void) return -1; } +/* + * Is pci device bound to any kdrv + */ +static inline int +pci_device_is_bound(void) +{ + struct rte_pci_device *dev = NULL; + int ret = 0; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_UNKNOWN || + dev->kdrv == RTE_KDRV_NONE) { + continue; + } else { + ret = 1; + break; + } + } + return ret; +} + +/* + * Any one of the device bound to uio + */ +static inline int +pci_device_bound_uio(void) +{ + struct rte_pci_device *dev = NULL; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_IGB_UIO || + dev->kdrv == RTE_KDRV_UIO_GENERIC) { + return 1; + } + } + return 0; +} + +/* + * Any one of the device has iova as va + */ +static inline int +pci_device_has_iova_va(void) +{ + struct rte_pci_device *dev = NULL; + struct rte_pci_driver *drv = NULL; + + FOREACH_DRIVER_ON_PCIBUS(drv) { + if (drv && drv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) { + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_VFIO && + rte_pci_match(drv, dev)) + return 1; + } + } + } + return 0; +} + +/* + * Get iommu class of PCI devices on the bus. + */ +enum rte_iova_mode +rte_pci_get_iommu_class(void) +{ + bool is_bound; + bool is_vfio_noiommu_enabled = true; + bool has_iova_va; + bool is_bound_uio; + + is_bound = pci_device_is_bound(); + if (!is_bound) + return RTE_IOVA_DC; + + has_iova_va = pci_device_has_iova_va(); + is_bound_uio = pci_device_bound_uio(); +#ifdef VFIO_PRESENT + is_vfio_noiommu_enabled = vfio_noiommu_is_enabled() == 1 ? 1 : 0; +#endif + + if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled) + return RTE_IOVA_VA; + + if (has_iova_va) { + RTE_LOG(WARNING, EAL, "Some devices want iova as va but pa will be used because.. "); + if (is_vfio_noiommu_enabled) + RTE_LOG(WARNING, EAL, "vfio-noiommu mode configured\n"); + if (is_bound_uio) + RTE_LOG(WARNING, EAL, "few device bound to UIO\n"); + } + + return RTE_IOVA_PA; +} + /* Read PCI config space. */ int rte_pci_read_config(const struct rte_pci_device *device, void *buf, size_t len, off_t offset) diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c index 946df7e31..c8a97b7e7 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c @@ -816,4 +816,23 @@ vfio_noiommu_dma_map(int __rte_unused vfio_container_fd) return 0; } +int +vfio_noiommu_is_enabled(void) +{ + int fd, ret, cnt __rte_unused; + char c; + + ret = -1; + fd = open(VFIO_NOIOMMU_MODE, O_RDONLY); + if (fd < 0) + return -1; + + cnt = read(fd, &c, 1); + if (c == 'Y') + ret = 1; + + close(fd); + return ret; +} + #endif diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h index 5ff63e5d7..26ea8e119 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.h +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h @@ -150,6 +150,8 @@ struct vfio_config { #define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" #define VFIO_GET_REGION_ADDR(x) ((uint64_t) x << 40ULL) #define VFIO_GET_REGION_IDX(x) (x >> 40) +#define VFIO_NOIOMMU_MODE \ + "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode" /* DMA mapping function prototype. * Takes VFIO container fd as a parameter. @@ -210,6 +212,8 @@ int pci_vfio_is_enabled(void); int vfio_mp_sync_setup(void); +int vfio_noiommu_is_enabled(void); + #define SOCKET_REQ_CONTAINER 0x100 #define SOCKET_REQ_GROUP 0x200 #define SOCKET_CLR_GROUP 0x300 diff --git a/lib/librte_eal/linuxapp/eal/rte_eal_version.map b/lib/librte_eal/linuxapp/eal/rte_eal_version.map index a15b382ff..40420ded3 100644 --- a/lib/librte_eal/linuxapp/eal/rte_eal_version.map +++ b/lib/librte_eal/linuxapp/eal/rte_eal_version.map @@ -247,5 +247,6 @@ DPDK_17.11 { global: rte_pci_match; + rte_pci_get_iommu_class; } DPDK_17.08;