From patchwork Thu Aug 31 03:26:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Shukla X-Patchwork-Id: 28103 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 018A4377E; Thu, 31 Aug 2017 05:27:16 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on0082.outbound.protection.outlook.com [104.47.36.82]) by dpdk.org (Postfix) with ESMTP id D66B3326B for ; Thu, 31 Aug 2017 05:27:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=KeVUfuJJg9Bt+i3tNI4iqJn9eKLu90TR4NU3ijucHP8=; b=XyhuRXPMBkk2F7/pF+y1hTYoP5gpH6mW09BZSWkokY0qMAlbcBVRPhAEZX/IFJJS2m05M1bjuqEzcKUJzs66JEsou+w88t2tjXYQfhiIYbhwa8yu1J3nHiZhegD9K+GHrzsbOlffuqHGrh750i8z0vLLmoK1JiA8nahMvPVe4Pk= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; Received: from localhost.localdomain (111.93.218.67) by BN6PR07MB3089.namprd07.prod.outlook.com (10.172.105.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1385.9; Thu, 31 Aug 2017 03:27:06 +0000 From: Santosh Shukla To: dev@dpdk.org Cc: thomas@monjalon.net, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, olivier.matz@6wind.com, maxime.coquelin@redhat.com, sergio.gonzalez.monroy@intel.com, bruce.richardson@intel.com, shreyansh.jain@nxp.com, gaetan.rivet@6wind.com, anatoly.burakov@intel.com, stephen@networkplumber.org, aconole@redhat.com, Santosh Shukla Date: Thu, 31 Aug 2017 03:26:12 +0000 Message-Id: <20170831032618.7120-4-santosh.shukla@caviumnetworks.com> X-Mailer: git-send-email 2.13.0 In-Reply-To: <20170831032618.7120-1-santosh.shukla@caviumnetworks.com> References: <20170814161059.6684-1-santosh.shukla@caviumnetworks.com> <20170831032618.7120-1-santosh.shukla@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: PN1PR01CA0118.INDPRD01.PROD.OUTLOOK.COM (10.174.144.34) To BN6PR07MB3089.namprd07.prod.outlook.com (10.172.105.7) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 48267e18-15fd-4bdf-04b6-08d4f0202a35 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:BN6PR07MB3089; X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB3089; 3:CGFX9uz5vHJKT7kx/A0TctWhM9amifEN+nquRqtN+tznAtc4GXplp16n0ZpAW9YmaYpCqjiL5SF1W70M9ZtVQNSJikfWmlEK83qLN3yXOOrdmp4nmOKvpgyNpySn5K9hEVhX5YMThkn4zAU9q9HCcPnX4NyChFHL/7kxK8F77Cz2lNe0CvD6q5xdNh+xefTUbs82INQVenyekeEYCGx3AKtBe6h3FNIjkvuNU2E6u1ZGtoap28X5hndJ5mYy8ZD7; 25:zWPpYkAVD4zUEcPwwKPrB8VhrQZaV7PvxxFqEs4Hnd1KduFPWV3xzuGv48lrUJBZzurMuWs2wtp2HkvDvaNL2cO2zahrLqCaQTYniC+l/lgfsZoIGrcWzS9gWxBJTc/On7BEXaI+OjLC4oIlNig62P0Su4AAAgnGBPhKQfGkn2IU7r3SdMQgrAc3s2M7ylT/64WRxhnDcfdnvdIIo0QxEjSepuIhSk8juWRCv+dpebUySaTAwZeCF7PcFPAeZQjzudnQPEeVifJ1eJlr1w05d9jm6PxTw7B8QaGpatN9grILoLeFzjf1+N97/DkTLNpC5GBrkhmhV0wJunZHtX2cxA==; 31:qHmj+oGyL7LhiLC/p1aq86jUKCiN3fUgcHLvYSQfCN4p4HmKEq3Cj39raZ4/3xr93JiSByotIrLVMCrsBsMrD75QwtSYkxA5DuOzXBDMz5zx0PDP1ZvOkBSOrj3G/Tw5SVJyVs8tpZr/WE9+sooSi/4wrnwjSRateM4VAHVMmC2GphixG8qFuWlmQFMkHLAa4SCns1O29+P0zcLIm1PJDT7T/2ncBXRu327bFvFyqVQ= X-MS-TrafficTypeDiagnostic: BN6PR07MB3089: X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB3089; 20:z0U2ADtolEAcnFmB8lY9+ADmL3ZBMfBlw+WAQYF56H3bu5o3FxysMQS1eijn9Or//gMe0oL8yk4TcHTDmyvRnTeT547wgeqKWIHIEhr457n2Tx/h+mJIzIVvQen2ibY0taxANkHP/Uuj5pn68/d/ihEQ2BWjFgZbXoiZpe9nso8mnZMICoaL69MZ/IakSZZs8oXy5nFzT05mzyLYcaI3K8BtwWOf4RpMLjpp17+jYHU+k/WRqB6zMFX2FG5kvDTrlsaIsdQzI7PGxPp4LOhAR9lt9rnU96Ef/0V0XEJyoqQAJgVDZpdwszNKaWpz2b0Hoi4aFDVFgS0lMU/g1Xkoev6qsfZNWkIEmqTLoz8gzQy+o7BJKJoOI2Dqo08qlmuI85/4dcqck9ba+wG8Pf1HEvTVMliejphFL46LhlVWVgFnXqRgrbAqCkyGn8Eeq2zN2qrTSHBw/WDfrzgG1YEYrFjPeUYgG1GPIEkMRIbnZlrNPdOLC4/1y8XUsHccRZCO7frDTtw4UMmadqyHlAmLt3pPKOCI54eMKbljjTZdCI0YHpWm6+MJI++WvLNsdLp8pkpNGMe4DYTwMfPd0oGFt8ZvxP9eY082Z6kT+Wbxbgg=; 4:6kp1z7kNvOTANAr5FatrX9KVP5v2oJ0MHtPyKPlCfSzePJOZ4XJ0SYuWlkL/HLvCtEu2xJRI2ofsZwwbvgXbRKZ0c5JuWZvBexZERuO/ZUEDtKGbxfSWFn1KxKdgbX40BkK4DHRmrrqKesq3ImGJwkdfVA9wO5vksYAjyoBAhAiyMDN/gAhYHjis4fiT2luIEo0yw/RQZjOM8OQJcnPrTP7FPF+ru2OC8R5l5I00E/Kk2fscsF8mUM5z4jAQHwngtKTPIE+NW6+Awl3P0M6CIhy/jPm/pH4L4yy4JrKg+TU= X-Exchange-Antispam-Report-Test: UriScan:(185117386973197); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(100000703101)(100105400095)(93006095)(6041248)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN6PR07MB3089; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN6PR07MB3089; X-Forefront-PRVS: 04163EF38A X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(6009001)(199003)(189002)(81156014)(8936002)(81166006)(8676002)(7736002)(6666003)(5003940100001)(305945005)(2950100002)(7416002)(2906002)(6916009)(42882006)(76176999)(50986999)(101416001)(50226002)(5660300001)(6486002)(6506006)(48376002)(50466002)(72206003)(5890100001)(36756003)(97736004)(478600001)(189998001)(42186005)(2361001)(2351001)(33646002)(106356001)(105586002)(110136004)(107886003)(5009440100003)(8656003)(6512007)(53936002)(4326008)(68736007)(1076002)(6116002)(3846002)(66066001)(47776003)(25786009); DIR:OUT; SFP:1101; SCL:1; SRVR:BN6PR07MB3089; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR07MB3089; 23:BXE32lo6qiu34CNdaVghlrcE0ns8ThOaWGWmHnm+j?= sGQhyzDve7LOeWMCv6epq2aOa7mlS1ExemX3R2RJ61arEZiiINSxi2+T6q+iPIZnLIWer2uk5uYVDqI9W6dPn/1st+U7p3xHb4jIau8buw5SW1gGFhDYjqfGDAd8/nGSF5NbhEXZd+qlM9O1RmXEgO6kCVDycsJQ1UYlBxym2wT3+oRbaBTh2Iay7NFOt/NIYsyxrS6lt7SauCa2wGrcigsD3YqScr0qvS2qymsEksJ6yqWeBmLCgPmdVmiTOUOFHMKu8J7jWxy2Dw5LV/peJCPQ6wT6dvXo5JD0tvGD6ojYJVWnpD/WHXaPBwUOvwLySRxKthjE34eR9ORtqZrkIHqpqlMXYDdHVxaNlVxZx+seBMkPnUSLRG8nSIMfeoalKl2vWLSXFqm/mnFjtneozbu3GIJiBqae7xzHOW+H1nLFVDd2OtcbAmloRbMXCul6cKty2lTrR/i94aANK5WXaSHCa1Vto233aHsATkgsgaU7YdSc9j6AHiYoB3yGWQFngONUW1K5521q6yCc/20o90OtkHYO/k42zFzxNpJJhdz86fjv6F/8+QNdWUvHPul6v3iPbviWd2C/YJkv49B69Qstxc/HdQc9VMIgm6t2tplU3k1CPPAO/U7i6V7biKfUyytw+ZwO35u9mQxO0SmH6FhJRYz/FnhbP4oxbkxsiJ/xlwZq/t6VELayQjWZ3rfCo//guywtOt7c0EkQK212jYgRQnb3CtxspZSSv3rBP4z7T2/+rdqW6/jZMrvDcyLFOOyIfSb08PdrKZpymL1DTtHUz31KV1CfyzsgKluBaJb03EeU6Pa1CI+Gi6OPhagI61+n1HEPqbj/GpAgL/VP4qb3s02A5OItehXyCSrewaHhhoo8d8OSYAU1cBhUDybke33bubn9H2BGBvnMDox2/z0jEgiPtYKNVz3Vm7IcnyEa30GNrvbnBfEUxfPzdVNcwmdE5pL1vjO8OM6hIjV2shzHlrhkpfXrvqCcY1/tjB6/4GDixa9psC7Lh7dx3jsML6ZMS+N18HRhoPZFQ2t790rxx8gLSdKDQcU6or10TsaBdV09fyfa1U8el2DDBggfKcCzPPejJao82VeQiicvwBWvudrOp145bfb4I9EP9JCtn5mwwUm3jXMiQ0YaBcRnMiYXYMdKL+nWT0t3RlSnT5GvcIMsUneSirEQC1Cj4iWOA== X-Microsoft-Exchange-Diagnostics: 1; BN6PR07MB3089; 6:8mUuyujik9nDyNDUZ/OztydJC/PZI3LGEELuypvH4PjEQBkO7Jt19D/NF0aUu0hdExFMb2rPplOMpvQvIgafKrg98u1akhBcyKnIOZl7yjuJ9m8dKEbxL0ZAQ6wmIxuqilpH809vb4Eb083bAFNxOhN3Ac9Zrc8Dj95eXK9mwHbyAs49165zMIHHuW9U+69/mxMlaMHRbagn/O3HZYqvK2/DNEwTdpA0oYwAhxmnyuUEoksZzS2e+tB604Il+EPboUMgP4FLxWxUMIYZHSQvv4QPCGokxHPZ3W+i/mrrIuFnZDwqL3969aatE2py22CJxIX8WArgmdfn7YFAPbYUEw==; 5:h+l9amefeRywFytMjxXzCHfc46Pv6jJSihaGIguMBrhdJXuvHXG/Hdp/yxuoz28cGyk5dz7A6dBDMFa9W+wshjQf2WGTuaRQGyQ5synf9ywGZCRy6bxPxciMpFfvj9GgZMh0PIvO4qBqoYO06Y5JtQ==; 24:I9UXH4un/176RfId2H3brvtAkIpD2Fhdy9rk9x1RfXGYznP6Lp2zr2gzpFbZtaWSE+vCxCNu6kKZmwmDlDWO5Uz0ZMg3dEVJtxWqaOBBl5U=; 7:Lys1z/xorX+XtPQPglUCCUSHeniQDjhqXPfsCR3ibBsFU4KqshS8vqAepLFmola09E4+cxgxoSVixhhOIqvshKkipmKNMShQlFUL73jdNDej6ctgyn5XTm6+C8YYJbS60SozDhlMyT4d1OVZsxKyZ04pCKnBRfwgufco1RJj7zuH1cSsU2J9DWaC4VUbvrOL7rx9855zXuHTgwxgYQ6BnYCA0l4pDX9y2AyMSTI0yGY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2017 03:27:06.7129 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR07MB3089 Subject: [dpdk-dev] [PATCH v7 3/9] linuxapp/eal_pci: get iommu class X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Get iommu class of PCI device on the bus and returns preferred iova mapping mode for that bus. Patch also introduces RTE_PCI_DRV_IOVA_AS_VA drv flag. Flag used when driver needs to operate in iova=va mode. Algorithm for iova scheme selection for PCI bus: 0. If no device bound then return with RTE_IOVA_DC mapping mode, else goto 1). 1. Look for device attached to vfio kdrv and has .drv_flag set to RTE_PCI_DRV_IOVA_AS_VA. 2. Look for any device attached to UIO class of driver. 3. Check for vfio-noiommu mode enabled. If 2) & 3) is false and 1) is true then select mapping scheme as RTE_IOVA_VA. Otherwise use default mapping scheme (RTE_IOVA_PA). Signed-off-by: Santosh Shukla Signed-off-by: Jerin Jacob Reviewed-by: Maxime Coquelin Acked-by: Hemant Agrawal Reviewed-by: Anatoly Burakov --- v6 --> v7: - squashed v6 series patch no [01/12] & [05/12].. i.e.. moved RTE_PCI_DRV_IOVA_AS_VA flag into this patch. (Aaron comment). lib/librte_eal/common/include/rte_pci.h | 2 + lib/librte_eal/linuxapp/eal/eal_pci.c | 95 +++++++++++++++++++++++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 19 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.h | 4 ++ lib/librte_eal/linuxapp/eal/rte_eal_version.map | 1 + 5 files changed, 121 insertions(+) diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h index 0e36de093..a67d77f22 100644 --- a/lib/librte_eal/common/include/rte_pci.h +++ b/lib/librte_eal/common/include/rte_pci.h @@ -202,6 +202,8 @@ struct rte_pci_bus { #define RTE_PCI_DRV_INTR_RMV 0x0010 /** Device driver needs to keep mapped resources if unsupported dev detected */ #define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020 +/** Device driver supports iova as va */ +#define RTE_PCI_DRV_IOVA_AS_VA 0X0040 /** * A structure describing a PCI mapping. diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index 8951ce742..9725fd493 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -45,6 +45,7 @@ #include "eal_filesystem.h" #include "eal_private.h" #include "eal_pci_init.h" +#include "eal_vfio.h" /** * @file @@ -487,6 +488,100 @@ rte_pci_scan(void) return -1; } +/* + * Is pci device bound to any kdrv + */ +static inline int +pci_device_is_bound(void) +{ + struct rte_pci_device *dev = NULL; + int ret = 0; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_UNKNOWN || + dev->kdrv == RTE_KDRV_NONE) { + continue; + } else { + ret = 1; + break; + } + } + return ret; +} + +/* + * Any one of the device bound to uio + */ +static inline int +pci_device_bound_uio(void) +{ + struct rte_pci_device *dev = NULL; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_IGB_UIO || + dev->kdrv == RTE_KDRV_UIO_GENERIC) { + return 1; + } + } + return 0; +} + +/* + * Any one of the device has iova as va + */ +static inline int +pci_device_has_iova_va(void) +{ + struct rte_pci_device *dev = NULL; + struct rte_pci_driver *drv = NULL; + + FOREACH_DRIVER_ON_PCIBUS(drv) { + if (drv && drv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) { + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_VFIO && + rte_pci_match(drv, dev)) + return 1; + } + } + } + return 0; +} + +/* + * Get iommu class of PCI devices on the bus. + */ +enum rte_iova_mode +rte_pci_get_iommu_class(void) +{ + bool is_bound; + bool is_vfio_noiommu_enabled = true; + bool has_iova_va; + bool is_bound_uio; + + is_bound = pci_device_is_bound(); + if (!is_bound) + return RTE_IOVA_DC; + + has_iova_va = pci_device_has_iova_va(); + is_bound_uio = pci_device_bound_uio(); +#ifdef VFIO_PRESENT + is_vfio_noiommu_enabled = vfio_noiommu_is_enabled() == 1 ? 1 : 0; +#endif + + if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled) + return RTE_IOVA_VA; + + if (has_iova_va) { + RTE_LOG(WARNING, EAL, "Some devices want iova as va but pa will be used because.. "); + if (is_vfio_noiommu_enabled) + RTE_LOG(WARNING, EAL, "vfio-noiommu mode configured\n"); + if (is_bound_uio) + RTE_LOG(WARNING, EAL, "few device bound to UIO\n"); + } + + return RTE_IOVA_PA; +} + /* Read PCI config space. */ int rte_pci_read_config(const struct rte_pci_device *device, void *buf, size_t len, off_t offset) diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c index 946df7e31..c8a97b7e7 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c @@ -816,4 +816,23 @@ vfio_noiommu_dma_map(int __rte_unused vfio_container_fd) return 0; } +int +vfio_noiommu_is_enabled(void) +{ + int fd, ret, cnt __rte_unused; + char c; + + ret = -1; + fd = open(VFIO_NOIOMMU_MODE, O_RDONLY); + if (fd < 0) + return -1; + + cnt = read(fd, &c, 1); + if (c == 'Y') + ret = 1; + + close(fd); + return ret; +} + #endif diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h index 5ff63e5d7..26ea8e119 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.h +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h @@ -150,6 +150,8 @@ struct vfio_config { #define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" #define VFIO_GET_REGION_ADDR(x) ((uint64_t) x << 40ULL) #define VFIO_GET_REGION_IDX(x) (x >> 40) +#define VFIO_NOIOMMU_MODE \ + "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode" /* DMA mapping function prototype. * Takes VFIO container fd as a parameter. @@ -210,6 +212,8 @@ int pci_vfio_is_enabled(void); int vfio_mp_sync_setup(void); +int vfio_noiommu_is_enabled(void); + #define SOCKET_REQ_CONTAINER 0x100 #define SOCKET_REQ_GROUP 0x200 #define SOCKET_CLR_GROUP 0x300 diff --git a/lib/librte_eal/linuxapp/eal/rte_eal_version.map b/lib/librte_eal/linuxapp/eal/rte_eal_version.map index a15b382ff..40420ded3 100644 --- a/lib/librte_eal/linuxapp/eal/rte_eal_version.map +++ b/lib/librte_eal/linuxapp/eal/rte_eal_version.map @@ -247,5 +247,6 @@ DPDK_17.11 { global: rte_pci_match; + rte_pci_get_iommu_class; } DPDK_17.08;