From patchwork Mon Sep 18 10:42:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Santosh Shukla X-Patchwork-Id: 28836 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 574711B19C; Mon, 18 Sep 2017 12:43:39 +0200 (CEST) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0081.outbound.protection.outlook.com [104.47.32.81]) by dpdk.org (Postfix) with ESMTP id 5E8A5199AE for ; Mon, 18 Sep 2017 12:43:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=W2UBySud1JMnexJ/jq0lhJsh09UH32ZfJdu0hlU1gRg=; b=kBxLyGEYQBrVEFZcPWoeow0zZmHvic9aED03jkRxvy++UxmzioCiMDSFi2Eu2lxEADNdPxY2j7d5wTnq2N9dEQ2OCj6T9DdNGwSYkJh/x2/wx8JGCx6SkD+6NYxa2LWEvhnXIwWr+qtS6QOMo+zlkw4LZxlK8P0VDLBqP6hxXRs= Received: from localhost.localdomain (14.140.2.178) by DM5PR07MB3097.namprd07.prod.outlook.com (10.172.85.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.56.9; Mon, 18 Sep 2017 10:43:29 +0000 From: Santosh Shukla To: dev@dpdk.org Cc: olivier.matz@6wind.com, thomas@monjalon.net, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, aconole@redhat.com, stephen@networkplumber.org, anatoly.burakov@intel.com, gaetan.rivet@6wind.com, shreyansh.jain@nxp.com, bruce.richardson@intel.com, sergio.gonzalez.monroy@intel.com, maxime.coquelin@redhat.com, Santosh Shukla Date: Mon, 18 Sep 2017 16:12:28 +0530 Message-Id: <20170918104234.9149-4-santosh.shukla@caviumnetworks.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20170918104234.9149-1-santosh.shukla@caviumnetworks.com> References: <20170831032618.7120-1-santosh.shukla@caviumnetworks.com> <20170918104234.9149-1-santosh.shukla@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: PN1PR01CA0093.INDPRD01.PROD.OUTLOOK.COM (10.174.144.161) To DM5PR07MB3097.namprd07.prod.outlook.com (10.172.85.7) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8cecf8a7-1931-4a2d-99b6-08d4fe821d47 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:DM5PR07MB3097; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3097; 3:m1m7Qr7JQ0EHXeAEvgDhYkiY0JDQAigK18qiHf7iEhXtyPmtzXHyBtynBsP46Ca2PEgZN/WsQG4B6RJ/BDCmUOc5MrCopefCox+hfdNPc6KJ7b1uK08tTE/vpX1+L89CIDH/xPLj/CQXMs2kLxlz+7ZUg/GxAd7Y7bWGdhZBqobG9JLk/GZcjrJs/p+1qzRGRlxqAGV/ovobDf+5dhZQaPwLaqwo70/e4Hg5v5N83mNPM/micLO3pBd4tsrBRGqz; 25:yvxx91ejeY8U/y1D45qRfefWJpSp7PE3Ljxj9JyrppakdLcFd6pwNEzq/ei7gYQuH6lL5Rd6UFOXEWrBwqQBsHmbCgUsgQjY+ZEuXBY7sXaZKVt/zqR2Nnp4N8hBL59SiLzhn2Rm1wTJYXQtekVUXDVsMu8So+DAc1pBT+OUjLR41SWBCN0AlL6m2GfDfsq5kgCByc1ovxpHZfrquqAxKJj7hl6CpK9ZgOfj/zxgbRI+9/FwEsVY90J2oso3l+16lGqEoFteGWNgUOlP98QV8LhiUPzU5+CfnAqORiPuLYS9+IazJWRzgPRNmN38PD2Mk7+w3Ek6qPXxZrXcENHACA==; 31:fq2iMPj56cGTSxY1l1s5+/cYlc3xOldxlM6dKYausYQseoZGFGzw4kD/1EDN2PPlRlWMV3oPHlMeDfSqnEgaY/XHyPJZz5RjYtjTqNWKwrGn4vWDvhS3GSxJ5x5Qv8OWRC36U2jd8xN7/kbeDmmmNhRkjN+cDfaL1rl72aeKpXQ65SaKP908RwdwN0c9sU1kuTQfClUT3uEttUTHcpE4zR7ocBrUz+nC2PZkXbfKOw8= X-MS-TrafficTypeDiagnostic: DM5PR07MB3097: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3097; 20:zYHIvKSg6f7152rF4EMPWzrQ9t/YgxexJEW1KWsi9TOAhzpWh9gDcX3BNIWRsDLh86kK/50whYIwS/T2jSrkkQdRlKcZ0BM4YO1X9oxpMeL41LWBZUpbIjErNtLtImqHfLcplQWb/MpLZYJ2MYdQgipQEexMu4ZJm6z2wpzB5Xpe6YQHzGfboYU4Ubtz4ep4Mb9YO7gk1Mgr4Z5h/7YaX4ZjmeXhPVTFhjJOUz+rmeEd404iz/CRku+Y1tHJvGEOJW9TGSTdNmqKu9HGpOoMps8w/ssUKC9hKlXEqikcJ9TrOL3UwXHSims7OHS9Db+jnWrVrc4z9UV9dqr9R0iN1bNN7X/747O6dYa0k3W4jPZwoKnfuRl2mhrToWvTOoLh2dc5dBbUQyX5LeRNms1argqvELBaaOALPgDesWf6+xsfCKLgrBFEQwFCnw8gczs2yKR3u1hYp51YnJu6FtS9z+UkqEsWiIzHGp7Ly1Q+zlmFlaNifk/HrdLOrhbLfSNCkiTlpeLlhjbkFB/XiuMYiK5ofrOeh8gw5JlLlQAvXXSmL/zFeXssjvkZ3OGYptAz/2amX/sJ4Tj10Y5/Ha8wGH/0g/t9PLNgBOjVgqgaoDU= X-Exchange-Antispam-Report-Test: UriScan:(185117386973197)(228905959029699); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3002001)(93006095)(10201501046)(100000703101)(100105400095)(6041248)(20161123560025)(20161123558100)(20161123564025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR07MB3097; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR07MB3097; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3097; 4:Nv3vyh6ZYSPCzxz8L7AjY6wHfT8ZPFb71MxpL5qmrVsY2N4/nhmmp3wWinSwPdVwFxQiuJ4IQmbb3m0VeNQUJdCFcHUXPuGIrC/LzNJIvSAfbbvJoB3siKlI94GVvUbeOOLmRPYvTZd64vKK6FGzmA7d5Ey4kyNOHdsdJo3O8dtr1PcXRXeR/9SdZzK0Xbu6iey9Wdv+s5K3Ll9qJVTpYcIVKyNYtX8i7PVRAdfhWfxqS9+AWZPpQvg38DNiojn+RnBwOFHsOWh//9unA+q1k6lDb8ZpdMjguGllq2WZGaGorkqC//XX5bPYpcOOGiiIS+fLQx9m0P0fezwHrBX0qA== X-Forefront-PRVS: 04347F8039 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(6009001)(346002)(376002)(199003)(189002)(6486002)(76176999)(6506006)(48376002)(50466002)(25786009)(5003940100001)(107886003)(110136004)(53936002)(4326008)(101416001)(97736004)(6512007)(16586007)(106356001)(105586002)(2351001)(47776003)(305945005)(189998001)(33646002)(478600001)(5660300001)(2361001)(42882006)(6916009)(2950100002)(7416002)(5890100001)(36756003)(66066001)(2906002)(7736002)(81156014)(68736007)(50986999)(8656003)(81166006)(50226002)(316002)(8676002)(3846002)(5009440100003)(8936002)(16526017)(6116002)(1076002)(72206003)(110426004); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3097; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3097; 23:2O1dHq+fLhAlfpzFNM2uFQqFmnySWMZbrogefpL7N?= RpsL3fCRaHohMf3noeI/Iaa1jcAswVgjYbuOCZokHtfUNuwG1m/y3LjT8VSdZF4mvCpUGlxzaNbunNnpWJINpj/eBjwbWuTwqxE0tXCEOzi4ee5lF5qvNqk0RKxI9iqVsmgCETtH/1W0l5Gyzb8n60RbOjJFeWBDLjb/otRJPzOnidVxhqlNqAxLrSn7JiA/rfq3vE+vMVydSY/cenNDRAukHLuscT3xjudhL8qg5cVdeDI7rtEgX4Ne2tkRpAolty4w6YoIJKDm+gkz840J61bIBvKU85cG8XC5EDFx+Y9gSbIZqF1/7RX66p/kZo3r0kkh8nCtYIxAvw5aJXzzkWSRhZEm7oEXwV6xXD9hHNXAdL9DDEKhV1rEK+jWFoN7atbY4UCBkwOk8UZSgBsvhwBgWRISQW4BoeA1rzgqH2FQUrrIxC1Uqk2+17WI7rhTq2nEjjPKXHwJl6IByj0WYAGVKi4TTGZpVhC4EVbOu54OU7VxBRD2iY2lFga1Uk9OxjIwjE7ueeJYAYGJqZcgpoB97DQ0bQMlIyQfAP3rzAw3BSah7dxKl2uE0ba0JugyHfkCYxl+PXpN83+9MOzvo90atcu+x/Mfpg0PnYhFTwpAmzCR0rLbtoWWaN/nQcvANNQLAGGzvpBa9ltm/KjwAXI6o9IET2Dq5H/s9lXu8oeWi4WtEktIp/sHVoPIr1tyoybbKP37DZDGbJIV3DSLESdFhM4Vz2RV6qTc5o/xQ6QtwEOGpNbp46479xNQNlLXlVdHDvbRH6+b5hZodtAkgx8r6vw2acgJzhGb07D0REmzTgw1U75DHg6Y7KK0z4o0JB5FkfCGD6dG5S9jL+/CWV09bRb0U5u1hDWecTD2+pFBvm3jdyadZXWM5ipmSJ2H3yR5cR/9xFyF+LrGy3WFz6KS0ocCxEV07aNJ9rK9DcPE48ArQduaU8oiof5OGsAjIdkwbplsqGQnZTb9uTOJ5KzE0+8qqh5Nh3ZOtlGakmj6nPatYe57GBkfBeq2LxZ8HB8Dj2GJP7ET/1SDN9i8QAq1JRC6jHaR9pgopZ3k11NyEzeeb78HoIg74ki8dln4mQdxFlwmEpOE+OKimN7rfZ1itz5gSpqb0XCIJ8DiOHa6dben44vJZ3gnBLl/A2WcMutqCxSdubPpqASKfS9OZjHgAitSgIW63oODjBeaxtRANAf0LyyKVkbPfK63sMIYqv9FbPuxwCWQL+RmREvvx/uq387psgYIeN+PQC63fuaEE07h4iKq5caqEzXLt9b+J8= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3097; 6:6bCqRt3/GrekFaNsA8PnAokNKDyJsMuTBR01q+m+dtbyIZicwWeWuWW/MKbrB/a5eznGMb60qdYGqmAebwfK99ff2ZoSICTjS62twTu7gc/qPx2vZ2OXH8Oh1cZCRMXxUMtqESEMGdkyaeWZh3crCNfB3/uxOIkIRB71oPs+riCkPVmC9Ssf/euSAklXjnVNg0HMjhGatNQ0G8Cnvi4zdnuDZjqrCbrFraJYJM7SYp4NrjepFCMSr2o56WWyLZxpXlwLgSZcwMZ8NxReEWJ6dh13b+SNbTarZi6ZyPtmL3o7gccZneBK19EAJXhHLGjKnS6hO2C3UldInDhHKE1WAw==; 5:/hN4HX1YIpjsVqtr7vSuyN5ibPRWigKfGna282rZPsMZavLxu9bEyrhBtI1BCNmc2XSrdGBRh1mk8/mHOBwrE42HOE0veZlWrfZDTwgxJZyAfmKlLHSzah6ig8IaAibpiLmMNu1OtX9w+AZMB1gwhw==; 24:7+HcPQ6uq7y+4eT7IHVEtFiAXOJA5y87I+SdAnrpx+nF9x2GeYGAU7ybCSz+6ELUZkPL0Glq1ZP8QoW4lx15zStXkmo96W2xib0Uyb0+qtw=; 7:TRXe55HpmOJLq31rDrMaE7bEJUzeOv0Rb2wQqjXQCw/SRTv5AVoBwkLod5PmUP88CZUaY1xM5C1xRJyjm74pECA8KUlJBuG4jWsl2afX+c+q27HlKbz0wTw22NLFf1v2zBSdtscDz6cjX+AgevHlIKrS48ARIK9B1PUW3CyXbm3R8GloYrisPZwlGfKaOm73HcniwxLLmUYnkK+HvlSM+XiN6u5gaotv5Ye7KULYytc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2017 10:43:29.2298 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3097 Subject: [dpdk-dev] [PATCH v8 3/9] linuxapp/eal_pci: get iommu class X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Get iommu class of PCI device on the bus and returns preferred iova mapping mode for that bus. Patch also introduces RTE_PCI_DRV_IOVA_AS_VA drv flag. Flag used when driver needs to operate in iova=va mode. Algorithm for iova scheme selection for PCI bus: 0. If no device bound then return with RTE_IOVA_DC mapping mode, else goto 1). 1. Look for device attached to vfio kdrv and has .drv_flag set to RTE_PCI_DRV_IOVA_AS_VA. 2. Look for any device attached to UIO class of driver. 3. Check for vfio-noiommu mode enabled. If 2) & 3) is false and 1) is true then select mapping scheme as RTE_IOVA_VA. Otherwise use default mapping scheme (RTE_IOVA_PA). Signed-off-by: Santosh Shukla Signed-off-by: Jerin Jacob Reviewed-by: Maxime Coquelin Acked-by: Hemant Agrawal Reviewed-by: Anatoly Burakov --- v7 --> v8: - Replaced 0/1 with false/true boolean value (Suggested by Anatoly) v6 --> v7: - squashed v6 series patch no [01/12] & [05/12].. i.e.. moved RTE_PCI_DRV_IOVA_AS_VA flag into this patch. (Aaron comment). lib/librte_eal/common/include/rte_pci.h | 2 + lib/librte_eal/linuxapp/eal/eal_pci.c | 96 +++++++++++++++++++++++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 19 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.h | 4 ++ lib/librte_eal/linuxapp/eal/rte_eal_version.map | 1 + 5 files changed, 122 insertions(+) diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h index 0e36de093..a67d77f22 100644 --- a/lib/librte_eal/common/include/rte_pci.h +++ b/lib/librte_eal/common/include/rte_pci.h @@ -202,6 +202,8 @@ struct rte_pci_bus { #define RTE_PCI_DRV_INTR_RMV 0x0010 /** Device driver needs to keep mapped resources if unsupported dev detected */ #define RTE_PCI_DRV_KEEP_MAPPED_RES 0x0020 +/** Device driver supports iova as va */ +#define RTE_PCI_DRV_IOVA_AS_VA 0X0040 /** * A structure describing a PCI mapping. diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c index 8951ce742..2971f1d4f 100644 --- a/lib/librte_eal/linuxapp/eal/eal_pci.c +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c @@ -45,6 +45,7 @@ #include "eal_filesystem.h" #include "eal_private.h" #include "eal_pci_init.h" +#include "eal_vfio.h" /** * @file @@ -487,6 +488,101 @@ rte_pci_scan(void) return -1; } +/* + * Is pci device bound to any kdrv + */ +static inline int +pci_device_is_bound(void) +{ + struct rte_pci_device *dev = NULL; + int ret = 0; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_UNKNOWN || + dev->kdrv == RTE_KDRV_NONE) { + continue; + } else { + ret = 1; + break; + } + } + return ret; +} + +/* + * Any one of the device bound to uio + */ +static inline int +pci_device_bound_uio(void) +{ + struct rte_pci_device *dev = NULL; + + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_IGB_UIO || + dev->kdrv == RTE_KDRV_UIO_GENERIC) { + return 1; + } + } + return 0; +} + +/* + * Any one of the device has iova as va + */ +static inline int +pci_device_has_iova_va(void) +{ + struct rte_pci_device *dev = NULL; + struct rte_pci_driver *drv = NULL; + + FOREACH_DRIVER_ON_PCIBUS(drv) { + if (drv && drv->drv_flags & RTE_PCI_DRV_IOVA_AS_VA) { + FOREACH_DEVICE_ON_PCIBUS(dev) { + if (dev->kdrv == RTE_KDRV_VFIO && + rte_pci_match(drv, dev)) + return 1; + } + } + } + return 0; +} + +/* + * Get iommu class of PCI devices on the bus. + */ +enum rte_iova_mode +rte_pci_get_iommu_class(void) +{ + bool is_bound; + bool is_vfio_noiommu_enabled = true; + bool has_iova_va; + bool is_bound_uio; + + is_bound = pci_device_is_bound(); + if (!is_bound) + return RTE_IOVA_DC; + + has_iova_va = pci_device_has_iova_va(); + is_bound_uio = pci_device_bound_uio(); +#ifdef VFIO_PRESENT + is_vfio_noiommu_enabled = vfio_noiommu_is_enabled() == true ? + true : false; +#endif + + if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled) + return RTE_IOVA_VA; + + if (has_iova_va) { + RTE_LOG(WARNING, EAL, "Some devices want iova as va but pa will be used because.. "); + if (is_vfio_noiommu_enabled) + RTE_LOG(WARNING, EAL, "vfio-noiommu mode configured\n"); + if (is_bound_uio) + RTE_LOG(WARNING, EAL, "few device bound to UIO\n"); + } + + return RTE_IOVA_PA; +} + /* Read PCI config space. */ int rte_pci_read_config(const struct rte_pci_device *device, void *buf, size_t len, off_t offset) diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c index 946df7e31..c8a97b7e7 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.c +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c @@ -816,4 +816,23 @@ vfio_noiommu_dma_map(int __rte_unused vfio_container_fd) return 0; } +int +vfio_noiommu_is_enabled(void) +{ + int fd, ret, cnt __rte_unused; + char c; + + ret = -1; + fd = open(VFIO_NOIOMMU_MODE, O_RDONLY); + if (fd < 0) + return -1; + + cnt = read(fd, &c, 1); + if (c == 'Y') + ret = 1; + + close(fd); + return ret; +} + #endif diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h index 5ff63e5d7..26ea8e119 100644 --- a/lib/librte_eal/linuxapp/eal/eal_vfio.h +++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h @@ -150,6 +150,8 @@ struct vfio_config { #define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" #define VFIO_GET_REGION_ADDR(x) ((uint64_t) x << 40ULL) #define VFIO_GET_REGION_IDX(x) (x >> 40) +#define VFIO_NOIOMMU_MODE \ + "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode" /* DMA mapping function prototype. * Takes VFIO container fd as a parameter. @@ -210,6 +212,8 @@ int pci_vfio_is_enabled(void); int vfio_mp_sync_setup(void); +int vfio_noiommu_is_enabled(void); + #define SOCKET_REQ_CONTAINER 0x100 #define SOCKET_REQ_GROUP 0x200 #define SOCKET_CLR_GROUP 0x300 diff --git a/lib/librte_eal/linuxapp/eal/rte_eal_version.map b/lib/librte_eal/linuxapp/eal/rte_eal_version.map index 287cc75cd..a8c8ea4f4 100644 --- a/lib/librte_eal/linuxapp/eal/rte_eal_version.map +++ b/lib/librte_eal/linuxapp/eal/rte_eal_version.map @@ -248,5 +248,6 @@ DPDK_17.11 { global: rte_pci_match; + rte_pci_get_iommu_class; } DPDK_17.08;