From patchwork Fri Dec 30 09:58:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Gupta, Nipun" X-Patchwork-Id: 121479 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A9A1A0032; Fri, 30 Dec 2022 10:59:05 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 136DA40DD8; Fri, 30 Dec 2022 10:59:05 +0100 (CET) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2068.outbound.protection.outlook.com [40.107.101.68]) by mails.dpdk.org (Postfix) with ESMTP id 03AE74067B for ; Fri, 30 Dec 2022 10:59:03 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E6uVmQ7y3wmO+D/VQKTasdPe2cZWXkdXO9jUljCgTWcGsTCs8o2ZdnvtAyfVvddUwQBpKIJ1ZrGrNewIYsMaYA+5hoXaKtUGoMt0LP4F/KcYwEaR8zbDnUMdxF/3jMNnu3iRXodyFxvx2sMh5PZ11XFqd0vVfJDF2KXn4Yzo2AynDXo9Rfkr3XADsCVUFAShD1tdhH+yyy3zIKI8bf5A2BndJiw84FUFk/FWAPR3Es+AVnIkF1POXFiUE/KN94kBamDAWwTrJod3uXE3Znr0H35TvYCoT13qsyLpQ90Ls92aj9FGLrQgSTV1gZ3yG7i8sNdnt8ySVSpgQYkJBnnCmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CLSbebH20rrqspeCSgRKDl2OO5fbuLRNZd1IZgMCwHk=; b=TdJIVkxHB5V78Uc2aMP1EuryxNG0VJS2MDIOOa5566J7GhOKQCOBkux4hwL1VdRYtwySebB4uHETnqo81UDE66UqAC6DOiOqZOHDy9j9WdOAAUpYbqfmqAR6uEQanv7YJA4H2k4TCd1knOsknxhY24P3oK8rm/aHjhz1sbLj4Fykx/2+/x9h4y/1m5ms2eniIa943ni3wbNnRNUqWqFfcB08K4vF74nc7MgtCP9MIKl381ftOD4MGRuMqAdGJN/AoOBR3JGL2JBrpk6ASh3Su6BuoOBgHu4oYPh1qxXY4yw3dbO7ercXOol0NnBxN4kJE2FrTXFYkn6lxi1NnSH40Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=temperror (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=temperror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CLSbebH20rrqspeCSgRKDl2OO5fbuLRNZd1IZgMCwHk=; b=1uP9BCEh4YXc9nsY5DJFeG8/crY4dO/kWK5XKUIHzTdePuGhQusdSGdccq1AY6hnbQ1OgqARpSXxxuZrDeyipnZMkVAwxgPeqwR/J+cfHiVh40Wp8Ouf6VXacSemT7ikbD3/bOToL6XM5sDvS7VMAfT6eAJir6oBu8eUvbrhHQA= Received: from CY5PR04CA0004.namprd04.prod.outlook.com (2603:10b6:930:1e::26) by SJ0PR12MB5439.namprd12.prod.outlook.com (2603:10b6:a03:3ae::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.16; Fri, 30 Dec 2022 09:59:01 +0000 Received: from CY4PEPF0000C979.namprd02.prod.outlook.com (2603:10b6:930:1e:cafe::39) by CY5PR04CA0004.outlook.office365.com (2603:10b6:930:1e::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5966.17 via Frontend Transport; Fri, 30 Dec 2022 09:59:01 +0000 X-MS-Exchange-Authentication-Results: spf=temperror (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=temperror action=none header.from=amd.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of amd.com: DNS Timeout) Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000C979.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5944.10 via Frontend Transport; Fri, 30 Dec 2022 09:59:00 +0000 Received: from SATLEXMB07.amd.com (10.181.41.45) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Fri, 30 Dec 2022 03:58:59 -0600 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB07.amd.com (10.181.41.45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Fri, 30 Dec 2022 01:58:59 -0800 Received: from xhdipdslab49.xilinx.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server id 15.1.2375.34 via Frontend Transport; Fri, 30 Dec 2022 03:58:57 -0600 From: Nipun Gupta To: , , , CC: , Nipun Gupta Subject: [PATCH] vfio: do not coalesce DMA mappings Date: Fri, 30 Dec 2022 15:28:53 +0530 Message-ID: <20221230095853.1323616-1-nipun.gupta@amd.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000C979:EE_|SJ0PR12MB5439:EE_ X-MS-Office365-Filtering-Correlation-Id: cf577b14-e109-41cc-418d-08daea4c79a6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: z3p+pybk6ArNSzrBehtVx6hlLl/rny3/Nr7TkxwHmd4evnzolmF1SC+j9S1DxXbsNLbQqfFHYSvH1v08RkIArLNwvAF6d5HgHpH47NGbG/TyTBQ8fFqh30c8oFfLS8iWjy4PqsHaItzcwTlhvtGmCqAyo3ilbXfVX/zwvOY8OQXpKCa5sVoK9khjBLuJiL2gse5SmhPzGc5A2MhY2bVNSSsULFj5WjIBcvSHdvM2QeKj4HZm/uwdQmdRTYB1PoBlia5GoLUHQxsVxWCDB6bt0O7sXSl1X29QCstQMSo1GWUNXz83yEh4l8yW/dyuYj1PHr3Ov+2KIwto4AXraAVEHp88GRm0rlkX4BfkjBm3M6hENSJQXzrMwJqayG5nuFGVkrVN5nIPeJwMs6To7j/2UUvocF8cwGrL3+jg+h3qR+z3xa9yCAw+rBGqsXyb4fD+r3h7diqSyJ2dgV20JKppFJfF0y4uQQtMeCTS2KtcDJeDPsQSh/k0TuB28A9kWZzXVOD+YDx1W8KQgVJwE3VTVq9QkfiJkrxOQbf9fzjQZylha+UhYWFWfetVBY1m8SOMsKaI3qO9yZMo7REfnLg3KAgLoKcnt+qZfwxMYsweExjb26mqfa961W56/Am3Bd2yO6o/v7C2j0DrYxq9wxmrWbU1MbdJRmDP8sAK/rd34PzxX0brZ9RNZqRjw1RiGWu9n0QglczICqgSaLrzQ6KNY4lE5anNzz1Iw+8p/qVD8FZpj7flphA3x46pLiq0ArxMzH+eZeUBd7aWDg3WwBSn6GHS0kwCsHwsIwZj8F1XIuNQcEaub67DwIXGpRIGdKZU X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB03.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230022)(4636009)(376002)(396003)(39860400002)(136003)(346002)(451199015)(40470700004)(46966006)(36840700001)(4326008)(8676002)(70206006)(356005)(8936002)(70586007)(5660300002)(41300700001)(110136005)(2906002)(82310400005)(316002)(36756003)(40460700003)(54906003)(44832011)(86362001)(6636002)(82740400003)(6666004)(478600001)(336012)(966005)(81166007)(186003)(26005)(2616005)(40480700001)(36860700001)(1076003)(426003)(63370400001)(83380400001)(63350400001)(47076005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Dec 2022 09:59:00.4605 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cf577b14-e109-41cc-418d-08daea4c79a6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000C979.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5439 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org At the cleanup time when dma unmap is done, linux kernel does not allow unmap of individual segments which were coalesced together while creating the DMA map for type1 IOMMU mappings. So, this change updates the mapping of the memory segments(hugepages) on a per-page basis. Signed-off-by: Nipun Gupta --- When hotplug of devices is used, multiple pages gets colaeced and a single mapping gets created for these pages (using APIs rte_memseg_contig_walk() and type1_map_contig(). On the cleanup time when the memory is released, the VFIO does not cleans up that memory and following error is observed in the eal for 2MB hugepages: EAL: Unexpected size 0 of DMA remapping cleared instead of 2097152 This is because VFIO does not clear the DMA (refer API vfio_dma_do_unmap() - https://elixir.bootlin.com/linux/latest/source/drivers/vfio/vfio_iommu_type1.c#L1330), where it checks the dma mapping where it checks for IOVA to free: https://elixir.bootlin.com/linux/latest/source/drivers/vfio/vfio_iommu_type1.c#L1418. Thus this change updates the mapping to be created individually instead of colaecing them. lib/eal/linux/eal_vfio.c | 29 ----------------------------- 1 file changed, 29 deletions(-) diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c index 549b86ae1d..56edccb0db 100644 --- a/lib/eal/linux/eal_vfio.c +++ b/lib/eal/linux/eal_vfio.c @@ -1369,19 +1369,6 @@ rte_vfio_get_group_num(const char *sysfs_base, return 1; } -static int -type1_map_contig(const struct rte_memseg_list *msl, const struct rte_memseg *ms, - size_t len, void *arg) -{ - int *vfio_container_fd = arg; - - if (msl->external) - return 0; - - return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova, - len, 1); -} - static int type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms, void *arg) @@ -1396,10 +1383,6 @@ type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms, if (ms->iova == RTE_BAD_IOVA) return 0; - /* if IOVA mode is VA, we've already mapped the internal segments */ - if (!msl->external && rte_eal_iova_mode() == RTE_IOVA_VA) - return 0; - return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, ms->iova, ms->len, 1); } @@ -1464,18 +1447,6 @@ vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova, static int vfio_type1_dma_map(int vfio_container_fd) { - if (rte_eal_iova_mode() == RTE_IOVA_VA) { - /* with IOVA as VA mode, we can get away with mapping contiguous - * chunks rather than going page-by-page. - */ - int ret = rte_memseg_contig_walk(type1_map_contig, - &vfio_container_fd); - if (ret) - return ret; - /* we have to continue the walk because we've skipped the - * external segments during the config walk. - */ - } return rte_memseg_walk(type1_map, &vfio_container_fd); }