From patchwork Tue Oct 24 02:28:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 133225 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9DDAF431EB; Tue, 24 Oct 2023 04:32:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6CD6242E02; Tue, 24 Oct 2023 04:30:20 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2100.outbound.protection.outlook.com [40.107.236.100]) by mails.dpdk.org (Postfix) with ESMTP id 70C9C42D76 for ; Tue, 24 Oct 2023 04:30:16 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fyTT3cbC+nP+cpnR0Hyzf5m7Cne3vGbtA5046558Se8LwP1+snxZZwnAKiQ3BTfd/FyVpW2lJctSZ/2kisHydF0FI9eab44kwZttDY5hggFpHQWdfjSXu7wezvQSe5SRL4MMhwFq18ky6YzrFmFh1ZwBsmukVLmwERPwdOzmw5kBNly8sevjUObleq/8ZMRMh5n+XCsRkz06uAphEF1W7nLZwQkeK7gS5NVGm2wvyntZOi2HK556JW199p0wWSLpODAWNM9dw7Cd/qQ8oDdYQCDtYSZXfSKLf0JnRwtGH7ijyeRoTZVpxmZ8wdiCc80A7ggfJAGgnmGY3WBn320zEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=CrQxgSJf1+g5PyrFwEHqnBZzb/TtizLecfLmtPtvqd9K0qufWq4EoacR65+EAdiqnzU+QchA6PLVrpvLVEXvhFnZUAVZpI9NQ4ALNtRQO8iEO8WLIlEox5BFjFI5a4XB0lVU1qIyGGAOAmnbl5n5IaOI5aI5fcjdQrKSHdFo7U7S9CjdE9EfuJ8nLWKmTfEUfCDsiyUuYYAtEBUfkzMPC1ONU07vhXrBWRG/1PJHpXwp7kddOEXemyQLJBYA6nYFW81nuGRlKbXaAUuk44gML6U00K9pshdhpZG7hDfTEE6iSdEBHA97vMAt51H8HSmJHzgSSZnR36AHkj/yBvEcYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=ix152++hrLBSiaUkY5Y6qrd/pETI/tdegzytawA0b2ZJupTDEbK1c5E5WIGIok71cW78S0H5dmCWVRq9zDFLy8KWCAMNqWal2IsZu9IK3ePo/THpM/mto+IiAFFAicdW2E6idxxnu7mnfay9sy4AWaQwkZzmEhm4xEHpZrNLocY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by PH8PR13MB6182.namprd13.prod.outlook.com (2603:10b6:510:259::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Tue, 24 Oct 2023 02:30:15 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::142e:d47c:7320:8a9d]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::142e:d47c:7320:8a9d%4]) with mapi id 15.20.6907.021; Tue, 24 Oct 2023 02:30:15 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v2 20/25] vdpa/nfp: add the logic of remap PCI memory Date: Tue, 24 Oct 2023 10:28:44 +0800 Message-Id: <20231024022849.3092506-21-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231024022849.3092506-1-chaoyong.he@corigine.com> References: <20231017054545.1692509-1-chaoyong.he@corigine.com> <20231024022849.3092506-1-chaoyong.he@corigine.com> X-ClientProxiedBy: BYAPR04CA0015.namprd04.prod.outlook.com (2603:10b6:a03:40::28) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|PH8PR13MB6182:EE_ X-MS-Office365-Filtering-Correlation-Id: 46b6e814-2b1f-40e7-ff13-08dbd43927b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KvAwQ689BBvW19hfBkiNS2rD/572yYMYI1HpzfzuH2P63Hhz5g3sZfSJ0d897OuUEPoumIWSdNMo1Ol1D3JL2tFgxKS4FiFCdhzNgK3/RcsZdlgvtL4jLSvxA3gSwyvX+UrqE//rdEnl8A1SAWOyyd8+NOaGJuXy9xwTN50ERE8WlwN9Y9ea91FtCnLQTeVd1G58AxrbfJllwuS+dvzL2nePNckZkiltdHBunpqOMb+5ghQ4RxI7U5MG2s31DG8x6MIqyxHQuCmB7umVqMSJ6jlW5BlWuf+V4qpC/TJMl9ZsHPlYgdVRFTaTjp/Z4AGsxh4ErwwEcl0sGD8eFKyaBoUirKMhNTsL8ZRq+UH/2at6IMc9g1jGsnXeoXUyE8KEYT7ZHIANHnOvEVVs3/08hzUwkgBj3T+K//QJ+fNWrn2ZT2asNg/GEp0eMSvT7MS7cCxCciAPyPTHIz7C4XITyEBq19cxt0fWx3KMF5kFgvbFXaJoQo1Ah3cWHX/lXBji4K6fpVfIHyQliG0KTKT2Ao+llUcGX0YslhUGa6HlB+G89asKv3NZ+3QHGKhQhpNncfU1mGGUYbUuNWIbnfNMrR3jwRCxwxupa19AgcdEAz99WstQrBF5n57XVsFkbofeI56zTeNvh9+qAzi7RO2UAnySxdPFv/IRxVyebh9FhLY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(396003)(376002)(39830400003)(346002)(136003)(230922051799003)(1800799009)(64100799003)(186009)(451199024)(86362001)(66946007)(316002)(2906002)(54906003)(66476007)(6916009)(66556008)(5660300002)(8936002)(44832011)(8676002)(4326008)(36756003)(41300700001)(2616005)(38100700002)(478600001)(1076003)(52116002)(6506007)(107886003)(6666004)(38350700005)(6512007)(26005)(83380400001)(6486002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: G+WX9BwsgVkmmvFwSp+phVz3q6GUG+fzh+0qCIMn5eCpGIVlNxYfAZUQc4rNcWyEiKCdSlleU6U0xyOExu8eElxTvgky18JJ6wg8xXkhFrOf/1b3ji+C+eD42s867QgteTw46ZRfL88NJ1sibLIBntB068xzHFv1pTBnl+kFqdk1ZYMzfvNGb2YiMxlE7CO077K4N05AwTzSg3rREsD61ObEFTldeTl20nLGrMwp6CyG0lxnnAykJph4f2ALY0kcDM2OXbdbBFaCFTi3rzs5hmdsn9Dz035ufY1o4+lOp5qN6uQf5JFQDojsvH1EIFyxkBYJ08mzEHKOAV699fMOfTic1XASThjWf6OTuV49DALvyishjz9FPJU+dek3IVzUQuv/4mTtG54tUaxuvQS4cEVzC/a+AJxzBTi2gLu1mnjJM2ENCtS7UAlZMpwiorRuAa3NkiD8L07bnJ2qSrEHLuPuNRUnspiYAB7f56JxhxBsggh0kR3EnnpbirXBvY9h1k7BIa6uRGmWr9HxsJDTfQ8STATtdbeRwG2Q5SkoTfnSIpNEwmigHTXH0Jo08CFE+Naxo0u/gSaKw44ERpON28kODVFOrdFRtRjaR05ItmJW7CFCWpS1i1Az+TsxgwfYOKKOXNDGBvho/AlNzNQ51wJBkmTMIKENW7IbzGu8hTkSqpFfpdHshOK8wvdFqYFiugpTb2Q8+D79ASHSBTWqjnIGwKTzu1Rq8RfunkP76d3W+ENpdv2FkfJrIKSVPCBWeXvkh9Nzn2EnTCzcpZCg40J2vGSqlZDDdjV9LsrEi8hnIBlTv7vGjh6JtPbBC/pwCvRPGvsaHC4oe3heHnjdf6aTeaeAX8O8zYEVaiQYHdk8U33XfdOIJn5L58uzTglkvPfuGbV0oPGJowaU0OEhvIE8N8IfTaCZwIhM4im5ScItZVnuiV1vcPROy2Z8FZszjtPHMLgpTGHDGfQiyibgM0945CX9Xljes9CVQ9dBDgcvmvNAIz1Zq+c2DnxtbCfkqEN30CN/xoHk7JXL8GuXw8TII23SStempT0GrScMl1v52rZ7m8S1gSr0tcd5zKFpVaeHkL0xBB1+M2OPcu+cRj4YRspa09tSZPkIlCDc61vtgkcfO1IrOWL6nD7qiZ+1+W3keEStAskB4axMZyfHbUXumi7lBwoPmXDuwKROsT5U0EvdOesRVXO6bKDKKsN5yjvLpHlQkcOUXoGX4AQOuDAjUeDYqOSH/MgqJRvCQIxDZ1NrrT3vulVczH+SOeUenUG8FAvkybbgo/74W9ZYDBLVMlH3B81Z4yJrCrwHUq5CDVDDfXBeI6+hKGI720nJdbuiZak25MDNNu2/YWthJ7pri74QNzh4rT3eRHLaP4dupW1xcgBVt6cNbxuL/6SKUAfZn1JzAtO8F9Sh4P74MpfID45PYHay5QtRzVIGlxWUC33FaZfpkWiqkopQEI7ZGqNeG1FV0zBkLcVegerrMg6zGrgJAP0e9oR2hyr9+YQU6fYH1FDQ8ahwCTK0A2KnQvxWGwN4JutLQ2YQOFenQgMqNGKlWGRz0z2ITNbLy8fnM9+565zEl+X7c6/xfRLpSF2jO7dUv0IQzUteXslaaw== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 46b6e814-2b1f-40e7-ff13-08dbd43927b9 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Oct 2023 02:30:15.0331 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wfRf/Daz0FQ0K9kmXnhWzCXv7t4iQkRYdmVnJPJOp5nbf2Zd51bDgA9U4znKtTcptwAaj/utfdouKeGgLYFog3rmgC/v9yAMVU4SGYXF5Bo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR13MB6182 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the logic of remap PCI memory. Signed-off-by: Chaoyong He Signed-off-by: Shujing Dong Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 65 ++++++++++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index ef4ae4946a..824b0d681a 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -7,6 +7,7 @@ #include #include +#include #include #include "nfp_vdpa_log.h" @@ -16,6 +17,11 @@ struct nfp_vdpa_dev { struct rte_pci_device *pci_dev; struct rte_vdpa_device *vdev; + + int vfio_container_fd; + int vfio_group_fd; + int vfio_dev_fd; + int iommu_group; }; struct nfp_vdpa_dev_node { @@ -53,12 +59,62 @@ nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) return NULL; } +static int +nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device) +{ + int ret; + char dev_name[RTE_DEV_NAME_MAX_LEN] = {0}; + struct rte_pci_device *pci_dev = device->pci_dev; + + rte_pci_unmap_device(pci_dev); + + rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN); + rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name, + &device->iommu_group); + + device->vfio_container_fd = rte_vfio_container_create(); + if (device->vfio_container_fd < 0) + return -1; + + device->vfio_group_fd = rte_vfio_container_group_bind( + device->vfio_container_fd, device->iommu_group); + if (device->vfio_group_fd < 0) + goto container_destroy; + + DRV_VDPA_LOG(DEBUG, "container_fd=%d, group_fd=%d,\n", + device->vfio_container_fd, device->vfio_group_fd); + + ret = rte_pci_map_device(pci_dev); + if (ret != 0) + goto group_unbind; + + device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle); + + return 0; + +group_unbind: + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); +container_destroy: + rte_vfio_container_destroy(device->vfio_container_fd); + + return -1; +} + +static void +nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device) +{ + rte_pci_unmap_device(device->pci_dev); + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); + rte_vfio_container_destroy(device->vfio_container_fd); +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { }; static int nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) { + int ret; struct nfp_vdpa_dev *device; struct nfp_vdpa_dev_node *node; @@ -75,10 +131,14 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) device->pci_dev = pci_dev; + ret = nfp_vdpa_vfio_setup(device); + if (ret != 0) + goto free_device; + device->vdev = rte_vdpa_register_device(&pci_dev->device, &nfp_vdpa_ops); if (device->vdev == NULL) { DRV_VDPA_LOG(ERR, "Failed to register device %s", pci_dev->name); - goto free_device; + goto vfio_teardown; } node->device = device; @@ -88,6 +148,8 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) return 0; +vfio_teardown: + nfp_vdpa_vfio_teardown(device); free_device: free(device); free_node: @@ -118,6 +180,7 @@ nfp_vdpa_pci_remove(struct rte_pci_device *pci_dev) pthread_mutex_unlock(&vdpa_list_lock); rte_vdpa_unregister_device(device->vdev); + nfp_vdpa_vfio_teardown(device); free(device); free(node);