From patchwork Fri Aug 11 16:14:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anatoly Burakov X-Patchwork-Id: 42 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4299A43036; Fri, 11 Aug 2023 18:14:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C29A040E03; Fri, 11 Aug 2023 18:14:51 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 7E5E840144 for ; Fri, 11 Aug 2023 18:14:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691770490; x=1723306490; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=0bhbZ/mZJioAiwyJIbJH05C/x3zSf9Qtz8VI1K4WSHU=; b=EebTOAmh31oHnRYWMFSA1+Uk7+zTgGVEi1bXYkSKhlyfVn3QhjCd+ODH W32CsLmh7fR/sxHE1sEB7s3lMNXpCQkZlfcmitRK5oIT0vV84d2TOOAlo RydD0TGTohuO4rMAn2V297hq9STK28ahCuPi7gt2WlVbmlPwKsnCWSD8o JG6q/C5n3sDvZE0qaqB68MAEcwOBUubzQUShLHLJwX58dO4G+TOOpbgEe CD0dgU4LvZSBc29hWHoeR22ByVbHeY9fidQvqMMkTrp6Ir9Quvxy4dVLu aiQBvhMEm4nQddU+Gp7veZQb9beEboWOp0LitSagkcI/LqKSJaAXqp6Ft g==; X-IronPort-AV: E=McAfee;i="6600,9927,10799"; a="351312909" X-IronPort-AV: E=Sophos;i="6.01,166,1684825200"; d="scan'208";a="351312909" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Aug 2023 09:14:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10799"; a="906499469" X-IronPort-AV: E=Sophos;i="6.01,166,1684825200"; d="scan'208";a="906499469" Received: from silpixa00401191.ir.intel.com ([10.55.128.139]) by orsmga005.jf.intel.com with ESMTP; 11 Aug 2023 09:14:48 -0700 From: Anatoly Burakov To: dev@dpdk.org Cc: bruce.richardson@intel.com Subject: [PATCH v1 0/3] Add support for inter-domain DMA operations Date: Fri, 11 Aug 2023 16:14:43 +0000 Message-Id: X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patchset adds inter-domain DMA operations, and implements driver support for them in Intel(R) IDXD driver. Inter-domain DMA operations are similar to regular DMA operations, except that source and/or destination addresses will be in virtual address space of another process. In this patchset, DMA device is extended to support two new data plane operations: inter-domain copy, and inter-domain fill. No control plane API is provided for dmadev to set up inter-domain communication (see below for more info). DMA device API is extended with inter-domain operations, along with their respective capability flag. Two new op flags are also added to allow for inter-domain operations to select whether the source and/or destination address is in an address space of another process. Finally, the `rte_dma_info` struct is extended with a "controller ID" value (set to -1 by default for all drivers that don't implement it), representing a hardware DMA controller ID. This is because under current IDXD implementation the IDPTE (Inter-Domain Permission Table Entry) table is global to each device. That is, even though there may be multiple dmadev devices used by IDXD driver, they will all share their IDPTE entries if they belong to the same hardware controller, so some sort of value indicating where each dmadev belongs was needed. Similarly, IDXD driver is extended to support the new dmadev API, as well as use the new "controller ID" value. IDXD driver is also extended to have a private API for control-plane operations related to creating/attaching to memory regions which are shared between processes. In the current implementation, control-plane operations were made as a private API, instead of extending the DMA device API. This is because technically, only the submitter (a process which is using IDXD driver to perform inter-domain operations) has to have a DMA device available, while the owner (a process which shares its memory regions with the submitter) does not have to manage a DMA device to give access to its memory to another process. Another consideration is that currently, this API is Linux*-specific and relies on passing file descriptors over IPC, and this process, if implemented on other vendors' hardware, may not map to the same scheme. NOTE: currently, no publicly released hardware is available to test this feature or this patchset We are seeking community review on the following aspects of the patchset: - The fact that control-plane API is supposed to be private to specific drivers - The design of inter-domain data-plane operations API with respect to how "inter-domain handles" are being used and whether it's possible to make the API more vendor-neutral - New data-plane ops in dmadev will extend the data plane struct into the second cache line - this should not be an issue since non-inter-domain operations are still in the first cache line, and thus existing fast path is not affected - Any other feedback is welcome as well! Anatoly Burakov (3): dmadev: add inter-domain operations dma/idxd: implement inter-domain operations dma/idxd: add API to create and attach to window doc/guides/dmadevs/idxd.rst | 52 ++++++++ doc/guides/prog_guide/dmadev.rst | 22 ++++ drivers/dma/idxd/idxd_bus.c | 35 ++++++ drivers/dma/idxd/idxd_common.c | 123 ++++++++++++++++--- drivers/dma/idxd/idxd_hw_defs.h | 14 ++- drivers/dma/idxd/idxd_inter_dom.c | 166 ++++++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 7 ++ drivers/dma/idxd/meson.build | 7 +- drivers/dma/idxd/rte_idxd_inter_dom.h | 79 ++++++++++++ drivers/dma/idxd/version.map | 11 ++ lib/dmadev/rte_dmadev.c | 2 + lib/dmadev/rte_dmadev.h | 133 +++++++++++++++++++++ lib/dmadev/rte_dmadev_core.h | 12 ++ 13 files changed, 644 insertions(+), 19 deletions(-) create mode 100644 drivers/dma/idxd/idxd_inter_dom.c create mode 100644 drivers/dma/idxd/rte_idxd_inter_dom.h create mode 100644 drivers/dma/idxd/version.map