From patchwork Fri Aug 27 17:20:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97471 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 202E0A0548; Fri, 27 Aug 2021 19:21:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 425E541134; Fri, 27 Aug 2021 19:21:13 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 3D0F840150 for ; Fri, 27 Aug 2021 19:21:10 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235889" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235889" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009358" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:08 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com Date: Fri, 27 Aug 2021 17:20:36 +0000 Message-Id: <20210827172048.558704-2-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 01/13] raw/ioat: only build if dmadev not present X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Bruce Richardson Only build the rawdev IDXD/IOAT drivers if the dmadev drivers are not present. Signed-off-by: Bruce Richardson --- drivers/raw/ioat/meson.build | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build index 0e81cb5951..7bd9ac912b 100644 --- a/drivers/raw/ioat/meson.build +++ b/drivers/raw/ioat/meson.build @@ -2,14 +2,31 @@ # Copyright 2019 Intel Corporation build = dpdk_conf.has('RTE_ARCH_X86') +# only use ioat rawdev driver if we don't have the equivalent dmadev ones +if not dpdk_conf.has('RTE_DMA_IDXD') and not dpdk_conf.has('RTE_DMA_IOAT') + build = false + subdir_done() +endif + reason = 'only supported on x86' sources = files( - 'idxd_bus.c', - 'idxd_pci.c', 'ioat_common.c', - 'ioat_rawdev.c', 'ioat_rawdev_test.c', ) + +if not dpdk_conf.has('RTE_DMA_IDXD') + sources += files( + 'idxd_bus.c', + 'idxd_pci.c', + ) +endif + +if not dpdk_conf.has('RTE_DMA_IOAT') + sources += files ( + 'ioat_rawdev.c', + ) +endif + deps += ['bus_pci', 'mbuf', 'rawdev'] headers = files( 'rte_ioat_rawdev.h', From patchwork Fri Aug 27 17:20:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97472 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9147FA0548; Fri, 27 Aug 2021 19:21:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 78C20411E9; Fri, 27 Aug 2021 19:21:14 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 6C2C8410E9 for ; Fri, 27 Aug 2021 19:21:12 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235891" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235891" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009382" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:09 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:37 +0000 Message-Id: <20210827172048.558704-3-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 02/13] doc: initial commit for dmadevs section X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add new section to the programmer's guide for dmadev devices. Signed-off-by: Kevin Laatz --- doc/guides/dmadevs/index.rst | 14 ++++++++++++++ doc/guides/index.rst | 1 + 2 files changed, 15 insertions(+) create mode 100644 doc/guides/dmadevs/index.rst diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst new file mode 100644 index 0000000000..b30004fd65 --- /dev/null +++ b/doc/guides/dmadevs/index.rst @@ -0,0 +1,14 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +DMA Device Drivers +================== + +The following are a list of DMA device PMDs, which can be used from an +application through DMAdev API. + +.. toctree:: + :maxdepth: 2 + :numbered: + + idxd diff --git a/doc/guides/index.rst b/doc/guides/index.rst index 857f0363d3..ccb71640dd 100644 --- a/doc/guides/index.rst +++ b/doc/guides/index.rst @@ -19,6 +19,7 @@ DPDK documentation bbdevs/index cryptodevs/index compressdevs/index + dmadevs/index vdpadevs/index regexdevs/index eventdevs/index From patchwork Fri Aug 27 17:20:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97473 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EC98BA0548; Fri, 27 Aug 2021 19:21:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5A0ED41257; Fri, 27 Aug 2021 19:21:17 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id AD2FF411D3 for ; Fri, 27 Aug 2021 19:21:13 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235901" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235901" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009398" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:11 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:38 +0000 Message-Id: <20210827172048.558704-4-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 03/13] dma/idxd: add skeleton for VFIO based DSA device X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the basic device probe/remove skeleton code for DSA device bound to the vfio pci driver. Relevant documentation and MAINTAINERS update also included. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- MAINTAINERS | 10 +++++ doc/guides/dmadevs/idxd.rst | 58 ++++++++++++++++++++++++++ doc/guides/rel_notes/release_21_11.rst | 5 +++ drivers/dma/idxd/idxd_internal.h | 27 ++++++++++++ drivers/dma/idxd/idxd_pci.c | 55 ++++++++++++++++++++++++ drivers/dma/idxd/meson.build | 7 ++++ drivers/dma/idxd/version.map | 3 ++ drivers/dma/meson.build | 1 + 8 files changed, 166 insertions(+) create mode 100644 doc/guides/dmadevs/idxd.rst create mode 100644 drivers/dma/idxd/idxd_internal.h create mode 100644 drivers/dma/idxd/idxd_pci.c create mode 100644 drivers/dma/idxd/meson.build create mode 100644 drivers/dma/idxd/version.map diff --git a/MAINTAINERS b/MAINTAINERS index 1661428a02..3d275c6bb4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1199,6 +1199,16 @@ F: doc/guides/compressdevs/zlib.rst F: doc/guides/compressdevs/features/zlib.ini +DMAdev Drivers +-------------- + +Intel IDXD - EXPERIMENTAL +M: Bruce Richardson +M: Kevin Laatz +F: drivers/dma/idxd/ +F: doc/guides/dmadevs/idxd.rst + + RegEx Drivers ------------- diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst new file mode 100644 index 0000000000..924700d17e --- /dev/null +++ b/doc/guides/dmadevs/idxd.rst @@ -0,0 +1,58 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Intel Corporation. + +.. include:: + +IDXD DMA Device Driver +====================== + +The ``idxd`` dmadev driver provides a poll-mode driver (PMD) for Intel\ |reg| +Data Streaming Accelerator `(Intel DSA) +`_. +This PMD can be used in conjunction with Intel\ |reg| DSA devices to offload +data operations, such as data copies, to hardware, freeing up CPU cycles for +other tasks. + +Hardware Requirements +---------------------- + +The ``dpdk-devbind.py`` script, included with DPDK, can be used to show the +presence of supported hardware. Running ``dpdk-devbind.py --status-dev dma`` +will show all the DMA devices on the system, including IDXD supported devices. +Intel\ |reg| DSA devices, are currently (at time of writing) appearing +as devices with type “0b25”, due to the absence of pci-id database entries for +them at this point. + +Compilation +------------ + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. No additional compilation steps are necessary. + +Device Setup +------------- + +Devices using VFIO/UIO drivers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The HW devices to be used will need to be bound to a user-space IO driver for use. +The ``dpdk-devbind.py`` script can be used to view the state of the devices +and to bind them to a suitable DPDK-supported driver, such as ``vfio-pci``. +For example:: + + $ dpdk-devbind.py -b vfio-pci 6a:01.0 + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For devices bound to a suitable DPDK-supported VFIO/UIO driver, the HW devices will +be found as part of the device scan done at application initialization time without +the need to pass parameters to the application. + +For Intel\ |reg| DSA devices, DPDK will automatically configure the device with the +maximum number of workqueues available on it, partitioning all resources equally +among the queues. +If fewer workqueues are required, then the ``max_queues`` parameter may be passed to +the device driver on the EAL commandline, via the ``allowlist`` or ``-a`` flag e.g.:: + + $ dpdk-test -a ,max_queues=4 diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 0d3c38f479..9235ee1847 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -61,6 +61,11 @@ New Features provisioning of hardware and software DMA poll mode drivers, defining generic APIs which support a number of different DMA operations. +* **Added IDXD dmadev driver implementation.** + + The IDXD dmadev driver provide device drivers for the Intel DSA devices. + This device driver can be used through the generic dmadev API. + Removed Items ------------- diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h new file mode 100644 index 0000000000..c6a7dcd72f --- /dev/null +++ b/drivers/dma/idxd/idxd_internal.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IDXD_INTERNAL_H_ +#define _IDXD_INTERNAL_H_ + +/** + * @file idxd_internal.h + * + * Internal data structures for the idxd/DSA driver for dmadev + * + * @warning + * @b EXPERIMENTAL: these structures and APIs may change without prior notice + */ + +extern int idxd_pmd_logtype; + +#define IDXD_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + idxd_pmd_logtype, "IDXD: %s(): " fmt "\n", __func__, ##args) + +#define IDXD_PMD_DEBUG(fmt, args...) IDXD_PMD_LOG(DEBUG, fmt, ## args) +#define IDXD_PMD_INFO(fmt, args...) IDXD_PMD_LOG(INFO, fmt, ## args) +#define IDXD_PMD_ERR(fmt, args...) IDXD_PMD_LOG(ERR, fmt, ## args) +#define IDXD_PMD_WARN(fmt, args...) IDXD_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IDXD_INTERNAL_H_ */ diff --git a/drivers/dma/idxd/idxd_pci.c b/drivers/dma/idxd/idxd_pci.c new file mode 100644 index 0000000000..79e4aadcab --- /dev/null +++ b/drivers/dma/idxd/idxd_pci.c @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include + +#include "idxd_internal.h" + +#define IDXD_VENDOR_ID 0x8086 +#define IDXD_DEVICE_ID_SPR 0x0B25 + +#define IDXD_PMD_DMADEV_NAME_PCI dmadev_idxd_pci + +const struct rte_pci_id pci_id_idxd_map[] = { + { RTE_PCI_DEVICE(IDXD_VENDOR_ID, IDXD_DEVICE_ID_SPR) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static int +idxd_dmadev_probe_pci(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + int ret = 0; + char name[PCI_PRI_STR_SIZE]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IDXD_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + dev->device.driver = &drv->driver; + + return ret; +} + +static int +idxd_dmadev_remove_pci(struct rte_pci_device *dev) +{ + char name[PCI_PRI_STR_SIZE]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IDXD_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return 0; +} + +struct rte_pci_driver idxd_pmd_drv_pci = { + .id_table = pci_id_idxd_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING, + .probe = idxd_dmadev_probe_pci, + .remove = idxd_dmadev_remove_pci, +}; + +RTE_PMD_REGISTER_PCI(IDXD_PMD_DMADEV_NAME_PCI, idxd_pmd_drv_pci); +RTE_PMD_REGISTER_PCI_TABLE(IDXD_PMD_DMADEV_NAME_PCI, pci_id_idxd_map); +RTE_PMD_REGISTER_KMOD_DEP(IDXD_PMD_DMADEV_NAME_PCI, "vfio-pci"); +RTE_PMD_REGISTER_PARAM_STRING(dmadev_idxd_pci, "max_queues=0"); diff --git a/drivers/dma/idxd/meson.build b/drivers/dma/idxd/meson.build new file mode 100644 index 0000000000..24c0f3106e --- /dev/null +++ b/drivers/dma/idxd/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2021 Intel Corporation + +deps += ['bus_pci'] +sources = files( + 'idxd_pci.c' +) \ No newline at end of file diff --git a/drivers/dma/idxd/version.map b/drivers/dma/idxd/version.map new file mode 100644 index 0000000000..4a76d1d52d --- /dev/null +++ b/drivers/dma/idxd/version.map @@ -0,0 +1,3 @@ +DPDK_21 { + local: *; +}; diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build index 0c2c34cd00..0b01b6a8ab 100644 --- a/drivers/dma/meson.build +++ b/drivers/dma/meson.build @@ -6,6 +6,7 @@ if is_windows endif drivers = [ + 'idxd', 'skeleton', ] std_deps = ['dmadev'] From patchwork Fri Aug 27 17:20:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97474 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 17E4EA0548; Fri, 27 Aug 2021 19:21:38 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 888364125D; Fri, 27 Aug 2021 19:21:18 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id C0FDD4124A for ; Fri, 27 Aug 2021 19:21:15 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235909" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235909" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009435" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:13 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:39 +0000 Message-Id: <20210827172048.558704-5-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 04/13] dma/idxd: add bus device probing X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the basic device probing for DSA devices bound to the IDXD kernel driver. These devices can be configured via sysfs and made available to DPDK if they are found during bus scan. Relevant documentation is included. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- doc/guides/dmadevs/idxd.rst | 64 +++++++ drivers/dma/idxd/idxd_bus.c | 351 +++++++++++++++++++++++++++++++++++ drivers/dma/idxd/meson.build | 1 + 3 files changed, 416 insertions(+) create mode 100644 drivers/dma/idxd/idxd_bus.c diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst index 924700d17e..ce33e2857a 100644 --- a/doc/guides/dmadevs/idxd.rst +++ b/doc/guides/dmadevs/idxd.rst @@ -32,6 +32,56 @@ target platform is x86-based. No additional compilation steps are necessary. Device Setup ------------- +Intel\ |reg| DSA devices can use the IDXD kernel driver or DPDK-supported drivers, +such as ``vfio-pci``. Both are supported by the IDXD PMD. + +Intel\ |reg| DSA devices using IDXD kernel driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To use an Intel\ |reg| DSA device bound to the IDXD kernel driver, the device must first be configured. +The `accel-config `_ utility library can be used for configuration. + +.. note:: + The device configuration can also be done by directly interacting with the sysfs nodes. + An example of how this may be done can be seen in the script ``dpdk_idxd_cfg.py`` + included in the driver source directory. + +There are some mandatory configuration steps before being able to use a device with an application. +The internal engines, which do the copies or other operations, +and the work-queues, which are used by applications to assign work to the device, +need to be assigned to groups, and the various other configuration options, +such as priority or queue depth, need to be set for each queue. + +To assign an engine to a group:: + + $ accel-config config-engine dsa0/engine0.0 --group-id=0 + $ accel-config config-engine dsa0/engine0.1 --group-id=1 + +To assign work queues to groups for passing descriptors to the engines a similar accel-config command can be used. +However, the work queues also need to be configured depending on the use case. +Some configuration options include: + +* mode (Dedicated/Shared): Indicates whether a WQ may accept jobs from multiple queues simultaneously. +* priority: WQ priority between 1 and 15. Larger value means higher priority. +* wq-size: the size of the WQ. Sum of all WQ sizes must be less that the total-size defined by the device. +* type: WQ type (kernel/mdev/user). Determines how the device is presented. +* name: identifier given to the WQ. + +Example configuration for a work queue:: + + $ accel-config config-wq dsa0/wq0.0 --group-id=0 \ + --mode=dedicated --priority=10 --wq-size=8 \ + --type=user --name=dpdk_app1 + +Once the devices have been configured, they need to be enabled:: + + $ accel-config enable-device dsa0 + $ accel-config enable-wq dsa0/wq0.0 + +Check the device configuration:: + + $ accel-config list + Devices using VFIO/UIO drivers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -56,3 +106,17 @@ If fewer workqueues are required, then the ``max_queues`` parameter may be passe the device driver on the EAL commandline, via the ``allowlist`` or ``-a`` flag e.g.:: $ dpdk-test -a ,max_queues=4 + +For devices bound to the IDXD kernel driver, +the DPDK IDXD driver will automatically perform a scan for available workqueues +to use. Any workqueues found listed in ``/dev/dsa`` on the system will be checked +in ``/sys``, and any which have ``dpdk_`` prefix in their name will be automatically +probed by the driver to make them available to the application. +Alternatively, to support use by multiple DPDK processes simultaneously, +the value used as the DPDK ``--file-prefix`` parameter may be used as a workqueue +name prefix, instead of ``dpdk_``, allowing each DPDK application instance to only +use a subset of configured queues. + +Once probed successfully, irrespective of kernel driver, the device will appear as a ``dmadev``, +that is a "DMA device type" inside DPDK, and can be accessed using APIs from the +``rte_dmadev`` library. diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c new file mode 100644 index 0000000000..c08f0f473b --- /dev/null +++ b/drivers/dma/idxd/idxd_bus.c @@ -0,0 +1,351 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "idxd_internal.h" + +/* default value for DSA paths, but allow override in environment for testing */ +#define DSA_DEV_PATH "/dev/dsa" +#define DSA_SYSFS_PATH "/sys/bus/dsa/devices" + +static unsigned int devcount; + +/** unique identifier for a DSA device/WQ instance */ +struct dsa_wq_addr { + uint16_t device_id; + uint16_t wq_id; +}; + +/** a DSA device instance */ +struct rte_dsa_device { + struct rte_device device; /**< Inherit core device */ + TAILQ_ENTRY(rte_dsa_device) next; /**< next dev in list */ + + char wq_name[32]; /**< the workqueue name/number e.g. wq0.1 */ + struct dsa_wq_addr addr; /**< Identifies the specific WQ */ +}; + +/* forward prototypes */ +struct dsa_bus; +static int dsa_scan(void); +static int dsa_probe(void); +static struct rte_device *dsa_find_device(const struct rte_device *start, + rte_dev_cmp_t cmp, const void *data); +static enum rte_iova_mode dsa_get_iommu_class(void); +static int dsa_addr_parse(const char *name, void *addr); + +/** List of devices */ +TAILQ_HEAD(dsa_device_list, rte_dsa_device); + +/** + * Structure describing the DSA bus + */ +struct dsa_bus { + struct rte_bus bus; /**< Inherit the generic class */ + struct rte_driver driver; /**< Driver struct for devices to point to */ + struct dsa_device_list device_list; /**< List of PCI devices */ +}; + +struct dsa_bus dsa_bus = { + .bus = { + .scan = dsa_scan, + .probe = dsa_probe, + .find_device = dsa_find_device, + .get_iommu_class = dsa_get_iommu_class, + .parse = dsa_addr_parse, + }, + .driver = { + .name = "dmadev_idxd" + }, + .device_list = TAILQ_HEAD_INITIALIZER(dsa_bus.device_list), +}; + +static inline const char * +dsa_get_dev_path(void) +{ + const char *path = getenv("DSA_DEV_PATH"); + return path ? path : DSA_DEV_PATH; +} + +static inline const char * +dsa_get_sysfs_path(void) +{ + const char *path = getenv("DSA_SYSFS_PATH"); + return path ? path : DSA_SYSFS_PATH; +} + +static void * +idxd_vdev_mmap_wq(struct rte_dsa_device *dev) +{ + void *addr; + char path[PATH_MAX]; + int fd; + + snprintf(path, sizeof(path), "%s/%s", dsa_get_dev_path(), dev->wq_name); + fd = open(path, O_RDWR); + if (fd < 0) { + IDXD_PMD_ERR("Failed to open device path: %s", path); + return NULL; + } + + addr = mmap(NULL, 0x1000, PROT_WRITE, MAP_SHARED, fd, 0); + close(fd); + if (addr == MAP_FAILED) { + IDXD_PMD_ERR("Failed to mmap device %s", path); + return NULL; + } + + return addr; +} + +static int +read_wq_string(struct rte_dsa_device *dev, const char *filename, + char *value, size_t valuelen) +{ + char sysfs_node[PATH_MAX]; + int len; + int fd; + + snprintf(sysfs_node, sizeof(sysfs_node), "%s/%s/%s", + dsa_get_sysfs_path(), dev->wq_name, filename); + fd = open(sysfs_node, O_RDONLY); + if (fd < 0) { + IDXD_PMD_ERR("%s(): opening file '%s' failed: %s", + __func__, sysfs_node, strerror(errno)); + return -1; + } + + len = read(fd, value, valuelen - 1); + close(fd); + if (len < 0) { + IDXD_PMD_ERR("%s(): error reading file '%s': %s", + __func__, sysfs_node, strerror(errno)); + return -1; + } + value[len] = '\0'; + return 0; +} + +static int +read_wq_int(struct rte_dsa_device *dev, const char *filename, + int *value) +{ + char sysfs_node[PATH_MAX]; + FILE *f; + int ret = 0; + + snprintf(sysfs_node, sizeof(sysfs_node), "%s/%s/%s", + dsa_get_sysfs_path(), dev->wq_name, filename); + f = fopen(sysfs_node, "r"); + if (f == NULL) { + IDXD_PMD_ERR("%s(): opening file '%s' failed: %s", + __func__, sysfs_node, strerror(errno)); + return -1; + } + + if (fscanf(f, "%d", value) != 1) { + IDXD_PMD_ERR("%s(): error reading file '%s': %s", + __func__, sysfs_node, strerror(errno)); + ret = -1; + } + + fclose(f); + return ret; +} + +static int +read_device_int(struct rte_dsa_device *dev, const char *filename, + int *value) +{ + char sysfs_node[PATH_MAX]; + FILE *f; + int ret = 0; + + snprintf(sysfs_node, sizeof(sysfs_node), "%s/dsa%d/%s", + dsa_get_sysfs_path(), dev->addr.device_id, filename); + f = fopen(sysfs_node, "r"); + if (f == NULL) { + IDXD_PMD_ERR("%s(): opening file '%s' failed: %s", + __func__, sysfs_node, strerror(errno)); + return -1; + } + + if (fscanf(f, "%d", value) != 1) { + IDXD_PMD_ERR("%s(): error reading file '%s': %s", + __func__, sysfs_node, strerror(errno)); + ret = -1; + } + + fclose(f); + return ret; +} + +static int +idxd_probe_dsa(struct rte_dsa_device *dev) +{ + struct idxd_dmadev idxd = {0}; + int ret = 0; + + IDXD_PMD_INFO("Probing device %s on numa node %d", + dev->wq_name, dev->device.numa_node); + if (read_wq_int(dev, "size", &ret) < 0) + return -1; + idxd.max_batches = ret; + if (read_wq_int(dev, "max_batch_size", &ret) < 0) + return -1; + idxd.max_batch_size = ret; + idxd.qid = dev->addr.wq_id; + idxd.u.vdev.dsa_id = dev->addr.device_id; + idxd.sva_support = 1; + + idxd.portal = idxd_vdev_mmap_wq(dev); + if (idxd.portal == NULL) { + IDXD_PMD_ERR("WQ mmap failed"); + return -ENOENT; + } + + return 0; +} + +static int +is_for_this_process_use(const char *name) +{ + char *runtime_dir = strdup(rte_eal_get_runtime_dir()); + char *prefix = basename(runtime_dir); + int prefixlen = strlen(prefix); + int retval = 0; + + if (strncmp(name, "dpdk_", 5) == 0) + retval = 1; + if (strncmp(name, prefix, prefixlen) == 0 && name[prefixlen] == '_') + retval = 1; + + free(runtime_dir); + return retval; +} + +static int +dsa_probe(void) +{ + struct rte_dsa_device *dev; + + TAILQ_FOREACH(dev, &dsa_bus.device_list, next) { + char type[64], name[64]; + + if (read_wq_string(dev, "type", type, sizeof(type)) < 0 || + read_wq_string(dev, "name", name, sizeof(name)) < 0) + continue; + + if (strncmp(type, "user", 4) == 0 && is_for_this_process_use(name)) { + dev->device.driver = &dsa_bus.driver; + idxd_probe_dsa(dev); + continue; + } + IDXD_PMD_DEBUG("WQ '%s', not allocated to DPDK", dev->wq_name); + } + + return 0; +} + +static int +dsa_scan(void) +{ + const char *path = dsa_get_dev_path(); + struct dirent *wq; + DIR *dev_dir; + + dev_dir = opendir(path); + if (dev_dir == NULL) { + if (errno == ENOENT) + return 0; /* no bus, return without error */ + IDXD_PMD_ERR("%s(): opendir '%s' failed: %s", + __func__, path, strerror(errno)); + return -1; + } + + while ((wq = readdir(dev_dir)) != NULL) { + struct rte_dsa_device *dev; + int numa_node = -1; + + if (strncmp(wq->d_name, "wq", 2) != 0) + continue; + if (strnlen(wq->d_name, sizeof(dev->wq_name)) == sizeof(dev->wq_name)) { + IDXD_PMD_ERR("%s(): wq name too long: '%s', skipping", + __func__, wq->d_name); + continue; + } + IDXD_PMD_DEBUG("%s(): found %s/%s", __func__, path, wq->d_name); + + dev = malloc(sizeof(*dev)); + if (dsa_addr_parse(wq->d_name, &dev->addr) < 0) { + IDXD_PMD_ERR("Error parsing WQ name: %s", wq->d_name); + free(dev); + continue; + } + dev->device.bus = &dsa_bus.bus; + strlcpy(dev->wq_name, wq->d_name, sizeof(dev->wq_name)); + TAILQ_INSERT_TAIL(&dsa_bus.device_list, dev, next); + devcount++; + + read_device_int(dev, "numa_node", &numa_node); + dev->device.numa_node = numa_node; + dev->device.name = dev->wq_name; + } + + closedir(dev_dir); + return 0; +} + +static struct rte_device * +dsa_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, + const void *data) +{ + struct rte_dsa_device *dev = TAILQ_FIRST(&dsa_bus.device_list); + + /* the rte_device struct must be at start of dsa structure */ + RTE_BUILD_BUG_ON(offsetof(struct rte_dsa_device, device) != 0); + + if (start != NULL) /* jump to start point if given */ + dev = TAILQ_NEXT((const struct rte_dsa_device *)start, next); + while (dev != NULL) { + if (cmp(&dev->device, data) == 0) + return &dev->device; + dev = TAILQ_NEXT(dev, next); + } + return NULL; +} + +static enum rte_iova_mode +dsa_get_iommu_class(void) +{ + /* if there are no devices, report don't care, otherwise VA mode */ + return devcount > 0 ? RTE_IOVA_VA : RTE_IOVA_DC; +} + +static int +dsa_addr_parse(const char *name, void *addr) +{ + struct dsa_wq_addr *wq = addr; + unsigned int device_id, wq_id; + + if (sscanf(name, "wq%u.%u", &device_id, &wq_id) != 2) { + IDXD_PMD_DEBUG("Parsing WQ name failed: %s", name); + return -1; + } + + wq->device_id = device_id; + wq->wq_id = wq_id; + return 0; +} + +RTE_REGISTER_BUS(dsa, dsa_bus.bus); diff --git a/drivers/dma/idxd/meson.build b/drivers/dma/idxd/meson.build index 24c0f3106e..f1fea000a7 100644 --- a/drivers/dma/idxd/meson.build +++ b/drivers/dma/idxd/meson.build @@ -3,5 +3,6 @@ deps += ['bus_pci'] sources = files( + 'idxd_bus.c', 'idxd_pci.c' ) \ No newline at end of file From patchwork Fri Aug 27 17:20:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97475 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 51D8DA0548; Fri, 27 Aug 2021 19:21:44 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BA4E941267; Fri, 27 Aug 2021 19:21:19 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 921A44125D for ; Fri, 27 Aug 2021 19:21:17 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235915" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235915" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009465" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:15 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:40 +0000 Message-Id: <20210827172048.558704-6-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 05/13] dma/idxd: create dmadev instances on bus probe X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When a suitable device is found during the bus scan/probe, create a dmadev instance for each HW queue. Internal structures required for device creation are also added. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- drivers/dma/idxd/idxd_bus.c | 20 ++++++++- drivers/dma/idxd/idxd_common.c | 75 ++++++++++++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 40 +++++++++++++++++ drivers/dma/idxd/meson.build | 1 + 4 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 drivers/dma/idxd/idxd_common.c diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c index c08f0f473b..0f33500dfc 100644 --- a/drivers/dma/idxd/idxd_bus.c +++ b/drivers/dma/idxd/idxd_bus.c @@ -84,6 +84,18 @@ dsa_get_sysfs_path(void) return path ? path : DSA_SYSFS_PATH; } +static int +idxd_dev_close(struct rte_dmadev *dev) +{ + struct idxd_dmadev *idxd = dev->data->dev_private; + munmap(idxd->portal, 0x1000); + return 0; +} + +static const struct rte_dmadev_ops idxd_vdev_ops = { + .dev_close = idxd_dev_close, +}; + static void * idxd_vdev_mmap_wq(struct rte_dsa_device *dev) { @@ -205,7 +217,7 @@ idxd_probe_dsa(struct rte_dsa_device *dev) return -1; idxd.max_batch_size = ret; idxd.qid = dev->addr.wq_id; - idxd.u.vdev.dsa_id = dev->addr.device_id; + idxd.u.bus.dsa_id = dev->addr.device_id; idxd.sva_support = 1; idxd.portal = idxd_vdev_mmap_wq(dev); @@ -214,6 +226,12 @@ idxd_probe_dsa(struct rte_dsa_device *dev) return -ENOENT; } + ret = idxd_dmadev_create(dev->wq_name, &dev->device, &idxd, &idxd_vdev_ops); + if (ret) { + IDXD_PMD_ERR("Failed to create rawdev %s", dev->wq_name); + return ret; + } + return 0; } diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c new file mode 100644 index 0000000000..7770b2e264 --- /dev/null +++ b/drivers/dma/idxd/idxd_common.c @@ -0,0 +1,75 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#include +#include +#include + +#include "idxd_internal.h" + +#define IDXD_PMD_NAME_STR "dmadev_idxd" + +int +idxd_dmadev_create(const char *name, struct rte_device *dev, + const struct idxd_dmadev *base_idxd, + const struct rte_dmadev_ops *ops) +{ + struct idxd_dmadev *idxd; + struct rte_dmadev *dmadev = NULL; + int ret = 0; + + if (!name) { + IDXD_PMD_ERR("Invalid name of the device!"); + ret = -EINVAL; + goto cleanup; + } + + /* Allocate device structure */ + dmadev = rte_dmadev_pmd_allocate(name); + if (dmadev == NULL) { + IDXD_PMD_ERR("Unable to allocate raw device"); + ret = -ENOMEM; + goto cleanup; + } + dmadev->dev_ops = ops; + dmadev->device = dev; + + idxd = rte_malloc_socket(NULL, sizeof(struct idxd_dmadev), 0, dev->numa_node); + if (idxd == NULL) { + IDXD_PMD_ERR("Unable to allocate memory for device"); + ret = -ENOMEM; + goto cleanup; + } + dmadev->data->dev_private = idxd; + dmadev->dev_private = idxd; + *idxd = *base_idxd; /* copy over the main fields already passed in */ + idxd->dmadev = dmadev; + + /* allocate batch index ring and completion ring. + * The +1 is because we can never fully use + * the ring, otherwise read == write means both full and empty. + */ + idxd->batch_comp_ring = rte_zmalloc(NULL, (sizeof(idxd->batch_idx_ring[0]) + + sizeof(idxd->batch_comp_ring[0])) * (idxd->max_batches + 1), + sizeof(idxd->batch_comp_ring[0])); + if (idxd->batch_comp_ring == NULL) { + IDXD_PMD_ERR("Unable to reserve memory for batch data\n"); + ret = -ENOMEM; + goto cleanup; + } + idxd->batch_idx_ring = (void *)&idxd->batch_comp_ring[idxd->max_batches+1]; + idxd->batch_iova = rte_mem_virt2iova(idxd->batch_comp_ring); + + return 0; + +cleanup: + if (dmadev) + rte_dmadev_pmd_release(dmadev); + + return ret; +} + +int idxd_pmd_logtype; + +RTE_LOG_REGISTER_DEFAULT(idxd_pmd_logtype, WARNING); diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index c6a7dcd72f..99ab2df925 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -24,4 +24,44 @@ extern int idxd_pmd_logtype; #define IDXD_PMD_ERR(fmt, args...) IDXD_PMD_LOG(ERR, fmt, ## args) #define IDXD_PMD_WARN(fmt, args...) IDXD_PMD_LOG(WARNING, fmt, ## args) +struct idxd_dmadev { + /* counters to track the batches */ + unsigned short max_batches; + unsigned short batch_idx_read; + unsigned short batch_idx_write; + + /* track descriptors and handles */ + unsigned short desc_ring_mask; + unsigned short ids_avail; /* handles for ops completed */ + unsigned short ids_returned; /* the read pointer for hdls/desc rings */ + unsigned short batch_start; /* start+size == write pointer for hdls/desc */ + unsigned short batch_size; + + void *portal; /* address to write the batch descriptor */ + + struct idxd_completion *batch_comp_ring; + unsigned short *batch_idx_ring; /* store where each batch ends */ + + struct rte_dmadev_stats stats; + + rte_iova_t batch_iova; /* base address of the batch comp ring */ + rte_iova_t desc_iova; /* base address of desc ring, needed for completions */ + + unsigned short max_batch_size; + + struct rte_dmadev *dmadev; + struct rte_dmadev_vchan_conf qcfg; + uint8_t sva_support; + uint8_t qid; + + union { + struct { + unsigned int dsa_id; + } bus; + } u; +}; + +int idxd_dmadev_create(const char *name, struct rte_device *dev, + const struct idxd_dmadev *base_idxd, const struct rte_dmadev_ops *ops); + #endif /* _IDXD_INTERNAL_H_ */ diff --git a/drivers/dma/idxd/meson.build b/drivers/dma/idxd/meson.build index f1fea000a7..81150e6f25 100644 --- a/drivers/dma/idxd/meson.build +++ b/drivers/dma/idxd/meson.build @@ -4,5 +4,6 @@ deps += ['bus_pci'] sources = files( 'idxd_bus.c', + 'idxd_common.c', 'idxd_pci.c' ) \ No newline at end of file From patchwork Fri Aug 27 17:20:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97476 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5238CA0548; Fri, 27 Aug 2021 19:21:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D8C694126C; Fri, 27 Aug 2021 19:21:21 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 996D241262 for ; Fri, 27 Aug 2021 19:21:19 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235919" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235919" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009487" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:17 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:41 +0000 Message-Id: <20210827172048.558704-7-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 06/13] dma/idxd: create dmadev instances on pci probe X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When a suitable device is found during the PCI probe, create a dmadev instance for each HW queue. HW definitions required are also included. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- drivers/dma/idxd/idxd_hw_defs.h | 71 ++++++++ drivers/dma/idxd/idxd_internal.h | 14 ++ drivers/dma/idxd/idxd_pci.c | 268 ++++++++++++++++++++++++++++++- 3 files changed, 350 insertions(+), 3 deletions(-) create mode 100644 drivers/dma/idxd/idxd_hw_defs.h diff --git a/drivers/dma/idxd/idxd_hw_defs.h b/drivers/dma/idxd/idxd_hw_defs.h new file mode 100644 index 0000000000..ea627cba6d --- /dev/null +++ b/drivers/dma/idxd/idxd_hw_defs.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021 Intel Corporation + */ + +#ifndef _IDXD_HW_DEFS_H_ +#define _IDXD_HW_DEFS_H_ + +/*** Definitions for Intel(R) Data Streaming Accelerator ***/ + +#define IDXD_CMD_SHIFT 20 +enum rte_idxd_cmds { + idxd_enable_dev = 1, + idxd_disable_dev, + idxd_drain_all, + idxd_abort_all, + idxd_reset_device, + idxd_enable_wq, + idxd_disable_wq, + idxd_drain_wq, + idxd_abort_wq, + idxd_reset_wq, +}; + +/* General bar0 registers */ +struct rte_idxd_bar0 { + uint32_t __rte_cache_aligned version; /* offset 0x00 */ + uint64_t __rte_aligned(0x10) gencap; /* offset 0x10 */ + uint64_t __rte_aligned(0x10) wqcap; /* offset 0x20 */ + uint64_t __rte_aligned(0x10) grpcap; /* offset 0x30 */ + uint64_t __rte_aligned(0x08) engcap; /* offset 0x38 */ + uint64_t __rte_aligned(0x10) opcap; /* offset 0x40 */ + uint64_t __rte_aligned(0x20) offsets[2]; /* offset 0x60 */ + uint32_t __rte_aligned(0x20) gencfg; /* offset 0x80 */ + uint32_t __rte_aligned(0x08) genctrl; /* offset 0x88 */ + uint32_t __rte_aligned(0x10) gensts; /* offset 0x90 */ + uint32_t __rte_aligned(0x08) intcause; /* offset 0x98 */ + uint32_t __rte_aligned(0x10) cmd; /* offset 0xA0 */ + uint32_t __rte_aligned(0x08) cmdstatus; /* offset 0xA8 */ + uint64_t __rte_aligned(0x20) swerror[4]; /* offset 0xC0 */ +}; + +/* workqueue config is provided by array of uint32_t. */ +enum rte_idxd_wqcfg { + wq_size_idx, /* size is in first 32-bit value */ + wq_threshold_idx, /* WQ threshold second 32-bits */ + wq_mode_idx, /* WQ mode and other flags */ + wq_sizes_idx, /* WQ transfer and batch sizes */ + wq_occ_int_idx, /* WQ occupancy interrupt handle */ + wq_occ_limit_idx, /* WQ occupancy limit */ + wq_state_idx, /* WQ state and occupancy state */ +}; + +#define WQ_MODE_SHARED 0 +#define WQ_MODE_DEDICATED 1 +#define WQ_PRIORITY_SHIFT 4 +#define WQ_BATCH_SZ_SHIFT 5 +#define WQ_STATE_SHIFT 30 +#define WQ_STATE_MASK 0x3 + +struct rte_idxd_grpcfg { + uint64_t grpwqcfg[4] __rte_cache_aligned; /* 64-byte register set */ + uint64_t grpengcfg; /* offset 32 */ + uint32_t grpflags; /* offset 40 */ +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + +#endif diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index 99ab2df925..85c400c9ec 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -5,6 +5,8 @@ #ifndef _IDXD_INTERNAL_H_ #define _IDXD_INTERNAL_H_ +#include "idxd_hw_defs.h" + /** * @file idxd_internal.h * @@ -24,6 +26,16 @@ extern int idxd_pmd_logtype; #define IDXD_PMD_ERR(fmt, args...) IDXD_PMD_LOG(ERR, fmt, ## args) #define IDXD_PMD_WARN(fmt, args...) IDXD_PMD_LOG(WARNING, fmt, ## args) +struct idxd_pci_common { + rte_spinlock_t lk; + + uint8_t wq_cfg_sz; + volatile struct rte_idxd_bar0 *regs; + volatile uint32_t *wq_regs_base; + volatile struct rte_idxd_grpcfg *grp_regs; + volatile void *portals; +}; + struct idxd_dmadev { /* counters to track the batches */ unsigned short max_batches; @@ -58,6 +70,8 @@ struct idxd_dmadev { struct { unsigned int dsa_id; } bus; + + struct idxd_pci_common *pci; } u; }; diff --git a/drivers/dma/idxd/idxd_pci.c b/drivers/dma/idxd/idxd_pci.c index 79e4aadcab..d13bcea4d2 100644 --- a/drivers/dma/idxd/idxd_pci.c +++ b/drivers/dma/idxd/idxd_pci.c @@ -3,6 +3,9 @@ */ #include +#include +#include +#include #include "idxd_internal.h" @@ -16,17 +19,276 @@ const struct rte_pci_id pci_id_idxd_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static inline int +idxd_pci_dev_command(struct idxd_dmadev *idxd, enum rte_idxd_cmds command) +{ + uint8_t err_code; + uint16_t qid = idxd->qid; + int i = 0; + + if (command >= idxd_disable_wq && command <= idxd_reset_wq) + qid = (1 << qid); + rte_spinlock_lock(&idxd->u.pci->lk); + idxd->u.pci->regs->cmd = (command << IDXD_CMD_SHIFT) | qid; + + do { + rte_pause(); + err_code = idxd->u.pci->regs->cmdstatus; + if (++i >= 1000) { + IDXD_PMD_ERR("Timeout waiting for command response from HW"); + rte_spinlock_unlock(&idxd->u.pci->lk); + return err_code; + } + } while (idxd->u.pci->regs->cmdstatus & CMDSTATUS_ACTIVE_MASK); + rte_spinlock_unlock(&idxd->u.pci->lk); + + return err_code & CMDSTATUS_ERR_MASK; +} + +static uint32_t * +idxd_get_wq_cfg(struct idxd_pci_common *pci, uint8_t wq_idx) +{ + return RTE_PTR_ADD(pci->wq_regs_base, + (uintptr_t)wq_idx << (5 + pci->wq_cfg_sz)); +} + +static int +idxd_is_wq_enabled(struct idxd_dmadev *idxd) +{ + uint32_t state = idxd_get_wq_cfg(idxd->u.pci, idxd->qid)[wq_state_idx]; + return ((state >> WQ_STATE_SHIFT) & WQ_STATE_MASK) == 0x1; +} + +static const struct rte_dmadev_ops idxd_pci_ops = { + +}; + +/* each portal uses 4 x 4k pages */ +#define IDXD_PORTAL_SIZE (4096 * 4) + +static int +init_pci_device(struct rte_pci_device *dev, struct idxd_dmadev *idxd, + unsigned int max_queues) +{ + struct idxd_pci_common *pci; + uint8_t nb_groups, nb_engines, nb_wqs; + uint16_t grp_offset, wq_offset; /* how far into bar0 the regs are */ + uint16_t wq_size, total_wq_size; + uint8_t lg2_max_batch, lg2_max_copy_size; + unsigned int i, err_code; + + pci = malloc(sizeof(*pci)); + if (pci == NULL) { + IDXD_PMD_ERR("%s: Can't allocate memory", __func__); + goto err; + } + rte_spinlock_init(&pci->lk); + + /* assign the bar registers, and then configure device */ + pci->regs = dev->mem_resource[0].addr; + grp_offset = (uint16_t)pci->regs->offsets[0]; + pci->grp_regs = RTE_PTR_ADD(pci->regs, grp_offset * 0x100); + wq_offset = (uint16_t)(pci->regs->offsets[0] >> 16); + pci->wq_regs_base = RTE_PTR_ADD(pci->regs, wq_offset * 0x100); + pci->portals = dev->mem_resource[2].addr; + pci->wq_cfg_sz = (pci->regs->wqcap >> 24) & 0x0F; + + /* sanity check device status */ + if (pci->regs->gensts & GENSTS_DEV_STATE_MASK) { + /* need function-level-reset (FLR) or is enabled */ + IDXD_PMD_ERR("Device status is not disabled, cannot init"); + goto err; + } + if (pci->regs->cmdstatus & CMDSTATUS_ACTIVE_MASK) { + /* command in progress */ + IDXD_PMD_ERR("Device has a command in progress, cannot init"); + goto err; + } + + /* read basic info about the hardware for use when configuring */ + nb_groups = (uint8_t)pci->regs->grpcap; + nb_engines = (uint8_t)pci->regs->engcap; + nb_wqs = (uint8_t)(pci->regs->wqcap >> 16); + total_wq_size = (uint16_t)pci->regs->wqcap; + lg2_max_copy_size = (uint8_t)(pci->regs->gencap >> 16) & 0x1F; + lg2_max_batch = (uint8_t)(pci->regs->gencap >> 21) & 0x0F; + + IDXD_PMD_DEBUG("nb_groups = %u, nb_engines = %u, nb_wqs = %u", + nb_groups, nb_engines, nb_wqs); + + /* zero out any old config */ + for (i = 0; i < nb_groups; i++) { + pci->grp_regs[i].grpengcfg = 0; + pci->grp_regs[i].grpwqcfg[0] = 0; + } + for (i = 0; i < nb_wqs; i++) + idxd_get_wq_cfg(pci, i)[0] = 0; + + /* limit queues if necessary */ + if (max_queues != 0 && nb_wqs > max_queues) { + nb_wqs = max_queues; + if (nb_engines > max_queues) + nb_engines = max_queues; + if (nb_groups > max_queues) + nb_engines = max_queues; + IDXD_PMD_DEBUG("Limiting queues to %u", nb_wqs); + } + + /* put each engine into a separate group to avoid reordering */ + if (nb_groups > nb_engines) + nb_groups = nb_engines; + if (nb_groups < nb_engines) + nb_engines = nb_groups; + + /* assign engines to groups, round-robin style */ + for (i = 0; i < nb_engines; i++) { + IDXD_PMD_DEBUG("Assigning engine %u to group %u", + i, i % nb_groups); + pci->grp_regs[i % nb_groups].grpengcfg |= (1ULL << i); + } + + /* now do the same for queues and give work slots to each queue */ + wq_size = total_wq_size / nb_wqs; + IDXD_PMD_DEBUG("Work queue size = %u, max batch = 2^%u, max copy = 2^%u", + wq_size, lg2_max_batch, lg2_max_copy_size); + for (i = 0; i < nb_wqs; i++) { + /* add engine "i" to a group */ + IDXD_PMD_DEBUG("Assigning work queue %u to group %u", + i, i % nb_groups); + pci->grp_regs[i % nb_groups].grpwqcfg[0] |= (1ULL << i); + /* now configure it, in terms of size, max batch, mode */ + idxd_get_wq_cfg(pci, i)[wq_size_idx] = wq_size; + idxd_get_wq_cfg(pci, i)[wq_mode_idx] = (1 << WQ_PRIORITY_SHIFT) | + WQ_MODE_DEDICATED; + idxd_get_wq_cfg(pci, i)[wq_sizes_idx] = lg2_max_copy_size | + (lg2_max_batch << WQ_BATCH_SZ_SHIFT); + } + + /* dump the group configuration to output */ + for (i = 0; i < nb_groups; i++) { + IDXD_PMD_DEBUG("## Group %d", i); + IDXD_PMD_DEBUG(" GRPWQCFG: %"PRIx64, pci->grp_regs[i].grpwqcfg[0]); + IDXD_PMD_DEBUG(" GRPENGCFG: %"PRIx64, pci->grp_regs[i].grpengcfg); + IDXD_PMD_DEBUG(" GRPFLAGS: %"PRIx32, pci->grp_regs[i].grpflags); + } + + idxd->u.pci = pci; + idxd->max_batches = wq_size; + + /* enable the device itself */ + err_code = idxd_pci_dev_command(idxd, idxd_enable_dev); + if (err_code) { + IDXD_PMD_ERR("Error enabling device: code %#x", err_code); + return err_code; + } + IDXD_PMD_DEBUG("IDXD Device enabled OK"); + + return nb_wqs; + +err: + free(pci); + return -1; +} + static int idxd_dmadev_probe_pci(struct rte_pci_driver *drv, struct rte_pci_device *dev) { - int ret = 0; + struct idxd_dmadev idxd = {0}; + uint8_t nb_wqs; + int qid, ret = 0; char name[PCI_PRI_STR_SIZE]; + unsigned int max_queues = 0; rte_pci_device_name(&dev->addr, name, sizeof(name)); IDXD_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); dev->device.driver = &drv->driver; - return ret; + if (dev->device.devargs && dev->device.devargs->args[0] != '\0') { + /* if the number of devargs grows beyond just 1, use rte_kvargs */ + if (sscanf(dev->device.devargs->args, + "max_queues=%u", &max_queues) != 1) { + IDXD_PMD_ERR("Invalid device parameter: '%s'", + dev->device.devargs->args); + return -1; + } + } + + ret = init_pci_device(dev, &idxd, max_queues); + if (ret < 0) { + IDXD_PMD_ERR("Error initializing PCI hardware"); + return ret; + } + nb_wqs = (uint8_t)ret; + + /* set up one device for each queue */ + for (qid = 0; qid < nb_wqs; qid++) { + char qname[32]; + + /* add the queue number to each device name */ + snprintf(qname, sizeof(qname), "%s-q%d", name, qid); + idxd.qid = qid; + idxd.portal = RTE_PTR_ADD(idxd.u.pci->portals, + qid * IDXD_PORTAL_SIZE); + if (idxd_is_wq_enabled(&idxd)) + IDXD_PMD_ERR("Error, WQ %u seems enabled", qid); + ret = idxd_dmadev_create(qname, &dev->device, + &idxd, &idxd_pci_ops); + if (ret != 0) { + IDXD_PMD_ERR("Failed to create dmadev %s", name); + if (qid == 0) /* if no devices using this, free pci */ + free(idxd.u.pci); + return ret; + } + } + + return 0; +} + +static int +idxd_dmadev_destroy(const char *name) +{ + int ret; + uint8_t err_code; + struct rte_dmadev *rdev; + struct idxd_dmadev *idxd; + + if (!name) { + IDXD_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + rdev = rte_dmadev_get_device_by_name(name); + if (!rdev) { + IDXD_PMD_ERR("Invalid device name (%s)", name); + return -EINVAL; + } + + idxd = rdev->dev_private; + if (!idxd) { + IDXD_PMD_ERR("Error getting dev_private"); + return -EINVAL; + } + + /* disable the device */ + err_code = idxd_pci_dev_command(idxd, idxd_disable_dev); + if (err_code) { + IDXD_PMD_ERR("Error disabling device: code %#x", err_code); + return err_code; + } + IDXD_PMD_DEBUG("IDXD Device disabled OK"); + + /* free device memory */ + IDXD_PMD_DEBUG("Freeing device driver memory"); + rdev->dev_private = NULL; + rte_free(idxd->batch_idx_ring); + rte_free(idxd->desc_ring); + + /* rte_dmadev_close is called by pmd_release */ + ret = rte_dmadev_pmd_release(rdev); + if (ret) + IDXD_PMD_DEBUG("Device cleanup failed"); + + return 0; } static int @@ -39,7 +301,7 @@ idxd_dmadev_remove_pci(struct rte_pci_device *dev) IDXD_PMD_INFO("Closing %s on NUMA node %d", name, dev->device.numa_node); - return 0; + return idxd_dmadev_destroy(name); } struct rte_pci_driver idxd_pmd_drv_pci = { From patchwork Fri Aug 27 17:20:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97477 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 47830A0548; Fri, 27 Aug 2021 19:21:58 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5849B4127B; Fri, 27 Aug 2021 19:21:23 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id A1011411D3 for ; Fri, 27 Aug 2021 19:21:21 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235924" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235924" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009510" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:19 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:42 +0000 Message-Id: <20210827172048.558704-8-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 07/13] dma/idxd: add datapath structures X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add data structures required for the data path for IDXD devices. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- drivers/dma/idxd/idxd_bus.c | 1 + drivers/dma/idxd/idxd_common.c | 33 ++++++++++++++++++ drivers/dma/idxd/idxd_hw_defs.h | 59 ++++++++++++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 3 ++ drivers/dma/idxd/idxd_pci.c | 2 +- 5 files changed, 97 insertions(+), 1 deletion(-) diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c index 0f33500dfc..dc11f829fd 100644 --- a/drivers/dma/idxd/idxd_bus.c +++ b/drivers/dma/idxd/idxd_bus.c @@ -94,6 +94,7 @@ idxd_dev_close(struct rte_dmadev *dev) static const struct rte_dmadev_ops idxd_vdev_ops = { .dev_close = idxd_dev_close, + .dev_dump = idxd_dump, }; static void * diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c index 7770b2e264..9490439fdc 100644 --- a/drivers/dma/idxd/idxd_common.c +++ b/drivers/dma/idxd/idxd_common.c @@ -10,6 +10,35 @@ #define IDXD_PMD_NAME_STR "dmadev_idxd" +int +idxd_dump(const struct rte_dmadev *dev, FILE *f) +{ + struct idxd_dmadev *idxd = dev->dev_private; + unsigned int i; + + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Portal: %p\n", idxd->portal); + fprintf(f, " Config: { ring_size: %u }\n", + idxd->qcfg.nb_desc); + fprintf(f, " Batch ring (sz = %u, max_batches = %u):\n\t", + idxd->max_batches + 1, idxd->max_batches); + for (i = 0; i <= idxd->max_batches; i++) { + fprintf(f, " %u ", idxd->batch_idx_ring[i]); + if (i == idxd->batch_idx_read && i == idxd->batch_idx_write) + fprintf(f, "[rd ptr, wr ptr] "); + else if (i == idxd->batch_idx_read) + fprintf(f, "[rd ptr] "); + else if (i == idxd->batch_idx_write) + fprintf(f, "[wr ptr] "); + if (i == idxd->max_batches) + fprintf(f, "\n"); + } + + fprintf(f, " Curr batch: start = %u, size = %u\n", idxd->batch_start, idxd->batch_size); + fprintf(f, " IDS: avail = %u, returned: %u\n", idxd->ids_avail, idxd->ids_returned); + return 0; +} + int idxd_dmadev_create(const char *name, struct rte_device *dev, const struct idxd_dmadev *base_idxd, @@ -19,6 +48,10 @@ idxd_dmadev_create(const char *name, struct rte_device *dev, struct rte_dmadev *dmadev = NULL; int ret = 0; + RTE_BUILD_BUG_ON(sizeof(struct idxd_hw_desc) != 64); + RTE_BUILD_BUG_ON(offsetof(struct idxd_hw_desc, size) != 32); + RTE_BUILD_BUG_ON(sizeof(struct idxd_completion) != 32); + if (!name) { IDXD_PMD_ERR("Invalid name of the device!"); ret = -EINVAL; diff --git a/drivers/dma/idxd/idxd_hw_defs.h b/drivers/dma/idxd/idxd_hw_defs.h index ea627cba6d..f6b9c25981 100644 --- a/drivers/dma/idxd/idxd_hw_defs.h +++ b/drivers/dma/idxd/idxd_hw_defs.h @@ -5,6 +5,65 @@ #ifndef _IDXD_HW_DEFS_H_ #define _IDXD_HW_DEFS_H_ +/* + * Defines used in the data path for interacting with IDXD hardware. + */ +#define IDXD_CMD_OP_SHIFT 24 +enum rte_idxd_ops { + idxd_op_nop = 0, + idxd_op_batch, + idxd_op_drain, + idxd_op_memmove, + idxd_op_fill +}; + +#define IDXD_FLAG_FENCE (1 << 0) +#define IDXD_FLAG_COMPLETION_ADDR_VALID (1 << 2) +#define IDXD_FLAG_REQUEST_COMPLETION (1 << 3) +#define IDXD_FLAG_CACHE_CONTROL (1 << 8) + +/** + * Hardware descriptor used by DSA hardware, for both bursts and + * for individual operations. + */ +struct idxd_hw_desc { + uint32_t pasid; + uint32_t op_flags; + rte_iova_t completion; + + RTE_STD_C11 + union { + rte_iova_t src; /* source address for copy ops etc. */ + rte_iova_t desc_addr; /* descriptor pointer for batch */ + }; + rte_iova_t dst; + + uint32_t size; /* length of data for op, or batch size */ + + uint16_t intr_handle; /* completion interrupt handle */ + + /* remaining 26 bytes are reserved */ + uint16_t __reserved[13]; +} __rte_aligned(64); + +#define IDXD_COMP_STATUS_INCOMPLETE 0 +#define IDXD_COMP_STATUS_SUCCESS 1 +#define IDXD_COMP_STATUS_INVALID_SIZE 0x13 +#define IDXD_COMP_STATUS_SKIPPED 0xFF /* not official IDXD error, but needed as placeholder */ + +/** + * Completion record structure written back by DSA + */ +struct idxd_completion { + uint8_t status; + uint8_t result; + /* 16-bits pad here */ + uint32_t completed_size; /* data length, or descriptors for batch */ + + rte_iova_t fault_address; + uint32_t invalid_flags; +} __rte_aligned(32); + /*** Definitions for Intel(R) Data Streaming Accelerator ***/ #define IDXD_CMD_SHIFT 20 diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index 85c400c9ec..09285b4e96 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -37,6 +37,8 @@ struct idxd_pci_common { }; struct idxd_dmadev { + struct idxd_hw_desc *desc_ring; + /* counters to track the batches */ unsigned short max_batches; unsigned short batch_idx_read; @@ -77,5 +79,6 @@ struct idxd_dmadev { int idxd_dmadev_create(const char *name, struct rte_device *dev, const struct idxd_dmadev *base_idxd, const struct rte_dmadev_ops *ops); +int idxd_dump(const struct rte_dmadev *dev, FILE *f); #endif /* _IDXD_INTERNAL_H_ */ diff --git a/drivers/dma/idxd/idxd_pci.c b/drivers/dma/idxd/idxd_pci.c index d13bcea4d2..ddb4e90447 100644 --- a/drivers/dma/idxd/idxd_pci.c +++ b/drivers/dma/idxd/idxd_pci.c @@ -60,7 +60,7 @@ idxd_is_wq_enabled(struct idxd_dmadev *idxd) } static const struct rte_dmadev_ops idxd_pci_ops = { - + .dev_dump = idxd_dump, }; /* each portal uses 4 x 4k pages */ From patchwork Fri Aug 27 17:20:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97478 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1D536A0548; Fri, 27 Aug 2021 19:22:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 82D2441281; Fri, 27 Aug 2021 19:21:24 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 59E1D4127C for ; Fri, 27 Aug 2021 19:21:23 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235932" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235932" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009536" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:21 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:43 +0000 Message-Id: <20210827172048.558704-9-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 08/13] dma/idxd: add configure and info_get functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add functions for device configuration. The info_get function is included here since it can be useful for checking successful configuration. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- doc/guides/dmadevs/idxd.rst | 33 ++++++++++++++++ drivers/dma/idxd/idxd_bus.c | 3 ++ drivers/dma/idxd/idxd_common.c | 67 ++++++++++++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 5 +++ drivers/dma/idxd/idxd_pci.c | 3 ++ 5 files changed, 111 insertions(+) diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst index ce33e2857a..66bc9fe744 100644 --- a/doc/guides/dmadevs/idxd.rst +++ b/doc/guides/dmadevs/idxd.rst @@ -120,3 +120,36 @@ use a subset of configured queues. Once probed successfully, irrespective of kernel driver, the device will appear as a ``dmadev``, that is a "DMA device type" inside DPDK, and can be accessed using APIs from the ``rte_dmadev`` library. + +Using IDXD DMAdev Devices +-------------------------- + +To use the devices from an application, the dmadev API can be used. + +Getting Device Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Basic information about each dmadev device can be queried using the +``rte_dmadev_info_get()`` API. This will return basic device information such as +the ``rte_device`` structure, device capabilities and other device specific values. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IDXD dmadev device is done using the ``rte_dmadev_configure()`` and +``rte_dmadev_vchan_setup`` APIs. The configurations are passed to these APIs using +the ``rte_dmadev_conf`` and ``rte_dmadev_vchan_conf`` structures, respectively. For +example, these can be used to configure the number of ``vchans`` per device, the +ring size, etc. The ring size must be a power of two, between 64 and 4096. + +The following code shows how the device is configured in +``test_dmadev.c``: + +.. literalinclude:: ../../../app/test/test_dmadev.c + :language: c + :start-after: Setup of the dmadev device. 8< + :end-before: >8 End of setup of the dmadev device. + :dedent: 1 + +Once configured, the device can then be made ready for use by calling the +``rte_dmadev_start()`` API. diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c index dc11f829fd..ad4f076bad 100644 --- a/drivers/dma/idxd/idxd_bus.c +++ b/drivers/dma/idxd/idxd_bus.c @@ -95,6 +95,9 @@ idxd_dev_close(struct rte_dmadev *dev) static const struct rte_dmadev_ops idxd_vdev_ops = { .dev_close = idxd_dev_close, .dev_dump = idxd_dump, + .dev_configure = idxd_configure, + .vchan_setup = idxd_vchan_setup, + .dev_info_get = idxd_info_get, }; static void * diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c index 9490439fdc..ea2c0b7f19 100644 --- a/drivers/dma/idxd/idxd_common.c +++ b/drivers/dma/idxd/idxd_common.c @@ -39,6 +39,73 @@ idxd_dump(const struct rte_dmadev *dev, FILE *f) return 0; } +int +idxd_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) +{ + struct idxd_dmadev *idxd = dev->dev_private; + + if (size < sizeof(*info)) + return -EINVAL; + + *info = (struct rte_dmadev_info) { + .device = dev->device, + .dev_capa = RTE_DMADEV_CAPA_MEM_TO_MEM | + RTE_DMADEV_CAPA_OPS_COPY | RTE_DMADEV_CAPA_OPS_FILL, + .max_vchans = 1, + .max_desc = 4096, + .min_desc = 64, + .nb_vchans = (idxd->desc_ring != NULL), /* returns 1 or 0 */ + }; + if (idxd->sva_support) + info->dev_capa |= RTE_DMADEV_CAPA_SVA; + return 0; +} + +int +idxd_configure(struct rte_dmadev *dev __rte_unused, const struct rte_dmadev_conf *dev_conf) +{ + if (dev_conf->nb_vchans != 1) + return -EINVAL; + return 0; +} + +int +idxd_vchan_setup(struct rte_dmadev *dev, uint16_t vchan __rte_unused, + const struct rte_dmadev_vchan_conf *qconf) +{ + struct idxd_dmadev *idxd = dev->dev_private; + uint16_t max_desc = qconf->nb_desc; + uint16_t i; + + idxd->qcfg = *qconf; + + if (!rte_is_power_of_2(max_desc)) + max_desc = rte_align32pow2(max_desc); + IDXD_PMD_DEBUG("DMA dev %u using %u descriptors", dev->data->dev_id, max_desc); + idxd->desc_ring_mask = max_desc - 1; + idxd->qcfg.nb_desc = max_desc; + + /* in case we are reconfiguring a device, free any existing memory */ + rte_free(idxd->desc_ring); + + /* allocate the descriptor ring at 2x size as batches can't wrap */ + idxd->desc_ring = rte_zmalloc(NULL, sizeof(*idxd->desc_ring) * max_desc * 2, 0); + if (idxd->desc_ring == NULL) + return -ENOMEM; + idxd->desc_iova = rte_mem_virt2iova(idxd->desc_ring); + + idxd->batch_idx_read = 0; + idxd->batch_idx_write = 0; + idxd->batch_start = 0; + idxd->batch_size = 0; + idxd->ids_returned = 0; + idxd->ids_avail = 0; + + for (i = 0; i < max_desc * 2; i++) + idxd->desc_ring[i].completion = __desc_idx_to_iova(idxd, i & (max_desc - 1)); + return 0; +} + int idxd_dmadev_create(const char *name, struct rte_device *dev, const struct idxd_dmadev *base_idxd, diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index 09285b4e96..18fc65d00c 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -80,5 +80,10 @@ struct idxd_dmadev { int idxd_dmadev_create(const char *name, struct rte_device *dev, const struct idxd_dmadev *base_idxd, const struct rte_dmadev_ops *ops); int idxd_dump(const struct rte_dmadev *dev, FILE *f); +int idxd_configure(struct rte_dmadev *dev, const struct rte_dmadev_conf *dev_conf); +int idxd_vchan_setup(struct rte_dmadev *dev, uint16_t vchan, + const struct rte_dmadev_vchan_conf *qconf); +int idxd_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *dev_info, + uint32_t size); #endif /* _IDXD_INTERNAL_H_ */ diff --git a/drivers/dma/idxd/idxd_pci.c b/drivers/dma/idxd/idxd_pci.c index ddb4e90447..46daa13e69 100644 --- a/drivers/dma/idxd/idxd_pci.c +++ b/drivers/dma/idxd/idxd_pci.c @@ -61,6 +61,9 @@ idxd_is_wq_enabled(struct idxd_dmadev *idxd) static const struct rte_dmadev_ops idxd_pci_ops = { .dev_dump = idxd_dump, + .dev_configure = idxd_configure, + .vchan_setup = idxd_vchan_setup, + .dev_info_get = idxd_info_get, }; /* each portal uses 4 x 4k pages */ From patchwork Fri Aug 27 17:20:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97479 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B20C7A0548; Fri, 27 Aug 2021 19:22:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AB9F44124D; Fri, 27 Aug 2021 19:21:26 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id DAD9341274 for ; Fri, 27 Aug 2021 19:21:24 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235939" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235939" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009555" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:23 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:44 +0000 Message-Id: <20210827172048.558704-10-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 09/13] dma/idxd: add start and stop functions for pci devices X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add device start/stop functions for DSA devices bound to vfio. For devices bound to the IDXD kernel driver, these are not required since the IDXD kernel driver takes care of this. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- drivers/dma/idxd/idxd_pci.c | 52 +++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/drivers/dma/idxd/idxd_pci.c b/drivers/dma/idxd/idxd_pci.c index 46daa13e69..9959c81a1e 100644 --- a/drivers/dma/idxd/idxd_pci.c +++ b/drivers/dma/idxd/idxd_pci.c @@ -59,11 +59,63 @@ idxd_is_wq_enabled(struct idxd_dmadev *idxd) return ((state >> WQ_STATE_SHIFT) & WQ_STATE_MASK) == 0x1; } +static int +idxd_pci_dev_stop(struct rte_dmadev *dev) +{ + struct idxd_dmadev *idxd = dev->dev_private; + uint8_t err_code; + + if (!idxd_is_wq_enabled(idxd)) { + IDXD_PMD_ERR("Work queue %d already disabled", idxd->qid); + return -EALREADY; + } + + err_code = idxd_pci_dev_command(idxd, idxd_disable_wq); + if (err_code || idxd_is_wq_enabled(idxd)) { + IDXD_PMD_ERR("Failed disabling work queue %d, error code: %#x", + idxd->qid, err_code); + return -err_code; + } + IDXD_PMD_DEBUG("Work queue %d disabled OK", idxd->qid); + + return 0; +} + +static int +idxd_pci_dev_start(struct rte_dmadev *dev) +{ + struct idxd_dmadev *idxd = dev->dev_private; + uint8_t err_code; + + if (idxd_is_wq_enabled(idxd)) { + IDXD_PMD_WARN("WQ %d already enabled", idxd->qid); + return 0; + } + + if (idxd->desc_ring == NULL) { + IDXD_PMD_ERR("WQ %d has not been fully configured", idxd->qid); + return -EINVAL; + } + + err_code = idxd_pci_dev_command(idxd, idxd_enable_wq); + if (err_code || !idxd_is_wq_enabled(idxd)) { + IDXD_PMD_ERR("Failed enabling work queue %d, error code: %#x", + idxd->qid, err_code); + return err_code == 0 ? -1 : err_code; + } + + IDXD_PMD_DEBUG("Work queue %d enabled OK", idxd->qid); + + return 0; +} + static const struct rte_dmadev_ops idxd_pci_ops = { .dev_dump = idxd_dump, .dev_configure = idxd_configure, .vchan_setup = idxd_vchan_setup, .dev_info_get = idxd_info_get, + .dev_start = idxd_pci_dev_start, + .dev_stop = idxd_pci_dev_stop, }; /* each portal uses 4 x 4k pages */ From patchwork Fri Aug 27 17:20:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97480 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2EE5DA0548; Fri, 27 Aug 2021 19:22:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 23BCF4128E; Fri, 27 Aug 2021 19:21:29 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id E1FDF41286 for ; Fri, 27 Aug 2021 19:21:26 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235946" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235946" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009573" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:24 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:45 +0000 Message-Id: <20210827172048.558704-11-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 10/13] dma/idxd: add data-path job submission functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add data path functions for enqueuing and submitting operations to DSA devices. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- doc/guides/dmadevs/idxd.rst | 64 ++++++++++++++ drivers/dma/idxd/idxd_common.c | 138 +++++++++++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 5 ++ drivers/dma/idxd/meson.build | 1 + 4 files changed, 208 insertions(+) diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst index 66bc9fe744..0c4c105e0f 100644 --- a/doc/guides/dmadevs/idxd.rst +++ b/doc/guides/dmadevs/idxd.rst @@ -153,3 +153,67 @@ The following code shows how the device is configured in Once configured, the device can then be made ready for use by calling the ``rte_dmadev_start()`` API. + +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +To perform data copies using IDXD dmadev devices, descriptors should be enqueued +using the ``rte_dmadev_copy()`` API. The HW can be triggered to perform the copy +in two ways, either via a ``RTE_DMA_OP_FLAG_SUBMIT`` flag or by calling +``rte_dmadev_submit()``. Once copies have been completed, the completion will +be reported back when the application calls ``rte_dmadev_completed()`` or +``rte_dmadev_completed_status()``. The latter will also report the status of each +completed operation. + +The ``rte_dmadev_copy()`` function enqueues a single copy to the device ring for +copying at a later point. The parameters to that function include the IOVA addresses +of both the source and destination buffers, as well as the length of the copy. + +The ``rte_dmadev_copy()`` function enqueues a copy operation on the device ring. +If the ``RTE_DMA_OP_FLAG_SUBMIT`` flag is set when calling ``rte_dmadev_copy()``, +the device hardware will be informed of the elements. Alternatively, if the flag +is not set, the application need to call the ``rte_dmadev_submit()`` function to +notify the device hardware. Once the device hardware is informed of the elements +enqueued on the ring, and the device will begin to process them. It is expected +that, for efficiency reasons, a burst of operations will be enqueued to the +device via multiple enqueue calls between calls to the ``rte_dmadev_submit()`` +function. + +The following code from demonstrates how to enqueue a burst of copies to the +device and start the hardware processing of them: + +.. code-block:: C + + struct rte_mbuf *srcs[COMP_BURST_SZ], *dsts[COMP_BURST_SZ]; + unsigned int i; + + for (i = 0; i < RTE_DIM(srcs); i++) { + uint64_t *src_data; + + srcs[i] = rte_pktmbuf_alloc(pool); + dsts[i] = rte_pktmbuf_alloc(pool); + src_data = rte_pktmbuf_mtod(srcs[i], uint64_t *); + if (srcs[i] == NULL || dsts[i] == NULL) { + PRINT_ERR("Error allocating buffers\n"); + return -1; + } + + for (j = 0; j < COPY_LEN/sizeof(uint64_t); j++) + src_data[j] = rte_rand(); + + if (rte_dmadev_copy(dev_id, vchan, srcs[i]->buf_iova + srcs[i]->data_off, + dsts[i]->buf_iova + dsts[i]->data_off, COPY_LEN, 0) < 0) { + PRINT_ERR("Error with rte_dmadev_copy for buffer %u\n", i); + return -1; + } + } + rte_dmadev_submit(dev_id, vchan); + +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The IDXD driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_dmadev_fill()`` function rather than the +``rte_dmadev_copy()`` function. diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c index ea2c0b7f19..e2ef7b3b95 100644 --- a/drivers/dma/idxd/idxd_common.c +++ b/drivers/dma/idxd/idxd_common.c @@ -2,14 +2,148 @@ * Copyright 2021 Intel Corporation */ +#include + #include #include #include +#include #include "idxd_internal.h" #define IDXD_PMD_NAME_STR "dmadev_idxd" +static __rte_always_inline rte_iova_t +__desc_idx_to_iova(struct idxd_dmadev *idxd, uint16_t n) +{ + return idxd->desc_iova + (n * sizeof(struct idxd_hw_desc)); +} + +static __rte_always_inline void +__idxd_movdir64b(volatile void *dst, const struct idxd_hw_desc *src) +{ + asm volatile (".byte 0x66, 0x0f, 0x38, 0xf8, 0x02" + : + : "a" (dst), "d" (src) + : "memory"); +} + +static __rte_always_inline void +__submit(struct idxd_dmadev *idxd) +{ + /* TODO have flag setting indicating polling on same core as submission */ + rte_prefetch1(&idxd->batch_comp_ring[idxd->batch_idx_read]); + + if (idxd->batch_size == 0) + return; + + /* write completion to batch comp ring */ + rte_iova_t comp_addr = idxd->batch_iova + + (idxd->batch_idx_write * sizeof(struct idxd_completion)); + + if (idxd->batch_size == 1) { + /* submit batch directly */ + struct idxd_hw_desc desc = + idxd->desc_ring[idxd->batch_start & idxd->desc_ring_mask]; + desc.completion = comp_addr; + desc.op_flags |= IDXD_FLAG_REQUEST_COMPLETION; + _mm_sfence(); /* fence before writing desc to device */ + __idxd_movdir64b(idxd->portal, &desc); + } else { + const struct idxd_hw_desc batch_desc = { + .op_flags = (idxd_op_batch << IDXD_CMD_OP_SHIFT) | + IDXD_FLAG_COMPLETION_ADDR_VALID | + IDXD_FLAG_REQUEST_COMPLETION, + .desc_addr = __desc_idx_to_iova(idxd, + idxd->batch_start & idxd->desc_ring_mask), + .completion = comp_addr, + .size = idxd->batch_size, + }; + _mm_sfence(); /* fence before writing desc to device */ + __idxd_movdir64b(idxd->portal, &batch_desc); + } + + if (++idxd->batch_idx_write > idxd->max_batches) + idxd->batch_idx_write = 0; + + idxd->batch_start += idxd->batch_size; + idxd->batch_size = 0; + idxd->batch_idx_ring[idxd->batch_idx_write] = idxd->batch_start; + _mm256_store_si256((void *)&idxd->batch_comp_ring[idxd->batch_idx_write], + _mm256_setzero_si256()); +} + +static __rte_always_inline int +__idxd_write_desc(struct rte_dmadev *dev, + const uint32_t op_flags, + const rte_iova_t src, + const rte_iova_t dst, + const uint32_t size, + const uint32_t flags) +{ + struct idxd_dmadev *idxd = dev->dev_private; + uint16_t mask = idxd->desc_ring_mask; + uint16_t job_id = idxd->batch_start + idxd->batch_size; + /* we never wrap batches, so we only mask the start and allow start+size to overflow */ + uint16_t write_idx = (idxd->batch_start & mask) + idxd->batch_size; + + /* first check batch ring space then desc ring space */ + if ((idxd->batch_idx_read == 0 && idxd->batch_idx_write == idxd->max_batches) || + idxd->batch_idx_write + 1 == idxd->batch_idx_read) + goto failed; + if (((write_idx + 1) & mask) == (idxd->ids_returned & mask)) + goto failed; + + /* write desc. Note: descriptors don't wrap, but the completion address does */ + const uint64_t op_flags64 = (uint64_t)(op_flags | IDXD_FLAG_COMPLETION_ADDR_VALID) << 32; + const uint64_t comp_addr = __desc_idx_to_iova(idxd, write_idx & mask); + _mm256_store_si256((void *)&idxd->desc_ring[write_idx], + _mm256_set_epi64x(dst, src, comp_addr, op_flags64)); + _mm256_store_si256((void *)&idxd->desc_ring[write_idx].size, + _mm256_set_epi64x(0, 0, 0, size)); + + idxd->batch_size++; + + rte_prefetch0_write(&idxd->desc_ring[write_idx + 1]); + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) + __submit(idxd); + + return job_id; + +failed: + return -1; +} + +int +idxd_enqueue_copy(struct rte_dmadev *dev, uint16_t qid __rte_unused, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + /* we can take advantage of the fact that the fence flag in dmadev and DSA are the same, + * but check it at compile time to be sure. + */ + RTE_BUILD_BUG_ON(RTE_DMA_OP_FLAG_FENCE != IDXD_FLAG_FENCE); + uint32_t memmove = (idxd_op_memmove << IDXD_CMD_OP_SHIFT) | + IDXD_FLAG_CACHE_CONTROL | (flags & IDXD_FLAG_FENCE); + return __idxd_write_desc(dev, memmove, src, dst, length, flags); +} + +int +idxd_enqueue_fill(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags) +{ + uint32_t fill = (idxd_op_fill << IDXD_CMD_OP_SHIFT) | + IDXD_FLAG_CACHE_CONTROL | (flags & IDXD_FLAG_FENCE); + return __idxd_write_desc(dev, fill, pattern, dst, length, flags); +} + +int +idxd_submit(struct rte_dmadev *dev, uint16_t qid __rte_unused) +{ + __submit(dev->dev_private); + return 0; +} + int idxd_dump(const struct rte_dmadev *dev, FILE *f) { @@ -135,6 +269,10 @@ idxd_dmadev_create(const char *name, struct rte_device *dev, dmadev->dev_ops = ops; dmadev->device = dev; + dmadev->copy = idxd_enqueue_copy; + dmadev->fill = idxd_enqueue_fill; + dmadev->submit = idxd_submit; + idxd = rte_malloc_socket(NULL, sizeof(struct idxd_dmadev), 0, dev->numa_node); if (idxd == NULL) { IDXD_PMD_ERR("Unable to allocate memory for device"); diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index 18fc65d00c..6a6c69fd61 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -85,5 +85,10 @@ int idxd_vchan_setup(struct rte_dmadev *dev, uint16_t vchan, const struct rte_dmadev_vchan_conf *qconf); int idxd_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *dev_info, uint32_t size); +int idxd_enqueue_copy(struct rte_dmadev *dev, uint16_t qid, rte_iova_t src, + rte_iova_t dst, unsigned int length, uint64_t flags); +int idxd_enqueue_fill(struct rte_dmadev *dev, uint16_t qid, uint64_t pattern, + rte_iova_t dst, unsigned int length, uint64_t flags); +int idxd_submit(struct rte_dmadev *dev, uint16_t qid); #endif /* _IDXD_INTERNAL_H_ */ diff --git a/drivers/dma/idxd/meson.build b/drivers/dma/idxd/meson.build index 81150e6f25..2de5130fd2 100644 --- a/drivers/dma/idxd/meson.build +++ b/drivers/dma/idxd/meson.build @@ -2,6 +2,7 @@ # Copyright(c) 2021 Intel Corporation deps += ['bus_pci'] +cflags += '-mavx2' # all platforms with idxd HW support AVX sources = files( 'idxd_bus.c', 'idxd_common.c', From patchwork Fri Aug 27 17:20:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97481 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6BC30A0548; Fri, 27 Aug 2021 19:22:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4D9B541289; Fri, 27 Aug 2021 19:21:31 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id E3ADF41258 for ; Fri, 27 Aug 2021 19:21:28 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235954" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235954" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009593" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:26 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:46 +0000 Message-Id: <20210827172048.558704-12-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 11/13] dma/idxd: add data-path job completion functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the data path functions for gathering completed operations. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- doc/guides/dmadevs/idxd.rst | 25 ++++ drivers/dma/idxd/idxd_common.c | 235 +++++++++++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 5 + 3 files changed, 265 insertions(+) diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst index 0c4c105e0f..8bf99ef453 100644 --- a/doc/guides/dmadevs/idxd.rst +++ b/doc/guides/dmadevs/idxd.rst @@ -209,6 +209,31 @@ device and start the hardware processing of them: } rte_dmadev_submit(dev_id, vchan); +To retrieve information about completed copies, ``rte_dmadev_completed()`` and +``rte_dmadev_completed_status()`` APIs should be used. ``rte_dmadev_completed()`` +will return the number of completed operations, along with the index of the last +successful completed operation and whether or not an error was encountered. If an +error was encounted, ``rte_dmadev_completed_status()`` must be used to kick the +device off to continue processing operations and also to gather the status of each +individual operations which is filled in to the ``status`` array provided as +parameter by the application. + +The following code shows how to retrieve the number of successfully completed +copies within a burst and then using ``rte_dmadev_completed_status()`` to check +which operation failed and kick off the device to continue processing operations: + +.. code-block:: C + + enum rte_dma_status_code status[COMP_BURST_SZ]; + uint16_t count, idx, status_count; + bool error = 0; + + count = rte_dmadev_completed(dev_id, vchan, COMP_BURST_SZ, &idx, &error); + + if (error){ + status_count = rte_dmadev_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status); + } + Filling an Area of Memory ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c index e2ef7b3b95..50b205d92f 100644 --- a/drivers/dma/idxd/idxd_common.c +++ b/drivers/dma/idxd/idxd_common.c @@ -144,6 +144,239 @@ idxd_submit(struct rte_dmadev *dev, uint16_t qid __rte_unused) return 0; } +static enum rte_dma_status_code +get_comp_status(struct idxd_completion *c) +{ + uint8_t st = c->status; + switch (st) { + /* successful descriptors are not written back normally */ + case IDXD_COMP_STATUS_INCOMPLETE: + case IDXD_COMP_STATUS_SUCCESS: + return RTE_DMA_STATUS_SUCCESSFUL; + case IDXD_COMP_STATUS_INVALID_SIZE: + return RTE_DMA_STATUS_INVALID_LENGTH; + case IDXD_COMP_STATUS_SKIPPED: + return RTE_DMA_STATUS_NOT_ATTEMPTED; + default: /* TODO - get more detail */ + return RTE_DMA_STATUS_ERROR_UNKNOWN; + } +} + +static __rte_always_inline int +batch_ok(struct idxd_dmadev *idxd, uint8_t max_ops, enum rte_dma_status_code *status) +{ + uint16_t ret; + uint8_t bstatus; + + if (max_ops == 0) + return 0; + + /* first check if there are any unreturned handles from last time */ + if (idxd->ids_avail != idxd->ids_returned) { + ret = RTE_MIN((uint16_t)(idxd->ids_avail - idxd->ids_returned), max_ops); + idxd->ids_returned += ret; + if (status) + memset(status, RTE_DMA_STATUS_SUCCESSFUL, ret * sizeof(*status)); + return ret; + } + + if (idxd->batch_idx_read == idxd->batch_idx_write) + return 0; + + bstatus = idxd->batch_comp_ring[idxd->batch_idx_read].status; + /* now check if next batch is complete and successful */ + if (bstatus == IDXD_COMP_STATUS_SUCCESS) { + /* since the batch idx ring stores the start of each batch, pre-increment to lookup + * start of next batch. + */ + if (++idxd->batch_idx_read > idxd->max_batches) + idxd->batch_idx_read = 0; + idxd->ids_avail = idxd->batch_idx_ring[idxd->batch_idx_read]; + + ret = RTE_MIN((uint16_t)(idxd->ids_avail - idxd->ids_returned), max_ops); + idxd->ids_returned += ret; + if (status) + memset(status, RTE_DMA_STATUS_SUCCESSFUL, ret * sizeof(*status)); + return ret; + } + /* check if batch is incomplete */ + else if (bstatus == IDXD_COMP_STATUS_INCOMPLETE) + return 0; + + return -1; /* error case */ +} + +static inline uint16_t +batch_completed(struct idxd_dmadev *idxd, uint8_t max_ops, bool *has_error) +{ + uint16_t i; + uint16_t b_start, b_end, next_batch; + + int ret = batch_ok(idxd, max_ops, NULL); + if (ret >= 0) + return ret; + + /* ERROR case, not successful, not incomplete */ + /* Get the batch size, and special case size 1. + * once we identify the actual failure job, return other jobs, then update + * the batch ring indexes to make it look like the first job of the batch has failed. + * Subsequent calls here will always return zero packets, and the error must be cleared by + * calling the completed_status() function. + */ + next_batch = (idxd->batch_idx_read + 1); + if (next_batch > idxd->max_batches) + next_batch = 0; + b_start = idxd->batch_idx_ring[idxd->batch_idx_read]; + b_end = idxd->batch_idx_ring[next_batch]; + + if (b_end - b_start == 1) { /* not a batch */ + *has_error = true; + return 0; + } + + for (i = b_start; i < b_end; i++) { + struct idxd_completion *c = (void *)&idxd->desc_ring[i & idxd->desc_ring_mask]; + if (c->status > IDXD_COMP_STATUS_SUCCESS) /* ignore incomplete(0) and success(1) */ + break; + } + ret = RTE_MIN((uint16_t)(i - idxd->ids_returned), max_ops); + if (ret < max_ops) + *has_error = true; /* we got up to the point of error */ + idxd->ids_avail = idxd->ids_returned += ret; + + /* to ensure we can call twice and just return 0, set start of batch to where we finished */ + idxd->batch_comp_ring[idxd->batch_idx_read].completed_size -= ret; + idxd->batch_idx_ring[idxd->batch_idx_read] += ret; + if (idxd->batch_idx_ring[next_batch] - idxd->batch_idx_ring[idxd->batch_idx_read] == 1) { + /* copy over the descriptor status to the batch ring as if no batch */ + uint16_t d_idx = idxd->batch_idx_ring[idxd->batch_idx_read] & idxd->desc_ring_mask; + struct idxd_completion *desc_comp = (void *)&idxd->desc_ring[d_idx]; + idxd->batch_comp_ring[idxd->batch_idx_read].status = desc_comp->status; + } + + return ret; +} + +static uint16_t +batch_completed_status(struct idxd_dmadev *idxd, uint16_t max_ops, enum rte_dma_status_code *status) +{ + uint16_t next_batch; + + int ret = batch_ok(idxd, max_ops, status); + if (ret >= 0) + return ret; + + /* ERROR case, not successful, not incomplete */ + /* Get the batch size, and special case size 1. + */ + next_batch = (idxd->batch_idx_read + 1); + if (next_batch > idxd->max_batches) + next_batch = 0; + const uint16_t b_start = idxd->batch_idx_ring[idxd->batch_idx_read]; + const uint16_t b_end = idxd->batch_idx_ring[next_batch]; + const uint16_t b_len = b_end - b_start; + if (b_len == 1) {/* not a batch */ + *status = get_comp_status(&idxd->batch_comp_ring[idxd->batch_idx_read]); + idxd->ids_avail++; + idxd->ids_returned++; + idxd->batch_idx_read = next_batch; + return 1; + } + + /* not a single-element batch, need to process more. + * Scenarios: + * 1. max_ops >= batch_size - can fit everything, simple case + * - loop through completed ops and then add on any not-attempted ones + * 2. max_ops < batch_size - can't fit everything, more complex case + * - loop through completed/incomplete and stop when hit max_ops + * - adjust the batch descriptor to update where we stopped, with appropriate bcount + * - if bcount is to be exactly 1, update the batch descriptor as it will be treated as + * non-batch next time. + */ + const uint16_t bcount = idxd->batch_comp_ring[idxd->batch_idx_read].completed_size; + for (ret = 0; ret < b_len && ret < max_ops; ret++) { + struct idxd_completion *c = (void *) + &idxd->desc_ring[(b_start + ret) & idxd->desc_ring_mask]; + status[ret] = (ret < bcount) ? get_comp_status(c) : RTE_DMA_STATUS_NOT_ATTEMPTED; + } + idxd->ids_avail = idxd->ids_returned += ret; + + /* everything fit */ + if (ret == b_len) { + idxd->batch_idx_read = next_batch; + return ret; + } + + /* set up for next time, update existing batch descriptor & start idx at batch_idx_read */ + idxd->batch_idx_ring[idxd->batch_idx_read] += ret; + if (ret > bcount) { + /* we have only incomplete ones - set batch completed size to 0 */ + struct idxd_completion *comp = &idxd->batch_comp_ring[idxd->batch_idx_read]; + comp->completed_size = 0; + /* if there is only one descriptor left, job skipped so set flag appropriately */ + if (b_len - ret == 1) + comp->status = IDXD_COMP_STATUS_SKIPPED; + } else { + struct idxd_completion *comp = &idxd->batch_comp_ring[idxd->batch_idx_read]; + comp->completed_size -= ret; + /* if there is only one descriptor left, copy status info straight to desc */ + if (comp->completed_size == 1) { + struct idxd_completion *c = (void *) + &idxd->desc_ring[(b_start + ret) & idxd->desc_ring_mask]; + comp->status = c->status; + /* individual descs can be ok without writeback, but not batches */ + if (comp->status == IDXD_COMP_STATUS_INCOMPLETE) + comp->status = IDXD_COMP_STATUS_SUCCESS; + } else if (bcount == b_len) { + /* check if we still have an error, and clear flag if not */ + uint16_t i; + for (i = b_start + ret; i < b_end; i++) { + struct idxd_completion *c = (void *) + &idxd->desc_ring[i & idxd->desc_ring_mask]; + if (c->status > IDXD_COMP_STATUS_SUCCESS) + break; + } + if (i == b_end) /* no errors */ + comp->status = IDXD_COMP_STATUS_SUCCESS; + } + } + + return ret; +} + +uint16_t +idxd_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint16_t max_ops, + uint16_t *last_idx, bool *has_error) +{ + struct idxd_dmadev *idxd = dev->dev_private; + uint16_t batch, ret = 0; + + do { + batch = batch_completed(idxd, max_ops - ret, has_error); + ret += batch; + } while (batch > 0 && *has_error == false); + + *last_idx = idxd->ids_returned - 1; + return ret; +} + +uint16_t +idxd_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint16_t max_ops, + uint16_t *last_idx, enum rte_dma_status_code *status) +{ + struct idxd_dmadev *idxd = dev->dev_private; + + uint16_t batch, ret = 0; + + do { + batch = batch_completed_status(idxd, max_ops - ret, &status[ret]); + ret += batch; + } while (batch > 0); + + *last_idx = idxd->ids_returned - 1; + return ret; +} + int idxd_dump(const struct rte_dmadev *dev, FILE *f) { @@ -272,6 +505,8 @@ idxd_dmadev_create(const char *name, struct rte_device *dev, dmadev->copy = idxd_enqueue_copy; dmadev->fill = idxd_enqueue_fill; dmadev->submit = idxd_submit; + dmadev->completed = idxd_completed; + dmadev->completed_status = idxd_completed_status; idxd = rte_malloc_socket(NULL, sizeof(struct idxd_dmadev), 0, dev->numa_node); if (idxd == NULL) { diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index 6a6c69fd61..4bcfe5372b 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -90,5 +90,10 @@ int idxd_enqueue_copy(struct rte_dmadev *dev, uint16_t qid, rte_iova_t src, int idxd_enqueue_fill(struct rte_dmadev *dev, uint16_t qid, uint64_t pattern, rte_iova_t dst, unsigned int length, uint64_t flags); int idxd_submit(struct rte_dmadev *dev, uint16_t qid); +uint16_t idxd_completed(struct rte_dmadev *dev, uint16_t qid, uint16_t max_ops, + uint16_t *last_idx, bool *has_error); +uint16_t idxd_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, + uint16_t max_ops, uint16_t *last_idx, + enum rte_dma_status_code *status); #endif /* _IDXD_INTERNAL_H_ */ From patchwork Fri Aug 27 17:20:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97482 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2B5BA0548; Fri, 27 Aug 2021 19:22:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F3E941296; Fri, 27 Aug 2021 19:21:33 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id D0166411E7 for ; Fri, 27 Aug 2021 19:21:30 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235964" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235964" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009616" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:28 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:47 +0000 Message-Id: <20210827172048.558704-13-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 12/13] dma/idxd: add operation statistic tracking X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add statistic tracking for DSA devices. Signed-off-by: Bruce Richardson Signed-off-by: Kevin Laatz --- doc/guides/dmadevs/idxd.rst | 11 +++++++++++ drivers/dma/idxd/idxd_bus.c | 2 ++ drivers/dma/idxd/idxd_common.c | 23 +++++++++++++++++++++++ drivers/dma/idxd/idxd_internal.h | 3 +++ 4 files changed, 39 insertions(+) diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst index 8bf99ef453..ba8bbac8b7 100644 --- a/doc/guides/dmadevs/idxd.rst +++ b/doc/guides/dmadevs/idxd.rst @@ -242,3 +242,14 @@ of memory is overwritten, or filled, with a short pattern of data. Fill operations can be performed in much the same was as copy operations described above, just using the ``rte_dmadev_fill()`` function rather than the ``rte_dmadev_copy()`` function. + +Querying Device Statistics +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The statistics from the IDXD dmadev device can be got via the stats functions in +the ``rte_dmadev`` library, i.e. ``rte_dmadev_stats_get()``. The statistics +returned for each device instance are: + +* ``submitted_count`` +* ``completed_fail_count`` +* ``completed_count`` diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c index ad4f076bad..9482dab9be 100644 --- a/drivers/dma/idxd/idxd_bus.c +++ b/drivers/dma/idxd/idxd_bus.c @@ -98,6 +98,8 @@ static const struct rte_dmadev_ops idxd_vdev_ops = { .dev_configure = idxd_configure, .vchan_setup = idxd_vchan_setup, .dev_info_get = idxd_info_get, + .stats_get = idxd_stats_get, + .stats_reset = idxd_stats_reset, }; static void * diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c index 50b205d92f..2f5addb359 100644 --- a/drivers/dma/idxd/idxd_common.c +++ b/drivers/dma/idxd/idxd_common.c @@ -66,6 +66,8 @@ __submit(struct idxd_dmadev *idxd) if (++idxd->batch_idx_write > idxd->max_batches) idxd->batch_idx_write = 0; + idxd->stats.submitted += idxd->batch_size; + idxd->batch_start += idxd->batch_size; idxd->batch_size = 0; idxd->batch_idx_ring[idxd->batch_idx_write] = idxd->batch_start; @@ -356,6 +358,7 @@ idxd_completed(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint16_t max_o ret += batch; } while (batch > 0 && *has_error == false); + idxd->stats.completed += ret; *last_idx = idxd->ids_returned - 1; return ret; } @@ -373,6 +376,7 @@ idxd_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint16_ ret += batch; } while (batch > 0); + idxd->stats.completed += ret; *last_idx = idxd->ids_returned - 1; return ret; } @@ -406,6 +410,25 @@ idxd_dump(const struct rte_dmadev *dev, FILE *f) return 0; } +int +idxd_stats_get(const struct rte_dmadev *dev, uint16_t vchan __rte_unused, + struct rte_dmadev_stats *stats, uint32_t stats_sz) +{ + struct idxd_dmadev *idxd = dev->dev_private; + if (stats_sz < sizeof(*stats)) + return -EINVAL; + *stats = idxd->stats; + return 0; +} + +int +idxd_stats_reset(struct rte_dmadev *dev, uint16_t vchan __rte_unused) +{ + struct idxd_dmadev *idxd = dev->dev_private; + idxd->stats = (struct rte_dmadev_stats){0}; + return 0; +} + int idxd_info_get(const struct rte_dmadev *dev, struct rte_dmadev_info *info, uint32_t size) { diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h index 4bcfe5372b..5079b4207d 100644 --- a/drivers/dma/idxd/idxd_internal.h +++ b/drivers/dma/idxd/idxd_internal.h @@ -95,5 +95,8 @@ uint16_t idxd_completed(struct rte_dmadev *dev, uint16_t qid, uint16_t max_ops, uint16_t idxd_completed_status(struct rte_dmadev *dev, uint16_t qid __rte_unused, uint16_t max_ops, uint16_t *last_idx, enum rte_dma_status_code *status); +int idxd_stats_get(const struct rte_dmadev *dev, uint16_t vchan, + struct rte_dmadev_stats *stats, uint32_t stats_sz); +int idxd_stats_reset(struct rte_dmadev *dev, uint16_t vchan); #endif /* _IDXD_INTERNAL_H_ */ From patchwork Fri Aug 27 17:20:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Laatz X-Patchwork-Id: 97483 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 90765A0548; Fri, 27 Aug 2021 19:22:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E4E0341270; Fri, 27 Aug 2021 19:21:34 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 019554126B for ; Fri, 27 Aug 2021 19:21:32 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10089"; a="198235967" X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="198235967" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 10:21:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,357,1620716400"; d="scan'208";a="445009647" Received: from silpixa00401122.ir.intel.com ([10.55.128.10]) by orsmga002.jf.intel.com with ESMTP; 27 Aug 2021 10:21:30 -0700 From: Kevin Laatz To: dev@dpdk.org Cc: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, conor.walsh@intel.com, Kevin Laatz Date: Fri, 27 Aug 2021 17:20:48 +0000 Message-Id: <20210827172048.558704-14-kevin.laatz@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210827172048.558704-1-kevin.laatz@intel.com> References: <20210827172048.558704-1-kevin.laatz@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 13/13] dma/idxd: move dpdk_idxd_cfg.py from raw to dma X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Conor Walsh Move the example script for configuring IDXD devices bound to the IDXD kernel driver from raw to dma, and create a symlink to still allow use from raw. Signed-off-by: Conor Walsh Signed-off-by: Kevin Laatz --- drivers/dma/idxd/dpdk_idxd_cfg.py | 117 +++++++++++++++++++++++++++++ drivers/raw/ioat/dpdk_idxd_cfg.py | 118 +----------------------------- 2 files changed, 118 insertions(+), 117 deletions(-) create mode 100755 drivers/dma/idxd/dpdk_idxd_cfg.py mode change 100755 => 120000 drivers/raw/ioat/dpdk_idxd_cfg.py diff --git a/drivers/dma/idxd/dpdk_idxd_cfg.py b/drivers/dma/idxd/dpdk_idxd_cfg.py new file mode 100755 index 0000000000..fcc27822ef --- /dev/null +++ b/drivers/dma/idxd/dpdk_idxd_cfg.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Intel Corporation + +""" +Configure an entire Intel DSA instance, using idxd kernel driver, for DPDK use +""" + +import sys +import argparse +import os +import os.path + + +class SysfsDir: + "Used to read/write paths in a sysfs directory" + def __init__(self, path): + self.path = path + + def read_int(self, filename): + "Return a value from sysfs file" + with open(os.path.join(self.path, filename)) as f: + return int(f.readline()) + + def write_values(self, values): + "write dictionary, where key is filename and value is value to write" + for filename, contents in values.items(): + with open(os.path.join(self.path, filename), "w") as f: + f.write(str(contents)) + + +def reset_device(dsa_id): + "Reset the DSA device and all its queues" + drv_dir = SysfsDir("/sys/bus/dsa/drivers/dsa") + drv_dir.write_values({"unbind": f"dsa{dsa_id}"}) + + +def get_pci_dir(pci): + "Search for the sysfs directory of the PCI device" + base_dir = '/sys/bus/pci/devices/' + for path, dirs, files in os.walk(base_dir): + for dir in dirs: + if pci in dir: + return os.path.join(base_dir, dir) + sys.exit(f"Could not find sysfs directory for device {pci}") + + +def get_dsa_id(pci): + "Get the DSA instance ID using the PCI address of the device" + pci_dir = get_pci_dir(pci) + for path, dirs, files in os.walk(pci_dir): + for dir in dirs: + if dir.startswith('dsa') and 'wq' not in dir: + return int(dir[3:]) + sys.exit(f"Could not get device ID for device {pci}") + + +def configure_dsa(dsa_id, queues, prefix): + "Configure the DSA instance with appropriate number of queues" + dsa_dir = SysfsDir(f"/sys/bus/dsa/devices/dsa{dsa_id}") + drv_dir = SysfsDir("/sys/bus/dsa/drivers/dsa") + + max_groups = dsa_dir.read_int("max_groups") + max_engines = dsa_dir.read_int("max_engines") + max_queues = dsa_dir.read_int("max_work_queues") + max_work_queues_size = dsa_dir.read_int("max_work_queues_size") + + nb_queues = min(queues, max_queues) + if queues > nb_queues: + print(f"Setting number of queues to max supported value: {max_queues}") + + # we want one engine per group, and no more engines than queues + nb_groups = min(max_engines, max_groups, nb_queues) + for grp in range(nb_groups): + dsa_dir.write_values({f"engine{dsa_id}.{grp}/group_id": grp}) + + # configure each queue + for q in range(nb_queues): + wq_dir = SysfsDir(os.path.join(dsa_dir.path, f"wq{dsa_id}.{q}")) + wq_dir.write_values({"group_id": q % nb_groups, + "type": "user", + "mode": "dedicated", + "name": f"{prefix}_wq{dsa_id}.{q}", + "priority": 1, + "size": int(max_work_queues_size / nb_queues)}) + + # enable device and then queues + drv_dir.write_values({"bind": f"dsa{dsa_id}"}) + for q in range(nb_queues): + drv_dir.write_values({"bind": f"wq{dsa_id}.{q}"}) + + +def main(args): + "Main function, does arg parsing and calls config function" + arg_p = argparse.ArgumentParser( + description="Configure whole DSA device instance for DPDK use") + arg_p.add_argument('dsa_id', + help="Specify DSA instance either via DSA instance number or PCI address") + arg_p.add_argument('-q', metavar='queues', type=int, default=255, + help="Number of queues to set up") + arg_p.add_argument('--name-prefix', metavar='prefix', dest='prefix', + default="dpdk", + help="Prefix for workqueue name to mark for DPDK use [default: 'dpdk']") + arg_p.add_argument('--reset', action='store_true', + help="Reset DSA device and its queues") + parsed_args = arg_p.parse_args(args[1:]) + + dsa_id = parsed_args.dsa_id + dsa_id = get_dsa_id(dsa_id) if ':' in dsa_id else dsa_id + if parsed_args.reset: + reset_device(dsa_id) + else: + configure_dsa(dsa_id, parsed_args.q, parsed_args.prefix) + + +if __name__ == "__main__": + main(sys.argv) diff --git a/drivers/raw/ioat/dpdk_idxd_cfg.py b/drivers/raw/ioat/dpdk_idxd_cfg.py deleted file mode 100755 index fcc27822ef..0000000000 --- a/drivers/raw/ioat/dpdk_idxd_cfg.py +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2020 Intel Corporation - -""" -Configure an entire Intel DSA instance, using idxd kernel driver, for DPDK use -""" - -import sys -import argparse -import os -import os.path - - -class SysfsDir: - "Used to read/write paths in a sysfs directory" - def __init__(self, path): - self.path = path - - def read_int(self, filename): - "Return a value from sysfs file" - with open(os.path.join(self.path, filename)) as f: - return int(f.readline()) - - def write_values(self, values): - "write dictionary, where key is filename and value is value to write" - for filename, contents in values.items(): - with open(os.path.join(self.path, filename), "w") as f: - f.write(str(contents)) - - -def reset_device(dsa_id): - "Reset the DSA device and all its queues" - drv_dir = SysfsDir("/sys/bus/dsa/drivers/dsa") - drv_dir.write_values({"unbind": f"dsa{dsa_id}"}) - - -def get_pci_dir(pci): - "Search for the sysfs directory of the PCI device" - base_dir = '/sys/bus/pci/devices/' - for path, dirs, files in os.walk(base_dir): - for dir in dirs: - if pci in dir: - return os.path.join(base_dir, dir) - sys.exit(f"Could not find sysfs directory for device {pci}") - - -def get_dsa_id(pci): - "Get the DSA instance ID using the PCI address of the device" - pci_dir = get_pci_dir(pci) - for path, dirs, files in os.walk(pci_dir): - for dir in dirs: - if dir.startswith('dsa') and 'wq' not in dir: - return int(dir[3:]) - sys.exit(f"Could not get device ID for device {pci}") - - -def configure_dsa(dsa_id, queues, prefix): - "Configure the DSA instance with appropriate number of queues" - dsa_dir = SysfsDir(f"/sys/bus/dsa/devices/dsa{dsa_id}") - drv_dir = SysfsDir("/sys/bus/dsa/drivers/dsa") - - max_groups = dsa_dir.read_int("max_groups") - max_engines = dsa_dir.read_int("max_engines") - max_queues = dsa_dir.read_int("max_work_queues") - max_work_queues_size = dsa_dir.read_int("max_work_queues_size") - - nb_queues = min(queues, max_queues) - if queues > nb_queues: - print(f"Setting number of queues to max supported value: {max_queues}") - - # we want one engine per group, and no more engines than queues - nb_groups = min(max_engines, max_groups, nb_queues) - for grp in range(nb_groups): - dsa_dir.write_values({f"engine{dsa_id}.{grp}/group_id": grp}) - - # configure each queue - for q in range(nb_queues): - wq_dir = SysfsDir(os.path.join(dsa_dir.path, f"wq{dsa_id}.{q}")) - wq_dir.write_values({"group_id": q % nb_groups, - "type": "user", - "mode": "dedicated", - "name": f"{prefix}_wq{dsa_id}.{q}", - "priority": 1, - "size": int(max_work_queues_size / nb_queues)}) - - # enable device and then queues - drv_dir.write_values({"bind": f"dsa{dsa_id}"}) - for q in range(nb_queues): - drv_dir.write_values({"bind": f"wq{dsa_id}.{q}"}) - - -def main(args): - "Main function, does arg parsing and calls config function" - arg_p = argparse.ArgumentParser( - description="Configure whole DSA device instance for DPDK use") - arg_p.add_argument('dsa_id', - help="Specify DSA instance either via DSA instance number or PCI address") - arg_p.add_argument('-q', metavar='queues', type=int, default=255, - help="Number of queues to set up") - arg_p.add_argument('--name-prefix', metavar='prefix', dest='prefix', - default="dpdk", - help="Prefix for workqueue name to mark for DPDK use [default: 'dpdk']") - arg_p.add_argument('--reset', action='store_true', - help="Reset DSA device and its queues") - parsed_args = arg_p.parse_args(args[1:]) - - dsa_id = parsed_args.dsa_id - dsa_id = get_dsa_id(dsa_id) if ':' in dsa_id else dsa_id - if parsed_args.reset: - reset_device(dsa_id) - else: - configure_dsa(dsa_id, parsed_args.q, parsed_args.prefix) - - -if __name__ == "__main__": - main(sys.argv) diff --git a/drivers/raw/ioat/dpdk_idxd_cfg.py b/drivers/raw/ioat/dpdk_idxd_cfg.py new file mode 120000 index 0000000000..85545548d1 --- /dev/null +++ b/drivers/raw/ioat/dpdk_idxd_cfg.py @@ -0,0 +1 @@ +../../dma/idxd/dpdk_idxd_cfg.py \ No newline at end of file