From patchwork Tue Jun 11 08:50:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Xiaoyun" X-Patchwork-Id: 54642 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 15D361C2DF; Tue, 11 Jun 2019 10:50:52 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id AFDF71C2D9 for ; Tue, 11 Jun 2019 10:50:47 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jun 2019 01:50:47 -0700 X-ExtLoop1: 1 Received: from dpdk-xiaoyun3.sh.intel.com ([10.67.119.132]) by fmsmga008.fm.intel.com with ESMTP; 11 Jun 2019 01:50:45 -0700 From: Xiaoyun Li To: jingjing.wu@intel.com, keith.wiles@intel.com, cunming.liang@intel.com, omkar.maslekar@intel.com Cc: dev@dpdk.org, Xiaoyun Li Date: Tue, 11 Jun 2019 16:50:13 +0800 Message-Id: <20190611085018.56386-2-xiaoyun.li@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611085018.56386-1-xiaoyun.li@intel.com> References: <20190606074303.104108-1-xiaoyun.li@intel.com> <20190611085018.56386-1-xiaoyun.li@intel.com> Subject: [dpdk-dev] [PATCH v3 1/6] raw/ntb: introduce ntb rawdev driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other. Signed-off-by: Xiaoyun Li --- config/common_base | 5 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb_rawdev/Makefile | 27 + drivers/raw/ntb_rawdev/meson.build | 7 + drivers/raw/ntb_rawdev/ntb_rawdev.c | 500 ++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_rawdev.h | 158 ++++++ .../ntb_rawdev/rte_pmd_ntb_rawdev_version.map | 4 + mk/rte.app.mk | 1 + 9 files changed, 704 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/Makefile create mode 100644 drivers/raw/ntb_rawdev/meson.build create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.c create mode 100644 drivers/raw/ntb_rawdev/ntb_rawdev.h create mode 100644 drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map diff --git a/config/common_base b/config/common_base index e406e7836..45e403130 100644 --- a/config/common_base +++ b/config/common_base @@ -746,6 +746,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring # diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile index 8e29b4a56..efe61f451 100644 --- a/drivers/raw/Makefile +++ b/drivers/raw/Makefile @@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_rawdev include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build index a61cdccef..6abf659d0 100644 --- a/drivers/raw/meson.build +++ b/drivers/raw/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev', 'ntb_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@' diff --git a/drivers/raw/ntb_rawdev/Makefile b/drivers/raw/ntb_rawdev/Makefile new file mode 100644 index 000000000..fb40204c1 --- /dev/null +++ b/drivers/raw/ntb_rawdev/Makefile @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb_rawdev.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_rawdev_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/raw/ntb_rawdev/meson.build b/drivers/raw/ntb_rawdev/meson.build new file mode 100644 index 000000000..ca905049d --- /dev/null +++ b/drivers/raw/ntb_rawdev/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb_rawdev.c') +allow_experimental_apis = true diff --git a/drivers/raw/ntb_rawdev/ntb_rawdev.c b/drivers/raw/ntb_rawdev/ntb_rawdev.c new file mode 100644 index 000000000..518373f8f --- /dev/null +++ b/drivers/raw/ntb_rawdev/ntb_rawdev.c @@ -0,0 +1,500 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ntb_rawdev.h" + +int ntb_logtype; + +static const struct rte_pci_id pci_id_ntb_map[] = { + { .vendor_id = 0, /* sentinel */ }, +}; + +static void +ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused, + rte_rawdev_obj_t queue_conf __rte_unused) +{ +} + +static int +ntb_queue_setup(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused, + rte_rawdev_obj_t queue_conf __rte_unused) +{ + return 0; +} + +static int +ntb_queue_release(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused) +{ + return 0; +} + +static uint16_t +ntb_queue_count(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + return hw->queue_pairs; +} + +static int +ntb_enqueue_bufs(struct rte_rawdev *dev, + struct rte_rawdev_buf **buffers, + unsigned int count, + rte_rawdev_obj_t context) +{ + RTE_SET_USED(dev); + RTE_SET_USED(buffers); + RTE_SET_USED(count); + RTE_SET_USED(context); + + return 0; +} + +static int +ntb_dequeue_bufs(struct rte_rawdev *dev, + struct rte_rawdev_buf **buffers, + unsigned int count, + rte_rawdev_obj_t context) +{ + RTE_SET_USED(dev); + RTE_SET_USED(buffers); + RTE_SET_USED(count); + RTE_SET_USED(context); + + return 0; +} + +static void +ntb_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info) +{ + struct ntb_hw *hw = dev->dev_private; + struct ntb_attr *ntb_attrs = dev_info; + + strncpy(ntb_attrs[NTB_TOPO_ID].name, NTB_TOPO_NAME, NTB_ATTR_NAME_LEN); + switch (hw->topo) { + case NTB_TOPO_B2B_DSD: + strncpy(ntb_attrs[NTB_TOPO_ID].value, "B2B DSD", + NTB_ATTR_NAME_LEN); + break; + case NTB_TOPO_B2B_USD: + strncpy(ntb_attrs[NTB_TOPO_ID].value, "B2B USD", + NTB_ATTR_NAME_LEN); + break; + default: + strncpy(ntb_attrs[NTB_TOPO_ID].value, "Unsupported", + NTB_ATTR_NAME_LEN); + } + + strncpy(ntb_attrs[NTB_LINK_STATUS_ID].name, NTB_LINK_STATUS_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_LINK_STATUS_ID].value, NTB_ATTR_NAME_LEN, + "%d", hw->link_status); + + strncpy(ntb_attrs[NTB_SPEED_ID].name, NTB_SPEED_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_SPEED_ID].value, NTB_ATTR_NAME_LEN, + "%d", hw->link_speed); + + strncpy(ntb_attrs[NTB_WIDTH_ID].name, NTB_WIDTH_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_WIDTH_ID].value, NTB_ATTR_NAME_LEN, + "%d", hw->link_width); + + strncpy(ntb_attrs[NTB_MW_CNT_ID].name, NTB_MW_CNT_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_MW_CNT_ID].value, NTB_ATTR_NAME_LEN, + "%d", hw->mw_cnt); + + strncpy(ntb_attrs[NTB_DB_CNT_ID].name, NTB_DB_CNT_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_DB_CNT_ID].value, NTB_ATTR_NAME_LEN, + "%d", hw->db_cnt); + + strncpy(ntb_attrs[NTB_SPAD_CNT_ID].name, NTB_SPAD_CNT_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_SPAD_CNT_ID].value, NTB_ATTR_NAME_LEN, + "%d", hw->spad_cnt); +} + +static int +ntb_dev_configure(const struct rte_rawdev *dev __rte_unused, + rte_rawdev_obj_t config __rte_unused) +{ + return 0; +} + +static int +ntb_dev_start(struct rte_rawdev *dev) +{ + /* TODO: init queues and start queues. */ + dev->started = 1; + + return 0; +} + +static void +ntb_dev_stop(struct rte_rawdev *dev) +{ + /* TODO: stop rx/tx queues. */ + dev->started = 0; +} + +static int +ntb_dev_close(struct rte_rawdev *dev) +{ + int ret = 0; + + if (dev->started) + ntb_dev_stop(dev); + + /* TODO: free queues. */ + + return ret; +} + +static int +ntb_dev_reset(struct rte_rawdev *rawdev __rte_unused) +{ + return 0; +} + +static int +ntb_attr_set(struct rte_rawdev *dev, const char *attr_name, + uint64_t attr_value) +{ + struct ntb_hw *hw = dev->dev_private; + + if (dev == NULL || attr_name == NULL) { + NTB_LOG(ERR, "Invalid arguments for setting attributes"); + return -EINVAL; + } + + if (!strncmp(attr_name, NTB_PEER_SPAD_14, NTB_ATTR_NAME_LEN)) { + if (hw->ntb_ops->spad_write == NULL) + return -ENOTSUP; + (*hw->ntb_ops->spad_write)(dev, 14, 1, attr_value); + NTB_LOG(INFO, "Set attribute (%s) Value (%" PRIu64 ")", + attr_name, attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_PEER_SPAD_15, NTB_ATTR_NAME_LEN)) { + if (hw->ntb_ops->spad_write == NULL) + return -ENOTSUP; + (*hw->ntb_ops->spad_write)(dev, 15, 1, attr_value); + NTB_LOG(INFO, "Set attribute (%s) Value (%" PRIu64 ")", + attr_name, attr_value); + return 0; + } + + /* Attribute not found. */ + return -EINVAL; +} + +static int +ntb_attr_get(struct rte_rawdev *dev, const char *attr_name, + uint64_t *attr_value) +{ + struct ntb_hw *hw = dev->dev_private; + + if (dev == NULL || attr_name == NULL || attr_value == NULL) { + NTB_LOG(ERR, "Invalid arguments for getting attributes"); + return -EINVAL; + } + + if (!strncmp(attr_name, NTB_TOPO_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->topo; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_LINK_STATUS_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->link_status; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_SPEED_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->link_speed; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_WIDTH_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->link_width; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_MW_CNT_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->mw_cnt; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_DB_CNT_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->db_cnt; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_SPAD_CNT_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->spad_cnt; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_PEER_SPAD_14, NTB_ATTR_NAME_LEN)) { + if (hw->ntb_ops->spad_read == NULL) + return -ENOTSUP; + *attr_value = (*hw->ntb_ops->spad_read)(dev, 14, 0); + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_PEER_SPAD_15, NTB_ATTR_NAME_LEN)) { + if (hw->ntb_ops->spad_read == NULL) + return -ENOTSUP; + *attr_value = (*hw->ntb_ops->spad_read)(dev, 15, 0); + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + /* Attribute not found. */ + return -EINVAL; +} + +static int +ntb_xstats_get(const struct rte_rawdev *dev __rte_unused, + const unsigned int ids[] __rte_unused, + uint64_t values[] __rte_unused, + unsigned int n __rte_unused) +{ + return 0; +} + +static int +ntb_xstats_get_names(const struct rte_rawdev *dev __rte_unused, + struct rte_rawdev_xstats_name *xstats_names __rte_unused, + unsigned int size __rte_unused) +{ + return 0; +} + +static uint64_t +ntb_xstats_get_by_name(const struct rte_rawdev *dev __rte_unused, + const char *name __rte_unused, + unsigned int *id __rte_unused) +{ + return 0; +} + +static int +ntb_xstats_reset(struct rte_rawdev *dev __rte_unused, + const uint32_t ids[] __rte_unused, + uint32_t nb_ids __rte_unused) +{ + return 0; +} + +static const struct rte_rawdev_ops ntb_rawdev_ops = { + .dev_info_get = ntb_dev_info_get, + .dev_configure = ntb_dev_configure, + .dev_start = ntb_dev_start, + .dev_stop = ntb_dev_stop, + .dev_close = ntb_dev_close, + .dev_reset = ntb_dev_reset, + + .queue_def_conf = ntb_queue_conf_get, + .queue_setup = ntb_queue_setup, + .queue_release = ntb_queue_release, + .queue_count = ntb_queue_count, + + .enqueue_bufs = ntb_enqueue_bufs, + .dequeue_bufs = ntb_dequeue_bufs, + + .attr_get = ntb_attr_get, + .attr_set = ntb_attr_set, + + .xstats_get = ntb_xstats_get, + .xstats_get_names = ntb_xstats_get_names, + .xstats_get_by_name = ntb_xstats_get_by_name, + .xstats_reset = ntb_xstats_reset, +}; + +static int +ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) +{ + struct ntb_hw *hw = dev->dev_private; + int ret; + + hw->pci_dev = pci_dev; + hw->peer_dev_up = 0; + hw->link_status = 0; + hw->link_speed = NTB_SPEED_NONE; + hw->link_width = NTB_WIDTH_NONE; + + switch (pci_dev->id.device_id) { + default: + NTB_LOG(ERR, "Not supported device."); + return -EINVAL; + } + + if (hw->ntb_ops->ntb_dev_init == NULL) + return -ENOTSUP; + ret = (*hw->ntb_ops->ntb_dev_init)(dev); + if (ret) { + NTB_LOG(ERR, "Unanle to init ntb dev."); + return ret; + } + + if (hw->ntb_ops->set_link == NULL) + return -ENOTSUP; + ret = (*hw->ntb_ops->set_link)(dev, 1); + if (ret) + return ret; + + return ret; +} + +static int +ntb_rawdev_create(struct rte_pci_device *pci_dev, int socket_id) +{ + char name[RTE_RAWDEV_NAME_MAX_LEN]; + struct rte_rawdev *rawdev = NULL; + int ret; + + if (pci_dev == NULL) { + NTB_LOG(ERR, "Invalid pci_dev."); + ret = -EINVAL; + goto fail; + } + + memset(name, 0, sizeof(name)); + snprintf(name, RTE_RAWDEV_NAME_MAX_LEN, "NTB:%x:%02x.%x", + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + + NTB_LOG(INFO, "Init %s on NUMA node %d", name, rte_socket_id()); + + /* Allocate device structure. */ + rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct ntb_hw), + socket_id); + if (rawdev == NULL) { + NTB_LOG(ERR, "Unable to allocate rawdev."); + ret = -EINVAL; + goto fail; + } + + rawdev->dev_ops = &ntb_rawdev_ops; + rawdev->device = &pci_dev->device; + rawdev->driver_name = pci_dev->driver->driver.name; + + ret = ntb_init_hw(rawdev, pci_dev); + if (ret < 0) { + NTB_LOG(ERR, "Unable to init ntb hw."); + goto fail; + } + + return ret; + +fail: + if (rawdev) + rte_rawdev_pmd_release(rawdev); + + return ret; +} + +static int +ntb_rawdev_destroy(struct rte_pci_device *pci_dev) +{ + char name[RTE_RAWDEV_NAME_MAX_LEN]; + struct rte_rawdev *rawdev; + int ret; + + if (pci_dev == NULL) { + NTB_LOG(ERR, "Invalid pci_dev."); + ret = -EINVAL; + return ret; + } + + memset(name, 0, sizeof(name)); + snprintf(name, RTE_RAWDEV_NAME_MAX_LEN, "NTB:%x:%02x.%x", + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + + NTB_LOG(INFO, "Closing %s on NUMA node %d", name, rte_socket_id()); + + rawdev = rte_rawdev_pmd_get_named_dev(name); + if (rawdev == NULL) { + NTB_LOG(ERR, "Invalid device name (%s)", name); + ret = -EINVAL; + return ret; + } + + ret = rte_rawdev_pmd_release(rawdev); + if (ret) + NTB_LOG(ERR, "Failed to destroy ntb rawdev."); + + return ret; +} + +static int +ntb_rawdev_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + return ntb_rawdev_create(pci_dev, rte_socket_id()); +} + +static int +ntb_rawdev_remove(struct rte_pci_device *pci_dev) +{ + return ntb_rawdev_destroy(pci_dev); +} + + +static struct rte_pci_driver rte_ntb_pmd = { + .id_table = pci_id_ntb_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING, + .probe = ntb_rawdev_probe, + .remove = ntb_rawdev_remove, +}; + +RTE_PMD_REGISTER_PCI(raw_ntb, rte_ntb_pmd); +RTE_PMD_REGISTER_PCI_TABLE(raw_ntb, pci_id_ntb_map); +RTE_PMD_REGISTER_KMOD_DEP(raw_ntb, "* igb_uio | uio_pci_generic | vfio-pci"); + +RTE_INIT(ntb_init_log) +{ + ntb_logtype = rte_log_register("pmd.raw.ntb"); + if (ntb_logtype >= 0) + rte_log_set_level(ntb_logtype, RTE_LOG_DEBUG); +} diff --git a/drivers/raw/ntb_rawdev/ntb_rawdev.h b/drivers/raw/ntb_rawdev/ntb_rawdev.h new file mode 100644 index 000000000..a13815a1d --- /dev/null +++ b/drivers/raw/ntb_rawdev/ntb_rawdev.h @@ -0,0 +1,158 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ + +#ifndef _NTB_RAWDEV_H_ +#define _NTB_RAWDEV_H_ + +#include + +extern int ntb_logtype; + +#define NTB_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, ntb_logtype, "%s(): " fmt "\n", \ + __func__, ##args) + +/* Vendor ID */ +#define NTB_INTEL_VENDOR_ID 0x8086 + +/* Device IDs */ +#define NTB_INTEL_DEV_ID_B2B_SKX 0x201C + +#define NTB_TOPO_NAME "topo" +#define NTB_LINK_STATUS_NAME "link_status" +#define NTB_SPEED_NAME "speed" +#define NTB_WIDTH_NAME "width" +#define NTB_MW_CNT_NAME "mw_count" +#define NTB_DB_CNT_NAME "db_count" +#define NTB_SPAD_CNT_NAME "spad_count" +/* Reserved to app to use. */ +#define NTB_PEER_SPAD_14 "spad14" +#define NTB_PEER_SPAD_15 "spad15" +#define NTB_ATTR_NAME_LEN 30 +#define NTB_ATTR_MAX 20 + +/* NTB Attributes */ +struct ntb_attr { + /**< Name of the attribute */ + char name[NTB_ATTR_NAME_LEN]; + /**< Value or reference of value of attribute */ + char value[NTB_ATTR_NAME_LEN]; +}; + +enum ntb_attr_idx { + NTB_TOPO_ID = 0, + NTB_LINK_STATUS_ID, + NTB_SPEED_ID, + NTB_WIDTH_ID, + NTB_MW_CNT_ID, + NTB_DB_CNT_ID, + NTB_SPAD_CNT_ID, +}; + +enum ntb_topo { + NTB_TOPO_NONE = 0, + NTB_TOPO_B2B_USD, + NTB_TOPO_B2B_DSD, +}; + +enum ntb_link { + NTB_LINK_DOWN = 0, + NTB_LINK_UP, +}; + +enum ntb_speed { + NTB_SPEED_NONE = 0, + NTB_SPEED_GEN1 = 1, + NTB_SPEED_GEN2 = 2, + NTB_SPEED_GEN3 = 3, + NTB_SPEED_GEN4 = 4, +}; + +enum ntb_width { + NTB_WIDTH_NONE = 0, + NTB_WIDTH_1 = 1, + NTB_WIDTH_2 = 2, + NTB_WIDTH_4 = 4, + NTB_WIDTH_8 = 8, + NTB_WIDTH_12 = 12, + NTB_WIDTH_16 = 16, + NTB_WIDTH_32 = 32, +}; + +/* Define spad registers usage. 0 is reserved. */ +enum ntb_spad_idx { + SPAD_NUM_MWS = 1, + SPAD_NUM_QPS, + SPAD_Q_SZ, + SPAD_MW0_SZ_H, + SPAD_MW0_SZ_L, + SPAD_MW1_SZ_H, + SPAD_MW1_SZ_L, +}; + +/** + * NTB device operations + * @ntb_dev_init: Init ntb dev. + * @get_peer_mw_addr: To get the addr of peer mw[mw_idx]. + * @mw_set_trans: Set translation of internal memory that remote can access. + * @get_link_status: get link status, link speed and link width. + * @set_link: Set local side up/down. + * @spad_read: Read local/peer spad register val. + * @spad_write: Write val to local/peer spad register. + * @db_read: Read doorbells status. + * @db_clear: Clear local doorbells. + * @db_set_mask: Set bits in db mask, preventing db interrpts generated + * for those db bits. + * @peer_db_set: Set doorbell bit to generate peer interrupt for that bit. + * @vector_bind: Bind vector source [intr] to msix vector [msix]. + */ +struct ntb_dev_ops { + int (*ntb_dev_init)(struct rte_rawdev *dev); + void *(*get_peer_mw_addr)(struct rte_rawdev *dev, int mw_idx); + int (*mw_set_trans)(struct rte_rawdev *dev, int mw_idx, + uint64_t addr, uint64_t size); + int (*get_link_status)(struct rte_rawdev *dev); + int (*set_link)(struct rte_rawdev *dev, bool up); + uint32_t (*spad_read)(struct rte_rawdev *dev, int spad, bool peer); + int (*spad_write)(struct rte_rawdev *dev, int spad, + bool peer, uint32_t spad_v); + uint64_t (*db_read)(struct rte_rawdev *dev); + int (*db_clear)(struct rte_rawdev *dev, uint64_t db_bits); + int (*db_set_mask)(struct rte_rawdev *dev, uint64_t db_mask); + int (*peer_db_set)(struct rte_rawdev *dev, uint8_t db_bit); + int (*vector_bind)(struct rte_rawdev *dev, uint8_t intr, uint8_t msix); +}; + +/* ntb private data. */ +struct ntb_hw { + uint8_t mw_cnt; + uint8_t peer_mw_cnt; + uint8_t db_cnt; + uint8_t spad_cnt; + + uint64_t db_valid_mask; + uint64_t db_mask; + + enum ntb_topo topo; + + enum ntb_link link_status; + enum ntb_speed link_speed; + enum ntb_width link_width; + + const struct ntb_dev_ops *ntb_ops; + + struct rte_pci_device *pci_dev; + + uint64_t *mw_size; + uint64_t *peer_mw_size; + uint8_t peer_dev_up; + + uint16_t queue_pairs; + uint16_t queue_size; + + /**< mem zone to populate RX ring. */ + const struct rte_memzone **mz; +}; + +#endif /* _NTB_RAWDEV_H_ */ diff --git a/drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map b/drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map new file mode 100644 index 000000000..8861484fb --- /dev/null +++ b/drivers/raw/ntb_rawdev/rte_pmd_ntb_rawdev_version.map @@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +}; diff --git a/mk/rte.app.mk b/mk/rte.app.mk index d0df0b023..ff17bef46 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -301,6 +301,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS From patchwork Tue Jun 11 08:50:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Xiaoyun" X-Patchwork-Id: 54643 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 17A871C2E7; Tue, 11 Jun 2019 10:50:55 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 795781C2D9 for ; Tue, 11 Jun 2019 10:50:49 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jun 2019 01:50:49 -0700 X-ExtLoop1: 1 Received: from dpdk-xiaoyun3.sh.intel.com ([10.67.119.132]) by fmsmga008.fm.intel.com with ESMTP; 11 Jun 2019 01:50:48 -0700 From: Xiaoyun Li To: jingjing.wu@intel.com, keith.wiles@intel.com, cunming.liang@intel.com, omkar.maslekar@intel.com Cc: dev@dpdk.org, Xiaoyun Li Date: Tue, 11 Jun 2019 16:50:14 +0800 Message-Id: <20190611085018.56386-3-xiaoyun.li@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611085018.56386-1-xiaoyun.li@intel.com> References: <20190606074303.104108-1-xiaoyun.li@intel.com> <20190611085018.56386-1-xiaoyun.li@intel.com> Subject: [dpdk-dev] [PATCH v3 2/6] raw/ntb: add intel ntb support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add in the list of registers for the device. And enable ntb device ops for intel skylake platform. Signed-off-by: Xiaoyun Li --- drivers/raw/ntb_rawdev/Makefile | 1 + drivers/raw/ntb_rawdev/meson.build | 3 +- drivers/raw/ntb_rawdev/ntb_hw_intel.c | 368 ++++++++++++++++++++++++++ drivers/raw/ntb_rawdev/ntb_hw_intel.h | 86 ++++++ drivers/raw/ntb_rawdev/ntb_rawdev.c | 5 + 5 files changed, 462 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.c create mode 100644 drivers/raw/ntb_rawdev/ntb_hw_intel.h diff --git a/drivers/raw/ntb_rawdev/Makefile b/drivers/raw/ntb_rawdev/Makefile index fb40204c1..337311ea4 100644 --- a/drivers/raw/ntb_rawdev/Makefile +++ b/drivers/raw/ntb_rawdev/Makefile @@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_SKELETON_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/raw/ntb_rawdev/meson.build b/drivers/raw/ntb_rawdev/meson.build index ca905049d..c696f60b3 100644 --- a/drivers/raw/ntb_rawdev/meson.build +++ b/drivers/raw/ntb_rawdev/meson.build @@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb_rawdev.c') +sources = files('ntb_rawdev.c', + 'ntb_hw_intel.c') allow_experimental_apis = true diff --git a/drivers/raw/ntb_rawdev/ntb_hw_intel.c b/drivers/raw/ntb_rawdev/ntb_hw_intel.c new file mode 100644 index 000000000..8a1e9be2a --- /dev/null +++ b/drivers/raw/ntb_rawdev/ntb_hw_intel.c @@ -0,0 +1,368 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "ntb_rawdev.h" +#include "ntb_hw_intel.h" + +enum xeon_ntb_bar { + XEON_NTB_BAR23 = 2, + XEON_NTB_BAR45 = 4, +}; + +static enum xeon_ntb_bar intel_ntb_bar[] = { + XEON_NTB_BAR23, + XEON_NTB_BAR45, +}; + +static int +intel_ntb_dev_init(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + uint8_t reg_val, bar; + int ret, i; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return -EINVAL; + } + + ret = rte_pci_read_config(hw->pci_dev, ®_val, + sizeof(reg_val), XEON_PPD_OFFSET); + if (ret < 0) { + NTB_LOG(ERR, "Cannot get NTB PPD (PCIe port definition)."); + return -EIO; + } + + /* Check connection topo type. Only support B2B. */ + switch (reg_val & XEON_PPD_CONN_MASK) { + case XEON_PPD_CONN_B2B: + NTB_LOG(INFO, "Topo B2B (back to back) is using."); + break; + case XEON_PPD_CONN_TRANSPARENT: + case XEON_PPD_CONN_RP: + NTB_LOG(ERR, "Not supported conn topo. Please use B2B."); + return -EINVAL; + } + + /* Check device type. */ + if (reg_val & XEON_PPD_DEV_DSD) { + NTB_LOG(INFO, "DSD, Downstream Device."); + hw->topo = NTB_TOPO_B2B_DSD; + } else { + NTB_LOG(INFO, "USD, Upstream device."); + hw->topo = NTB_TOPO_B2B_USD; + } + + /* Check if bar4 is split. Do not support split bar. */ + if (reg_val & XEON_PPD_SPLIT_BAR_MASK) { + NTB_LOG(ERR, "Do not support split bar."); + return -EINVAL; + } + + hw->mw_cnt = XEON_MW_COUNT; + hw->db_cnt = XEON_DB_COUNT; + hw->spad_cnt = XEON_SPAD_COUNT; + + hw->mw_size = rte_zmalloc("uint64_t", + hw->mw_cnt * sizeof(uint64_t), 0); + for (i = 0; i < hw->mw_cnt; i++) { + bar = intel_ntb_bar[i]; + hw->mw_size[i] = hw->pci_dev->mem_resource[bar].len; + } + + return 0; +} + +static void * +intel_ntb_get_peer_mw_addr(struct rte_rawdev *dev, int mw_idx) +{ + struct ntb_hw *hw = dev->dev_private; + uint8_t bar; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return 0; + } + + if (mw_idx < 0 || mw_idx > hw->mw_cnt) { + NTB_LOG(ERR, "Invalid memory window index (0 - %u).", + hw->mw_cnt - 1); + return 0; + } + + bar = intel_ntb_bar[mw_idx]; + + return hw->pci_dev->mem_resource[bar].addr; +} + +static int +intel_ntb_mw_set_trans(struct rte_rawdev *dev, int mw_idx, + uint64_t addr, uint64_t size) +{ + struct ntb_hw *hw = dev->dev_private; + void *xlat_addr, *limit_addr; + uint64_t xlat_off, limit_off; + uint64_t base, limit; + uint8_t bar; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return -EINVAL; + } + + if (mw_idx < 0 || mw_idx > hw->mw_cnt) { + NTB_LOG(ERR, "Invalid memory window index (0 - %u).", + hw->mw_cnt - 1); + return -EINVAL; + } + + bar = intel_ntb_bar[mw_idx]; + + xlat_off = XEON_IMBAR1XBASE_OFFSET + mw_idx * XEON_BAR_INTERVAL_OFFSET; + limit_off = XEON_IMBAR1XLMT_OFFSET + mw_idx * XEON_BAR_INTERVAL_OFFSET; + xlat_addr = (char *)hw->pci_dev->mem_resource[0].addr + xlat_off; + limit_addr = (char *)hw->pci_dev->mem_resource[0].addr + limit_off; + + /* Limit reg val should be EMBAR base address plus MW size. */ + base = addr; + limit = hw->pci_dev->mem_resource[bar].phys_addr + size; + *((volatile uint64_t *)xlat_addr) = base; + *((volatile uint64_t *)limit_addr) = limit; + + /* Setup the external point so that remote can access. */ + xlat_off = XEON_EMBAR1_OFFSET + 8 * mw_idx; + xlat_addr = (char *)hw->pci_dev->mem_resource[0].addr + xlat_off; + limit_off = XEON_EMBAR1XLMT_OFFSET + mw_idx * XEON_BAR_INTERVAL_OFFSET; + limit_addr = (char *)hw->pci_dev->mem_resource[0].addr + limit_off; + base = *((volatile uint64_t *)xlat_addr); + base &= ~0xf; + limit = base + size; + *((volatile uint64_t *)limit_addr) = limit; + + return 0; +} + +static int +intel_ntb_get_link_status(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + uint16_t reg_val; + int ret; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return -EINVAL; + } + + ret = rte_pci_read_config(hw->pci_dev, ®_val, + sizeof(reg_val), XEON_LINK_STATUS_OFFSET); + if (ret < 0) { + NTB_LOG(ERR, "Unable to get link status."); + return -EIO; + } + + hw->link_status = NTB_LNK_STA_ACTIVE(reg_val); + + if (hw->link_status) { + hw->link_speed = NTB_LNK_STA_SPEED(reg_val); + hw->link_width = NTB_LNK_STA_WIDTH(reg_val); + } else { + hw->link_speed = NTB_SPEED_NONE; + hw->link_width = NTB_WIDTH_NONE; + } + + return 0; +} + +static int +intel_ntb_set_link(struct rte_rawdev *dev, bool up) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t ntb_ctrl, reg_off; + void *reg_addr; + + reg_off = XEON_NTBCNTL_OFFSET; + reg_addr = (char *)hw->pci_dev->mem_resource[0].addr + reg_off; + ntb_ctrl = *((volatile uint32_t *)reg_addr); + + if (up) { + ntb_ctrl &= ~(NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK); + ntb_ctrl |= NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP; + ntb_ctrl |= NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP; + } else { + ntb_ctrl &= ~(NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP); + ntb_ctrl &= ~(NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP); + ntb_ctrl |= NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK; + } + + *((volatile uint32_t *)reg_addr) = ntb_ctrl; + + return 0; +} + +static uint32_t +intel_ntb_spad_read(struct rte_rawdev *dev, int spad, bool peer) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t spad_v, reg_off; + void *reg_addr; + + if (spad < 0 || spad >= hw->spad_cnt) { + NTB_LOG(ERR, "Invalid spad reg index."); + return 0; + } + + /* When peer is true, read peer spad reg */ + if (peer) + reg_off = XEON_B2B_SPAD_OFFSET; + else + reg_off = XEON_IM_SPAD_OFFSET; + reg_addr = (char *)hw->pci_dev->mem_resource[0].addr + + reg_off + (spad << 2); + spad_v = *((volatile uint32_t *)reg_addr); + + return spad_v; +} + +static int +intel_ntb_spad_write(struct rte_rawdev *dev, int spad, + bool peer, uint32_t spad_v) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t reg_off; + void *reg_addr; + + if (spad < 0 || spad >= hw->spad_cnt) { + NTB_LOG(ERR, "Invalid spad reg index."); + return -EINVAL; + } + + /* When peer is true, write peer spad reg */ + if (peer) + reg_off = XEON_B2B_SPAD_OFFSET; + else + reg_off = XEON_IM_SPAD_OFFSET; + reg_addr = (char *)hw->pci_dev->mem_resource[0].addr + + reg_off + (spad << 2); + + *((volatile uint32_t *)reg_addr) = spad_v; + + return 0; +} + +static uint64_t +intel_ntb_db_read(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + uint64_t db_off, db_bits; + void *db_addr; + + db_off = XEON_IM_INT_STATUS_OFFSET; + db_addr = (char *)hw->pci_dev->mem_resource[0].addr + db_off; + + db_bits = *((volatile uint64_t *)db_addr); + + return db_bits; +} + +static int +intel_ntb_db_clear(struct rte_rawdev *dev, uint64_t db_bits) +{ + struct ntb_hw *hw = dev->dev_private; + uint64_t db_off; + void *db_addr; + + db_off = XEON_IM_INT_STATUS_OFFSET; + db_addr = (char *)hw->pci_dev->mem_resource[0].addr + db_off; + + *((volatile uint64_t *)db_addr) = db_bits; + + return 0; +} + +static int +intel_ntb_db_set_mask(struct rte_rawdev *dev, uint64_t db_mask) +{ + struct ntb_hw *hw = dev->dev_private; + uint64_t db_m_off; + void *db_m_addr; + + db_m_off = XEON_IM_INT_DISABLE_OFFSET; + db_m_addr = (char *)hw->pci_dev->mem_resource[0].addr + db_m_off; + + db_mask |= hw->db_mask; + + *((volatile uint64_t *)db_m_addr) = db_mask; + + hw->db_mask = db_mask; + + return 0; +} + +static int +intel_ntb_peer_db_set(struct rte_rawdev *dev, uint8_t db_idx) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t db_off; + void *db_addr; + + if (((uint64_t)1 << db_idx) & ~hw->db_valid_mask) { + NTB_LOG(ERR, "Invalid doorbell."); + return -EINVAL; + } + + db_off = XEON_IM_DOORBELL_OFFSET + db_idx * 4; + db_addr = (char *)hw->pci_dev->mem_resource[0].addr + db_off; + + *((volatile uint32_t *)db_addr) = 1; + + return 0; +} + +static int +intel_ntb_vector_bind(struct rte_rawdev *dev, uint8_t intr, uint8_t msix) +{ + struct ntb_hw *hw = dev->dev_private; + uint8_t reg_off; + void *reg_addr; + + if (intr >= hw->db_cnt) { + NTB_LOG(ERR, "Invalid intr source."); + return -EINVAL; + } + + /* Bind intr source to msix vector */ + reg_off = XEON_INTVEC_OFFSET; + reg_addr = (char *)hw->pci_dev->mem_resource[0].addr + + reg_off + intr; + + *((volatile uint8_t *)reg_addr) = msix; + + return 0; +} + +/* operations for primary side of local ntb */ +const struct ntb_dev_ops intel_ntb_ops = { + .ntb_dev_init = intel_ntb_dev_init, + .get_peer_mw_addr = intel_ntb_get_peer_mw_addr, + .mw_set_trans = intel_ntb_mw_set_trans, + .get_link_status = intel_ntb_get_link_status, + .set_link = intel_ntb_set_link, + .spad_read = intel_ntb_spad_read, + .spad_write = intel_ntb_spad_write, + .db_read = intel_ntb_db_read, + .db_clear = intel_ntb_db_clear, + .db_set_mask = intel_ntb_db_set_mask, + .peer_db_set = intel_ntb_peer_db_set, + .vector_bind = intel_ntb_vector_bind, +}; diff --git a/drivers/raw/ntb_rawdev/ntb_hw_intel.h b/drivers/raw/ntb_rawdev/ntb_hw_intel.h new file mode 100644 index 000000000..4d1e64504 --- /dev/null +++ b/drivers/raw/ntb_rawdev/ntb_hw_intel.h @@ -0,0 +1,86 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ + +#ifndef _NTB_HW_INTEL_H_ +#define _NTB_HW_INTEL_H_ + +/* Ntb control and link status */ +#define NTB_CTL_CFG_LOCK 1 +#define NTB_CTL_DISABLE 2 +#define NTB_CTL_S2P_BAR2_SNOOP (1 << 2) +#define NTB_CTL_P2S_BAR2_SNOOP (1 << 4) +#define NTB_CTL_S2P_BAR4_SNOOP (1 << 6) +#define NTB_CTL_P2S_BAR4_SNOOP (1 << 8) +#define NTB_CTL_S2P_BAR5_SNOOP (1 << 12) +#define NTB_CTL_P2S_BAR5_SNOOP (1 << 14) + +#define NTB_LNK_STA_ACTIVE_BIT 0x2000 +#define NTB_LNK_STA_SPEED_MASK 0x000f +#define NTB_LNK_STA_WIDTH_MASK 0x03f0 +#define NTB_LNK_STA_ACTIVE(x) (!!((x) & NTB_LNK_STA_ACTIVE_BIT)) +#define NTB_LNK_STA_SPEED(x) ((x) & NTB_LNK_STA_SPEED_MASK) +#define NTB_LNK_STA_WIDTH(x) (((x) & NTB_LNK_STA_WIDTH_MASK) >> 4) + +/* Intel Skylake Xeon hardware */ +#define XEON_IMBAR1SZ_OFFSET 0x00d0 +#define XEON_IMBAR2SZ_OFFSET 0x00d1 +#define XEON_EMBAR1SZ_OFFSET 0x00d2 +#define XEON_EMBAR2SZ_OFFSET 0x00d3 +#define XEON_DEVCTRL_OFFSET 0x0098 +#define XEON_DEVSTS_OFFSET 0x009a +#define XEON_UNCERRSTS_OFFSET 0x014c +#define XEON_CORERRSTS_OFFSET 0x0158 +#define XEON_LINK_STATUS_OFFSET 0x01a2 + +#define XEON_NTBCNTL_OFFSET 0x0000 +#define XEON_BAR_INTERVAL_OFFSET 0x0010 +#define XEON_IMBAR1XBASE_OFFSET 0x0010 /* SBAR2XLAT */ +#define XEON_IMBAR1XLMT_OFFSET 0x0018 /* SBAR2LMT */ +#define XEON_IMBAR2XBASE_OFFSET 0x0020 /* SBAR4XLAT */ +#define XEON_IMBAR2XLMT_OFFSET 0x0028 /* SBAR4LMT */ +#define XEON_IM_INT_STATUS_OFFSET 0x0040 +#define XEON_IM_INT_DISABLE_OFFSET 0x0048 +#define XEON_IM_SPAD_OFFSET 0x0080 /* SPAD */ +#define XEON_USMEMMISS_OFFSET 0x0070 +#define XEON_INTVEC_OFFSET 0x00d0 +#define XEON_IM_DOORBELL_OFFSET 0x0100 /* SDOORBELL0 */ +#define XEON_B2B_SPAD_OFFSET 0x0180 /* B2B SPAD */ +#define XEON_EMBAR0XBASE_OFFSET 0x4008 /* B2B_XLAT */ +#define XEON_EMBAR1XBASE_OFFSET 0x4010 /* PBAR2XLAT */ +#define XEON_EMBAR1XLMT_OFFSET 0x4018 /* PBAR2LMT */ +#define XEON_EMBAR2XBASE_OFFSET 0x4020 /* PBAR4XLAT */ +#define XEON_EMBAR2XLMT_OFFSET 0x4028 /* PBAR4LMT */ +#define XEON_EM_INT_STATUS_OFFSET 0x4040 +#define XEON_EM_INT_DISABLE_OFFSET 0x4048 +#define XEON_EM_SPAD_OFFSET 0x4080 /* remote SPAD */ +#define XEON_EM_DOORBELL_OFFSET 0x4100 /* PDOORBELL0 */ +#define XEON_SPCICMD_OFFSET 0x4504 /* SPCICMD */ +#define XEON_EMBAR0_OFFSET 0x4510 /* SBAR0BASE */ +#define XEON_EMBAR1_OFFSET 0x4518 /* SBAR23BASE */ +#define XEON_EMBAR2_OFFSET 0x4520 /* SBAR45BASE */ + +#define XEON_PPD_OFFSET 0x00d4 +#define XEON_PPD_CONN_MASK 0x03 +#define XEON_PPD_CONN_TRANSPARENT 0x00 +#define XEON_PPD_CONN_B2B 0x01 +#define XEON_PPD_CONN_RP 0x02 +#define XEON_PPD_DEV_MASK 0x10 +#define XEON_PPD_DEV_USD 0x00 +#define XEON_PPD_DEV_DSD 0x10 +#define XEON_PPD_SPLIT_BAR_MASK 0x40 + + +#define XEON_MW_COUNT 2 + +#define XEON_DB_COUNT 32 +#define XEON_DB_LINK 32 +#define XEON_DB_LINK_BIT (1ULL << XEON_DB_LINK) +#define XEON_DB_MSIX_VECTOR_COUNT 33 +#define XEON_DB_MSIX_VECTOR_SHIFT 1 +#define XEON_DB_TOTAL_SHIFT 33 +#define XEON_SPAD_COUNT 16 + +extern const struct ntb_dev_ops intel_ntb_ops; + +#endif /* _NTB_HW_INTEL_H_ */ diff --git a/drivers/raw/ntb_rawdev/ntb_rawdev.c b/drivers/raw/ntb_rawdev/ntb_rawdev.c index 518373f8f..a03decd55 100644 --- a/drivers/raw/ntb_rawdev/ntb_rawdev.c +++ b/drivers/raw/ntb_rawdev/ntb_rawdev.c @@ -18,11 +18,13 @@ #include #include +#include "ntb_hw_intel.h" #include "ntb_rawdev.h" int ntb_logtype; static const struct rte_pci_id pci_id_ntb_map[] = { + { RTE_PCI_DEVICE(NTB_INTEL_VENDOR_ID, NTB_INTEL_DEV_ID_B2B_SKX) }, { .vendor_id = 0, /* sentinel */ }, }; @@ -363,6 +365,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width = NTB_WIDTH_NONE; switch (pci_dev->id.device_id) { + case NTB_INTEL_DEV_ID_B2B_SKX: + hw->ntb_ops = &intel_ntb_ops; + break; default: NTB_LOG(ERR, "Not supported device."); return -EINVAL; From patchwork Tue Jun 11 08:50:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Xiaoyun" X-Patchwork-Id: 54644 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 74F651C2F3; Tue, 11 Jun 2019 10:50:58 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 55AAC1C2DD for ; Tue, 11 Jun 2019 10:50:51 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jun 2019 01:50:51 -0700 X-ExtLoop1: 1 Received: from dpdk-xiaoyun3.sh.intel.com ([10.67.119.132]) by fmsmga008.fm.intel.com with ESMTP; 11 Jun 2019 01:50:50 -0700 From: Xiaoyun Li To: jingjing.wu@intel.com, keith.wiles@intel.com, cunming.liang@intel.com, omkar.maslekar@intel.com Cc: dev@dpdk.org, Xiaoyun Li Date: Tue, 11 Jun 2019 16:50:15 +0800 Message-Id: <20190611085018.56386-4-xiaoyun.li@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611085018.56386-1-xiaoyun.li@intel.com> References: <20190606074303.104108-1-xiaoyun.li@intel.com> <20190611085018.56386-1-xiaoyun.li@intel.com> Subject: [dpdk-dev] [PATCH v3 3/6] raw/ntb: add handshake process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add handshake process using doorbell so that two hosts can communicate to start and stop. Signed-off-by: Xiaoyun Li --- drivers/raw/ntb_rawdev/ntb_rawdev.c | 336 +++++++++++++++++++++++++++- 1 file changed, 335 insertions(+), 1 deletion(-) diff --git a/drivers/raw/ntb_rawdev/ntb_rawdev.c b/drivers/raw/ntb_rawdev/ntb_rawdev.c index a03decd55..d9088e825 100644 --- a/drivers/raw/ntb_rawdev/ntb_rawdev.c +++ b/drivers/raw/ntb_rawdev/ntb_rawdev.c @@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw(struct rte_rawdev *dev, int mw_idx, uint64_t mw_size) +{ + struct ntb_hw *hw = dev->dev_private; + char mw_name[RTE_MEMZONE_NAMESIZE]; + const struct rte_memzone *mz; + int ret = 0; + + if (hw->ntb_ops->mw_set_trans == NULL) { + NTB_LOG(ERR, "Not supported to set mw."); + return -ENOTSUP; + } + + snprintf(mw_name, sizeof(mw_name), "ntb_%d_mw_%d", + dev->dev_id, mw_idx); + + mz = rte_memzone_lookup(mw_name); + if (mz) + return 0; + + /** + * Hardware requires that mapped memory base address should be + * aligned with EMBARSZ and needs continuous memzone. + */ + mz = rte_memzone_reserve_aligned(mw_name, mw_size, dev->socket_id, + RTE_MEMZONE_IOVA_CONTIG, hw->mw_size[mw_idx]); + if (!mz) { + NTB_LOG(ERR, "Cannot allocate aligned memzone."); + return -EIO; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); + if (ret) { + NTB_LOG(ERR, "Cannot set mw translation."); + return ret; + } + + return ret; +} + +static void +ntb_link_cleanup(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + int status, i; + + if (hw->ntb_ops->spad_write == NULL || + hw->ntb_ops->mw_set_trans == NULL) { + NTB_LOG(ERR, "Not supported to clean up link."); + return; + } + + /* Clean spad registers. */ + for (i = 0; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i, 0, 0); + if (status) + NTB_LOG(ERR, "Failed to clean local spad."); + } + + /* Clear mw so that peer cannot access local memory.*/ + for (i = 0; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i, 0, 0); + if (status) + NTB_LOG(ERR, "Failed to clean mw."); + } +} + +static void +ntb_dev_intr_handler(void *param) +{ + struct rte_rawdev *dev = (struct rte_rawdev *)param; + struct ntb_hw *hw = dev->dev_private; + uint32_t mw_size_h, mw_size_l; + uint64_t db_bits = 0; + int i = 0; + + if (hw->ntb_ops->db_read == NULL || + hw->ntb_ops->db_clear == NULL || + hw->ntb_ops->peer_db_set == NULL) { + NTB_LOG(ERR, "Doorbell is not supported."); + return; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); + if (!db_bits) + NTB_LOG(ERR, "No doorbells"); + + /* Doorbell 0 is for peer device ready. */ + if (db_bits & 1) { + NTB_LOG(DEBUG, "DB0: Peer device is up."); + /* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, 1); + + /** + * Peer dev is already up. All mw settings are already done. + * Skip them. + */ + if (hw->peer_dev_up) + return; + + if (hw->ntb_ops->spad_read == NULL || + hw->ntb_ops->spad_write == NULL) { + NTB_LOG(ERR, "Scratchpad is not supported."); + return; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev, SPAD_NUM_MWS, 0); + hw->peer_mw_size = rte_zmalloc("uint64_t", + hw->peer_mw_cnt * sizeof(uint64_t), 0); + for (i = 0; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev, SPAD_MW0_SZ_H + 2 * i, 0); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev, SPAD_MW0_SZ_L + 2 * i, 0); + hw->peer_mw_size[i] = ((uint64_t)mw_size_h << 32) | + mw_size_l; + NTB_LOG(DEBUG, "Peer %u mw size: 0x%"PRIx64"", i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up = 1; + + /** + * Handshake with peer. Spad_write only works when both + * devices are up. So write spad again when db is received. + * And set db again for the later device who may miss + * the 1st db. + */ + for (i = 0; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev, SPAD_NUM_MWS, + 1, hw->mw_cnt); + mw_size_h = hw->mw_size[i] >> 32; + (*hw->ntb_ops->spad_write)(dev, SPAD_MW0_SZ_H + 2 * i, + 1, mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev, SPAD_MW0_SZ_L + 2 * i, + 1, mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev, 0); + + /* To get the link info. */ + if (hw->ntb_ops->get_link_status == NULL) { + NTB_LOG(ERR, "Not supported to get link status."); + return; + } + (*hw->ntb_ops->get_link_status)(dev); + NTB_LOG(INFO, "Link is up. Link speed: %u. Link width: %u", + hw->link_speed, hw->link_width); + return; + } + + if (db_bits & (1 << 1)) { + NTB_LOG(DEBUG, "DB1: Peer device is down."); + /* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, 2); + + /* Peer device will be down, So clean local side too. */ + ntb_link_cleanup(dev); + + hw->peer_dev_up = 0; + /* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev, 2); + return; + } + + if (db_bits & (1 << 2)) { + NTB_LOG(DEBUG, "DB2: Peer device agrees dev to be down."); + /* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 << 2)); + hw->peer_dev_up = 0; + return; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused, @@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused, static int ntb_dev_start(struct rte_rawdev *dev) { + struct ntb_hw *hw = dev->dev_private; + int ret, i; + /* TODO: init queues and start queues. */ + + /* Map memory of bar_size to remote. */ + hw->mz = rte_zmalloc("struct rte_memzone *", + hw->mw_cnt * sizeof(struct rte_memzone *), 0); + for (i = 0; i < hw->mw_cnt; i++) { + ret = ntb_set_mw(dev, i, hw->mw_size[i]); + if (ret) { + NTB_LOG(ERR, "Fail to set mw."); + return ret; + } + } + dev->started = 1; return 0; @@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev) static void ntb_dev_stop(struct rte_rawdev *dev) { + struct ntb_hw *hw = dev->dev_private; + uint32_t time_out; + int status; + /* TODO: stop rx/tx queues. */ + + if (!hw->peer_dev_up) + goto clean; + + ntb_link_cleanup(dev); + + /* Notify the peer that device will be down. */ + if (hw->ntb_ops->peer_db_set == NULL) { + NTB_LOG(ERR, "Peer doorbell setting is not supported."); + return; + } + status = (*hw->ntb_ops->peer_db_set)(dev, 1); + if (status) { + NTB_LOG(ERR, "Failed to tell peer device is down."); + return; + } + + /* + * Set time out as 1s in case that the peer is stopped accidently + * without any notification. + */ + time_out = 1000000; + + /* Wait for cleanup work down before db mask clear. */ + while (hw->peer_dev_up && time_out) { + time_out -= 10; + rte_delay_us(10); + } + +clean: + /* Clear doorbells mask. */ + if (hw->ntb_ops->db_set_mask == NULL) { + NTB_LOG(ERR, "Doorbell mask setting is not supported."); + return; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t)1 << hw->db_cnt) - 1)); + if (status) + NTB_LOG(ERR, "Failed to clear doorbells."); + dev->started = 0; } static int ntb_dev_close(struct rte_rawdev *dev) { + struct ntb_hw *hw = dev->dev_private; + struct rte_intr_handle *intr_handle; int ret = 0; if (dev->started) @@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev) /* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; + /* Clean datapath event and vec mapping */ + rte_intr_efd_disable(intr_handle); + if (intr_handle->intr_vec) { + rte_free(intr_handle->intr_vec); + intr_handle->intr_vec = NULL; + } + /* Disable uio intr before callback unregister */ + rte_intr_disable(intr_handle); + + /* Unregister callback func to eal lib */ + rte_intr_callback_unregister(intr_handle, + ntb_dev_intr_handler, dev); + return ret; } @@ -356,7 +608,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; - int ret; + struct rte_intr_handle *intr_handle; + uint32_t val; + int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up = 0; @@ -387,6 +641,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) if (ret) return ret; + /* Init doorbell. */ + hw->db_valid_mask = ((uint64_t)1 << hw->db_cnt) - 1; + + intr_handle = &pci_dev->intr_handle; + /* Register callback func to eal lib */ + rte_intr_callback_register(intr_handle, + ntb_dev_intr_handler, dev); + + ret = rte_intr_efd_enable(intr_handle, hw->db_cnt); + if (ret) + return ret; + + /* To clarify, the interrupt for each doorbell is already mapped + * by default for intel gen3. They are mapped to msix vec 1-32, + * and hardware intr is mapped to 0. Map all to 0 for uio. + */ + if (!rte_intr_cap_multiple(intr_handle)) { + for (i = 0; i < hw->db_cnt; i++) { + if (hw->ntb_ops->vector_bind == NULL) + return -ENOTSUP; + ret = (*hw->ntb_ops->vector_bind)(dev, i, 0); + if (ret) + return ret; + } + } + + if (hw->ntb_ops->db_set_mask == NULL || + hw->ntb_ops->peer_db_set == NULL) { + NTB_LOG(ERR, "Doorbell is not supported."); + return -ENOTSUP; + } + hw->db_mask = 0; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); + if (ret) { + NTB_LOG(ERR, "Unanle to enable intr for all dbs."); + return ret; + } + + /* enable uio intr after callback register */ + rte_intr_enable(intr_handle); + + if (hw->ntb_ops->spad_write == NULL) { + NTB_LOG(ERR, "Scratchpad is not supported."); + return -ENOTSUP; + } + /* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev, SPAD_NUM_MWS, 1, hw->mw_cnt); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer mw count."); + return ret; + } + + /* Tell peer each mw size on local side. */ + for (i = 0; i < hw->mw_cnt; i++) { + NTB_LOG(DEBUG, "Local %u mw size: 0x%"PRIx64"", i, + hw->mw_size[i]); + val = hw->mw_size[i] >> 32; + ret = (*hw->ntb_ops->spad_write) + (dev, SPAD_MW0_SZ_H + 2 * i, 1, val); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer mw size."); + return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev, SPAD_MW0_SZ_L + 2 * i, 1, val); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer mw size."); + return ret; + } + } + + /* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev, 0); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer device is probed."); + return ret; + } + return ret; } From patchwork Tue Jun 11 08:50:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Xiaoyun" X-Patchwork-Id: 54645 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 64CF81C2F8; Tue, 11 Jun 2019 10:51:01 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 474351C2E3 for ; Tue, 11 Jun 2019 10:50:53 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jun 2019 01:50:52 -0700 X-ExtLoop1: 1 Received: from dpdk-xiaoyun3.sh.intel.com ([10.67.119.132]) by fmsmga008.fm.intel.com with ESMTP; 11 Jun 2019 01:50:51 -0700 From: Xiaoyun Li To: jingjing.wu@intel.com, keith.wiles@intel.com, cunming.liang@intel.com, omkar.maslekar@intel.com Cc: dev@dpdk.org, Xiaoyun Li Date: Tue, 11 Jun 2019 16:50:16 +0800 Message-Id: <20190611085018.56386-5-xiaoyun.li@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611085018.56386-1-xiaoyun.li@intel.com> References: <20190606074303.104108-1-xiaoyun.li@intel.com> <20190611085018.56386-1-xiaoyun.li@intel.com> Subject: [dpdk-dev] [PATCH v3 4/6] examples/ntb: enable an example for ntb X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'. But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M. Signed-off-by: Xiaoyun Li --- drivers/raw/ntb_rawdev/ntb_rawdev.c | 28 +- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 +++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 387 ++++++++++++++++++++++++++++ 6 files changed, 493 insertions(+), 9 deletions(-) create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c diff --git a/drivers/raw/ntb_rawdev/ntb_rawdev.c b/drivers/raw/ntb_rawdev/ntb_rawdev.c index d9088e825..9d7b8c07b 100644 --- a/drivers/raw/ntb_rawdev/ntb_rawdev.c +++ b/drivers/raw/ntb_rawdev/ntb_rawdev.c @@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { - RTE_SET_USED(dev); - RTE_SET_USED(buffers); - RTE_SET_USED(count); - RTE_SET_USED(context); + /* Not FIFO right now. Just for testing memory write. */ + struct ntb_hw *hw = dev->dev_private; + unsigned int i; + void *bar_addr; + size_t size; + + if (hw->ntb_ops->get_peer_mw_addr == NULL) + return -ENOTSUP; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev, 0); + size = (size_t)context; + for (i = 0; i < count; i++) + rte_memcpy(bar_addr, buffers[i]->buf_addr, size); return 0; } @@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { - RTE_SET_USED(dev); - RTE_SET_USED(buffers); - RTE_SET_USED(count); - RTE_SET_USED(context); + /* Not FIFO. Just for testing memory read. */ + struct ntb_hw *hw = dev->dev_private; + unsigned int i; + size_t size; + + size = (size_t)context; + for (i = 0; i < count; i++) + rte_memcpy(buffers[i]->buf_addr, hw->mz[i]->addr, size); return 0; } diff --git a/examples/Makefile b/examples/Makefile index 7562424d9..de11dd487 100644 --- a/examples/Makefile +++ b/examples/Makefile @@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread diff --git a/examples/meson.build b/examples/meson.build index c695d52c9..2a4a084af 100644 --- a/examples/meson.build +++ b/examples/meson.build @@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks', diff --git a/examples/ntb/Makefile b/examples/ntb/Makefile new file mode 100644 index 000000000..5ddd9b95f --- /dev/null +++ b/examples/ntb/Makefile @@ -0,0 +1,68 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# binary name +APP = ntb_fwd + +# all source are stored in SRCS-y +SRCS-y := ntb_fwd.c + +# Build using pkg-config variables if possible +$(shell pkg-config --exists libdpdk) +ifeq ($(.SHELLSTATUS),0) + +all: shared +.PHONY: shared static +shared: build/$(APP)-shared + ln -sf $(APP)-shared build/$(APP) +static: build/$(APP)-static + ln -sf $(APP)-static build/$(APP) + +CFLAGS += -D_FILE_OFFSET_BITS=64 +LDFLAGS += -pthread + +PC_FILE := $(shell pkg-config --path libdpdk) +CFLAGS += -O3 $(shell pkg-config --cflags libdpdk) +LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk) +LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk) + +build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED) + +build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC) + +build: + @mkdir -p $@ + +.PHONY: clean +clean: + rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared + rmdir --ignore-fail-on-non-empty build + +else # Build using legacy build system + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +ifneq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) +$(info This application can only operate in a linuxapp environment, \ +please change the definition of the RTE_TARGET environment variable) +all: +else + +CFLAGS += -D_FILE_OFFSET_BITS=64 +CFLAGS += -O2 +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API + +include $(RTE_SDK)/mk/rte.extapp.mk + +endif +endif diff --git a/examples/ntb/meson.build b/examples/ntb/meson.build new file mode 100644 index 000000000..9a6288f4f --- /dev/null +++ b/examples/ntb/meson.build @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +) diff --git a/examples/ntb/ntb_fwd.c b/examples/ntb/ntb_fwd.c new file mode 100644 index 000000000..0f54d0e2d --- /dev/null +++ b/examples/ntb/ntb_fwd.c @@ -0,0 +1,387 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#define NTB_DRV_NAME_LEN 7 +static uint64_t max_file_size = 0x400000; +static uint8_t interactive = 1; +static uint16_t dev_id; + +/* *** Help command with introduction. *** */ +struct cmd_help_result { + cmdline_fixed_string_t help; +}; + +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result, + struct cmdline *cl, + __attribute__((unused)) void *data) +{ + cmdline_printf( + cl, + "\n" + "The following commands are currently available:\n\n" + "Control:\n" + " quit :" + " Quit the application.\n" + "\nFile transmit:\n" + " send [path] :" + " Send [path] file. (No more than %"PRIu64")\n" + " recv [path] :" + " Receive file to [path]. Make sure sending is done" + " on the other side.\n", + max_file_size + ); + +} + +cmdline_parse_token_string_t cmd_help_help = + TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help"); + +cmdline_parse_inst_t cmd_help = { + .f = cmd_help_parsed, + .data = NULL, + .help_str = "show help", + .tokens = { + (void *)&cmd_help_help, + NULL, + }, +}; + +/* *** QUIT *** */ +struct cmd_quit_result { + cmdline_fixed_string_t quit; +}; + +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result, + struct cmdline *cl, + __attribute__((unused)) void *data) +{ + /* Stop traffic and Close port. */ + rte_rawdev_stop(dev_id); + rte_rawdev_close(dev_id); + + cmdline_quit(cl); +} + +cmdline_parse_token_string_t cmd_quit_quit = + TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit"); + +cmdline_parse_inst_t cmd_quit = { + .f = cmd_quit_parsed, + .data = NULL, + .help_str = "exit application", + .tokens = { + (void *)&cmd_quit_quit, + NULL, + }, +}; + +/* *** SEND FILE PARAMETERS *** */ +struct cmd_sendfile_result { + cmdline_fixed_string_t send_string; + char filepath[]; +}; + +static void +cmd_sendfile_parsed(void *parsed_result, + __attribute__((unused)) struct cmdline *cl, + __attribute__((unused)) void *data) +{ + struct cmd_sendfile_result *res = parsed_result; + struct rte_rawdev_buf *pkts_send[1]; + uint64_t rsize, size, link; + uint8_t *buff; + uint32_t val; + FILE *file; + + if (!rte_rawdevs[dev_id].started) { + printf("Device needs to be up first. Try later.\n"); + return; + } + + rte_rawdev_get_attr(dev_id, "link_status", &link); + if (!link) { + printf("Link is not up, cannot send file.\n"); + return; + } + + if (res->filepath == NULL) { + printf("Fail to get filepath.\n"); + return; + } + + file = fopen(res->filepath, "r"); + if (file == NULL) { + printf("Fail to open the file.\n"); + return; + } + + fseek(file, 0, SEEK_END); + size = ftell(file); + fseek(file, 0, SEEK_SET); + + /** + * No FIFO now. Only test memory. Limit sending file + * size <= max_file_size. + */ + if (size > max_file_size) { + printf("Warning: The file is too large. Only send first" + " %"PRIu64" bits.\n", max_file_size); + size = max_file_size; + } + + buff = (uint8_t *)malloc(size); + rsize = fread(buff, size, 1, file); + if (rsize != 1) { + printf("Fail to read file.\n"); + fclose(file); + free(buff); + return; + } + + /* Tell remote about the file size. */ + val = size >> 32; + rte_rawdev_set_attr(dev_id, "spad14", val); + val = size; + rte_rawdev_set_attr(dev_id, "spad15", val); + + pkts_send[0] = (struct rte_rawdev_buf *)malloc + (sizeof(struct rte_rawdev_buf)); + pkts_send[0]->buf_addr = buff; + + if (rte_rawdev_enqueue_buffers(dev_id, pkts_send, 1, + (void *)(size_t)size)) { + printf("Fail to enqueue.\n"); + goto clean; + } + printf("Done sending file.\n"); + +clean: + fclose(file); + free(buff); + free(pkts_send[0]); +} + +cmdline_parse_token_string_t cmd_send_file_send = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string, + "send"); +cmdline_parse_token_string_t cmd_send_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_send_file = { + .f = cmd_sendfile_parsed, + .data = NULL, + .help_str = "send ", + .tokens = { + (void *)&cmd_send_file_send, + (void *)&cmd_send_file_filepath, + NULL, + }, +}; + +/* *** RECEIVE FILE PARAMETERS *** */ +struct cmd_recvfile_result { + cmdline_fixed_string_t recv_string; + char filepath[]; +}; + +static void +cmd_recvfile_parsed(void *parsed_result, + __attribute__((unused)) struct cmdline *cl, + __attribute__((unused)) void *data) +{ + struct cmd_sendfile_result *res = parsed_result; + struct rte_rawdev_buf *pkts_recv[1]; + uint8_t *buff; + uint64_t val; + size_t size; + FILE *file; + + if (!rte_rawdevs[dev_id].started) { + printf("Device needs to be up first. Try later.\n"); + return; + } + + rte_rawdev_get_attr(dev_id, "link_status", &val); + if (!val) { + printf("Link is not up, cannot receive file.\n"); + return; + } + + if (res->filepath == NULL) { + printf("Fail to get filepath.\n"); + return; + } + + file = fopen(res->filepath, "w"); + if (file == NULL) { + printf("Fail to open the file.\n"); + return; + } + + rte_rawdev_get_attr(dev_id, "spad14", &val); + size = val << 32; + rte_rawdev_get_attr(dev_id, "spad15", &val); + size |= val; + + buff = (uint8_t *)malloc(size); + pkts_recv[0] = (struct rte_rawdev_buf *)malloc + (sizeof(struct rte_rawdev_buf)); + pkts_recv[0]->buf_addr = buff; + + if (rte_rawdev_dequeue_buffers(dev_id, pkts_recv, 1, (void *)size)) { + printf("Fail to dequeue.\n"); + goto clean; + } + + fwrite(buff, size, 1, file); + printf("Done receiving to file.\n"); + +clean: + fclose(file); + free(buff); + free(pkts_recv[0]); +} + +cmdline_parse_token_string_t cmd_recv_file_recv = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, recv_string, + "recv"); +cmdline_parse_token_string_t cmd_recv_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_recv_file = { + .f = cmd_recvfile_parsed, + .data = NULL, + .help_str = "recv ", + .tokens = { + (void *)&cmd_recv_file_recv, + (void *)&cmd_recv_file_filepath, + NULL, + }, +}; + +/* list of instructions */ +cmdline_parse_ctx_t main_ctx[] = { + (cmdline_parse_inst_t *)&cmd_help, + (cmdline_parse_inst_t *)&cmd_send_file, + (cmdline_parse_inst_t *)&cmd_recv_file, + (cmdline_parse_inst_t *)&cmd_quit, + NULL, +}; + +/* prompt function, called from main on MASTER lcore */ +static void +prompt(void) +{ + struct cmdline *cl; + + cl = cmdline_stdin_new(main_ctx, "ntb> "); + if (cl == NULL) + return; + + cmdline_interact(cl); + cmdline_stdin_exit(cl); +} + +static void +signal_handler(int signum) +{ + if (signum == SIGINT || signum == SIGTERM) { + printf("\nSignal %d received, preparing to exit...\n", signum); + signal(signum, SIG_DFL); + kill(getpid(), signum); + } +} + +static void +ntb_usage(const char *prgname) +{ + printf("%s [EAL options] -- [options]\n" + "-i : run in interactive mode (default value is 1)\n", + prgname); +} + +static int +parse_args(int argc, char **argv) +{ + char *prgname = argv[0], **argvopt = argv; + int opt, ret; + + /* Only support interactive mode to send/recv file first. */ + while ((opt = getopt(argc, argvopt, "i")) != EOF) { + switch (opt) { + case 'i': + printf("Interactive-mode selected\n"); + interactive = 1; + break; + + default: + ntb_usage(prgname); + return -1; + } + } + + if (optind >= 0) + argv[optind-1] = prgname; + + ret = optind-1; + optind = 1; /* reset getopt lib */ + return ret; +} + +int +main(int argc, char **argv) +{ + int ret, i; + + signal(SIGINT, signal_handler); + signal(SIGTERM, signal_handler); + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization.\n"); + + /* Find 1st ntb rawdev. */ + for (i = 0; i < RTE_RAWDEV_MAX_DEVS; i++) + if (rte_rawdevs[i].driver_name && + (strncmp(rte_rawdevs[i].driver_name, "raw_ntb", + NTB_DRV_NAME_LEN) == 0) && (rte_rawdevs[i].attached == 1)) + break; + + if (i == RTE_RAWDEV_MAX_DEVS) + rte_exit(EXIT_FAILURE, "Cannot find any ntb device.\n"); + + dev_id = i; + + argc -= ret; + argv += ret; + + ret = parse_args(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid arguments\n"); + + rte_rawdev_start(dev_id); + + if (interactive) { + sleep(1); + prompt(); + } + + return 0; +} From patchwork Tue Jun 11 08:50:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Xiaoyun" X-Patchwork-Id: 54646 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 82D591C2FE; Tue, 11 Jun 2019 10:51:04 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 10FEB1C2E6 for ; Tue, 11 Jun 2019 10:50:54 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jun 2019 01:50:54 -0700 X-ExtLoop1: 1 Received: from dpdk-xiaoyun3.sh.intel.com ([10.67.119.132]) by fmsmga008.fm.intel.com with ESMTP; 11 Jun 2019 01:50:53 -0700 From: Xiaoyun Li To: jingjing.wu@intel.com, keith.wiles@intel.com, cunming.liang@intel.com, omkar.maslekar@intel.com Cc: dev@dpdk.org, Xiaoyun Li Date: Tue, 11 Jun 2019 16:50:17 +0800 Message-Id: <20190611085018.56386-6-xiaoyun.li@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611085018.56386-1-xiaoyun.li@intel.com> References: <20190606074303.104108-1-xiaoyun.li@intel.com> <20190611085018.56386-1-xiaoyun.li@intel.com> Subject: [dpdk-dev] [PATCH v3 5/6] usertools/dpdk-devbind.py: add support for ntb X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In order to allow binding/unbinding of devices for use by the ntb_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. And only support skx platform right now. Signed-off-by: Xiaoyun Li --- usertools/dpdk-devbind.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 9e79f0d28..6e6f64bd3 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -36,11 +36,15 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties @@ -595,6 +599,9 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices , "Misc") + def parse_args(): '''Parses the command-line arguments given by the user and takes the @@ -670,6 +677,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status() @@ -690,6 +698,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__": From patchwork Tue Jun 11 08:50:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Xiaoyun" X-Patchwork-Id: 54647 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2D1011C307; Tue, 11 Jun 2019 10:51:07 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id C1CFA1C2EA for ; Tue, 11 Jun 2019 10:50:56 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jun 2019 01:50:56 -0700 X-ExtLoop1: 1 Received: from dpdk-xiaoyun3.sh.intel.com ([10.67.119.132]) by fmsmga008.fm.intel.com with ESMTP; 11 Jun 2019 01:50:55 -0700 From: Xiaoyun Li To: jingjing.wu@intel.com, keith.wiles@intel.com, cunming.liang@intel.com, omkar.maslekar@intel.com Cc: dev@dpdk.org, Xiaoyun Li Date: Tue, 11 Jun 2019 16:50:18 +0800 Message-Id: <20190611085018.56386-7-xiaoyun.li@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190611085018.56386-1-xiaoyun.li@intel.com> References: <20190606074303.104108-1-xiaoyun.li@intel.com> <20190611085018.56386-1-xiaoyun.li@intel.com> Subject: [dpdk-dev] [PATCH v3 6/6] doc: update docs for ntb driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Update related documents for ntb pmd and example. Signed-off-by: Xiaoyun Li --- MAINTAINERS | 8 +++++ doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb_rawdev.rst | 41 ++++++++++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 15 ++++++++ doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++++++++++++++++++++++++ 6 files changed, 113 insertions(+) create mode 100644 doc/guides/rawdevs/ntb_rawdev.rst create mode 100644 doc/guides/sample_app_ug/ntb.rst diff --git a/MAINTAINERS b/MAINTAINERS index 0212fe6d0..b97cc18ba 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1056,6 +1056,10 @@ M: Nipun Gupta F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB Rawdev +M: Xiaoyun Li +F: drivers/raw/ntb_rawdev/ +F: doc/guides/rawdevs/ntb_rawdev.rst Packet processing ----------------- @@ -1432,3 +1436,7 @@ F: examples/tep_termination/ F: examples/vmdq/ F: examples/vmdq_dcb/ F: doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst + +M: Xiaoyun Li +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst index 7c3bd9586..cf6fcb06b 100644 --- a/doc/guides/rawdevs/index.rst +++ b/doc/guides/rawdevs/index.rst @@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ntb_rawdev diff --git a/doc/guides/rawdevs/ntb_rawdev.rst b/doc/guides/rawdevs/ntb_rawdev.rst new file mode 100644 index 000000000..429e2af3e --- /dev/null +++ b/doc/guides/rawdevs/ntb_rawdev.rst @@ -0,0 +1,41 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +NTB Rawdev Driver +================= + +The ``ntb`` rawdev driver provides a non-transparent bridge between two +separate hosts so that they can communicate with each other. Thus, many +user cases can benefit from this, such as fault tolerance and visual +acceleration. + +This PMD allows two hosts to handshake for device start and stop, memory +allocation for the peer to access and read/write allocated memory from peer. +Also, the PMD allows to use doorbell registers to notify the peer and share +some information by using scratchpad registers. + +But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. +And this PMD only supports intel skylake platform. + +BIOS setting on skylake platform +-------------------------------- + +Intel non-transparent bridge needs special BIOS setting. Since the PMD only +supports intel skylake platform, introduce BIOS setting here. The referencce +is https://www.intel.com/content/dam/support/us/en/documents/server-products/Intel_Xeon_Processor_Scalable_Family_BIOS_User_Guide.pdf + +- Set the needed PCIe port as NTB to NTB mode on both hosts. +- Enable NTB bars and set bar size of bar 23 and bar 45 as 12-29 (2K-512M) + on both hosts. Note that bar size on both hosts should be the same. +- Disable split bars for both hosts. +- Set crosslink control override as DSD/USP on one host, USD/DSP on + another host. +- Disable PCIe PII SSC (Spread Spectrum Clocking) for both hosts. This + is a hardware requirement. + +Build options +------------- + +- ``CONFIG_RTE_LIBRTE_IFPGA_RAWDEV`` (default ``y``) + + Toggle compilation of the ``ntb_rawdev`` driver. diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst index 575c590d9..a4c41a8c1 100644 --- a/doc/guides/rel_notes/release_19_08.rst +++ b/doc/guides/rel_notes/release_19_08.rst @@ -72,6 +72,21 @@ New Features Added the new Shared Memory Packet Interface (``memif``) PMD. See the :doc:`../nics/memif` guide for more details on this new driver. +* **Introduced NTB PMD.** + + The PMD provided a non-transparent bridge between two separate hosts so + that they can communicate with each other. Thus, many user cases can + benefit from this, such as fault tolerance and visual acceleration. + + This PMD implemented the following features: + * Handshake for device start and stop between two hosts. + * Memory allocation for the peer to access and read/write allocated + memory from peer. + * Use doorbell registers to notify the peer and share some information + by using scratchpad registers. + + But the PMD hasn't implemented FIFO. The FIFO will come in 19.11 release. + And this PMD only supports intel skylake platform. Removed Items ------------- diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index 2945be08f..f23f8f59e 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb diff --git a/doc/guides/sample_app_ug/ntb.rst b/doc/guides/sample_app_ug/ntb.rst new file mode 100644 index 000000000..079242175 --- /dev/null +++ b/doc/guides/sample_app_ug/ntb.rst @@ -0,0 +1,47 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +NTB Sample Application +====================== + +The ntb sample application shows how to use ntb rawdev driver. +This sample provides interactive mode to transmit file between +two hosts. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ntb`` sub-directory. + +Running the Application +----------------------- + +The application requires an available core for each port, plus one. +The only available options are the standard ones for the EAL: + +.. code-block:: console + + ./build/ntb_fwd -c 0xf -n 6 -- -i + +Refer to the *DPDK Getting Started Guide* for general information on +running applications and the Environment Abstraction Layer (EAL) +options. + +Using the application +--------------------- + +The application is console-driven using the cmdline DPDK interface: + +.. code-block:: console + + ntb> + +From this interface the available commands and descriptions of what +they do as as follows: + +* ``send [filepath]``: Send file to the peer host. +* ``receive [filepath]``: Receive file to [filepath]. Need the peer + to send file successfully first. +* ``quit``: Exit program