From patchwork Mon May 23 09:35:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111608 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B5164A04FD; Mon, 23 May 2022 12:22:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E3F32410E8; Mon, 23 May 2022 12:22:34 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 46EFB4067C for ; Mon, 23 May 2022 12:22:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301353; x=1684837353; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=TRTpkT90FRTJDNDeWIC6LYxeUfE7ZO2BJIXAQyfvuYk=; b=ZCT9N+FC7OwhlIgdnU19bZSa5ZyruEyrOgnCbdvbEZz59o17RquXHXA9 U0RVqK1pMynMpIJ3QR+Wf2quHW3IjSaQeyxoj3Jp+k9VGRWHYj1at6vhu u8t2LdWqYs9QbEpEryt4QYfrV2MuDObArAZ96PC9yhhf33Z3m0RdYkmyc a5unJyc79JzCdUakh9U5wSVLSXeV34UY4mhbnm0c9POTtYH/kcYGnVVLv Qpz8T9HrilmG1/62jH66Qjzwt/e5p0zjRnlciPbG4faDSZq4iqXmh/iQJ MB6MWdGOaJFYgpTWIdlXjU+BEroClAzV6OOqvrb35SCA+HCBNVWnf/Hty Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239266" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239266" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889349" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:30 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 01/13] vdpa/ifc: add support for virtio blk device Date: Mon, 23 May 2022 17:35:12 +0800 Message-Id: <1653298524-232138-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++--- drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 102 insertions(+), 15 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..a761d49 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,12 +5,19 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" -#define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_DEVICE_ID 0x1041 -#define IFCVF_SUBSYS_VENDOR_ID 0x8086 -#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +#define IFCVF_VENDOR_ID 0x1AF4 +#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_SUBSYS_VENDOR_ID 0x8086 +#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_BLK_DEVICE_ID 0x0002 #define IFCVF_MAX_QUEUES 1 @@ -126,13 +133,18 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + int device_type; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9f05595..1eed90b 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -75,6 +75,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1138,7 +1144,7 @@ struct internal_list { return 0; } -static struct rte_vdpa_dev_ops ifcvf_ops = { +static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, .get_protocol_features = ifcvf_get_protocol_features, @@ -1167,6 +1173,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_net_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1178,6 +1226,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1227,13 +1276,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->hw.device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->hw.device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1245,7 +1305,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->hw.device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1308,11 +1369,25 @@ struct internal_list { static const struct rte_pci_id pci_id_ifcvf_map[] = { { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, - .device_id = IFCVF_DEVICE_ID, + .device_id = IFCVF_NET_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Mon May 23 09:35:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111609 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F282DA04FD; Mon, 23 May 2022 12:22:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DF1E64161A; Mon, 23 May 2022 12:22:37 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 8FB834161A for ; Mon, 23 May 2022 12:22:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301355; x=1684837355; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2SO/NXrVgSokERE7L5aHhYOoSldzv4OKs1wMFicFg2A=; b=BhRAHkBBQJDWkPz2R+nAl0iNqCU57rH6+9mtPGozRl/8pk6SVLYmz1aM L/esAOo5XKb0Tk85wUIu4yQFtRZ8LeyF4op3W4wYi8GL1xyHNBT9CiFpj 5NaU7o0AeO7CwAJB6zBdFHgapR2XUqXBxhtVeX9Z/yYOXx+/WL6t9PPHc hn6qnPL5ncYEW7GMHmUBUyZ6Y7PlhE32rvEKRtfJYnn8sqMkFE5EZMfex OQ/tzisNwHMYEpdXlcCRjXisrmFCn+c7mMfhhYRYMkGvav44jbxImc71r Je6tJPHCuKGbQkAakO0Uf9IDk/FeBkY7aKBhz2/aRN+3ts5PxzGOqEmST Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239276" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239276" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889371" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:33 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 02/13] vhost: add vDPA ops for blk device Date: Mon, 23 May 2022 17:35:13 +0800 Message-Id: <1653298524-232138-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 88138be..c4233a6 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t size); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Mon May 23 09:35:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111610 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5B401A04FD; Mon, 23 May 2022 12:22:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2679D42685; Mon, 23 May 2022 12:22:40 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 2C2B442685 for ; Mon, 23 May 2022 12:22:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301358; x=1684837358; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bmUC8oiGHBtoKQtKt6JczWEAS2m3rLR/99gZZGUBd9E=; b=AzZJwUP6ipZLDGU1iX1zu3ISa3gm6ieyahr7FkuKjmB63jIxNnPoz/80 zjQEvyN8Kbw8tXkfUgJUaFhd4qVwu3OfBv6ra4rm8pDtXslaMvzjjwTJs dEyBqPIANVRgX2SYMoeT4c2iEbeYX0ZlWC76tPa5ujwFLKs5RXrnfi2So rOQcYoSW5M4pa1THr/OS/r1JzTK+2J7PWFukcjd4uAVDQ0uehM+Wb5odB JI0teChEApTQCYQQEItXf1zoUfXOFhZzFM3DEqAyzGorFqlgLVo8Uqa8d Hdf/lqqKWPBSuFZZmcAFkNb0xrScHJJQSJlCTYV8twXu4ZOFsDTt/zvho A==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239283" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239283" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889392" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:35 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 03/13] vhost: add vhost msg support for get/set config Date: Mon, 23 May 2022 17:35:14 +0800 Message-Id: <1653298524-232138-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 101 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 850848c..b37d814 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2468,6 +2468,92 @@ static int is_vring_iotlb(struct virtio_net *dev, } static int +vhost_user_get_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (!vdpa_dev) { + VHOST_LOG_CONFIG(ERR, + "(%s) is not vDPA device!\n", + dev->ifname); + goto out; + } + + if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); + if (ret != 0) { + ctx->msg.size = 0; + VHOST_LOG_CONFIG(ERR, + "(%s) get_config() return error!\n", + dev->ifname); + } + } else { + VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n", + dev->ifname); + } + +out: + return RTE_VHOST_MSG_RESULT_REPLY; +} + +static int +vhost_user_set_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) { + VHOST_LOG_CONFIG(ERR, + "(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, + VHOST_USER_MAX_CONFIG_SIZE); + goto out; + } + + if (!vdpa_dev) { + VHOST_LOG_CONFIG(ERR, + "(%s) is not vDPA device!\n", + dev->ifname); + goto out; + } + + if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); + if (ret) + VHOST_LOG_CONFIG(ERR, + "(%s) set_config() return error!\n", + dev->ifname); + } else { + VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_OK; + +out: + return RTE_VHOST_MSG_RESULT_ERR; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx, int main_fd __rte_unused) @@ -2686,6 +2772,8 @@ static int is_vring_iotlb(struct virtio_net *dev, VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end, false) \ diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index ba1c5c7..c4d091e 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -50,6 +50,8 @@ VHOST_USER_NET_SET_MTU = 20, VHOST_USER_SET_SLAVE_REQ_FD = 21, VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_GET_CONFIG = 24, + VHOST_USER_SET_CONFIG = 25, VHOST_USER_CRYPTO_CREATE_SESS = 26, VHOST_USER_CRYPTO_CLOSE_SESS = 27, VHOST_USER_POSTCOPY_ADVISE = 28, @@ -123,6 +125,16 @@ uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -146,6 +158,7 @@ VhostUserCryptoSessionParam crypto_session; VhostUserVringArea area; VhostUserInflight inflight; + struct vhost_user_config cfg; } payload; /* Nothing should be added after the payload */ } __rte_packed VhostUserMsg; From patchwork Mon May 23 09:35:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111611 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C8FCDA04FD; Mon, 23 May 2022 12:22:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F34E74280B; Mon, 23 May 2022 12:22:42 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id D7DA04280B for ; Mon, 23 May 2022 12:22:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301361; x=1684837361; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=qBLHs8POpWcarNalAnitgldUMnerM54MdLT+fI0Z2k4=; b=ej3XTktsLGItq+sQWdr66T2hx9NHYkT7JDqL6uL6pO+lWVenug746XTr ey0Ky1E+9/x696vYCvcRY75l9BiJuLslTLxPeIifsUTtTgt4YHE47Ds/B nW3k1Rpbw6NHKCIt8LJAMcKN4MVFKTcNwSP+3BPA4GQfDrL4yJomrauKe yTuELCJ47tICLKqTR2G3ua2Sx1e1knEgGWvzYZ4ihj8X+3Onx9e5g4iUt zcWojHbU0bceMWb4+mO7jwwrJi5mylz7+TRCA+k5NkGIb55/atQey8BJF UtT8Td8mwlGtHdDbmckVwEmVXThzIdiK6Fce4mBy8SsWMQTNoN9//vJBw A==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239297" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239297" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889420" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:38 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 04/13] vdpa/ifc: add blk ops for ifc device Date: Mon, 23 May 2022 17:35:15 +0800 Message-Id: <1653298524-232138-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index a761d49..b6fdfdb 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 1eed90b..c1fc1d7 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t size) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + uint64_t capacity = 0; + uint8_t *byte; + + if (size != sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %u", + size, (uint32_t)sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + /* The capacity is number of sectors in 512-byte. + * So right shift 1 bit we get in K, + * another right shift 10 bits we get in M, + * right shift 10 more bits, we get in G. + * To show capacity in G, we right shift 21 bits in total. + */ + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(DEBUG, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(DEBUG, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(DEBUG, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(DEBUG, "geometry"); + DRV_LOG(DEBUG, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(DEBUG, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(DEBUG, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Mon May 23 09:35:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111612 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 44A04A04FD; Mon, 23 May 2022 12:23:01 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3157A42B72; Mon, 23 May 2022 12:22:44 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 404064281C for ; Mon, 23 May 2022 12:22:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301363; x=1684837363; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=sVBFE3rhLiVJycha7dhdlb6HMYWMwQJlPEd4Vd1PKq8=; b=F0CzsOrepTfp4k6M21mWreu+TdBAxEzL/Rqq5FvMpFdLHjCJyXxJbA35 zDiR7cPwZIU9nvWquWh00LWpLXgD+A9k3Hwh2DShyEPzCNPPK54uVxEEA T8r6zYHlxK2eDK7QdKMNlluBbKdVl/Cg6SGwZ4hWcCHPVCy3dP3yWeCnB RF3H3It07XzVOMRE9q+qvFrd1tEEvIbticf6kq/NvM/6X/2lKESEEPrAj SWjO367/APFb3VVwEbymiO79HHN38WyZcmkYGY9GrJwiTBfJGBMbX3K8Q 1dSBeaa+AwXtpJANC6s+sXT4ZwnzUij5cq2LA5blJPI6OO149nk6QqbxB w==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239311" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239311" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889436" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:40 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 05/13] vdpa/ifc: add vDPA interrupt relay for blk device Date: Mon, 23 May 2022 17:35:16 +0800 Message-Id: <1653298524-232138-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index c1fc1d7..1d05529 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); @@ -378,7 +379,13 @@ struct rte_vdpa_dev_info { for (i = 0; i < nr_vring; i++) { rte_vhost_get_vhost_vring(internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { + if (m_rx == true && + ((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK)) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + * For the blk we need to relay all the read cmd + * of each queue + */ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); if (fd < 0) { DRV_LOG(ERR, "can't setup eventfd: %s", From patchwork Mon May 23 09:35:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111613 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9BDF0A04FD; Mon, 23 May 2022 12:23:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2282942824; Mon, 23 May 2022 12:22:47 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 0941340041 for ; Mon, 23 May 2022 12:22:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301366; x=1684837366; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9gtdDkzEKSRAJs8DgVKi2p/HiEojZu7hZPdWl2eRWlI=; b=DAISFspNIYbTGwtnF3xsBRzpU9bwX1mWWA6nVVwRUqnTUMkrmkVozWoz nqnnTtl1NVJLNhEIxRJ9nySfSs9XDKfM/hM2/qVQSbyt0wsCY7fO/5azB 2xiqtutTQvptZFA3ZZJjJBDSZK03ZFqk7gx5EGYLFYtwIt/6C/GXFeLhL VVvIJiCzyqvL9cwg6AEjqrInnVlUxLyPkEgLLzy2XXgewwwMcLWRmJYE0 zJ487SOfWvl5pXkyrolT+66hiXYH+IV0cKIt9f9llOFwIS0hgLsniJREc vbRQu+PEvmRelDj+GOdDvsKgMk5VcsfQuBLGSsonJPOLqTrykf8GSrhLz A==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239331" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239331" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889444" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:43 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 06/13] vdpa/ifc: add block device SW live-migration Date: Mon, 23 May 2022 17:35:17 +0800 Message-Id: <1653298524-232138-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is paused, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have received acks, and no inflight IO. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index b6fdfdb..9d95aac 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -63,6 +63,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 1d05529..7a0bdb4 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -316,8 +316,34 @@ struct rte_vdpa_dev_info { uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; + + /* to make sure no packet is lost for blk device + * do not stop until last_avail_idx == last_used_idx + */ + if (internal->hw.device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); + } + } + ifcvf_stop_hw(hw); for (i = 0; i < hw->nr_vring; i++) @@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /* NET: Direct I/O for Tx queue, relay for Rx queue + * BLK: relay every queue + */ + if ((internal->hw.device_type == IFCVF_NET) && (i & 1)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); @@ -692,8 +720,12 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (internal->hw.device_type == IFCVF_NET) { + if ((i & 1) == 0) + update_used_ring(internal, i); + } else if (internal->hw.device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -755,7 +787,9 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { + for (qid = 0; qid < q_num; qid += 1) { + if ((internal->hw.device_type == IFCVF_NET) && (qid & 1)) + continue; ev.events = EPOLLIN | EPOLLPRI; /* leave a flag to mark it's for interrupt */ ev.data.u64 = 1 | qid << 1 | From patchwork Mon May 23 09:35:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111614 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A628A04FD; Mon, 23 May 2022 12:23:13 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0665E42B76; Mon, 23 May 2022 12:22:51 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 0D58F42B70 for ; Mon, 23 May 2022 12:22:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301369; x=1684837369; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=yYpnzl1r2rXkJtvRax/1xLbxXIh3mfBKTZhIw6YEjlM=; b=LkQzizKktCWEYZGyICe6nyVK6zX4oMLIIMoVh1GXjK4ZY34uSPkF2hk3 H1xLZaPqHVOvsFPm2j0KahGlcFBn9PIogUuFC5nEONiSIGAMX+EsHR579 GZgjrF2/SMxUeMqr2vi9gSSpOR0M4w+Le29GExVigByyiRudKj/WvR33Z brIcZxxE5LIikhnNhAIC3OQ0koTyrk1fF/+rDlvGhVqeQ1wGnKkxKNFzm Sdgschj7A6AOAB+FjPlA0iBjldtADIBJTnxBETgbGjxeJfZrxt9X6miCj B2kspL4jnLTLY0fm6Pr2Kam/vY2dJ9qlAf3K+w35xTi1sMvexmD6nIvDQ w==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239351" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239351" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889458" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:45 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 07/13] vhost: add API to get vDPA device type Date: Mon, 23 May 2022 17:35:18 +0800 Message-Id: <1653298524-232138-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Vhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+) diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst index f287b76..0337b38 100644 --- a/doc/guides/prog_guide/vhost_lib.rst +++ b/doc/guides/prog_guide/vhost_lib.rst @@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions: Clear inflight packets which are submitted to DMA engine in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations -------------------------- diff --git a/doc/guides/rel_notes/release_22_07.rst b/doc/guides/rel_notes/release_22_07.rst index e49cace..63875b7 100644 --- a/doc/guides/rel_notes/release_22_07.rst +++ b/doc/guides/rel_notes/release_22_07.rst @@ -60,6 +60,10 @@ New Features Added an API which can get the number of in-flight packets in vhost async data path without using lock. +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. + * **Updated Intel iavf driver.** * Added Tx QoS queue rate limitation support. diff --git a/lib/vhost/rte_vhost.h b/lib/vhost/rte_vhost.h index c733f85..2f130ec 100644 --- a/lib/vhost/rte_vhost.h +++ b/lib/vhost/rte_vhost.h @@ -117,6 +117,9 @@ #define RTE_MAX_VHOST_DEVICE 1024 +#define RTE_VHOST_VDPA_DEVICE_TYPE_NET 0 +#define RTE_VHOST_VDPA_DEVICE_TYPE_BLK 1 + struct rte_vdpa_device; /** @@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path); /** + * Get the device type of the vdpa device. + * + * @param path + * The vhost-user socket file path + * @param type + * the device type of the vdpa device + * @return + * 0 on success, -1 on failure + */ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index b304339..baef4d2 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -619,6 +619,50 @@ struct rte_vdpa_device * } int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type) +{ + struct vhost_user_socket *vsocket; + struct rte_vdpa_device *vdpa_dev; + uint32_t vdpa_type = 0; + int ret = 0; + + pthread_mutex_lock(&vhost_user.mutex); + vsocket = find_vhost_user_socket(path); + if (!vsocket) { + VHOST_LOG_CONFIG(ERR, + "(%s) socket file is not registered yet.\n", + path); + ret = -1; + goto unlock_exit; + } + + vdpa_dev = vsocket->vdpa_dev; + if (!vdpa_dev) { + ret = -1; + goto unlock_exit; + } + + if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); + if (ret) { + VHOST_LOG_CONFIG(ERR, + "(%s) failed to get vdpa dev type for socket file.\n", + path); + ret = -1; + goto unlock_exit; + } + } else { + vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + } + + *type = vdpa_type; + +unlock_exit: + pthread_mutex_unlock(&vhost_user.mutex); + return ret; +} + +int rte_vhost_driver_disable_features(const char *path, uint64_t features) { struct vhost_user_socket *vsocket; diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index c4233a6..8b88a53 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops { /** Set the device configuration space */ int (*set_config)(int vid, uint8_t *config, uint32_t offset, uint32_t size, uint32_t flags); + + /** get device type: net device, blk device... */ + int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type); }; /** diff --git a/lib/vhost/version.map b/lib/vhost/version.map index 5841315..583b4f3 100644 --- a/lib/vhost/version.map +++ b/lib/vhost/version.map @@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; }; From patchwork Mon May 23 09:35:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111615 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 35C09A04FD; Mon, 23 May 2022 12:23:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DA0F242B70; Mon, 23 May 2022 12:22:53 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id DA19442B79 for ; Mon, 23 May 2022 12:22:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301372; x=1684837372; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Qu7juAmgtVYsisb6vCFxoUVwpeslaOFM73ygvAAh+tY=; b=IV1bnfdNQmQ4/0FWaUx7pSd/i3J7xiNXuSTVmDQfoxh0HRCDf2Nq778/ q1BhE+D/6Seo7/pbE2u8aUTaYUSujA4go8z0hl9exFPGhbZ9tmuIyuRP1 UFVCFZfh7/F2FwI1gTu6TeaxJw41yhk7Owp+A6do53UeNRLl+EfLo4MHl R/mRmOmNaHUuV02aC+kJC3oxQ0qmPg/J/dqWTGNeG9EjGh95Qr40lCuiw 7j/WjJQgvc62EvdfRFHEdD5NNPiXmwyizNftGcfaYUZoYJ2fxmJxPFIBn DI7FwPXLluZtYmytyiJ2x5zMpG3Y4Gi/2/Fi9HofHRY1NIPv6rYBfIxkM g==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239363" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239363" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889474" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:48 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 08/13] vdpa/ifc: add get device type ops to ifc driver Date: Mon, 23 May 2022 17:35:19 +0800 Message-Id: <1653298524-232138-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add get device type ops to ifc driver. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 7a0bdb4..1c5746a 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info { return 0; } +static int +ifcvf_get_device_type(struct rte_vdpa_device *vdev, + uint32_t *type) +{ + struct ifcvf_internal *internal; + struct internal_list *list; + + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + if (internal->hw.device_type == IFCVF_BLK) + *type = RTE_VHOST_VDPA_DEVICE_TYPE_BLK; + else + *type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + + return 0; +} + static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, @@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int @@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, }; struct rte_vdpa_dev_info dev_info[] = { From patchwork Mon May 23 09:35:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111616 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1519CA04FD; Mon, 23 May 2022 12:23:25 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B7BFD42B7D; Mon, 23 May 2022 12:22:56 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 0376B42B7B for ; Mon, 23 May 2022 12:22:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301375; x=1684837375; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=4BrY9sz/w3ltjbapBG7nfdew7ym66mv1zo884kKhm5Q=; b=EC0TpQDobeQScvGVmenR9/6NQgor7pBSQNWjgRL632VSdaNDoggqdyje QraozHTmCQaqUOos3BgE+V3hUqcCkQctluwbCkgCkbA7zi7EhQTPRS9Tm tNFwltzSVq0hBPSMQEa0A6YeGWXE/1ZxHWTJibLXlWVMtC27EAUW7FpVg 3USp3GDUqGNuKN+6hYDzSpSxh/GCRH18sGc5fyZitArOmG7vCanNaHrzn oV0e9jve8THukrz0I8Iv8I8fIClfEAaIK7o/iRrQDHYgF5kh02c8Px7P9 IPMgj0nSt60Ux4BPFC+QR5KnRtB0K5opww5Wv/drdTFOnVr9UVNFFJEIZ A==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239378" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239378" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889491" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:51 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 09/13] examples/vdpa: add virtio blk support Date: Mon, 23 May 2022 17:35:20 +0800 Message-Id: <1653298524-232138-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to vDPA example. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..7e11ef4 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -159,8 +160,53 @@ struct vdpa_port { }; static int +vdpa_blk_device_set_features_and_protocol(const char *path) +{ + uint64_t protocol_features = 0; + int ret; + + ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_disable_features(path, + VHOST_BLK_DISABLED_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_disable_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_get_protocol_features(path, &protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |= VHOST_BLK_PROTOCOL_FEATURES; + + ret = rte_vhost_driver_set_protocol_features(path, protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + } + +out: + return ret; +} + +static int start_vdpa(struct vdpa_port *vport) { + uint32_t device_type = 0; int ret; char *socket_path = vport->ifname; @@ -192,6 +238,16 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type); + if (ret == 0 && device_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path); + ret = vdpa_blk_device_set_features_and_protocol(socket_path); + if (ret != 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..143548e --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ + +/* Legacy feature bits */ +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ + +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ From patchwork Mon May 23 09:35:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111617 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9A5D3A04FD; Mon, 23 May 2022 12:23:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9863B4067C; Mon, 23 May 2022 12:22:58 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 5E8B542B81 for ; Mon, 23 May 2022 12:22:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301377; x=1684837377; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=+r6J8piZ0RfOHC/3sweboySSsMyjxytDBfbcG9Xta1k=; b=PemOAlx+SYTFtDD8yaU/z3WCe+hxXWnzsXJ1t4oN17ddYMp7XrJDMpD8 GGc67feD2IWs0ICg4/RhEbBAsXafkGD+gfnbn+S7ZIWTuutnQUs6Y62Wd BwYpC2m68nivWDS06fxPvN499JKkdKh5ZCwcgjKrkjATr+Szu1sTh1KgW x1n6iAzUcAayLRXe0rj6twIE5A1QmBkmp7FJQS2iq1P4GnOQ2NbmtIPmA JqGrexqRqxQNPMm4EjSv9W6fnuyt4qG0AnRQo/vmWw+tQEqR4fJm05R6y +ooQzOyeOS4FLLxSnW7cEWYKo6Al9YEUC7hI9R39KZyVJIxnMo8+Uyp6o g==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239386" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239386" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889503" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:54 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 10/13] usertools: add support for virtio blk device Date: Mon, 23 May 2022 17:35:21 +0800 Message-Id: <1653298524-232138-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to devbind. Signed-off-by: Andy Pei Acked-by: Chenbo Xia --- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..18c7d67 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties From patchwork Mon May 23 09:35:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111618 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C9DE6A04FD; Mon, 23 May 2022 12:23:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD17342B84; Mon, 23 May 2022 12:23:01 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id D479742B8E for ; Mon, 23 May 2022 12:22:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301380; x=1684837380; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=mnOOi7xfkNfV0i8YQK29UQMF9j86DxaWww6eJr5tNhs=; b=YHM3tiRU7CcIMxJCwA5DXAu4ibbVcM+Klz3nKo5MTFW4odaYStldXVzL Zqi79d1V+YPJrEZs67HK4Njyld5+JGbjhM3iP0ks3gk2KeH/QWko5HRGG dPWCXL/CV/DblwWvCQbGWiDGofcKi9Nr8dPfvmGxPaiyi9yStYw4Hp4it So9w59NnQFY6ef8aM6PEQqi3viEEC+suQdtwphif6Laz11x77NZ8GO+Xr T70TLil9pZ9Sek7YxM/e+LFgHYqVNawkuCbC9RDrUV8Zz5oWAQqFsBzS9 XaY0ZDot7+bweDEqZW4TJZD8BMA1Rvxcp8t4XEjEIwMKwbYpGRHU8a4hD w==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239395" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239395" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:22:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889520" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:57 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 11/13] vdpa/ifc: add log for config space of virtio blk Date: Mon, 23 May 2022 17:35:22 +0800 Message-Id: <1653298524-232138-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add some log of virtio blk device config space information at VDPA launch before qemu connects. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 1c5746a..80d6261 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + uint64_t capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /* cannot read 64-bit register in one attempt, + * so read byte by byte. + */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + /* The capacity is number of sectors in 512-byte. + * So right shift 1 bit we get in K, + * another right shift 10 bits we get in M, + * right shift 10 more bits, we get in G. + * To show capacity in G, we right shift 21 bits in total. + */ + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(DEBUG, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(DEBUG, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(DEBUG, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(DEBUG, "geometry"); + DRV_LOG(DEBUG, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(DEBUG, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(DEBUG, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(DEBUG, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Mon May 23 09:35:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111619 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 79DECA04FD; Mon, 23 May 2022 12:23:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C1B0C42B8F; Mon, 23 May 2022 12:23:03 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 7BE7540156 for ; Mon, 23 May 2022 12:23:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301382; x=1684837382; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2Kn5/lPhfCLBUjN3qkKYL6oh6Is+p5mBsvE9047AfUA=; b=Y5B8Sgb72KApSHGg/j0oR/u6Rv5DYdiLOwcw396mxTIiovSegWg3VIhv Q++DVTqIn94YYHgZnsQv15OKdtFM0sVhcqRvu5eKZwy2FsizrUs9ELp3M wiQBMqoy3J2sPryYHQUWo3CYOcVO6WqB/FGnChzSLQYAts2kDGRnfwcNl sbkZV+dDlbkbk7oEvbXTLkwZ1U82Q6YENczroFTxpEBf4SWhsIIPfcFwo VEqVBZh7VtDfbvuXP9NQopAc4nyHSJhDj/mvbIW2O1gTUXvIXZB9XEFLm 6Fbj62tpz7U4GljvIWXOF/0Q1n3Ig9Ico4lc8kqUO7fS+BjuQN9Lo4mEv Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239404" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239404" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:23:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889546" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:22:59 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 12/13] vdpa/ifc: add interrupt handling for config space Date: Mon, 23 May 2022 17:35:23 +0800 Message-Id: <1653298524-232138-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 80d6261..af1f192 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for config space change interrupt relay */ int epfd; + int csc_epfd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -566,6 +568,114 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_epfd, csc_val = 0; + + csc_epfd = epoll_create(1); + if (csc_epfd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + goto out; + } + + internal->csc_epfd = csc_epfd; + + for (;;) { + csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail."); + goto out; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || + errno == EWOULDBLOCK || + errno == EAGAIN) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + goto out; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + +out: + if (csc_epfd >= 0) + close(csc_epfd); + internal->csc_epfd = -1; + + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + char name[THREAD_NAME_LEN]; + int ret; + + snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid); + ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL, + intr_relay, (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static void +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_epfd >= 0) + close(internal->csc_epfd); + internal->csc_epfd = -1; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -592,10 +702,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -812,7 +928,7 @@ struct rte_vdpa_dev_info { if (nfds < 0) { if (errno == EINTR) continue; - DRV_LOG(ERR, "epoll_wait return fail\n"); + DRV_LOG(ERR, "epoll_wait return fail."); return NULL; } @@ -888,6 +1004,9 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); vdpa_ifcvf_stop(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Mon May 23 09:35:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111620 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B5BFEA04FD; Mon, 23 May 2022 12:23:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B731F42B94; Mon, 23 May 2022 12:23:05 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id A30F242B93 for ; Mon, 23 May 2022 12:23:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653301384; x=1684837384; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=7XyHxRdg3k3dWPi/bG+6PK7dnWbMDBPcqJF3TqK7BFY=; b=kJR4poR+C8owDNIPxvPA0LFcaLi8WIshO47bkjKgIROvaQxxfVPvR+vj w/MsRJkgvveBnA2rCsakpqNEMcO6NuF7XEW/4/1AEzxk40yEKPUDuo9Cq ezNDSabaN2zYpdoz3Q6X6cDyI7sAlrRHT7wkHbVWRrRG9xQ11b5u774tv 6fCPTuc/IhsKnV27le2NDfFJZY+yeA0QFMv2asD7JxOj2ieZu6FV6AD/3 sX5Ok1xgThlDv6GamBTx4L+ko9O7gws0mHQy1McU+yPRmngp/Ib2Vvviw wcBKU2SA1HnHcGRIHSr8fn3VixOUm2sjBPDYAmKkMEs3pRZNjk7gEN3RT w==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="255239410" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="255239410" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 03:23:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547889566" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 03:23:02 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v9 13/13] vdpa/ifc/base: access correct register for blk device Date: Mon, 23 May 2022 17:35:24 +0800 Message-Id: <1653298524-232138-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653298524-232138-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653298524-232138-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Register address is different between net and blk device. We are re-useing most of the code, when register address is different, we have to check net and blk device go through different code. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..dd475a7 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -218,10 +218,17 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->device_type == IFCVF_BLK) + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + else + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +261,22 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->device_type == IFCVF_BLK) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + else + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + + if (hw->device_type == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } }