From patchwork Tue May 24 02:48:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111673 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A5059A04FF; Tue, 24 May 2022 05:35:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0341640E78; Tue, 24 May 2022 05:35:32 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 253194067B for ; Tue, 24 May 2022 05:35:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363330; x=1684899330; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2Tdx7IwfX/Qw63wIoOlFxHv7o8sTmOGQ07N0SVk/roE=; b=LS8xMeTscn6z5v+IltsyVH6wJsLaTLAsY+VjwmiPoe/+uMsiAUODKsTi DPU5ZqPzMy0x1nYxMhU3HmnAswDCuNMbKs5rj9nGH24d8qDnzhPPOeZZ0 gIyDz+kvATUUAnA7ytzeO09AXRtSCE7VjC79J4Pf67SehFkr2/dYffGAX +RQPOIekkivkVgewxWNPzPmTiH94Ohef0/+5F52YvN9jzGVPSrFGZRK5T 6nNOtvrN7nFBAjMke8vojusD/MvL6RbPMxRWmlW7apBxxOVOgvqFZrCgB iq/FJljKLM7UVTRAHkXDUAdGrCJXmOPG/2Lgl87TX71961XbzK4CxBHLI w==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533596" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533596" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046897" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:27 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 01/13] vdpa/ifc: add support for virtio blk device Date: Tue, 24 May 2022 10:48:05 +0800 Message-Id: <1653360497-18080-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 22 +++++++--- drivers/vdpa/ifc/ifcvf_vdpa.c | 95 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 102 insertions(+), 15 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..a761d49 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,12 +5,19 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" -#define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_DEVICE_ID 0x1041 -#define IFCVF_SUBSYS_VENDOR_ID 0x8086 -#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +#define IFCVF_VENDOR_ID 0x1AF4 +#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_SUBSYS_VENDOR_ID 0x8086 +#define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_BLK_DEVICE_ID 0x0002 #define IFCVF_MAX_QUEUES 1 @@ -126,13 +133,18 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + int device_type; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9f05595..1eed90b 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -75,6 +75,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1138,7 +1144,7 @@ struct internal_list { return 0; } -static struct rte_vdpa_dev_ops ifcvf_ops = { +static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, .get_protocol_features = ifcvf_get_protocol_features, @@ -1167,6 +1173,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_net_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1178,6 +1226,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1227,13 +1276,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->hw.device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->hw.device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1245,7 +1305,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->hw.device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1308,11 +1369,25 @@ struct internal_list { static const struct rte_pci_id pci_id_ifcvf_map[] = { { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, - .device_id = IFCVF_DEVICE_ID, + .device_id = IFCVF_NET_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Tue May 24 02:48:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111674 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 243C0A04FF; Tue, 24 May 2022 05:35:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E5B1B42670; Tue, 24 May 2022 05:35:34 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 05A2342670 for ; Tue, 24 May 2022 05:35:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363332; x=1684899332; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=srDgBcwDeWe8TWKgo7H+aw94AQ9bN76kfEg63pw+G5c=; b=O1TbPsRAtX9aqs4S+Dn174rXp7FXRjGJCG7OwqPfn1bsfTUxqNOLvy4g usbZWwHV/nZtghCTn7TF24uIyjeFEAbmnoGCDKerA/cA9V7OpWmrGIhkx lhD1LUxRomfohqfpOU5kQpdpfv4zz8e7BtNPIsnVN0fGEcNPVDqJYajdB 4bMawMD5oyxhr+C162tz7TdXjM4whWMYkCuY10/5USHWh8fuEblw6s21o cJzNAVn8eYTZq5P67ntLiiMzLdW/Sxbh1AwhUS62avS6xNy1yI9xFJoBv 1VMal0pExRsmXCiXJhq25HPv3vyCzNdOhYoLPYomW2sa5RAIq/8Zt/aeV A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533598" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533598" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046909" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:29 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 02/13] vhost: add vDPA ops for blk device Date: Tue, 24 May 2022 10:48:06 +0800 Message-Id: <1653360497-18080-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 88138be..c4233a6 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t size); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Tue May 24 02:48:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111675 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3753DA04FF; Tue, 24 May 2022 05:35:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D815B427F3; Tue, 24 May 2022 05:35:37 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 27CC942670 for ; Tue, 24 May 2022 05:35:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363334; x=1684899334; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Xsa8luJn8gZWq6vnV3W1pGrvRgk4FSxK7QolrtpOReM=; b=eW9UXC2YTSSfCSP/eqXnOa0PFZauqESPB0pPy3qiLTcrBJADLLOBZ52K 2uQlHxuK+VPbuDwaWWmintwlVBdO6WmUKhZRYtUjt73GZhXPd9Hz14Vdb hZkLL7BxBlhnmM0+3wSBjM0H06SvIRXfgDPN/A5r+vnhcZB4EvNrRuOox sgLOeuIkLfJsjWAjk9aXYuTUHqPJ816dHDiMGK/PucBhQw+ccWs2tnJhB luUv/AMGYsjIpr+elUp8ldUrlqwh9jGz+ut/Se8fWnq9wrGSpan6yxiWo uxMwSn7j0f+qIMIDQx0nhu4zGWPNW94DoZs//bJqkCEsMZVc9qaua266W w==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533604" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533604" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046926" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:31 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 03/13] vhost: add vhost msg support for get/set config Date: Tue, 24 May 2022 10:48:07 +0800 Message-Id: <1653360497-18080-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- lib/vhost/vhost_user.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 98 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 850848c..4f2a777 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2468,6 +2468,89 @@ static int is_vring_iotlb(struct virtio_net *dev, } static int +vhost_user_get_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (!vdpa_dev) { + VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n", + dev->ifname); + return RTE_VHOST_MSG_RESULT_ERR; + } + + if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); + if (ret != 0) { + ctx->msg.size = 0; + VHOST_LOG_CONFIG(ERR, + "(%s) get_config() return error!\n", + dev->ifname); + } + } else { + VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_REPLY; +} + +static int +vhost_user_set_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) { + VHOST_LOG_CONFIG(ERR, + "(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, + VHOST_USER_MAX_CONFIG_SIZE); + goto out; + } + + if (!vdpa_dev) { + VHOST_LOG_CONFIG(ERR, "(%s) is not vDPA device!\n", + dev->ifname); + goto out; + } + + if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); + if (ret) + VHOST_LOG_CONFIG(ERR, + "(%s) set_config() return error!\n", + dev->ifname); + } else { + VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_OK; + +out: + return RTE_VHOST_MSG_RESULT_ERR; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx, int main_fd __rte_unused) @@ -2686,6 +2769,8 @@ static int is_vring_iotlb(struct virtio_net *dev, VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end, false) \ diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index ba1c5c7..c4d091e 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -50,6 +50,8 @@ VHOST_USER_NET_SET_MTU = 20, VHOST_USER_SET_SLAVE_REQ_FD = 21, VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_GET_CONFIG = 24, + VHOST_USER_SET_CONFIG = 25, VHOST_USER_CRYPTO_CREATE_SESS = 26, VHOST_USER_CRYPTO_CLOSE_SESS = 27, VHOST_USER_POSTCOPY_ADVISE = 28, @@ -123,6 +125,16 @@ uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -146,6 +158,7 @@ VhostUserCryptoSessionParam crypto_session; VhostUserVringArea area; VhostUserInflight inflight; + struct vhost_user_config cfg; } payload; /* Nothing should be added after the payload */ } __rte_packed VhostUserMsg; From patchwork Tue May 24 02:48:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111676 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 21E0DA04FF; Tue, 24 May 2022 05:35:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C33384281B; Tue, 24 May 2022 05:35:38 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 3CE56427F3 for ; Tue, 24 May 2022 05:35:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363336; x=1684899336; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=g3XfAaB74qczc1WZCxnMa6HfmBrd1A/JzKVDLqxxM0E=; b=LTiEix/2u89GDJR4s5QMmWrvt5dMp/ONWGeFFoZJqugtpkRCSQNBqZEh zMeQCeCGZ6PzIXygGTdYYYesEgRZsz2jkF8fLXtPCwWXPgoplTB3QH3w7 vkC/l8ArvXcScCG6iOL3aYfkY+wF/aKbqSGbYzRdtAAKsKHV6vrbMC49x Vo+j3i8SU71aBTP4zEJqqramNKURwH8tjQqi6URtuqAWPvxw2NgfBpn6I CNMdnOHlKM2NhIFr+DssfTQF1giD6lMiYo2NUmm0cs/naSgGz6gDCYOvk +BNCV8YmI8BeNg7cHVro+2mhV0yvDVus9JmurdRH8iQWTWosQWotE5HvR w==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533606" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533606" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046940" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:33 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 04/13] vdpa/ifc: add blk ops for ifc device Date: Tue, 24 May 2022 10:48:08 +0800 Message-Id: <1653360497-18080-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 96 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 99 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index a761d49..b6fdfdb 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 1eed90b..10a4a25 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1199,6 +1203,96 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t size) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + uint64_t capacity = 0; + uint8_t *byte; + + if (size != sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %u", + size, (uint32_t)sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + if (vdev == NULL) { + DRV_LOG(ERR, "Invalid vDPA device vid: %d", vid); + return -1; + } + + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + /* The capacity is number of sectors in 512-byte. + * So right shift 1 bit we get in K, + * another right shift 10 bits we get in M, + * right shift 10 more bits, we get in G. + * To show capacity in G, we right shift 21 bits in total. + */ + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(DEBUG, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(DEBUG, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(DEBUG, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(DEBUG, "geometry"); + DRV_LOG(DEBUG, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(DEBUG, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(DEBUG, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1211,7 +1305,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Tue May 24 02:48:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111677 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 42A0DA04FF; Tue, 24 May 2022 05:36:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 23F1F42B73; Tue, 24 May 2022 05:35:40 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 1ADC3427FF for ; Tue, 24 May 2022 05:35:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363338; x=1684899338; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=y9UEly9sJA1NJDD3iw4wrPO5vv2ZlBIXOio7m4/KTWs=; b=jBDNta+MEp3V2RQNzyb9xRB/xwIzHEhVVgyFpZEkhjorjkR/anPXrxqp drjjFJ31MjfTBkJkG8Koj+ggip/CIaNuTJOAhV1Sf1USlP4RnB2EPFQOj U8QeXLv4cL3hT46tvR9q22OrGhf2H0fWFukUv1RAyyK7/LqkVDR7wz6kR mPccrMKjULFtgxsaifRI2tS1f+yl/4ZUuHuY9eImufxCZgauXT7gfFRCc CW7orwMFLOF5J+Gfc9O/gcgjfRMAho8lMK8olMuJlYJ7654o7hhlymPki d0ggYwkjU+qrUhNeYmpEf4GwMYbSd5dWGmwA3A4tPFHXccY1P2b98vHyR g==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533610" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533610" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046955" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:35 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 05/13] vdpa/ifc: add vDPA interrupt relay for blk device Date: Tue, 24 May 2022 10:48:09 +0800 Message-Id: <1653360497-18080-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 10a4a25..61fb427 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); @@ -378,7 +379,13 @@ struct rte_vdpa_dev_info { for (i = 0; i < nr_vring; i++) { rte_vhost_get_vhost_vring(internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { + if (m_rx == true && + ((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK)) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + * For the blk we need to relay all the read cmd + * of each queue + */ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); if (fd < 0) { DRV_LOG(ERR, "can't setup eventfd: %s", From patchwork Tue May 24 02:48:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111678 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 25B54A04FF; Tue, 24 May 2022 05:36:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0E76742B6E; Tue, 24 May 2022 05:35:42 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 480DF42B75 for ; Tue, 24 May 2022 05:35:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363340; x=1684899340; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=jVIUbG9caZJ27p7oaD98uVLiyYuLObHsUW5q7s9tQSs=; b=mbmcCniE/hB2tWOwfUVVnhdLQJzf0oYA9KUfjarGs0BdH7hZmeqKNUGD +bGKjsyq+jta4Gxd0SHq/WKoqujCCBTFVuz6a2vjVTBj+Qfc52FhXK2DW cgcDsDryRcWetwv3mV58QQ2hSP52Ln1nybnlv0hkxbAQwp9j/CBYiL/Bh UwCiJrTMSBoCy8n2CM5bMkV+Rdk6nIj8RXT+ZW6SGi8jXD5kArVHYjQQ7 9s2l2MpIdKxd6he449t1HfqBpZhZpm8tp8V/DVBHssiMJZyAlsfdeKdUG Vvr+3AbBhjq7sIBRC18CcBUGsbZ0ph6UHhN9z472u+bKQJGcmJzDYltu6 A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533621" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533621" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046971" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:37 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 06/13] vdpa/ifc: add block device SW live-migration Date: Tue, 24 May 2022 10:48:10 +0800 Message-Id: <1653360497-18080-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is paused, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have received acks, and no inflight IO. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index b6fdfdb..9d95aac 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -63,6 +63,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 61fb427..0f9db8a 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -316,8 +316,34 @@ struct rte_vdpa_dev_info { uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; + + /* to make sure no packet is lost for blk device + * do not stop until last_avail_idx == last_used_idx + */ + if (internal->hw.device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); + } + } + ifcvf_stop_hw(hw); for (i = 0; i < hw->nr_vring; i++) @@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /* NET: Direct I/O for Tx queue, relay for Rx queue + * BLK: relay every queue + */ + if ((internal->hw.device_type == IFCVF_NET) && (i & 1)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); @@ -692,8 +720,12 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (internal->hw.device_type == IFCVF_NET) { + if ((i & 1) == 0) + update_used_ring(internal, i); + } else if (internal->hw.device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -755,7 +787,9 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { + for (qid = 0; qid < q_num; qid += 1) { + if ((internal->hw.device_type == IFCVF_NET) && (qid & 1)) + continue; ev.events = EPOLLIN | EPOLLPRI; /* leave a flag to mark it's for interrupt */ ev.data.u64 = 1 | qid << 1 | From patchwork Tue May 24 02:48:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111679 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5C96BA0501; Tue, 24 May 2022 05:36:13 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0EAA142B72; Tue, 24 May 2022 05:35:44 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 5F0944280C for ; Tue, 24 May 2022 05:35:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363342; x=1684899342; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=sgVma+HXcoxJrqlI6pz21FaQ3Wu2hXCwKZt7tOxrHDk=; b=DdBTS+CGSvWRVGfK1/vsdWWsPB//3j+WZb4/qm2mmczQAStuO5xvKvTH BJZlkuyFkSTbBrmxsUJlClbuxZlVDhlSd+Nskx+DeS/3kS2BpWfZWdbC0 K5cq+HT+nkfwgeSk1d9RuBziM17DS0MIL0yWYHpIRwP4xSbddkDSls6Sv to+iKUqU1o4DjZLY4rja4iZ1+Eg2AwUCvrW6dTn+cGZ8qXxLclTtgtt+P mdQJaFwpdmKS6Z6RQLyxsThSv6QP6dRrUzWuysPmsBU1dlsoMP0f1iqgH 0loZjvWTGRn1qPFmG5GDJQF0tIPmwYM+Ae3efCGKFhS8fmCnT15KEl/nI g==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533627" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533627" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745046987" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:40 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 07/13] vhost: add API to get vDPA device type Date: Tue, 24 May 2022 10:48:11 +0800 Message-Id: <1653360497-18080-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Vhost backend of different devices have different features. Add an API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+) diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst index f287b76..0337b38 100644 --- a/doc/guides/prog_guide/vhost_lib.rst +++ b/doc/guides/prog_guide/vhost_lib.rst @@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions: Clear inflight packets which are submitted to DMA engine in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations -------------------------- diff --git a/doc/guides/rel_notes/release_22_07.rst b/doc/guides/rel_notes/release_22_07.rst index e49cace..63875b7 100644 --- a/doc/guides/rel_notes/release_22_07.rst +++ b/doc/guides/rel_notes/release_22_07.rst @@ -60,6 +60,10 @@ New Features Added an API which can get the number of in-flight packets in vhost async data path without using lock. +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. + * **Updated Intel iavf driver.** * Added Tx QoS queue rate limitation support. diff --git a/lib/vhost/rte_vhost.h b/lib/vhost/rte_vhost.h index c733f85..2f130ec 100644 --- a/lib/vhost/rte_vhost.h +++ b/lib/vhost/rte_vhost.h @@ -117,6 +117,9 @@ #define RTE_MAX_VHOST_DEVICE 1024 +#define RTE_VHOST_VDPA_DEVICE_TYPE_NET 0 +#define RTE_VHOST_VDPA_DEVICE_TYPE_BLK 1 + struct rte_vdpa_device; /** @@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path); /** + * Get the device type of the vdpa device. + * + * @param path + * The vhost-user socket file path + * @param type + * the device type of the vdpa device + * @return + * 0 on success, -1 on failure + */ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index b304339..baef4d2 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -619,6 +619,50 @@ struct rte_vdpa_device * } int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type) +{ + struct vhost_user_socket *vsocket; + struct rte_vdpa_device *vdpa_dev; + uint32_t vdpa_type = 0; + int ret = 0; + + pthread_mutex_lock(&vhost_user.mutex); + vsocket = find_vhost_user_socket(path); + if (!vsocket) { + VHOST_LOG_CONFIG(ERR, + "(%s) socket file is not registered yet.\n", + path); + ret = -1; + goto unlock_exit; + } + + vdpa_dev = vsocket->vdpa_dev; + if (!vdpa_dev) { + ret = -1; + goto unlock_exit; + } + + if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); + if (ret) { + VHOST_LOG_CONFIG(ERR, + "(%s) failed to get vdpa dev type for socket file.\n", + path); + ret = -1; + goto unlock_exit; + } + } else { + vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + } + + *type = vdpa_type; + +unlock_exit: + pthread_mutex_unlock(&vhost_user.mutex); + return ret; +} + +int rte_vhost_driver_disable_features(const char *path, uint64_t features) { struct vhost_user_socket *vsocket; diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index c4233a6..8b88a53 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops { /** Set the device configuration space */ int (*set_config)(int vid, uint8_t *config, uint32_t offset, uint32_t size, uint32_t flags); + + /** get device type: net device, blk device... */ + int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type); }; /** diff --git a/lib/vhost/version.map b/lib/vhost/version.map index 5841315..583b4f3 100644 --- a/lib/vhost/version.map +++ b/lib/vhost/version.map @@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; }; From patchwork Tue May 24 02:48:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111680 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1A038A04FF; Tue, 24 May 2022 05:36:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 049F342B7F; Tue, 24 May 2022 05:35:46 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 3AE8342B7A for ; Tue, 24 May 2022 05:35:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363344; x=1684899344; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=EBPfx3LVHhAxsh8f83KDraPWghNdi4wlu0FOqf3a6XI=; b=hyGplbK972bvv2/MXfpgyD/iAeUxnnoXbrxyGn94rMU+ow0h/kVRepz0 NXRmLN4ZEE03/LRJLmsc8d9Zs9BmS+jqHwfm63KlXSrDttgvg3O1Y82qJ re3R5QNNw6H7p8sfY4Eo2TIIM6kUg96tQ+oUjEz5NbyTTezX8PZBiaaBN f1gfsCz0PKvezJeQxzvycyHmhbW2SOptp3H5ASNc150Ho8yiBdQ9Ke7YS 2vDQ8LJO/dduYIYIxtU+1RHJL6EtTaQoZc/tgYR2/YAhIYQSdf/DzONJ7 hrXgNDAp1qFW1/y58671DICMlmrq6posytK7AUEYUAxxVnE0qyJjoSQ9O w==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533631" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533631" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745047007" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:42 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 08/13] vdpa/ifc: add get device type ops to ifc driver Date: Tue, 24 May 2022 10:48:12 +0800 Message-Id: <1653360497-18080-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add get device type ops to ifc driver. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 0f9db8a..f4c6198 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info { return 0; } +static int +ifcvf_get_device_type(struct rte_vdpa_device *vdev, + uint32_t *type) +{ + struct ifcvf_internal *internal; + struct internal_list *list; + + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + if (internal->hw.device_type == IFCVF_BLK) + *type = RTE_VHOST_VDPA_DEVICE_TYPE_BLK; + else + *type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + + return 0; +} + static struct rte_vdpa_dev_ops ifcvf_net_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, @@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int @@ -1332,6 +1356,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, }; struct rte_vdpa_dev_info dev_info[] = { From patchwork Tue May 24 02:48:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111681 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 53068A04FF; Tue, 24 May 2022 05:36:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EA20542B82; Tue, 24 May 2022 05:35:47 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 7CB7342B81 for ; Tue, 24 May 2022 05:35:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363346; x=1684899346; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=FPsJGiFM70SNKVEuNQqDbe5xlnMDUXTHNdqeH2Oox1A=; b=jeI4Pf6gQc1nMBRCwgiWtJ8TRAUZ8SRAhwUWsUuyzltW5uXO3/2GUvyH cppD0OW4zbn3/zVKGSqJTn73yavs3aD9zX7DV0vUDUBbmncCRcb8LZlUh VHRU+CCQAZlJq89O7XgHqeoIlgmfhvf43/ROJPuOLHBazsclstq5Xx+Ai cgT1NCiKC2d0gVW3cZ8b0uecxKPsEoJJXv0ODN7jJcswcc3Ks6BWi+agD AIfTvWJfWmBy2mrmcr0pePvnMabcCRtHbu6ntfwN7ZzmwWfi2mqhr/af2 Kv1oT2ylYZ1yh0B4VmXDWOy4DpnZTNaX4RTYsb0wl5jxeaOiUttTq/lJD A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533635" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533635" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745047029" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:44 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 09/13] examples/vdpa: add virtio blk support Date: Tue, 24 May 2022 10:48:13 +0800 Message-Id: <1653360497-18080-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to vDPA example. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 50 +++++++++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..7e11ef4 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -159,8 +160,53 @@ struct vdpa_port { }; static int +vdpa_blk_device_set_features_and_protocol(const char *path) +{ + uint64_t protocol_features = 0; + int ret; + + ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_disable_features(path, + VHOST_BLK_DISABLED_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_disable_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_get_protocol_features(path, &protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |= VHOST_BLK_PROTOCOL_FEATURES; + + ret = rte_vhost_driver_set_protocol_features(path, protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + } + +out: + return ret; +} + +static int start_vdpa(struct vdpa_port *vport) { + uint32_t device_type = 0; int ret; char *socket_path = vport->ifname; @@ -192,6 +238,16 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type); + if (ret == 0 && device_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path); + ret = vdpa_blk_device_set_features_and_protocol(socket_path); + if (ret != 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..143548e --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ + +/* Legacy feature bits */ +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ + +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ From patchwork Tue May 24 02:48:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111682 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 829E0A04FF; Tue, 24 May 2022 05:36:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E631542B85; Tue, 24 May 2022 05:35:49 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id C6E3742B85 for ; Tue, 24 May 2022 05:35:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363349; x=1684899349; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=zF27mMlf/kUbDCdth9ErFujwal/D+Gybj91c7MRK63E=; b=Zm0MuUOwoxfhrIzO4i0KyWADxGA7JjnLnDpq/T/suJKRm6Fz8X7r1/qG 9iG8JxD63kCuRg2aDSopUZhZoxl4WLc2xX3aayhIoAHmUEvKQ/unVcFPI d4dVKmB3+D9ypeUAgXcTHVUaIUszyVtb/dz6bylHmrq8S7TuSlc9XGHEB tIkChMhNvoy85R/rURZ80lzp8AetvSwN/krbKNLHek6vkSH6KptuQ50Rf 5OFHQ2hH0TDXpGd/4Hs9JrYQoM+I7arnrpklrOkOT97apsW14NhBcVCsQ wwtlnFbAgrTov1WUbqrndwZ8Sbz9HskL6UvHfR6UeDFZFP00IPrEzNIrY A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533651" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533651" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745047037" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:46 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 10/13] usertools: add support for virtio blk device Date: Tue, 24 May 2022 10:48:14 +0800 Message-Id: <1653360497-18080-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to devbind. Signed-off-by: Andy Pei Acked-by: Chenbo Xia --- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..18c7d67 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001,1042', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties From patchwork Tue May 24 02:48:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111683 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 863BBA04FF; Tue, 24 May 2022 05:36:35 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 330BA427F2; Tue, 24 May 2022 05:35:56 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 1B6F8400D6 for ; Tue, 24 May 2022 05:35:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363354; x=1684899354; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=kXu/KjeWgEAoWO6Ull1YoPFIdIj2Pjdx+jnR8Pok04g=; b=M0arJUghqgEBF33eWgCH2l1+iua0jFdsDYHw8QZomwdmra9DQ4CnJLOn CjAc0Urx5+eidjJErnXXswa8swX0rbwLAaMjasFVYyJm7l0x/mZfL17lr 895TWfTO+3Xx66An4wBfVnk9/vnid7RgDsNIIqrxZOQmNAfEB8va9/k6/ R+Nq7DDYUcze2a8vc3e5wn29SSLXJ55V09F9tyoGBFJU8TDhdtOdxsuf6 0VS9+4UKH1kjR0MFilDvNN33RdaKmf8rvhvi0d0KTIE7FSq48uOZApo9B N7fYELtrXfaOPwKZ9KeDAGbc6IedSjeAbUiYYbFnXi22KlWVVcbm0Ov+Y g==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533676" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533676" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745047046" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:48 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 11/13] vdpa/ifc: add log for config space of virtio blk Date: Tue, 24 May 2022 10:48:15 +0800 Message-Id: <1653360497-18080-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add some log of virtio blk device config space information at VDPA launch before qemu connects. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index f4c6198..30850b0 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1387,6 +1387,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + uint64_t capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1453,6 +1456,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /* cannot read 64-bit register in one attempt, + * so read byte by byte. + */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + /* The capacity is number of sectors in 512-byte. + * So right shift 1 bit we get in K, + * another right shift 10 bits we get in M, + * right shift 10 more bits, we get in G. + * To show capacity in G, we right shift 21 bits in total. + */ + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(DEBUG, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(DEBUG, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(DEBUG, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(DEBUG, "geometry"); + DRV_LOG(DEBUG, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(DEBUG, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(DEBUG, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(DEBUG, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Tue May 24 02:48:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111684 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1BBE8A04FF; Tue, 24 May 2022 05:36:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3026E42B91; Tue, 24 May 2022 05:35:58 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id B40ED42B8A for ; Tue, 24 May 2022 05:35:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363356; x=1684899356; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=xhs2FkaARdzH0xgEqtriMy2SFfvMPDlW1YlSl4KKR+k=; b=l9sWtwqr1TUG0VgQEFqNWpkRlFmGKPfsHI80n41u1Ea8UfGwKRhcunuh 5OgIOMvkdm9smr9RMuzTZNJtzZkUqNcsgJGLmgShbJL+Rg2UQ6Hjd1VZL lTQ5zobqPyZtv4kBmUCsAbog+02Y7T4fIXgHpWd+s2V7L2rV5vKaWg2bJ byMwcxhTh8FZW63ImJkXiP+U+D07MdKflDFnXrxihziMw5jRKVE15j3Ty wfl4nJO5oSj7YVh9f7pobC3yBzj4+cbrplfH25NsTp4nMrjrm0wLse5ul 8We+dk7kgOm1qHyYbuM6TF1yU7JCDLAQoPMWxszfKYjKTJAV3cKifFodO A==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533721" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533721" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745047059" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:49 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 12/13] vdpa/ifc: add interrupt handling for config space Date: Tue, 24 May 2022 10:48:16 +0800 Message-Id: <1653360497-18080-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 121 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 30850b0..40a18b2 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for config space change interrupt relay */ int epfd; + int csc_epfd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -566,6 +568,114 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_epfd, csc_val = 0; + + csc_epfd = epoll_create(1); + if (csc_epfd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + goto out; + } + + internal->csc_epfd = csc_epfd; + + for (;;) { + csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail."); + goto out; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || + errno == EWOULDBLOCK || + errno == EAGAIN) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + goto out; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + +out: + if (csc_epfd >= 0) + close(csc_epfd); + internal->csc_epfd = -1; + + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + char name[THREAD_NAME_LEN]; + int ret; + + snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid); + ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL, + intr_relay, (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static void +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_epfd >= 0) + close(internal->csc_epfd); + internal->csc_epfd = -1; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -592,10 +702,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -812,7 +928,7 @@ struct rte_vdpa_dev_info { if (nfds < 0) { if (errno == EINTR) continue; - DRV_LOG(ERR, "epoll_wait return fail\n"); + DRV_LOG(ERR, "epoll_wait return fail."); return NULL; } @@ -888,6 +1004,9 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); vdpa_ifcvf_stop(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Tue May 24 02:48:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111685 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2B4BA04FF; Tue, 24 May 2022 05:36:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 106B842B93; Tue, 24 May 2022 05:36:01 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 9002F42B93 for ; Tue, 24 May 2022 05:35:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653363359; x=1684899359; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=tF9JTjICKDzvKL+Jmr7T+e9gbjlvuXdalsQvH9zFLbM=; b=ST0xwc8i8+XpKBNcc+ArWdXQN7uKyjre9y5F+VYGAjZLtoVDa3HnsB/U b74eFR1IhdY6wxSK3jy9iDal1p6I4GRjE9yqg9OJ8Uy019gq6KtJ32NDO axg435dLjgXBjm3WvV6ZvzVDv8NuCAgQKsJ5/pefrdiXSJF9VJ59seayU hDIXZkljkpf10RBtmIciz/t4cYWP48JGkNby689lxLUudc1jh9Q9dR0bH 6LYXKePhA5/LRoTF2Jm/OIQUyXsln1BcRd6gaQV7XU6wKVPA8k7Rqp0fK nHCIpluVuYW/ZkTtxD7IMh6JvvbpzeCU1epLnAD0E89uzbuj/AuXod4Rr Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10356"; a="273533735" X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="273533735" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 20:35:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,247,1647327600"; d="scan'208";a="745047091" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga005.jf.intel.com with ESMTP; 23 May 2022 20:35:52 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v10 13/13] vdpa/ifc/base: access correct register for blk device Date: Tue, 24 May 2022 10:48:17 +0800 Message-Id: <1653360497-18080-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1653360497-18080-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1653360497-18080-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Register address is different between net and blk device. We are re-using most of the code, when register address is different, we have to check net and blk device go through different code. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..dd475a7 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -218,10 +218,17 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->device_type == IFCVF_BLK) + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + else + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +261,22 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->device_type == IFCVF_BLK) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + else + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + + if (hw->device_type == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } }