From patchwork Tue Jan 25 09:37:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106479 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C5D24A04A7; Tue, 25 Jan 2022 10:42:46 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 36FAD426D5; Tue, 25 Jan 2022 10:42:45 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 82BEE4285F for ; Tue, 25 Jan 2022 10:42:43 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103763; x=1674639763; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rFUz/SVjhGgn20xUFjFFemaWcitUduNA2brEvZV/5+I=; b=e2dPbmIce7E39q4gzMExlHZwAUGFWldXN0xqA6bv4leMfCcZHbHctIIR KPmk+CS3CrLtHa6f0OHEXDzl9hsywnnKRMKi7wCAGnnMYy5JhWepaFwdS NlpzWZU3A5sOI7xmug+VBi9xwtez0mYy/ZfrhX9SZ/KGs4q02+5XQdhq2 vzdrXnAIAD+fDLPnTByaJlTeQtUkSbmuea2E6osU6lz5hFhu3CUCKsrt8 F+AWT950fvybqIHBnsVaaEku/mWbx/+RMP4GV0YNVFXEzt0jMe0YYnfbj 8omampOmfkbdYxYH7V8T4iT9u6y1j/0bfQxFzAEvGe2a50KBZngnHhSgF w==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840346" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840346" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656714" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:41 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 01/15] vdpa/ifc: add support for virtio blk device Date: Tue, 25 Jan 2022 17:37:03 +0800 Message-Id: <1643103437-118618-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 92 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..01522c6 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,8 +5,17 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 + #define IFCVF_VENDOR_ID 0x1AF4 #define IFCVF_DEVICE_ID 0x1041 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 @@ -57,7 +66,6 @@ #define IFCVF_32_BIT_MASK 0xffffffff - struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ @@ -126,7 +134,11 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3853c4c..96b67dd 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -58,6 +58,7 @@ struct ifcvf_internal { struct rte_vdpa_device *vdev; uint16_t max_queues; uint64_t features; + int device_type; rte_atomic32_t started; rte_atomic32_t dev_attached; rte_atomic32_t running; @@ -75,6 +76,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1170,6 +1177,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1181,6 +1230,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1230,13 +1280,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1248,7 +1309,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1316,6 +1378,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Tue Jan 25 09:37:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106480 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0B1F0A04A7; Tue, 25 Jan 2022 10:42:53 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7675D42889; Tue, 25 Jan 2022 10:42:46 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id C306242869 for ; Tue, 25 Jan 2022 10:42:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103765; x=1674639765; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=8QP2LyrrdZj62eIBJqZfLk3qWwIGPSMqMJr/mc509yA=; b=A49QGqkwfUhiyBH4LQ5g9gVV42jEwadF7ykBZkMqO33tpOFVYr13zept fT9zpPZhgsy4C5q+4SJBxqinsGPtInZ94ynNBNP7OVK2B4j2zrwFL0KkU I8K/iI7r/lFOHDp9uVagg3urlQkK25hQq2QvD2wwq1qagLQMbxb+Nl1E6 akExvLpF5QDfQF1BT3PAFroOn3CziYiq9cO1k9nDfagfnSRekm1D1EKhL 8gV0kSBoB2hcP5DOQh71p311mZDef+LBcSZdfZyBVq63lFO9R/F+FE9XG +EesIDdBZbimW4z9o5U88CIyo+rSyhA2FLQBtfbipJgZSe9ow6TR7ekeq g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840357" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840357" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656723" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:44 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 02/15] vhost: add vdpa ops for blk device Date: Tue, 25 Jan 2022 17:37:04 +0800 Message-Id: <1643103437-118618-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vdpa ops. Signed-off-by: Andy Pei --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index fc2d6ac..9a23db9 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -65,8 +65,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t len); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Tue Jan 25 09:37:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106481 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D0B97A04A7; Tue, 25 Jan 2022 10:42:57 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 712494287A; Tue, 25 Jan 2022 10:42:50 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 363264287A for ; Tue, 25 Jan 2022 10:42:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103768; x=1674639768; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ZG3ghIIeKjdm5110Qeehu88SuM5WF88WXKG9bdxruIU=; b=UfFK6fOXED7XudYF/k0pYvUCOdlpLvRumMldLpioAWMmCW5SXOptJtl3 7jZwG/bJ2e9a5sDFLIW6/vXsTnKemwAFYLEgqLQu5kNoBRZq5XAwnOrh3 9SqPSgPGOYd3TZPYUjW2EjnGF2NEBjrXD37NiqCRGDte5w/OJmngryyU0 90Z0gVKRR8F+Ah8F+fexQ6NRbHYT5/2YN8UVvNGy48jbaJ8mi1PBNgn4h KsTmrPOqhpzRdhZ9w9kk9Y4WBv88LrDeLiDlZsOZdBGLc8t93riX1TGMt fqg7zWRHnWvVl8XhwzwlAJxxrm8pD2fTHZmPLzmJnW2RZo81a6FzxXnWs Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840369" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840369" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656744" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:46 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 03/15] vdpa/ifc: add blk ops for ifc device Date: Tue, 25 Jan 2022 17:37:05 +0800 Message-Id: <1643103437-118618-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 85 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 88 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 01522c6..769c603 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -66,6 +66,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 96b67dd..57fdd2c 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1091,6 +1091,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1203,6 +1207,85 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + __u64 capacity = 0; + uint8_t *byte; + + if (len < sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %lu", + len, sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (__u64)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %quG", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(INFO, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = NULL, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1215,7 +1298,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Tue Jan 25 09:37:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106482 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 06DD1A04A7; Tue, 25 Jan 2022 10:43:03 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6A9EB4285D; Tue, 25 Jan 2022 10:42:52 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 6B8AB42877 for ; Tue, 25 Jan 2022 10:42:50 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103770; x=1674639770; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QJF/c0+1bOcY83ixPz7RkYMXpGDcDqYOZ43PB5YEmHI=; b=XtYBGirFRgQ35+EllZIVUxaYejAM+dhYkl7b1NxcgggRKEWwAWnQeclK /joTZZLFhPmfXVxS7IbaUUYDA225GH9r9gM2Ep1gwfof/FNgGU7c5xMWW b6mvgAzamwfjUBkkaoirZebPeHJczoZofXscXfvtUPilJm72uuzGNIbra ApR2Qs6PCgs27xdj0itHXyTcuAPwp7LvNgKfH7ABgNOj5R7bfpycPP5jK JgH5v+NDiGscDscOq1gRU8Kp+MpHXv2fFbLp26GU5BiA2S9iVPAwrNEYR XB/anjPpjJdrHYvco8pwnbdoN2X/FhWu3gkHCXfQGnLHF3QXYIgkhXUYX g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840383" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840383" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656754" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:48 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 04/15] vdpa/ifc: add vdpa interrupt for blk device Date: Tue, 25 Jan 2022 17:37:06 +0800 Message-Id: <1643103437-118618-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the blk we need to relay all the cmd of each queue. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 46 ++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 57fdd2c..ef5b36c 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -372,24 +372,48 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); for (i = 0; i < nr_vring; i++) internal->intr_fd[i] = -1; - for (i = 0; i < nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vring); - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { - fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); - if (fd < 0) { - DRV_LOG(ERR, "can't setup eventfd: %s", - strerror(errno)); - return -1; + if (internal->device_type == IFCVF_NET) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if ((i & 1) == 0 && m_rx == true) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + */ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; + } + } + } else if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if (m_rx == true) { + /* For the blk we need to relay all the read cmd + * of each queue + */ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } - internal->intr_fd[i] = fd; - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } } From patchwork Tue Jan 25 09:37:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106483 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2685DA04A7; Tue, 25 Jan 2022 10:43:08 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 77F7E428A0; Tue, 25 Jan 2022 10:42:55 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id DE3CC428A0 for ; Tue, 25 Jan 2022 10:42:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103773; x=1674639773; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=o5wfELVhov9kRlfMSj7L2VX6tYX6Y8/n6J/pN9iNnK8=; b=XvWBu701ruIUADV6Ry0gC+biCs7jWo4cqul2WiD0/AlDADDnfU/Gbj5l uxfsyrjMbj45G7SeURNcJZHxrh8f4/P1bXJgG2v+F3pVYF9AwwPIrbYGQ cAX+QyrinHT0hwYQRPour2ZQ2/fvo0FB7mXEKDdok0OkvTKjSP100KHBa ZIFiUheBHB4+/dQuSYaFG4qJ73BWhBHsWRXeFHCx5vC7dAIl1ieQqpe4V ud0/S89/nMAm1OUl1Rzv9QTsvi6KXs/2Eo+EWwt/IQuRm+iL2paTc9gjr vzCIsc6oLQRJjAGYWeTWAXSkO9WbIj9W2OrggFoAb1fXtlADwfipImOyI A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840389" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840389" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:52 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656761" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:50 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 05/15] vdpa/ifc: add blk dev sw live migration Date: Tue, 25 Jan 2022 17:37:07 +0800 Message-Id: <1643103437-118618-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable virtio blk sw live migration relay callfd and log the dirty page. In this version we ignore the write cmd and still mark it dirty. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 4 +- drivers/vdpa/ifc/base/ifcvf.h | 6 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 128 +++++++++++++++++++++++++++++++++++------- 3 files changed, 116 insertions(+), 22 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 721cb1d..3a69e53 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -189,7 +189,7 @@ IFCVF_WRITE_REG32(val >> 32, hi); } -STATIC int +int ifcvf_hw_enable(struct ifcvf_hw *hw) { struct ifcvf_pci_common_cfg *cfg; @@ -238,7 +238,7 @@ return 0; } -STATIC void +void ifcvf_hw_disable(struct ifcvf_hw *hw) { u32 i; diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 769c603..6dd7925 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -179,4 +179,10 @@ struct ifcvf_hw { u64 ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid); +int +ifcvf_hw_enable(struct ifcvf_hw *hw); + +void +ifcvf_hw_disable(struct ifcvf_hw *hw); + #endif /* _IFCVF_H_ */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index ef5b36c..14bc5c8 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -332,10 +332,67 @@ struct rte_vdpa_dev_info { rte_vhost_get_negotiated_features(vid, &features); if (RTE_VHOST_NEED_LOG(features)) { - ifcvf_disable_logging(hw); - rte_vhost_get_log_base(internal->vid, &log_base, &log_size); - rte_vfio_container_dma_unmap(internal->vfio_container_fd, - log_base, IFCVF_LOG_BASE, log_size); + if (internal->device_type == IFCVF_NET) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, + &log_size); + rte_vfio_container_dma_unmap( + internal->vfio_container_fd, log_base, + IFCVF_LOG_BASE, log_size); + } + /* IFCVF marks dirty memory pages for only packet buffer, + * SW helps to mark the used ring as dirty after device stops. + */ + for (i = 0; i < hw->nr_vring; i++) { + len = IFCVF_USED_RING_LEN(hw->vring[i].size); + rte_vhost_log_used_vring(vid, i, 0, len); + } + } +} + +static void +vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) +{ + struct ifcvf_hw *hw = &internal->hw; + struct rte_vhost_vring vq; + int i, vid; + uint64_t features = 0; + uint64_t log_base = 0, log_size = 0; + uint64_t len; + + vid = internal->vid; + + if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vq); + while (vq.avail->idx != vq.used->idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + hw->vring[i].last_avail_idx = vq.avail->idx; + hw->vring[i].last_used_idx = vq.used->idx; + } + } + + ifcvf_hw_disable(hw); + + for (i = 0; i < hw->nr_vring; i++) + rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, + hw->vring[i].last_used_idx); + + if (internal->sw_lm) + return; + + rte_vhost_get_negotiated_features(vid, &features); + if (RTE_VHOST_NEED_LOG(features)) { + if (internal->device_type == IFCVF_NET) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, + &log_size); + rte_vfio_container_dma_unmap( + internal->vfio_container_fd, log_base, + IFCVF_LOG_BASE, log_size); + } /* * IFCVF marks dirty memory pages for only packet buffer, * SW helps to mark the used ring as dirty after device stops. @@ -661,15 +718,17 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /* NETWORK: Direct I/O for Tx queue, relay for Rx queue + * BLK: relay every queue + */ + if ((i & 1) && (internal->device_type == IFCVF_NET)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); return -1; } hw->vring[i].used = gpa; - } else { + } else if (internal->device_type == IFCVF_BLK) { hw->vring[i].used = m_vring_iova + (char *)internal->m_vring[i].used - (char *)internal->m_vring[i].desc; @@ -688,7 +747,10 @@ struct rte_vdpa_dev_info { } hw->nr_vring = nr_vring; - return ifcvf_start_hw(&internal->hw); + if (internal->device_type == IFCVF_NET) + return ifcvf_start_hw(&internal->hw); + else if (internal->device_type == IFCVF_BLK) + return ifcvf_hw_enable(&internal->hw); error: for (i = 0; i < nr_vring; i++) @@ -713,8 +775,10 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (((i & 1) == 0 && internal->device_type == IFCVF_NET) || + internal->device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -726,6 +790,8 @@ struct rte_vdpa_dev_info { (uint64_t)(uintptr_t)internal->m_vring[i].desc, m_vring_iova, size); + hw->vring[i].last_avail_idx = vq.used->idx; + hw->vring[i].last_used_idx = vq.used->idx; rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, hw->vring[i].last_used_idx); rte_free(internal->m_vring[i].desc); @@ -776,17 +842,36 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { - ev.events = EPOLLIN | EPOLLPRI; - /* leave a flag to mark it's for interrupt */ - ev.data.u64 = 1 | qid << 1 | - (uint64_t)internal->intr_fd[qid] << 32; - if (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev) - < 0) { - DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); - return NULL; + if (internal->device_type == IFCVF_NET) { + for (qid = 0; qid < q_num; qid += 2) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); + } + } else if (internal->device_type == IFCVF_BLK) { + for (qid = 0; qid < q_num; qid += 1) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); } - update_used_ring(internal, qid); } /* start relay with a first kick */ @@ -874,7 +959,10 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_NET) + vdpa_ifcvf_stop(internal); + else if (internal->device_type == IFCVF_BLK) + vdpa_ifcvf_blk_pause(internal); vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Tue Jan 25 09:37:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106484 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A7FFA04A7; Tue, 25 Jan 2022 10:43:13 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 680D9428AE; Tue, 25 Jan 2022 10:42:58 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id BA82E428B0 for ; Tue, 25 Jan 2022 10:42:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103775; x=1674639775; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bzPqYXy00oDRSKcw+iAtEOQqzBSoo5eHcufTbT3b3Pc=; b=mNHQBDsKKmg2Yq9jxU5K9t+gIfUQOJDvO1vwPq8tJswbTiPuYB9/9DlC PaLUDCeSbufxsWs2unw3Hk6JZgCoxRi5YXJehn8VffUV532yxC/Qu16H+ 92phLcuzd4BTlPPKHTac3RpBbUKXrEXb6HBFgup+hmXCSyJVL01kTQi+v AIVPLKok2b1g/7A1Ul2PQqsh5zKaCjay6gJIAm5PfHZLF54MKLHvQEQOU vCLY3VI0UdSxMfR4LfwYCr2JICSJjETUUXhOXDzy4v58WJxE5OV4pa3Xo 26/Wi04d6xRZa6sOv9t0fj1YFV6FraVmDslWHU79TtTfeoRYIf1DGgOE8 g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840406" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840406" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656769" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:53 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 06/15] example/vdpa:add vdpa blk support in example Date: Tue, 25 Jan 2022 17:37:08 +0800 Message-Id: <1643103437-118618-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device suppoort to vdpa example. Signed-off-by: Andy Pei --- examples/vdpa/Makefile | 2 +- examples/vdpa/main.c | 8 ++ examples/vdpa/meson.build | 1 + examples/vdpa/vdpa_blk_compact.c | 150 +++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 117 ++++++++++++++++++++++++ examples/vdpa/vhost_user.h | 189 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 466 insertions(+), 1 deletion(-) create mode 100644 examples/vdpa/vdpa_blk_compact.c create mode 100644 examples/vdpa/vdpa_blk_compact.h create mode 100644 examples/vdpa/vhost_user.h diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile index d974db4..9d0479b 100644 --- a/examples/vdpa/Makefile +++ b/examples/vdpa/Makefile @@ -5,7 +5,7 @@ APP = vdpa # all source are stored in SRCS-y -SRCS-y := main.c +SRCS-y := main.c vdpa_blk_compact.c CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..924ad7b 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -156,6 +157,7 @@ struct vdpa_port { static const struct rte_vhost_device_ops vdpa_sample_devops = { .new_device = new_device, .destroy_device = destroy_device, + .new_connection = rte_vhost_blk_session_install_rte_compat_hooks, }; static int @@ -192,6 +194,12 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + if (vdpa_blk_device_set_features_and_protocol(socket_path, vport->dev) + < 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build index bd08605..f0d111c 100644 --- a/examples/vdpa/meson.build +++ b/examples/vdpa/meson.build @@ -15,4 +15,5 @@ deps += 'vhost' allow_experimental_apis = true sources = files( 'main.c', + 'vdpa_blk_compact.c', ) diff --git a/examples/vdpa/vdpa_blk_compact.c b/examples/vdpa/vdpa_blk_compact.c new file mode 100644 index 0000000..0c4d3ee --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.c @@ -0,0 +1,150 @@ +/* INTEL CONFIDENTIAL + * + * Copyright (c) Intel Corporation. + * All rights reserved. + * + * The source code contained or described herein and all documents related + * to the source code ("Material") are owned by Intel Corporation or its + * suppliers or licensors. Title to the Material remains with Intel + * Corporation or its suppliers and licensors. The Material contains trade + * secrets and proprietary and confidential information of Intel or its + * suppliers and licensors. The Material is protected by worldwide + * copyright and trade secret laws and treaty provisions. No part of the + * Material may be used, copied, reproduced, modified, published, uploaded, + * posted, transmitted, distributed, or disclosed in any way without Intel's + * prior express written permission. + * + * No license under any patent, copyright, trade secret or other + * intellectual property right is granted to or conferred upon you by + * disclosure or delivery of the Materials, either expressly, by + * implication, inducement, estoppel or otherwise. Any license under such + * intellectual property rights must be express and approved by Intel in + * writing. + */ + +/* @file + * + * Block device specific vhost lib + */ + +#include + +#include +#include +#include +#include "vdpa_blk_compact.h" +#include "vhost_user.h" + +#define VHOST_USER_GET_CONFIG 24 +#define VHOST_USER_SET_CONFIG 25 + +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + +/* + * Function to handle vhost user blk message + */ +static enum rte_vhost_msg_result +rte_vhost_blk_extern_vhost_pre_msg_handler(int vid, void *_msg) +{ + struct VhostUserMsg *msg = _msg; + struct rte_vdpa_device *vdev = NULL; + + vdev = rte_vhost_get_vdpa_device(vid); + if (vdev == NULL) + return RTE_VHOST_MSG_RESULT_ERR; + + fprintf(stderr, "msg is %d\n", msg->request.master); + switch (msg->request.master) { + case VHOST_USER_GET_CONFIG: { + int rc = 0; + + fprintf(stdout, "read message VHOST_USER_GET_CONFIG\n"); + + if (vdev->ops->get_config) { + fprintf(stdout, "get_config() function is valid!\n"); + rc = vdev->ops->get_config(vid, + msg->payload.cfg.region, + msg->payload.cfg.size); + if (rc != 0) { + msg->size = 0; + fprintf(stdout, "get_config() return error!\n"); + } + } else { + fprintf(stdout, "get_config() function is invalid!\n"); + } + + return RTE_VHOST_MSG_RESULT_REPLY; + } + case VHOST_USER_SET_CONFIG: { + int rc = 0; + + fprintf(stdout, + "read message VHOST_USER_SET_CONFIG\n"); + + if (vdev->ops->set_config) { + rc = vdev->ops->set_config(vid, + msg->payload.cfg.region, + msg->payload.cfg.offset, + msg->payload.cfg.size, + msg->payload.cfg.flags); + } + + return rc == 0 ? RTE_VHOST_MSG_RESULT_OK : RTE_VHOST_MSG_RESULT_ERR; + } + default: + break; + } + + return RTE_VHOST_MSG_RESULT_NOT_HANDLED; +} + +struct rte_vhost_user_extern_ops g_blk_extern_vhost_ops = { + .pre_msg_handle = rte_vhost_blk_extern_vhost_pre_msg_handler, + .post_msg_handle = NULL, +}; + +int +rte_vhost_blk_session_install_rte_compat_hooks(int vid) +{ + int rc; + + rc = rte_vhost_extern_callback_register(vid, + &g_blk_extern_vhost_ops, + NULL); + if (rc != 0) { + fprintf(stderr, "%s() failed for vid = %d\n", __func__, vid); + return -1; + } + fprintf(stdout, "register extern vhost ops on vid = %d\n", vid); + return 0; +} + + +int +vdpa_blk_device_set_features_and_protocol(const char *path, + struct rte_vdpa_device *vdev) +{ + uint64_t protocol_features = 0; + + if (!vdev) { + fprintf(stdout, "vdev is NULL.\n"); + return -EINVAL; + } + + /* vdpa net does not have the get_config */ + if (!vdev->ops->get_config) + return 0; + + rte_vhost_driver_set_features(path, SPDK_VHOST_BLK_FEATURES_BASE); + rte_vhost_driver_disable_features(path, + SPDK_VHOST_BLK_DISABLED_FEATURES); + + rte_vhost_driver_get_protocol_features(path, &protocol_features); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_CONFIG); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD); + rte_vhost_driver_set_protocol_features(path, protocol_features); + + return 0; +} diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..420d48e --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,117 @@ +/* INTEL CONFIDENTIAL + * + * Copyright (c) Intel Corporation. + * All rights reserved. + * + * The source code contained or described herein and all documents related + * to the source code ("Material") are owned by Intel Corporation or its + * suppliers or licensors. Title to the Material remains with Intel + * Corporation or its suppliers and licensors. The Material contains trade + * secrets and proprietary and confidential information of Intel or its + * suppliers and licensors. The Material is protected by worldwide + * copyright and trade secret laws and treaty provisions. No part of the + * Material may be used, copied, reproduced, modified, published, uploaded, + * posted, transmitted, distributed, or disclosed in any way without Intel's + * prior express written permission. + * + * No license under any patent, copyright, trade secret or other + * intellectual property right is granted to or conferred upon you by + * disclosure or delivery of the Materials, either expressly, by + * implication, inducement, estoppel or otherwise. Any license under such + * intellectual property rights must be express and approved by Intel in + * writing. + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** + * @file + * + * Device specific vhost lib + */ + +#include + +#include +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_RO 5 /* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ +#define VIRTIO_BLK_F_DISCARD 13 /* DISCARD is supported */ +#define VIRTIO_BLK_F_WRITE_ZEROES 14 /* WRITE ZEROES is supported */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_FLUSH 9 /* Flush command supported */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ + +/* Old (deprecated) name for VIRTIO_BLK_F_FLUSH. */ +#define VIRTIO_BLK_F_WCE VIRTIO_BLK_F_FLUSH +#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#endif + +#define SPDK_VHOST_FEATURES ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC)) + +#define SPDK_VHOST_DISABLED_FEATURES ((1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY)) + +#define SPDK_VHOST_BLK_FEATURES_BASE (SPDK_VHOST_FEATURES | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define SPDK_VHOST_BLK_DISABLED_FEATURES (SPDK_VHOST_DISABLED_FEATURES | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_BARRIER) | (1ULL << VIRTIO_BLK_F_SCSI)) + +/* Vhost-blk support protocol features */ +#define SPDK_VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_CONFIG) | \ + (1ULL << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This function will set vhost user block + * + * @param path + * socket path + */ +int +vdpa_blk_device_set_features_and_protocol(const char *path, + struct rte_vdpa_device *vdev); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Install external hook to handle vhost user block message + * + * @param vid + * vhost device id + */ +int +rte_vhost_blk_session_install_rte_compat_hooks(int vid); + +#endif /* _VDPA_BLK_COMPACT_H_ */ diff --git a/examples/vdpa/vhost_user.h b/examples/vdpa/vhost_user.h new file mode 100644 index 0000000..8b747d0 --- /dev/null +++ b/examples/vdpa/vhost_user.h @@ -0,0 +1,189 @@ +/* INTEL CONFIDENTIAL + * + * Copyright (c) Intel Corporation. + * All rights reserved. + * + * The source code contained or described herein and all documents related + * to the source code ("Material") are owned by Intel Corporation or its + * suppliers or licensors. Title to the Material remains with Intel + * Corporation or its suppliers and licensors. The Material contains trade + * secrets and proprietary and confidential information of Intel or its + * suppliers and licensors. The Material is protected by worldwide + * copyright and trade secret laws and treaty provisions. No part of the + * Material may be used, copied, reproduced, modified, published, uploaded, + * posted, transmitted, distributed, or disclosed in any way without Intel's + * prior express written permission. + * + * No license under any patent, copyright, trade secret or other + * intellectual property right is granted to or conferred upon you by + * disclosure or delivery of the Materials, either expressly, by + * implication, inducement, estoppel or otherwise. Any license under such + * intellectual property rights must be express and approved by Intel in + * writing. + */ + +#ifndef _VHOST_NET_USER_H +#define _VHOST_NET_USER_H + +#include +#include + +#include "rte_vhost.h" + +/* refer to hw/virtio/vhost-user.c */ + +#define VHOST_MEMORY_MAX_NREGIONS 8 + +#ifndef VHOST_USER_MAX_CONFIG_SIZE +#define VHOST_USER_MAX_CONFIG_SIZE 256 +#endif + +#define VHOST_USER_PROTOCOL_FEATURES ((1ULL << VHOST_USER_PROTOCOL_F_MQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) |\ + (1ULL << VHOST_USER_PROTOCOL_F_RARP) | \ + (1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \ + (1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CRYPTO_SESSION) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER) | \ + (1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT)) + +typedef enum VhostUserRequest { + VHOST_USER_NONE = 0, + VHOST_USER_GET_FEATURES = 1, + VHOST_USER_SET_FEATURES = 2, + VHOST_USER_SET_OWNER = 3, + VHOST_USER_RESET_OWNER = 4, + VHOST_USER_SET_MEM_TABLE = 5, + VHOST_USER_SET_LOG_BASE = 6, + VHOST_USER_SET_LOG_FD = 7, + VHOST_USER_SET_VRING_NUM = 8, + VHOST_USER_SET_VRING_ADDR = 9, + VHOST_USER_SET_VRING_BASE = 10, + VHOST_USER_GET_VRING_BASE = 11, + VHOST_USER_SET_VRING_KICK = 12, + VHOST_USER_SET_VRING_CALL = 13, + VHOST_USER_SET_VRING_ERR = 14, + VHOST_USER_GET_PROTOCOL_FEATURES = 15, + VHOST_USER_SET_PROTOCOL_FEATURES = 16, + VHOST_USER_GET_QUEUE_NUM = 17, + VHOST_USER_SET_VRING_ENABLE = 18, + VHOST_USER_SEND_RARP = 19, + VHOST_USER_NET_SET_MTU = 20, + VHOST_USER_SET_SLAVE_REQ_FD = 21, + VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_CRYPTO_CREATE_SESS = 26, + VHOST_USER_CRYPTO_CLOSE_SESS = 27, + VHOST_USER_POSTCOPY_ADVISE = 28, + VHOST_USER_POSTCOPY_LISTEN = 29, + VHOST_USER_POSTCOPY_END = 30, + VHOST_USER_GET_INFLIGHT_FD = 31, + VHOST_USER_SET_INFLIGHT_FD = 32, + VHOST_USER_MAX = 33 +} VhostUserRequest; + +typedef enum VhostUserSlaveRequest { + VHOST_USER_SLAVE_NONE = 0, + VHOST_USER_SLAVE_IOTLB_MSG = 1, + VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG = 3, + VHOST_USER_SLAVE_MAX +} VhostUserSlaveRequest; + +typedef struct VhostUserMemoryRegion { + uint64_t guest_phys_addr; + uint64_t memory_size; + uint64_t userspace_addr; + uint64_t mmap_offset; +} VhostUserMemoryRegion; + +typedef struct VhostUserMemory { + uint32_t nregions; + uint32_t padding; + VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS]; +} VhostUserMemory; + +typedef struct VhostUserLog { + uint64_t mmap_size; + uint64_t mmap_offset; +} VhostUserLog; + +/* Comply with Cryptodev-Linux */ +#define VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH 512 +#define VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH 64 + +/* Same structure as vhost-user backend session info */ +typedef struct VhostUserCryptoSessionParam { + int64_t session_id; + uint32_t op_code; + uint32_t cipher_algo; + uint32_t cipher_key_len; + uint32_t hash_algo; + uint32_t digest_len; + uint32_t auth_key_len; + uint32_t aad_len; + uint8_t op_type; + uint8_t dir; + uint8_t hash_mode; + uint8_t chaining_dir; + uint8_t *ciphe_key; + uint8_t *auth_key; + uint8_t cipher_key_buf[VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH]; + uint8_t auth_key_buf[VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH]; +} VhostUserCryptoSessionParam; + +typedef struct VhostUserVringArea { + uint64_t u64; + uint64_t size; + uint64_t offset; +} VhostUserVringArea; + +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint16_t num_queues; + uint16_t queue_size; +} VhostUserInflight; + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + +typedef struct VhostUserMsg { + union { + uint32_t master; /* a VhostUserRequest value */ + uint32_t slave; /* a VhostUserSlaveRequest value*/ + } request; + +#define VHOST_USER_VERSION_MASK 0x3 +#define VHOST_USER_REPLY_MASK (0x1 << 2) +#define VHOST_USER_NEED_REPLY (0x1 << 3) + uint32_t flags; + uint32_t size; /* the following payload size */ + union { +#define VHOST_USER_VRING_IDX_MASK 0xff +#define VHOST_USER_VRING_NOFD_MASK (0x1<<8) + uint64_t u64; + struct vhost_vring_state state; + struct vhost_vring_addr addr; + VhostUserMemory memory; + VhostUserLog log; + struct vhost_iotlb_msg iotlb; + VhostUserCryptoSessionParam crypto_session; + VhostUserVringArea area; + VhostUserInflight inflight; + struct vhost_user_config cfg; + } payload; + int fds[VHOST_MEMORY_MAX_NREGIONS]; + int fd_num; +} __attribute((packed)) VhostUserMsg; + +#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) + +/* The version of the protocol we support */ +#define VHOST_USER_VERSION 0x1 +#endif From patchwork Tue Jan 25 09:37:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106485 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 202F7A04A7; Tue, 25 Jan 2022 10:43:21 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D372E428B0; Tue, 25 Jan 2022 10:43:00 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id D716C4286A for ; Tue, 25 Jan 2022 10:42:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103778; x=1674639778; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Q4pWHhsiiMJon/PmG+i6zWiX4I7IA0g3rSbtXM7w4vE=; b=LjbkWAJ3ZKCBkYhMqaImJ4oKmtULD//12NXMvRzttGSRW4/PiIJBzKOj 0gwLHQEMC+QlvSGfTOWbQ9HpVtEOs80lANtCTM4rZTkZpbqKJpGMjXJWy 9Z8IXBEY/l3lVBC5UJQms86eQ/zPOYFreRlX0ydEAkqriaRHmPoVJ119x nxlfbF2mdbC5k2bgViqJlMIaX6fzjJnmEkAmkWF+asLTYNXsdisPLXMOW SNGBJoyqivJwe+tDT+AqmOgfDeVe9n4WYX7JFU5BFgtRWM+Qw9Jjch7/h Wa1ALEYutY2FzsgwHM6wmFXeDrLyfqa8qVBy7obiasmquDoPWvH+U8QzR g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840417" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840417" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656781" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:55 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 07/15] usertools: add support for virtio blk device Date: Tue, 25 Jan 2022 17:37:09 +0800 Message-Id: <1643103437-118618-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to devbind. Signed-off-by: Andy Pei --- usertools/dpdk-devbind.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..cbe336f 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -14,6 +14,8 @@ from os.path import join as path_join # The PCI base class for all devices +virtio_blk_class = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': '8086', 'SDevice': '0002'} network_class = {'Class': '02', 'Vendor': None, 'Device': None, 'SVendor': None, 'SDevice': None} acceleration_class = {'Class': '12', 'Vendor': None, 'Device': None, @@ -72,6 +74,7 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk_devices = [virtio_blk_class] network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -587,6 +590,9 @@ def show_status(): Displays to the user what devices are bound to the igb_uio driver, the kernel driver or to no driver''' + if status_dev in ["virtio_blk", "all"]: + show_device_status(virtio_blk_devices, "virtio_blk") + if status_dev in ["net", "all"]: show_device_status(network_devices, "Network", if_field=True) @@ -746,6 +752,7 @@ def do_arg_actions(): if b_flag is not None: clear_data() # refresh if we have changed anything + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) @@ -769,6 +776,7 @@ def main(): parse_args() check_modules() clear_data() + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) From patchwork Tue Jan 25 09:37:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106486 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52E08A04A7; Tue, 25 Jan 2022 10:43:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BC5AA428CE; Tue, 25 Jan 2022 10:43:01 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id F076A428C9 for ; Tue, 25 Jan 2022 10:42:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103780; x=1674639780; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=eFWSzjSflOt+73fmhqiqM+b7J/E9deaDTnp103Ddl54=; b=kL0AmAI/rLa3R8gog3c0ph9tFgXQLX6kvz2JBf1kD+23hTq2uNRDEoYs 4k2ugAZ/xEgnmRE4sN203UtJOqzHDEwvLPFOYjtRyJnDZAA6rPc0i7acx xco749NynKVG0U3Nm3GIEA9awYcs/94J93Fd7qDjo97RtSUZ+mSBsh/KE sLnst3cYAPLz826uLCR60lkZkEQRx6JCdWdqXM9xBiLLkJOZdffqJp0kK i76K0YFjwgW2544aboE6t6YgR6qitjt9ShJ/nhjiM3xyo5tYtPmi9ks+n lv8ZPANN2qITERSiXeNVrvtvw4qXCBVba/GC/1/4ZedEZ7+8BOY4L/a8B A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840429" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840429" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:42:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656802" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:42:58 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 08/15] vdpa/ifc: set_vring_state op is mandatory, add set_vring_state for blk device Date: Tue, 25 Jan 2022 17:37:10 +0800 Message-Id: <1643103437-118618-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Set_vring_state op is mandatory, add set_vring_state for blk device. Currently set_vring_state for blk device is not implemented. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 14bc5c8..00e7274 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1373,6 +1373,16 @@ struct rte_vdpa_dev_info { } static int +ifcvf_blk_set_vring_state(int vid, int vring, int state) +{ + RTE_SET_USED(vid); + RTE_SET_USED(vring); + RTE_SET_USED(state); + + return 0; +} + +static int ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1390,7 +1400,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, From patchwork Tue Jan 25 09:37:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106487 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1F651A04A7; Tue, 25 Jan 2022 10:43:31 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B918F428C9; Tue, 25 Jan 2022 10:43:05 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id D3A414285F for ; Tue, 25 Jan 2022 10:43:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103784; x=1674639784; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=I+hJ8ZTAlweT9fitcr7WwTIuTsq8PUZmX4NaH0xJHdY=; b=MwrJ2wxLPzH3pI0aYB04g/ixJMQyfNm7dV1O/9vwUnSW4XWVCOIcLwiH WYOnF7l590AEtleHxoQ0z43F/AiO/SkkIXbDByX9SfB9Myj3LoGB0PcIP g70uP4QZGcgOTMndDfrL5aTuU5fvF/0okoehNzXHpQQc5n6AmLLA7/DyI 4emVdEP/c7QaqGRoXB0EIF9nBZSLVqIw/bJSb6t+EfWk1fYweymMpF6/+ VxiZfI710RJ8aV0uaNuinIEIbpPbnuKB9hwIjsLlSklfGpG3E+ZGhJeel idbln7mP1OQhK/Dwbi3zZ/4g2R7ZU8m5QFWWaIvM93ttT+vbxaR4RNw8u A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840444" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840444" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656826" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:00 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 09/15] vdpa/ifc: add some log at VDPA launch before qemu connect Date: Tue, 25 Jan 2022 17:37:11 +0800 Message-Id: <1643103437-118618-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add some log of virtio blk device config space information at VDPA launch before qemu connects. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 00e7274..ff91e80 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1436,6 +1436,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + __u64 capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1502,6 +1505,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /* cannot read 64-bit register in one attempt, + * so read byte by byte. + */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (__u64)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %quG", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Tue Jan 25 09:37:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106488 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9E6A7A04A7; Tue, 25 Jan 2022 10:43:35 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B31B2428D9; Tue, 25 Jan 2022 10:43:07 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 9019E428BB for ; Tue, 25 Jan 2022 10:43:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103785; x=1674639785; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=oxieEmrNWEb5j5e3hOZSzxVfMAYa1ZcG7WMMC5Co0Nk=; b=nriaQK8glBKSPjlZEjF9QGuZrq4/W5OFXmP0qWSpsqEHW7NG6zRIv1kG owgk6tIz3Z0pDqSW6arro9BeEBmsAThNaJo5Jcu1c4zQ9W8UQTPXDlKj/ iNsYJX9LBxumGmQfsjBbq/tUJFem+Urh6ti+ixpaM8elZJDtDYKVU1LX1 WvbQhPvfKBoeVB2Uy13zpOzr4dRwocWX/X8xywCxy1C1p82WfJCLkJ/iB DkZgVStwl99huu6xsSCzEixrYtIATmeSpZH5X9rcDRvcpXx26TJHtnmTX euaphMZWKewQcwQF1nYmA4iAh2SRQW7y2hA2rsBSWxy87PZ6UrMCZLvZA g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840455" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840455" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656854" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:03 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 10/15] vdpa/ifc: read virtio max_queues from hardware Date: Tue, 25 Jan 2022 17:37:12 +0800 Message-Id: <1643103437-118618-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Original code max_queues is set to IFCVF_MAX_QUEUES. New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index ff91e80..d30c3fd 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1530,6 +1530,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(INFO, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + /* reset max_queue here, to minimum modification */ + internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Tue Jan 25 09:37:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106489 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 18B04A04A7; Tue, 25 Jan 2022 10:43:40 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ADF58428D4; Tue, 25 Jan 2022 10:43:09 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 0CE6E428DE for ; Tue, 25 Jan 2022 10:43:07 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103788; x=1674639788; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=6rijh25HyMqNHLj4jCVrLoLm/fWaoqRdGYk0CGg6qDY=; b=Lewe0505GCMp5gtIuomxWSfcKGBhDCG43dH2fPYE4QUMSSH6ONiXmJzl +ev5Zyx3ESsC1KQZTjgp6oVfV5vU8B3nkrkl3tsUdo0zhKs15zoEvlxdE rBsM7ooMovNweQcggx2KueO2iU9wRuPa89X9qwTbtK0pv81TX7mMTAqX8 vUx9H/MGAVriYzpiTvICYYz4hmNzB4M8QaHJX/tOZD4tx1HwS2VFAsm0q UthYMi/LS3Ze00KOiO+CMiSm7te2GjykDOTkYtbT/29YwxNyLIoEb0/eL 0KIZlrgp/7tww7rkcqCwqndg7QHXDZpBaaZ4SS/GXLGAQTKyqcU2tGCLM g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840467" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840467" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656877" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:06 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 11/15] vdpa: add config space change interrupt register and handle for virtio_blk Date: Tue, 25 Jan 2022 17:37:13 +0800 Message-Id: <1643103437-118618-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 111 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index d30c3fd..981cb26 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for intr relay */ int epfd; + int csc_fd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -617,6 +619,106 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_fd, csc_val = 0; + + csc_fd = epoll_create(1); + if (csc_fd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_fd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + return NULL; + } + + internal->csc_fd = csc_fd; + + for (;;) { + csc_val = epoll_wait(csc_fd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail\n"); + return NULL; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || errno == EWOULDBLOCK) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + return NULL; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + int ret; + + ret = pthread_create(&internal->intr_tid, NULL, intr_relay, + (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static int +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_fd >= 0) + close(internal->csc_fd); + internal->csc_fd = -1; + + return 0; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -643,10 +745,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + ret = unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -963,6 +1071,9 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(internal); else if (internal->device_type == IFCVF_BLK) vdpa_ifcvf_blk_pause(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Tue Jan 25 09:37:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106490 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A6035A04A7; Tue, 25 Jan 2022 10:43:44 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BB196428CD; Tue, 25 Jan 2022 10:43:11 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 4B235428E2 for ; Tue, 25 Jan 2022 10:43:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103790; x=1674639790; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=B+C9Vo/IQeNBdxaGgxc6sXLgGmjTuAoXmV8y22eVMOA=; b=TagZLUk2m0wqFU9KDCusTJ4IOCVo4P6tzhC6jWFbBPbz4Ujbkw87hlTa 7QEHXbHmrq6ANG+rs0PAaicKyYH8+s0MPdz7q64sgA7m5qOnNkJ79umuZ UOjBMpmfqGBR/8iWIfeALZkdCJ9LWsx01Gv6yoeSVAQetzvXKYxrzUML1 tZvYc3wiyZXxLGC+TrQMcjeVB61Kl5n1sVlD3qj2t73vxlK9ZonQxStQ/ pWVOrjYDBoQEYrwkLywV7OhOxnYDHE9pmQKI2tVF/JggAtU5LyqZbHd4Q 4jIXgc3VTpSYC10hcBGbV/nGpuIGadAq1lbe2p72OBBFo7LnuTK2KIY0o A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840474" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840474" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656886" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:08 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 12/15] vdpa/ifc: add is_blk flag to ifcvf_hw, and init is_blk during probe Date: Tue, 25 Jan 2022 17:37:14 +0800 Message-Id: <1643103437-118618-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add is_blk flag to ifcvf_hw, and init is_blk during probe. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 6dd7925..8e602af 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -149,6 +149,7 @@ struct ifcvf_hw { u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + u8 is_blk; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 981cb26..4eb8f98 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1608,11 +1608,13 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->device_type = IFCVF_NET; + internal->hw.is_blk = IFCVF_NET; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; } else if (device_id == VIRTIO_ID_BLOCK) { internal->device_type = IFCVF_BLK; + internal->hw.is_blk = IFCVF_BLK; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; From patchwork Tue Jan 25 09:37:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106491 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5C08AA04A7; Tue, 25 Jan 2022 10:43:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B6B28428E8; Tue, 25 Jan 2022 10:43:15 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 66DED428DA for ; Tue, 25 Jan 2022 10:43:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103793; x=1674639793; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=OfMQKUusCbeZdl+8sJdrAlffNHrBeBS10u16yx50v50=; b=NPxFTeQ1tSEtLQ3k3LhAyzHlPTl1vBeOtje0PcjBRKCSh9dZ42gxQeiO jOP/MZNFKpFSNOcVBJuBGMyUb3xehLmgj9tmR0iSzmfzLtszmhbcX9vmY lqVx4wssk9DERHK1PE4B/XJYD3utBPhDgJWtPuu14qF6MksrbsAqEcOJz 9UYajmFkN50AdGSREzKA2ZP6rO9dBzrSZA8j7BlxWIKCkjYuqioegE8xF /kJH6bM43BMxt8M9X8RQzuBM7jVfxGZ27v0T9yUKdCpltFcmBNUtWClMG HvqiY30SagcQbv+u50L83XFcRs1pB0otreYFw5KKBoBwoPAFRxMePd1Iq A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840481" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840481" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656897" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:10 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 13/15] vdpa/ifc/base: for blk device, live migration register is different from net device Date: Tue, 25 Jan 2022 17:37:15 +0800 Message-Id: <1643103437-118618-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 1.last_avail_idx is lower 16 bit of the register. 2.address of ring_state register is different between net and blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 3a69e53..a8a4728 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -216,10 +216,18 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->is_blk == IFCVF_BLK) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } else if (hw->is_blk == IFCVF_NET) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -252,9 +260,23 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + } else if (hw->is_blk == IFCVF_NET) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + } + + if (hw->is_blk == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else if (hw->is_blk == IFCVF_NET) + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } } diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 8e602af..7367094 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 From patchwork Tue Jan 25 09:37:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106492 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 79E96A04A7; Tue, 25 Jan 2022 10:43:55 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E2A01428ED; Tue, 25 Jan 2022 10:43:17 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 84901428DA for ; Tue, 25 Jan 2022 10:43:15 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103795; x=1674639795; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Ub3AZnnS8uIx3lW0N+A7Vw9tpP0UyirUaST2e+H5Tpc=; b=CRxdM35eW1+9iY1lRgIIY2Wi8BvYw1pB0L9w3Iv93Gzjfb5oPl+65ucN U+0sgVgoxKlW5t/hPx3mRhW9OGVueYgJ6OhSGAG80aDMvplrA/v+rBlmC rRz+sKwYdGPuj/COwj7molmFnhi8jxOHl8SEU88m/tHDsS5JCJHSqlAa+ Uj0533QbVeT0LYx0+rdizJqoycT9x0Hj6GdUh2f7qq1LNRU4vpJFUP6OK 69s/16iFtWk1/DrcXumocU4rWDr8sQXMZO+p7X/h/XaPpw6Dm1CNb5Tqf Z/u9lmm2xt99PB6qMII5ENw/QGm410y+feJgM48EJUyqNIwjoLxLWvc3c A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840488" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840488" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656907" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:13 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 14/15] vdpa/ifc: make sure hardware last_avail_idx and last_used_idx is the same when blk device pause Date: Tue, 25 Jan 2022 17:37:16 +0800 Message-Id: <1643103437-118618-15-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx is the same. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 2 +- drivers/vdpa/ifc/base/ifcvf.h | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 32 +++++++++++++++++++++++--------- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index a8a4728..7018048 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -116,7 +116,7 @@ IFCVF_WRITE_REG8(status, &hw->common_cfg->device_status); } -STATIC void +void ifcvf_reset(struct ifcvf_hw *hw) { ifcvf_set_status(hw, 0); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 7367094..f22d18b 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -157,6 +157,9 @@ struct ifcvf_hw { int ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev); +void +ifcvf_reset(struct ifcvf_hw *hw); + u64 ifcvf_get_features(struct ifcvf_hw *hw); diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 4eb8f98..b0b2859 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -356,23 +356,32 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) { struct ifcvf_hw *hw = &internal->hw; - struct rte_vhost_vring vq; int i, vid; uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; if (internal->device_type == IFCVF_BLK) { for (i = 0; i < hw->nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vq); - while (vq.avail->idx != vq.used->idx) { - ifcvf_notify_queue(hw, i); - usleep(10); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } @@ -759,7 +768,12 @@ struct rte_vdpa_dev_info { if (ret) goto err; - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_BLK) { + vdpa_ifcvf_blk_pause(internal); + ifcvf_reset(&internal->hw); + } else { + vdpa_ifcvf_stop(internal); + } ret = vdpa_disable_vfio_intr(internal); if (ret) From patchwork Tue Jan 25 09:37:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106493 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1F82EA04A7; Tue, 25 Jan 2022 10:44:00 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D1517428F0; Tue, 25 Jan 2022 10:43:19 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id EEDC2428FF for ; Tue, 25 Jan 2022 10:43:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643103798; x=1674639798; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=CYVxCE6tK347SJO9ema6bsNPo2bc/mSy5Uqe9fuByJc=; b=Q67Fb8I61pq0zkPQAzBcRyUMG/MlV/FOH/AUS8io3ju9fPwNxZsJxzzL gwX0JQyudtVAvSZL8yTU47GO8xcIpmHguc8GaCbr4wgOVsYh+doqA9kp5 yw+zeL7uyM8bni7IaWcGF6I+4GBRAJRq/g9T8Rrz3Pt8Jh1RMF14I2ED9 WiSFMP49pNVcjtleNJUHP3L5GbODVAamZpXEX13W2UbQQE1It06mKW6JU FfzmbPMReEzFkjsRExX2eWcnI3VPCgLmpbeaX1Yln1mUiaQLPgCQVUmxG EZsUEQCmo+00bjbLJ3Vc5/xeCHE+IYF7I80gE+fVy4ErX6SY7PXxD5FEN g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="229840493" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="229840493" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Jan 2022 01:43:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="534656915" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga008.jf.intel.com with ESMTP; 25 Jan 2022 01:43:15 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v2 15/15] vhost: make sure each queue callfd is configured Date: Tue, 25 Jan 2022 17:37:17 +0800 Message-Id: <1643103437-118618-16-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643103437-118618-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643103437-118618-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 5eb1dd6..b25b25f 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3137,12 +3137,26 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, if (!vdpa_dev) goto out; + if (request != VHOST_USER_SET_VRING_CALL) + goto out; + if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid)) VHOST_LOG_CONFIG(ERR, "Failed to configure vDPA device\n"); else dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED; + } else { + /* when VIRTIO_DEV_VDPA_CONFIGURED already configured + * close the device and config the device again, + * make sure the call fd of each queue is configured correctly. + */ + if (vdpa_dev->ops->dev_close(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to close vDPA device\n"); + if (vdpa_dev->ops->dev_conf(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to re-config vDPA device\n"); } out: