From patchwork Tue Jan 25 06:47:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106456 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C7C73A04A9; Tue, 25 Jan 2022 07:53:13 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7E67542836; Tue, 25 Jan 2022 07:53:13 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 2DF6E4282D for ; Tue, 25 Jan 2022 07:53:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093592; x=1674629592; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=/qh3YZ5eyOpRUU1K0kkvtTczrRdHwrkxQvKWeZB3xHk=; b=dlF764Jvm2clN9EfgodRj8kk67Hdi54jVVBZLw2UNCVUxdTCOW+VfEaF MQfEDGE7ER3b/aaA5Fv0S82/Yv9+5KIpd6THA4J3S2A4gwcfCQCGF8fOs 6oCGRF8Fwe62Ra3dSKaruc0pth0DfN43ZWHU+mk3WMI3x+oPpHI0R4txx lkZQuCY6N990yC0pIQCCaSOwc1grWHMfpHLtF9G4435SyTtNwXqUqo1Ex pYhYxtO5Q3i82jh2xvFLvIvpk0jwiK/xV2mMNwsqVj6WZg3y7jiDbTokd igOavSWRlXjy1JcgUrMQ7NbL8K3I7am952RH1h9S03Fw7UunKt/BZ7gPd w==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181297" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181297" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724232" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:09 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 01/15] vdpa/ifc: add support for virtio blk device Date: Tue, 25 Jan 2022 14:47:24 +0800 Message-Id: <1643093258-47258-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemeted with proper feature and ops. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 96 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 102 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..01522c6 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,8 +5,17 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 + #define IFCVF_VENDOR_ID 0x1AF4 #define IFCVF_DEVICE_ID 0x1041 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 @@ -57,7 +66,6 @@ #define IFCVF_32_BIT_MASK 0xffffffff - struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ @@ -126,7 +134,11 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3853c4c..48056d1 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -58,6 +58,7 @@ struct ifcvf_internal { struct rte_vdpa_device *vdev; uint16_t max_queues; uint64_t features; + int device_type; rte_atomic32_t started; rte_atomic32_t dev_attached; rte_atomic32_t running; @@ -75,6 +76,14 @@ struct internal_list { struct ifcvf_internal *internal; }; +/** +** vdpa decice info includes device features and devcic operation. +**/ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1170,6 +1179,50 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /** + ** Transitional devices: use the PCI subsystem device id as + ** virtio device id, same as legacy driver always did. + **/ + device_id = pci_dev->id.subsystem_device_id; + } else { + /** + ** Modern devices: simply use PCI device id, + ** but start from 0x1040. + **/ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1181,6 +1234,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1230,13 +1284,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1248,7 +1313,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1316,6 +1382,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Tue Jan 25 06:47:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106457 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2F563A04A9; Tue, 25 Jan 2022 07:53:19 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6911F42846; Tue, 25 Jan 2022 07:53:15 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 4A2274283E for ; Tue, 25 Jan 2022 07:53:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093594; x=1674629594; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=8QP2LyrrdZj62eIBJqZfLk3qWwIGPSMqMJr/mc509yA=; b=ckNGvdAr5wpXt2ji9xgkbqmEaTNrW1wgxN63EPHhRyuJJTOML2pPd99r pHlUuwDbbxyCJNw6EHExSgIWHuWyVrV76H0IlMdgwTk9Z1/r3mMzAIP19 k/K36IIr+SMUMrxo1rouDs3zxAMhVtQ5+Is4W/7ZFkGILtZ3Kai65cvTt uqq+gTSm3tMPaaYmJ0ghNY4WIKqv5Af3HOGENcgr68W+TMmeDhs2cEmKF LATrS+92uxJBSk6tJ6Gee3H/E08yAKJGYV2+EUJZx9q9kDwXOlyRwr4jW yHgysotGb398WwdNhmeaJv24369bLC08Wo6pMbb6Vk1M/G46fQxGSBMAX w==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181302" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181302" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724249" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:12 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 02/15] vhost: add vdpa ops for blk device Date: Tue, 25 Jan 2022 14:47:25 +0800 Message-Id: <1643093258-47258-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vdpa ops. Signed-off-by: Andy Pei --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index fc2d6ac..9a23db9 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -65,8 +65,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t len); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Tue Jan 25 06:47:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106458 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A6D80A04A9; Tue, 25 Jan 2022 07:53:24 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 69C1942848; Tue, 25 Jan 2022 07:53:18 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 259734284E for ; Tue, 25 Jan 2022 07:53:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093597; x=1674629597; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=slCZwNdaaV5VnUjkyD4UltzmfoJfBl56f6aUpURdLM4=; b=Hijf1lQ4m+EyOFnjjXcoWIjblkZjyNAljnIQzr1LOz9tNULdUElEZhO9 zkGrBjl56q1nxUWqCdR+xmLeY6PN6vjZTV+4A0NqJfOy33YCvBqZcfFQO q7hOvg/ep4OadfNbFVyaJPECTzvIFF/nm5rZCXWZ9kY7wCNjFupE4NH1i aHlUvOvJbgCuaw40TWzJa8sRhzvEZQgYSFtOgA7nHirqPM+wOyW+EfSgw MLBviXZgChmPZp0EO332ZjsF6kSsX0ESNV7UzXE96U6Yjnm7hAB4dzlGo UGnE6dQTMvAJw+utCFkyFWrd6kzMLNBaNMXjQI3JhgzwZ/Q+NOfjnd1q5 Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181306" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181306" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724261" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:15 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 03/15] vdpa/ifc: add blk ops for ifc device Date: Tue, 25 Jan 2022 14:47:26 +0800 Message-Id: <1643093258-47258-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 88 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 91 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 01522c6..769c603 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -66,6 +66,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 48056d1..965baa2 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1093,6 +1093,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1207,6 +1211,88 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + __u64 capacity = 0; + uint8_t *byte; + + if (len < sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %lu", + len, sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /** + ** cannot read 64-bit register in one attempt, + ** so read byte by byte. + **/ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (__u64)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %quG", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(INFO, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = NULL, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1219,7 +1305,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Tue Jan 25 06:47:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106459 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4BE80A04A9; Tue, 25 Jan 2022 07:53:32 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A0BEE42835; Tue, 25 Jan 2022 07:53:20 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 7ACD642863 for ; Tue, 25 Jan 2022 07:53:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093599; x=1674629599; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ETeqzt+QwqJql0eu6l5BgZJ6ZaBVQQmEgkrNgey+DII=; b=KgjKz9c17ojP5kbIWQ9sOeyCHBPFR0xbE9YWt3jglvho5s1zf2gn4Mrk BZr/4sefQN++Atuq2A09a1zrfUxyG/D5nWZ93xOoDT28U5McQE87PEu1k WItnXwH+yJhcBtz9S8bygTxgmw4TS2ofyEXyNBhuR02+SzL2BLxw3TBTp RK74SJ7SkczoqqKyThYY4hYZL3/Xg4DjjqV9sOAbVehahAyr1qo2EZmyU vJYSvOW9QHmxltPiaZI20mg9eGVP2T9l8KTPyQT8e0HEIrpmaOzkvH5WQ r/eztD5A5xTApnh0nd84sE4VRAc3aMkIl8xCVzHIVGIh9+jR81o9NdXmB A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181309" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181309" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724270" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:17 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 04/15] vdpa/ifc: add vdpa interrupt for blk device Date: Tue, 25 Jan 2022 14:47:27 +0800 Message-Id: <1643093258-47258-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the blk we need to relay all the cmd of each queue. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 48 +++++++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 965baa2..9729490 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -374,24 +374,50 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); for (i = 0; i < nr_vring; i++) internal->intr_fd[i] = -1; - for (i = 0; i < nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vring); - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { - fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); - if (fd < 0) { - DRV_LOG(ERR, "can't setup eventfd: %s", - strerror(errno)); - return -1; + if (internal->device_type == IFCVF_NET) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if ((i & 1) == 0 && m_rx == true) { + /** + ** For the net we only need to relay rx queue, + ** which will change the mem of VM. + **/ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; + } + } + } else if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if (m_rx == true) { + /** + ** For the blk we need to relay all the read cmd + ** of each queue + **/ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } - internal->intr_fd[i] = fd; - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } } From patchwork Tue Jan 25 06:47:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106460 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DB163A04A9; Tue, 25 Jan 2022 07:53:37 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8AA3A4285C; Tue, 25 Jan 2022 07:53:23 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 3DF254285C for ; Tue, 25 Jan 2022 07:53:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093602; x=1674629602; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ZlAhMqEI3synkfxgQEk4Lad+ED0fDu17W/+sgE/TXFk=; b=RMylkEmS27bEotAPfTAP7ZqQU9k6Bu/lW5LACdtAA/PPGmaCYYokiWib rWmLVBFbpnxwLl4418u0x0pf3So2u3esVjisFOXxGlkL0qm2hc8Ift3oY lpu04WPC79HYX+l9NEs8mI6ieCRRsCSc0SEyo/040Nwewet4TAK4DEUl4 xVXVcHpN0H57qRS/+Ep0tzZFnTh3xfkkyK/6amEZh4m8lJOOnk2vSbfCO r6iCghSCm7/bEaY4YpsnPrS3EVxcWAHqegz/Z/bRqs/KXKzDQoZkdoPlj wuD/OSqbghgdVb64vAE1LJF7rs0IqrfTFFCh/s9FwaPxXkrD4trasLtVi Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181314" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181314" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724282" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:20 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, Jin Yu Subject: [PATCH 05/15] vdpa/ifc: add blk dev sw live migration Date: Tue, 25 Jan 2022 14:47:28 +0800 Message-Id: <1643093258-47258-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable virtio blk sw live migration relay callfd and log the dirty page. In this version we ignore the write cmd and still mark it dirty. Maybe we can improve it later. Signed-off-by: Jin Yu Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 4 +- drivers/vdpa/ifc/base/ifcvf.h | 6 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 130 +++++++++++++++++++++++++++++++++++------- 3 files changed, 118 insertions(+), 22 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 721cb1d..3a69e53 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -189,7 +189,7 @@ IFCVF_WRITE_REG32(val >> 32, hi); } -STATIC int +int ifcvf_hw_enable(struct ifcvf_hw *hw) { struct ifcvf_pci_common_cfg *cfg; @@ -238,7 +238,7 @@ return 0; } -STATIC void +void ifcvf_hw_disable(struct ifcvf_hw *hw) { u32 i; diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 769c603..6dd7925 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -179,4 +179,10 @@ struct ifcvf_hw { u64 ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid); +int +ifcvf_hw_enable(struct ifcvf_hw *hw); + +void +ifcvf_hw_disable(struct ifcvf_hw *hw); + #endif /* _IFCVF_H_ */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9729490..1f832a3 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -334,10 +334,68 @@ struct rte_vdpa_dev_info { rte_vhost_get_negotiated_features(vid, &features); if (RTE_VHOST_NEED_LOG(features)) { - ifcvf_disable_logging(hw); - rte_vhost_get_log_base(internal->vid, &log_base, &log_size); - rte_vfio_container_dma_unmap(internal->vfio_container_fd, - log_base, IFCVF_LOG_BASE, log_size); + if (internal->device_type == IFCVF_NET) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, + &log_size); + rte_vfio_container_dma_unmap( + internal->vfio_container_fd, log_base, + IFCVF_LOG_BASE, log_size); + } + /** + ** IFCVF marks dirty memory pages for only packet buffer, + ** SW helps to mark the used ring as dirty after device stops. + **/ + for (i = 0; i < hw->nr_vring; i++) { + len = IFCVF_USED_RING_LEN(hw->vring[i].size); + rte_vhost_log_used_vring(vid, i, 0, len); + } + } +} + +static void +vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) +{ + struct ifcvf_hw *hw = &internal->hw; + struct rte_vhost_vring vq; + int i, vid; + uint64_t features = 0; + uint64_t log_base = 0, log_size = 0; + uint64_t len; + + vid = internal->vid; + + if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vq); + while (vq.avail->idx != vq.used->idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + hw->vring[i].last_avail_idx = vq.avail->idx; + hw->vring[i].last_used_idx = vq.used->idx; + } + } + + ifcvf_hw_disable(hw); + + for (i = 0; i < hw->nr_vring; i++) + rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, + hw->vring[i].last_used_idx); + + if (internal->sw_lm) + return; + + rte_vhost_get_negotiated_features(vid, &features); + if (RTE_VHOST_NEED_LOG(features)) { + if (internal->device_type == IFCVF_NET) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, + &log_size); + rte_vfio_container_dma_unmap( + internal->vfio_container_fd, log_base, + IFCVF_LOG_BASE, log_size); + } /* * IFCVF marks dirty memory pages for only packet buffer, * SW helps to mark the used ring as dirty after device stops. @@ -665,15 +723,18 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /** + ** NETWORK: Direct I/O for Tx queue, relay for Rx queue + ** BLK: relay every queue + **/ + if ((i & 1) && (internal->device_type == IFCVF_NET)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); return -1; } hw->vring[i].used = gpa; - } else { + } else if (internal->device_type == IFCVF_BLK) { hw->vring[i].used = m_vring_iova + (char *)internal->m_vring[i].used - (char *)internal->m_vring[i].desc; @@ -692,7 +753,10 @@ struct rte_vdpa_dev_info { } hw->nr_vring = nr_vring; - return ifcvf_start_hw(&internal->hw); + if (internal->device_type == IFCVF_NET) + return ifcvf_start_hw(&internal->hw); + else if (internal->device_type == IFCVF_BLK) + return ifcvf_hw_enable(&internal->hw); error: for (i = 0; i < nr_vring; i++) @@ -717,8 +781,10 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (((i & 1) == 0 && internal->device_type == IFCVF_NET) || + internal->device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -730,6 +796,8 @@ struct rte_vdpa_dev_info { (uint64_t)(uintptr_t)internal->m_vring[i].desc, m_vring_iova, size); + hw->vring[i].last_avail_idx = vq.used->idx; + hw->vring[i].last_used_idx = vq.used->idx; rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, hw->vring[i].last_used_idx); rte_free(internal->m_vring[i].desc); @@ -780,17 +848,36 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { - ev.events = EPOLLIN | EPOLLPRI; - /* leave a flag to mark it's for interrupt */ - ev.data.u64 = 1 | qid << 1 | - (uint64_t)internal->intr_fd[qid] << 32; - if (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev) - < 0) { - DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); - return NULL; + if (internal->device_type == IFCVF_NET) { + for (qid = 0; qid < q_num; qid += 2) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); + } + } else if (internal->device_type == IFCVF_BLK) { + for (qid = 0; qid < q_num; qid += 1) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); } - update_used_ring(internal, qid); } /* start relay with a first kick */ @@ -878,7 +965,10 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_NET) + vdpa_ifcvf_stop(internal); + else if (internal->device_type == IFCVF_BLK) + vdpa_ifcvf_blk_pause(internal); vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Tue Jan 25 06:47:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106461 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 60D77A04A9; Tue, 25 Jan 2022 07:53:43 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7858342873; Tue, 25 Jan 2022 07:53:26 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id D551B4286A for ; Tue, 25 Jan 2022 07:53:24 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093605; x=1674629605; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=eXXZ9bdJ8Dxt6weLQSDlujTia56xKnD4uXZ6229kWiA=; b=Cw9WUM575I5i2jKsI+cEMP3B9h9F1HeYfppvgNtLGCOa/d9SKPpazKC6 1PARowzjHbJ1vuWiVftHbk0raJHf0HKT4zZGuKBonykQhppLYBdpslFV5 D9HDZNSP2GcEPMOsrNHjD0qt3MkCvJI+Aw0EHkgA676FoXoSk3sXH9vAu KTlA5rfeHy/1cc8HPieEXxkbDPHfks7PPb2G2iGEh9N+lijzvA/dwnRfa C/rUHt+Ecl056icrYvYcCKfCkWSGUuWPZLXaCF6vRTSWCSJdqkQlVpsJy agVUz8Y+cWtKP0biAxifsKWTtDDLQVthB6F8pCbFKfRd4k9Fc092QQz81 w==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181322" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181322" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724289" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:22 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 06/15] example/vdpa:add vdpa blk support in example Date: Tue, 25 Jan 2022 14:47:29 +0800 Message-Id: <1643093258-47258-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- examples/vdpa/Makefile | 2 +- examples/vdpa/main.c | 8 ++ examples/vdpa/meson.build | 1 + examples/vdpa/vdpa_blk_compact.c | 152 +++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 118 ++++++++++++++++++++++++ examples/vdpa/vhost_user.h | 190 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 470 insertions(+), 1 deletion(-) create mode 100644 examples/vdpa/vdpa_blk_compact.c create mode 100644 examples/vdpa/vdpa_blk_compact.h create mode 100644 examples/vdpa/vhost_user.h diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile index d974db4..9d0479b 100644 --- a/examples/vdpa/Makefile +++ b/examples/vdpa/Makefile @@ -5,7 +5,7 @@ APP = vdpa # all source are stored in SRCS-y -SRCS-y := main.c +SRCS-y := main.c vdpa_blk_compact.c CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..3fa3d3a 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -156,6 +157,7 @@ struct vdpa_port { static const struct rte_vhost_device_ops vdpa_sample_devops = { .new_device = new_device, .destroy_device = destroy_device, + .new_connection = rte_vhost_blk_session_install_rte_compat_hooks, }; static int @@ -192,6 +194,12 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + if (vdpa_blk_device_set_features_and_protocol(socket_path, vport->dev) + < 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocal features failed: %s\n", + socket_path); + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build index bd08605..f0d111c 100644 --- a/examples/vdpa/meson.build +++ b/examples/vdpa/meson.build @@ -15,4 +15,5 @@ deps += 'vhost' allow_experimental_apis = true sources = files( 'main.c', + 'vdpa_blk_compact.c', ) diff --git a/examples/vdpa/vdpa_blk_compact.c b/examples/vdpa/vdpa_blk_compact.c new file mode 100644 index 0000000..7310ebb --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.c @@ -0,0 +1,152 @@ +/* +** INTEL CONFIDENTIAL +** +** Copyright (c) Intel Corporation. +** All rights reserved. +** +** The source code contained or described herein and all documents related +** to the source code ("Material") are owned by Intel Corporation or its +** suppliers or licensors. Title to the Material remains with Intel +** Corporation or its suppliers and licensors. The Material contains trade +** secrets and proprietary and confidential information of Intel or its +** suppliers and licensors. The Material is protected by worldwide +** copyright and trade secret laws and treaty provisions. No part of the +** Material may be used, copied, reproduced, modified, published, uploaded, +** posted, transmitted, distributed, or disclosed in any way without Intel's +** prior express written permission. +** +** No license under any patent, copyright, trade secret or other +** intellectual property right is granted to or conferred upon you by +** disclosure or delivery of the Materials, either expressly, by +** implication, inducement, estoppel or otherwise. Any license under such +** intellectual property rights must be express and approved by Intel in +** writing. +*/ + +/** +** @file +** +** Block device specific vhost lib +**/ + +#include + +#include +#include +#include +#include "vdpa_blk_compact.h" +#include "vhost_user.h" + +#define VHOST_USER_GET_CONFIG 24 +#define VHOST_USER_SET_CONFIG 25 + +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + +/* + * Function to handle vhost user blk message + */ +static enum rte_vhost_msg_result +rte_vhost_blk_extern_vhost_pre_msg_handler(int vid, void *_msg) +{ + struct VhostUserMsg *msg = _msg; + struct rte_vdpa_device *vdev = NULL; + + vdev = rte_vhost_get_vdpa_device(vid); + if (vdev == NULL) + return RTE_VHOST_MSG_RESULT_ERR; + + fprintf(stderr, "msg is %d\n", msg->request.master); + switch (msg->request.master) { + case VHOST_USER_GET_CONFIG: { + int rc = 0; + + fprintf(stdout, "read message VHOST_USER_GET_CONFIG\n"); + + if (vdev->ops->get_config) { + fprintf(stdout, "get_config() function is valid!\n"); + rc = vdev->ops->get_config(vid, + msg->payload.cfg.region, + msg->payload.cfg.size); + if (rc != 0) { + msg->size = 0; + fprintf(stdout, "get_config() return error!\n"); + } + } else { + fprintf(stdout, "get_config() function is invalid!\n"); + } + + return RTE_VHOST_MSG_RESULT_REPLY; + } + case VHOST_USER_SET_CONFIG: { + int rc = 0; + + fprintf(stdout, + "read message VHOST_USER_SET_CONFIG\n"); + + if (vdev->ops->set_config) { + rc = vdev->ops->set_config(vid, + msg->payload.cfg.region, + msg->payload.cfg.offset, + msg->payload.cfg.size, + msg->payload.cfg.flags); + } + + return rc == 0 ? RTE_VHOST_MSG_RESULT_OK : RTE_VHOST_MSG_RESULT_ERR; + } + default: + break; + } + + return RTE_VHOST_MSG_RESULT_NOT_HANDLED; +} + +struct rte_vhost_user_extern_ops g_blk_extern_vhost_ops = { + .pre_msg_handle = rte_vhost_blk_extern_vhost_pre_msg_handler, + .post_msg_handle = NULL, +}; + +int +rte_vhost_blk_session_install_rte_compat_hooks(int vid) +{ + int rc; + + rc = rte_vhost_extern_callback_register(vid, + &g_blk_extern_vhost_ops, + NULL); + if (rc != 0) { + fprintf(stderr, "%s() failed for vid = %d\n", __func__, vid); + return -1; + } + fprintf(stdout, "register extern vhost ops on vid = %d\n", vid); + return 0; +} + + +int +vdpa_blk_device_set_features_and_protocol(const char *path, + struct rte_vdpa_device *vdev) +{ + uint64_t protocol_features = 0; + + if (!vdev) { + fprintf(stdout, "vdev is NULL.\n"); + return -EINVAL; + } + + /* vdpa net does not have the get_config */ + if (!vdev->ops->get_config) + return 0; + + rte_vhost_driver_set_features(path, SPDK_VHOST_BLK_FEATURES_BASE); + rte_vhost_driver_disable_features(path, + SPDK_VHOST_BLK_DISABLED_FEATURES); + + rte_vhost_driver_get_protocol_features(path, &protocol_features); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_CONFIG); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD); + rte_vhost_driver_set_protocol_features(path, protocol_features); + + return 0; +} diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..94bd9c1 --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,118 @@ +/* +** INTEL CONFIDENTIAL +** +** Copyright (c) Intel Corporation. +** All rights reserved. +** +** The source code contained or described herein and all documents related +** to the source code ("Material") are owned by Intel Corporation or its +** suppliers or licensors. Title to the Material remains with Intel +** Corporation or its suppliers and licensors. The Material contains trade +** secrets and proprietary and confidential information of Intel or its +** suppliers and licensors. The Material is protected by worldwide +** copyright and trade secret laws and treaty provisions. No part of the +** Material may be used, copied, reproduced, modified, published, uploaded, +** posted, transmitted, distributed, or disclosed in any way without Intel's +** prior express written permission. +** +** No license under any patent, copyright, trade secret or other +** intellectual property right is granted to or conferred upon you by +** disclosure or delivery of the Materials, either expressly, by +** implication, inducement, estoppel or otherwise. Any license under such +** intellectual property rights must be express and approved by Intel in +** writing. +*/ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** + * @file + * + * Device specific vhost lib + */ + +#include + +#include +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_RO 5 /* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ +#define VIRTIO_BLK_F_DISCARD 13 /* DISCARD is supported */ +#define VIRTIO_BLK_F_WRITE_ZEROES 14 /* WRITE ZEROES is supported */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_FLUSH 9 /* Flush command supported */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ + +/* Old (deprecated) name for VIRTIO_BLK_F_FLUSH. */ +#define VIRTIO_BLK_F_WCE VIRTIO_BLK_F_FLUSH +#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#endif + +#define SPDK_VHOST_FEATURES ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC)) + +#define SPDK_VHOST_DISABLED_FEATURES ((1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY)) + +#define SPDK_VHOST_BLK_FEATURES_BASE (SPDK_VHOST_FEATURES | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define SPDK_VHOST_BLK_DISABLED_FEATURES (SPDK_VHOST_DISABLED_FEATURES | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_BARRIER) | (1ULL << VIRTIO_BLK_F_SCSI)) + +/* Vhost-blk support protocol features */ +#define SPDK_VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_CONFIG) | \ + (1ULL << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This function will set vhost user block + * + * @param path + * socket path + */ +int +vdpa_blk_device_set_features_and_protocol(const char *path, + struct rte_vdpa_device *vdev); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Install external hook to handle vhost user block message + * + * @param vid + * vhost device id + */ +int +rte_vhost_blk_session_install_rte_compat_hooks(int vid); + +#endif /* _VDPA_BLK_COMPACT_H_ */ diff --git a/examples/vdpa/vhost_user.h b/examples/vdpa/vhost_user.h new file mode 100644 index 0000000..b9e1be1 --- /dev/null +++ b/examples/vdpa/vhost_user.h @@ -0,0 +1,190 @@ +/* +** INTEL CONFIDENTIAL +** +** Copyright (c) Intel Corporation. +** All rights reserved. +** +** The source code contained or described herein and all documents related +** to the source code ("Material") are owned by Intel Corporation or its +** suppliers or licensors. Title to the Material remains with Intel +** Corporation or its suppliers and licensors. The Material contains trade +** secrets and proprietary and confidential information of Intel or its +** suppliers and licensors. The Material is protected by worldwide +** copyright and trade secret laws and treaty provisions. No part of the +** Material may be used, copied, reproduced, modified, published, uploaded, +** posted, transmitted, distributed, or disclosed in any way without Intel's +** prior express written permission. +** +** No license under any patent, copyright, trade secret or other +** intellectual property right is granted to or conferred upon you by +** disclosure or delivery of the Materials, either expressly, by +** implication, inducement, estoppel or otherwise. Any license under such +** intellectual property rights must be express and approved by Intel in +** writing. +*/ + +#ifndef _VHOST_NET_USER_H +#define _VHOST_NET_USER_H + +#include +#include + +#include "rte_vhost.h" + +/* refer to hw/virtio/vhost-user.c */ + +#define VHOST_MEMORY_MAX_NREGIONS 8 + +#ifndef VHOST_USER_MAX_CONFIG_SIZE +#define VHOST_USER_MAX_CONFIG_SIZE 256 +#endif + +#define VHOST_USER_PROTOCOL_FEATURES ((1ULL << VHOST_USER_PROTOCOL_F_MQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) |\ + (1ULL << VHOST_USER_PROTOCOL_F_RARP) | \ + (1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \ + (1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CRYPTO_SESSION) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER) | \ + (1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT)) + +typedef enum VhostUserRequest { + VHOST_USER_NONE = 0, + VHOST_USER_GET_FEATURES = 1, + VHOST_USER_SET_FEATURES = 2, + VHOST_USER_SET_OWNER = 3, + VHOST_USER_RESET_OWNER = 4, + VHOST_USER_SET_MEM_TABLE = 5, + VHOST_USER_SET_LOG_BASE = 6, + VHOST_USER_SET_LOG_FD = 7, + VHOST_USER_SET_VRING_NUM = 8, + VHOST_USER_SET_VRING_ADDR = 9, + VHOST_USER_SET_VRING_BASE = 10, + VHOST_USER_GET_VRING_BASE = 11, + VHOST_USER_SET_VRING_KICK = 12, + VHOST_USER_SET_VRING_CALL = 13, + VHOST_USER_SET_VRING_ERR = 14, + VHOST_USER_GET_PROTOCOL_FEATURES = 15, + VHOST_USER_SET_PROTOCOL_FEATURES = 16, + VHOST_USER_GET_QUEUE_NUM = 17, + VHOST_USER_SET_VRING_ENABLE = 18, + VHOST_USER_SEND_RARP = 19, + VHOST_USER_NET_SET_MTU = 20, + VHOST_USER_SET_SLAVE_REQ_FD = 21, + VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_CRYPTO_CREATE_SESS = 26, + VHOST_USER_CRYPTO_CLOSE_SESS = 27, + VHOST_USER_POSTCOPY_ADVISE = 28, + VHOST_USER_POSTCOPY_LISTEN = 29, + VHOST_USER_POSTCOPY_END = 30, + VHOST_USER_GET_INFLIGHT_FD = 31, + VHOST_USER_SET_INFLIGHT_FD = 32, + VHOST_USER_MAX = 33 +} VhostUserRequest; + +typedef enum VhostUserSlaveRequest { + VHOST_USER_SLAVE_NONE = 0, + VHOST_USER_SLAVE_IOTLB_MSG = 1, + VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG = 3, + VHOST_USER_SLAVE_MAX +} VhostUserSlaveRequest; + +typedef struct VhostUserMemoryRegion { + uint64_t guest_phys_addr; + uint64_t memory_size; + uint64_t userspace_addr; + uint64_t mmap_offset; +} VhostUserMemoryRegion; + +typedef struct VhostUserMemory { + uint32_t nregions; + uint32_t padding; + VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS]; +} VhostUserMemory; + +typedef struct VhostUserLog { + uint64_t mmap_size; + uint64_t mmap_offset; +} VhostUserLog; + +/* Comply with Cryptodev-Linux */ +#define VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH 512 +#define VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH 64 + +/* Same structure as vhost-user backend session info */ +typedef struct VhostUserCryptoSessionParam { + int64_t session_id; + uint32_t op_code; + uint32_t cipher_algo; + uint32_t cipher_key_len; + uint32_t hash_algo; + uint32_t digest_len; + uint32_t auth_key_len; + uint32_t aad_len; + uint8_t op_type; + uint8_t dir; + uint8_t hash_mode; + uint8_t chaining_dir; + uint8_t *ciphe_key; + uint8_t *auth_key; + uint8_t cipher_key_buf[VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH]; + uint8_t auth_key_buf[VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH]; +} VhostUserCryptoSessionParam; + +typedef struct VhostUserVringArea { + uint64_t u64; + uint64_t size; + uint64_t offset; +} VhostUserVringArea; + +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint16_t num_queues; + uint16_t queue_size; +} VhostUserInflight; + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + +typedef struct VhostUserMsg { + union { + uint32_t master; /* a VhostUserRequest value */ + uint32_t slave; /* a VhostUserSlaveRequest value*/ + } request; + +#define VHOST_USER_VERSION_MASK 0x3 +#define VHOST_USER_REPLY_MASK (0x1 << 2) +#define VHOST_USER_NEED_REPLY (0x1 << 3) + uint32_t flags; + uint32_t size; /* the following payload size */ + union { +#define VHOST_USER_VRING_IDX_MASK 0xff +#define VHOST_USER_VRING_NOFD_MASK (0x1<<8) + uint64_t u64; + struct vhost_vring_state state; + struct vhost_vring_addr addr; + VhostUserMemory memory; + VhostUserLog log; + struct vhost_iotlb_msg iotlb; + VhostUserCryptoSessionParam crypto_session; + VhostUserVringArea area; + VhostUserInflight inflight; + struct vhost_user_config cfg; + } payload; + int fds[VHOST_MEMORY_MAX_NREGIONS]; + int fd_num; +} __attribute((packed)) VhostUserMsg; + +#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) + +/* The version of the protocol we support */ +#define VHOST_USER_VERSION 0x1 +#endif From patchwork Tue Jan 25 06:47:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106462 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE933A04A9; Tue, 25 Jan 2022 07:53:50 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B33AC4288B; Tue, 25 Jan 2022 07:53:28 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 26EDC4286D for ; Tue, 25 Jan 2022 07:53:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093607; x=1674629607; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=f9iIhO50rLyIGKCTBiUd876H7y7ZRQkY7IipgetRm0o=; b=I0IM+A/ic+PZO9D5sh90VTACg41Voa7WDlE51HLicoQTv8NGNUR76lGA JUwl8KpuYzHgRlqyjHpOpX3OIr6Z9IdtOOTDLZALVAtOJwOb9qaQD5jZu YgcK5jYvC0g4quFZsiZFvclndq1AcvXcGFRzf9f4Tpq/Q9gj35eiZ2uVj jdPo7eXOXwiQp8M/4zNjUl7RXTMPT7acC9Rarcz+AbJwm8oQ10uBYhTCI GTztX1ioQOaqgMxdO5naLIDDbuWA0SUEXuxWasb2mi/Xna/4HcdHMs/m9 SU0j4BBG/ptuzEC2iDBA0BeGOqVIjAoVxTEGe5rxYIoa9xFR3lTw6fyeU Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181326" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181326" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724302" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:25 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 07/15] usertools: add support for virtio blk device Date: Tue, 25 Jan 2022 14:47:30 +0800 Message-Id: <1643093258-47258-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- usertools/dpdk-devbind.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..cbe336f 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -14,6 +14,8 @@ from os.path import join as path_join # The PCI base class for all devices +virtio_blk_class = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': '8086', 'SDevice': '0002'} network_class = {'Class': '02', 'Vendor': None, 'Device': None, 'SVendor': None, 'SDevice': None} acceleration_class = {'Class': '12', 'Vendor': None, 'Device': None, @@ -72,6 +74,7 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk_devices = [virtio_blk_class] network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -587,6 +590,9 @@ def show_status(): Displays to the user what devices are bound to the igb_uio driver, the kernel driver or to no driver''' + if status_dev in ["virtio_blk", "all"]: + show_device_status(virtio_blk_devices, "virtio_blk") + if status_dev in ["net", "all"]: show_device_status(network_devices, "Network", if_field=True) @@ -746,6 +752,7 @@ def do_arg_actions(): if b_flag is not None: clear_data() # refresh if we have changed anything + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) @@ -769,6 +776,7 @@ def main(): parse_args() check_modules() clear_data() + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) From patchwork Tue Jan 25 06:47:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106463 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 214EEA04A9; Tue, 25 Jan 2022 07:53:56 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A828C42864; Tue, 25 Jan 2022 07:53:31 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 9615A4286D for ; Tue, 25 Jan 2022 07:53:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093610; x=1674629610; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Lhdq/xvbv3NUIqAt2UMtdPlzcD1yb79uecQkP9ZKb7I=; b=P5H0dDu8C2JKaEaJbZ6kAX2DE4umCnE71c2mDzF69njmOcoFTRiR0mJN LRt8wwjZNrasBtkglBKcU+zUU+4pmfae4VhmAjUWWMpcIhdsyMV3FgIAv lVWDK9/3CVHTf1rXrK2h2kXgM+76Rc/neP1qidADNe6+rLbvLNStb1noG OUX853NmcHt5gBCUaqcFM+C6o2cwW3W2nG5VDR+CRwAII7Q+FB5nxvwEo EklZRynX8f703FWmvjxwSBqA59FO+aQLk7p/8EDQPaaVGIWxpNrJhuAM7 xxkTcOtdU+E9EIW1StXmYs4SQdSHfWARQWHOZWq7aofEdfCsOmv6sGb10 A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181330" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181330" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724316" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:27 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 08/15] vdpa/ifc: set_vring_state op is mandatory, add set_vring_state for blk device Date: Tue, 25 Jan 2022 14:47:31 +0800 Message-Id: <1643093258-47258-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 1f832a3..eff6ff3 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1384,6 +1384,16 @@ struct rte_vdpa_dev_info { } static int +ifcvf_blk_set_vring_state(int vid, int vring, int state) +{ + RTE_SET_USED(vid); + RTE_SET_USED(vring); + RTE_SET_USED(state); + + return 0; +} + +static int ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1401,7 +1411,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, From patchwork Tue Jan 25 06:47:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106464 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 93782A04A9; Tue, 25 Jan 2022 07:54:01 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9501F42878; Tue, 25 Jan 2022 07:53:33 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 2153342878 for ; Tue, 25 Jan 2022 07:53:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093612; x=1674629612; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=RP8MZzDlK94sqHiey5HuZVVinOlXw0d6JHwDfo1Thk8=; b=Re3hnIPPTHiW8usDkcRELcoVGwXs1GuHnSZVoPriwPhFEAxoskxyEZLQ IklI2YR1Yl78DSwhZW+31rscDv2m/HcQF2Bonneo1v2PAOxk+lu6xGQw+ aDIm4XRL0Qfhd90KyBxNaTWlFQyNgSORa0WX47ZskcjNUmVaTO2itU0jV +OOo76NKW8m2A96uauwt2sSspzMfGKcsUCa0DeXv5Ft/Q6S8JfK+zWicX tydUi1omSIjFVFfO85y5na4Tixrd/g7vJQ/E21nKE5tEZqUFaQ2BrBVNQ WQ07V3/fOqKfI6qwcENGvAyLRZ+bwn8r+TgzxNjUYbcpqDBHOr01RtA45 Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181335" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181335" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724330" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:30 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 09/15] vdpa/ifc: add some log at VDPA lauch before qemu connect Date: Tue, 25 Jan 2022 14:47:32 +0800 Message-Id: <1643093258-47258-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index eff6ff3..0b4b77f 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1447,6 +1447,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + __u64 capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1513,6 +1516,32 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /** + ** cannot read 64-bit register in one attempt, + ** so read byte by byte. + **/ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (__u64)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %quG", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Tue Jan 25 06:47:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106465 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0B55BA04A9; Tue, 25 Jan 2022 07:54:07 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8670D4286E; Tue, 25 Jan 2022 07:53:36 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id A0E8C4288C for ; Tue, 25 Jan 2022 07:53:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093614; x=1674629614; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=R4soTG88F7fmB5HFcY5JlL1xxLKfssfENGnWh3CDKUw=; b=kIcj5hxZV3SBMyx2iKNaiuynu91vGqvnv9xj7Znd1DAfoC9K8ZdHids1 U4xlqlWwIAl7q3wTyZCnHqvBQ6S3guZnw2R40MpHaHoOxOKnc9PLW1XLW XmTUKT5tlldy5Ltu7xxg3zhMEz3GfQDIgpAK2EH5+XGl2S8WWQFRBxXiF BcDV8G3fS/rn/VRpCGAA/NjOkW/U7nAD+KcdlZfTPTRmYMRNkdPCqeLRE ih3U9v+Y4b6wtBfuoNr1ua/2PB/BHy/RCQjehentSxlg9y0MMuH73owpK dRMxDdI9cmAGcCec7NGjcz+VBUZchayFHmD8cFXnfqI3JwdTR8WJpy5XG g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181342" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181342" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724346" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:32 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 10/15] vdpa/ifc: read virtio max_queues from hardware Date: Tue, 25 Jan 2022 14:47:33 +0800 Message-Id: <1643093258-47258-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org original code max_queues is set to IFCVF_MAX_QUEUES. New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 0b4b77f..f092aca 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1542,6 +1542,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(INFO, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + /* reset max_queue here, to minimum modification */ + internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Tue Jan 25 06:47:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106466 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 875AEA04A9; Tue, 25 Jan 2022 07:54:12 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 65B4542882; Tue, 25 Jan 2022 07:53:39 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 795754289C for ; Tue, 25 Jan 2022 07:53:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093617; x=1674629617; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=y6rcAdI9zXMDEW5jHIi8SQtr8ojYhewXUelobSRsLSw=; b=VZKZPjwt+o6ildeJQU4khC6SeQEPbtNZBmUpMXBRPnRrlPPeTY2p43mq 2N4jbYdjdC5WCgU0iTu8yNNkzkjDrRacn42RRpVZ4qjUws0RRAsgCiLtQ LhBTpsIDbe+9WutSEeyEw+m4aIA32Wgj/8+cLRnnfvu+opwmsrXp1ty12 ycLe9VPX6aC/0cJT/+qMRXUNHcgATOtsc+jmxdOJiwSUC8+4KNkw19Cg2 dzHScJaDY7rw9CjMIfhkLCSuyF1DvMF+ORGzn8FVc5Okvao6wisvofVaP cFcM7vChi0weMUwVJ3pK36bfsNe5l5kEtvIbWyzlq4j5md4iPA9BeTTRI g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181345" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181345" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724355" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:35 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 11/15] vdpa: add config space change interrupt register and handle for virtio_blk Date: Tue, 25 Jan 2022 14:47:34 +0800 Message-Id: <1643093258-47258-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 113 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index f092aca..2552375 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for intr relay */ int epfd; + int csc_fd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -622,6 +624,108 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); + + return; +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_fd, csc_val = 0; + + csc_fd = epoll_create(1); + if (csc_fd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_fd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + return NULL; + } + + internal->csc_fd = csc_fd; + + for (;;) { + csc_val = epoll_wait(csc_fd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail\n"); + return NULL; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || errno == EWOULDBLOCK) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + return NULL; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + int ret; + + ret = pthread_create(&internal->intr_tid, NULL, intr_relay, + (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static int +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_fd >= 0) + close(internal->csc_fd); + internal->csc_fd = -1; + + return 0; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -648,10 +752,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + ret = unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -969,6 +1079,9 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(internal); else if (internal->device_type == IFCVF_BLK) vdpa_ifcvf_blk_pause(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Tue Jan 25 06:47:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106467 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E8F40A04A9; Tue, 25 Jan 2022 07:54:17 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 55D0F428A4; Tue, 25 Jan 2022 07:53:41 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 00A56428A1 for ; Tue, 25 Jan 2022 07:53:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093620; x=1674629620; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=fzoVQeTfmJOOvSfP9KSLwJQ1n4KTLh0V4M2zfDlP3nI=; b=SkGd5jJX8bhxjFLuXqbrNw+aZlPoh4rZNl4vXRI/FA1GvB8ZgxKtbBXc SrzsfNrEUvhMtU0mfQ1MQIzqDpw+P2lqp8aYduJNMklNSTHaQ8NzZoFUw 7A+artfpKYLoCw65s99+hHf1enjmCgvXD6SUdLo7YF/eXr6dxesH9KhDB 91xHJTxrd8urt7TgPfxZRC6sZf0eRJb1+ZOwZ9J8T9bfhuVAah+m8FNtt RFNuyuokHqgjQWL5AndRuqCxe4z22P/zEIH5OnCucwkESXfNdNb2V/5Ub 0vXk6HgspM8dFlbJelIU9atOgLnltF9rvxyPbZVWmrx0qNiA+1by934fH Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181349" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181349" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724362" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:38 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 12/15] vdpa/ifc: add is_blk flag to ifcvf_hw, and init is_blk during probe Date: Tue, 25 Jan 2022 14:47:35 +0800 Message-Id: <1643093258-47258-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 6dd7925..8e602af 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -149,6 +149,7 @@ struct ifcvf_hw { u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + u8 is_blk; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 2552375..546f9bd 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1621,11 +1621,13 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->device_type = IFCVF_NET; + internal->hw.is_blk = IFCVF_NET; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; } else if (device_id == VIRTIO_ID_BLOCK) { internal->device_type = IFCVF_BLK; + internal->hw.is_blk = IFCVF_BLK; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; From patchwork Tue Jan 25 06:47:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106468 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07FDFA04A9; Tue, 25 Jan 2022 07:54:23 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3D7B242883; Tue, 25 Jan 2022 07:53:44 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id C5805428A1 for ; Tue, 25 Jan 2022 07:53:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093623; x=1674629623; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=OfMQKUusCbeZdl+8sJdrAlffNHrBeBS10u16yx50v50=; b=AmZeVIlbo9c46mvz3HeS4ny4MAVVIRF0WtJRwO0MpUtgf7AYSNg3erhc 6KAl7+MT+vrXOjH4HPqW6drFd+TKwggzcNG5lsPdGU+7yHdenBKA7LTLV BADSp7xHwV8n8Y1xfrESGxkK7fDcs+SXrWNw+8ZwDqGPbdoTntA3/mP0Q TMAwmRwZIHSXzum0K3KQwYhx0dgBIzw7eNgNigUh/C/tNgEVzUrasRkbN XQeJfMVSgYFjzCLBu8v1vj5vY9RT7CC6YiXbYug3BOwrJQilvYkAiZdyK A6bD5MHwJli+DhaT+FSPmyUp+f8sz9Gu0Sc1IdvMm7emvG0Dr6QiEc52h g==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181358" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181358" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:42 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724370" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:40 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 13/15] vdpa/ifc/base: for blk device, live migration register is different from net device Date: Tue, 25 Jan 2022 14:47:36 +0800 Message-Id: <1643093258-47258-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 1.last_avail_idx is lower 16 bit of the register. 2.address of ring_state register is different between net and blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 3a69e53..a8a4728 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -216,10 +216,18 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->is_blk == IFCVF_BLK) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } else if (hw->is_blk == IFCVF_NET) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -252,9 +260,23 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + } else if (hw->is_blk == IFCVF_NET) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + } + + if (hw->is_blk == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else if (hw->is_blk == IFCVF_NET) + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } } diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 8e602af..7367094 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 From patchwork Tue Jan 25 06:47:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106469 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 97088A04A9; Tue, 25 Jan 2022 07:54:28 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2A1DB428B3; Tue, 25 Jan 2022 07:53:47 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 6F3E7428B0 for ; Tue, 25 Jan 2022 07:53:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093625; x=1674629625; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=PzkWnA2tG6Y/NlQiseF5ZlBl65W8mq1mnzq6NKqVwrY=; b=MBfLe5OVo7zM6HJ4HBz1xRkZOvxGarStwjrFnCxGAAhk2U8grJ60fMGS WUQIhGzj2KUNJcbTY+OBfW8EXg8qFZWaDHIes86ZKOe1d+6hOgzyckXCb Y4pRPMTb9A27Zg9o72tsmCGy86xZi9jpdNCUBfrwJNa20a0FuQpD4Z5GR vCJDA6C2+6dEBuuTjSt1iInVEwSahmTiiFwH33ebglLUGpkcAeCzeP3o0 yRnpqhIIL6I8DLxIBmIWaTt6AU3f2IvfJWp+CQs7ka3gkHX0AgAmMS3GZ 56qapkxlKZTWsHJ8/jmoVuqQs51o/jivczz4Xrel1BE11ww8XvFztI7tZ A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181367" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181367" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724378" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:43 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 14/15] vdpa/ifc: make sure hardware last_avail_idx and last_used_idx is the same when blk device pause Date: Tue, 25 Jan 2022 14:47:37 +0800 Message-Id: <1643093258-47258-15-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 2 +- drivers/vdpa/ifc/base/ifcvf.h | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 32 +++++++++++++++++++++++--------- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index a8a4728..7018048 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -116,7 +116,7 @@ IFCVF_WRITE_REG8(status, &hw->common_cfg->device_status); } -STATIC void +void ifcvf_reset(struct ifcvf_hw *hw) { ifcvf_set_status(hw, 0); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 7367094..f22d18b 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -157,6 +157,9 @@ struct ifcvf_hw { int ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev); +void +ifcvf_reset(struct ifcvf_hw *hw); + u64 ifcvf_get_features(struct ifcvf_hw *hw); diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 546f9bd..ff233bc 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -359,23 +359,32 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) { struct ifcvf_hw *hw = &internal->hw; - struct rte_vhost_vring vq; int i, vid; uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; if (internal->device_type == IFCVF_BLK) { for (i = 0; i < hw->nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vq); - while (vq.avail->idx != vq.used->idx) { - ifcvf_notify_queue(hw, i); - usleep(10); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } @@ -766,7 +775,12 @@ struct rte_vdpa_dev_info { if (ret) goto err; - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_BLK) { + vdpa_ifcvf_blk_pause(internal); + ifcvf_reset(&internal->hw); + } else { + vdpa_ifcvf_stop(internal); + } ret = vdpa_disable_vfio_intr(internal); if (ret) From patchwork Tue Jan 25 06:47:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106470 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D916AA04A9; Tue, 25 Jan 2022 07:54:34 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7C60642837; Tue, 25 Jan 2022 07:54:04 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id E82F94282D for ; Tue, 25 Jan 2022 07:54:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643093643; x=1674629643; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=iiC9zjPCRWSFSUr3n/L661D8aNJGk/y8rMEVaa6naFk=; b=SurbhJvXfTc6Me3731VZ5XA34asgpzqkvQyVsaAAYxTdIA4pgSvJMmnR QMNbUKRsZ579L0Mn7YLD5eXCaMtipF2snvzPwG1Gn87L+7C5CuBgBE10q r92W/m0tPr4qmjoYyZRfxjKppgt20gOBL4HyCR9CksI4pXsVn3ib7Wd12 Is29LcQDxpEi+xDqjsSvoO0yiqJpKUduNHFCSzx2GxCq5G54hzorgfu2t B6XoKuCp7trb24Ds0jOOz1KDeammV11SrpRWHX5jM1n1BOEnQFulXbRsZ zy/7rMqfsOK7fyNOmJwgRRbiiI9DpZRmnfwJlUfPWbuvl3yd2XSuK6QSd A==; X-IronPort-AV: E=McAfee;i="6200,9189,10237"; a="246181375" X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="246181375" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2022 22:53:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,314,1635231600"; d="scan'208";a="695724385" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by orsmga005.jf.intel.com with ESMTP; 24 Jan 2022 22:53:46 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH 15/15] vhost: make sure each queue callfd is configured Date: Tue, 25 Jan 2022 14:47:38 +0800 Message-Id: <1643093258-47258-16-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643093258-47258-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 5eb1dd6..0be879a 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3137,12 +3137,27 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, if (!vdpa_dev) goto out; + if (request != VHOST_USER_SET_VRING_CALL) + goto out; + if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid)) VHOST_LOG_CONFIG(ERR, "Failed to configure vDPA device\n"); else dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED; + } else { + /** + ** when VIRTIO_DEV_VDPA_CONFIGURED already configured + ** close the device and config the device again, + ** make sure the call fd of each queue is configed correctly. + **/ + if (vdpa_dev->ops->dev_close(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to close vDPA device\n"); + if (vdpa_dev->ops->dev_conf(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to re-config vDPA device\n"); } out: