From patchwork Sat Jan 29 03:03:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106683 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9FB2CA00C4; Sat, 29 Jan 2022 04:09:36 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 89AEE426DB; Sat, 29 Jan 2022 04:09:36 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id A024B40143 for ; Sat, 29 Jan 2022 04:09:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425774; x=1674961774; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rFUz/SVjhGgn20xUFjFFemaWcitUduNA2brEvZV/5+I=; b=m7bRTxN+A0LUFDHU7z39YJJj0WqkaEDTmvz++Npw/dkasgKKKM1fxVpb BKwHhYyq5EWmnE9iwfT0RfvuRQoKqYftgZvHrkd9h3ymEuzw74h4kpc7P HiiIsQJTQGJbsPJMReDtpSo2C/8CCrGsK0uL2LHfmvjXg4dXAkKhN4URR EJg6JdB2QvSbrDpy2/KHff2h2/Zv4hAj/3dqU73+f5JVMKPTNZAzlk4t8 01Ltm33EAyuK4SYTx0AK+qnUk4qEakrZuGCyYMU6kkdc+lhIVg9ro0W43 vMRnB7EQizHvuyEVUiJH6jNNr2A0BHCXCSfepr2rmq3O6X0E3b/xlS26g w==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="271699050" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="271699050" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047757" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:32 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 01/15] vdpa/ifc: add support for virtio blk device Date: Sat, 29 Jan 2022 11:03:23 +0800 Message-Id: <1643425417-215270-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 92 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..01522c6 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,8 +5,17 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 + #define IFCVF_VENDOR_ID 0x1AF4 #define IFCVF_DEVICE_ID 0x1041 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 @@ -57,7 +66,6 @@ #define IFCVF_32_BIT_MASK 0xffffffff - struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ @@ -126,7 +134,11 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3853c4c..96b67dd 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -58,6 +58,7 @@ struct ifcvf_internal { struct rte_vdpa_device *vdev; uint16_t max_queues; uint64_t features; + int device_type; rte_atomic32_t started; rte_atomic32_t dev_attached; rte_atomic32_t running; @@ -75,6 +76,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1170,6 +1177,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1181,6 +1230,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1230,13 +1280,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1248,7 +1309,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1316,6 +1378,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Sat Jan 29 03:03:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106684 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF379A00C4; Sat, 29 Jan 2022 04:09:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C821D42797; Sat, 29 Jan 2022 04:09:39 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 5EA2740143 for ; Sat, 29 Jan 2022 04:09:36 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425776; x=1674961776; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=8QP2LyrrdZj62eIBJqZfLk3qWwIGPSMqMJr/mc509yA=; b=lZIOeFADVrqYLzH7vrZB8BrOaJQYJR7CgbRnvGFFvGs3eCWp4c9W/eKQ PWeH5TbPyiLtRAKAqrGrgYHkiwkP/8GRPvnd5uSG7x5l/v3936UU+Um4R 7X6MjOrOBoc7aUrDyxR76tziz4t/KPu+ovmMCAj//0JBNjRlArLjBfm1u l1G1/gaZ3srJCjvLNzi+0uDb6t5dhvG6bKtq2yyeZqRo/VrmW6RmCsOYZ BgoJ+C9+6AMjGwE2i5Pc9mbARyyJKC2WqrrRtr83eMKsR3VuTYvgL2L4x Taf6cISLUNL52TD0fXe7jRx0o851GiRjuQMM4L1J1Xenf39gIdTU/KZ0y A==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="271699051" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="271699051" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047764" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:34 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 02/15] vhost: add vdpa ops for blk device Date: Sat, 29 Jan 2022 11:03:24 +0800 Message-Id: <1643425417-215270-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vdpa ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index fc2d6ac..9a23db9 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -65,8 +65,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t len); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Sat Jan 29 03:03:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106685 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F128DA00C4; Sat, 29 Jan 2022 04:09:46 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B7669427CA; Sat, 29 Jan 2022 04:09:40 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 5CDF440041 for ; Sat, 29 Jan 2022 04:09:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425779; x=1674961779; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=yxEs1HTEx5hxacfPc0cRGZ0+6ahapOXRkIQvLgowQ0M=; b=BghxmVQa8X5qKo9iRXoj70pVb00128NMc//3MkdmaIQWazk46qW1RtaV MUNvJM/dErs6OIfYelWZAzbsvADsKYnAQxIQ6JddKzNl0VlrjslyHc8wP OTWfZLhtkViY6RcxyJELL/1XEtEJdzjojaOebQrx2t9BRJ8Tis0mG+vSO 7AdhWk4o1qZoMXtUXkLHve7wre5uf47Hyad9niTgwYIwXIyL6Xv2QAYj6 mCpwqbFgs3MUIrGMeEpYpBTaLH19r6TW2AfEvGGporC6M+Y7D8sz886z+ sYf2PjXS8cnW64zTHcNneJefFlXv3saDMrr/TBfsIO6hGaSx2xbDgEYkr Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="271699052" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="271699052" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047775" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:37 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 03/15] vdpa/ifc: add blk ops for ifc device Date: Sat, 29 Jan 2022 11:03:25 +0800 Message-Id: <1643425417-215270-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 85 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 88 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 01522c6..769c603 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -66,6 +66,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 96b67dd..778e1fd 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1091,6 +1091,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1203,6 +1207,85 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + uint64_t capacity = 0; + uint8_t *byte; + + if (len < sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %u", + len, (uint32_t)sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(INFO, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = NULL, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1215,7 +1298,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Sat Jan 29 03:03:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106686 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D32C6A00C4; Sat, 29 Jan 2022 04:09:52 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C3DF4427AB; Sat, 29 Jan 2022 04:09:43 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 0348F427E2 for ; Sat, 29 Jan 2022 04:09:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425782; x=1674961782; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=t4X6tW0rhGpYcgtGZzxyo0yrIsxpuFKGI9cxthcm0Ng=; b=JHPi1p8mLnsQKZid37AKRYkNLAK+cO0uayWH2Jn+rjzc3vWdLuYvVFgN A0WZ5xTSURmHJOB1NG7j6g47lLJO4gcvd5HcnEp3MZOsGvT40XYmm3OZ4 DjoXdRo8CJlVZlGD5nMZXEQ5TsyLJUZnKGYwmlr56D4MWal+371OkJQmn BgSnhNPWD+PS+F2RTMLhBYdmIQXSCR/2lgk1/lBRrYq1CS3kxx3+58SeV HDRMMTNOvG8/BgrYfx7L1EGjFYSSYY6pEFJ+mM2RfSNB7GegiMcWerIlh p7z7fWh0NmXu+57HGVHnvhTpDQ7/qPmxor/hoVRdqtEYfxsf9K67KolCc g==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="271699055" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="271699055" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047785" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:40 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 04/15] vdpa/ifc: add vdpa interrupt for blk device Date: Sat, 29 Jan 2022 11:03:26 +0800 Message-Id: <1643425417-215270-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the blk we need to relay all the cmd of each queue. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 46 ++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 778e1fd..4f99bb3 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -372,24 +372,48 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); for (i = 0; i < nr_vring; i++) internal->intr_fd[i] = -1; - for (i = 0; i < nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vring); - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { - fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); - if (fd < 0) { - DRV_LOG(ERR, "can't setup eventfd: %s", - strerror(errno)); - return -1; + if (internal->device_type == IFCVF_NET) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if ((i & 1) == 0 && m_rx == true) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + */ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; + } + } + } else if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if (m_rx == true) { + /* For the blk we need to relay all the read cmd + * of each queue + */ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } - internal->intr_fd[i] = fd; - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } } From patchwork Sat Jan 29 03:03:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106687 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 729D8A00C4; Sat, 29 Jan 2022 04:10:00 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0600D427A7; Sat, 29 Jan 2022 04:09:51 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 2405D4279D for ; Sat, 29 Jan 2022 04:09:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425789; x=1674961789; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=mmQdtEyWueR0IpOBtyGW8/NFuPKxBfvpQaweaOKnDns=; b=Q13V6agXL3DCv2ni38tZ2CeWvdRE31eMiJnXCGUl195dJDDt5unX/BKY Lut+JL0MApqqxTjyNpP5LlnEgrbKJxxpxlVu1Ve/ZKeQMkpBLK7FdvSyX ndB6pgXmEsVOibEZf1l2rQFe9rGGoJ1dGlBpJoOTNQhMH9vAzZ9r2mu3Y 4fE4dw3Dh1fOYUa/zR3ztcU7RCBRilL2qHJwLsRfBl4/0NUvAIwdzjJcF MAT7YAAcvr0X3g+1v+cnm35UKSnXex3h3cI9wUCEUMSLbWv8qxujIupLl GGf6OcedZqXlEtgto7A54YC/rbdzLUSyq/JicQcYn/opUCw3km3jeQDH/ g==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076619" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076619" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:48 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047799" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:42 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 05/15] vdpa/ifc: add blk dev sw live migration Date: Sat, 29 Jan 2022 11:03:27 +0800 Message-Id: <1643425417-215270-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable virtio blk sw live migration relay callfd and log the dirty page. In this version we ignore the write cmd and still mark it dirty. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 4 +- drivers/vdpa/ifc/base/ifcvf.h | 6 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 128 +++++++++++++++++++++++++++++++++++------- 3 files changed, 116 insertions(+), 22 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 721cb1d..3a69e53 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -189,7 +189,7 @@ IFCVF_WRITE_REG32(val >> 32, hi); } -STATIC int +int ifcvf_hw_enable(struct ifcvf_hw *hw) { struct ifcvf_pci_common_cfg *cfg; @@ -238,7 +238,7 @@ return 0; } -STATIC void +void ifcvf_hw_disable(struct ifcvf_hw *hw) { u32 i; diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 769c603..6dd7925 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -179,4 +179,10 @@ struct ifcvf_hw { u64 ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid); +int +ifcvf_hw_enable(struct ifcvf_hw *hw); + +void +ifcvf_hw_disable(struct ifcvf_hw *hw); + #endif /* _IFCVF_H_ */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 4f99bb3..a930825 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -332,10 +332,67 @@ struct rte_vdpa_dev_info { rte_vhost_get_negotiated_features(vid, &features); if (RTE_VHOST_NEED_LOG(features)) { - ifcvf_disable_logging(hw); - rte_vhost_get_log_base(internal->vid, &log_base, &log_size); - rte_vfio_container_dma_unmap(internal->vfio_container_fd, - log_base, IFCVF_LOG_BASE, log_size); + if (internal->device_type == IFCVF_NET) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, + &log_size); + rte_vfio_container_dma_unmap( + internal->vfio_container_fd, log_base, + IFCVF_LOG_BASE, log_size); + } + /* IFCVF marks dirty memory pages for only packet buffer, + * SW helps to mark the used ring as dirty after device stops. + */ + for (i = 0; i < hw->nr_vring; i++) { + len = IFCVF_USED_RING_LEN(hw->vring[i].size); + rte_vhost_log_used_vring(vid, i, 0, len); + } + } +} + +static void +vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) +{ + struct ifcvf_hw *hw = &internal->hw; + struct rte_vhost_vring vq; + int i, vid; + uint64_t features = 0; + uint64_t log_base = 0, log_size = 0; + uint64_t len; + + vid = internal->vid; + + if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vq); + while (vq.avail->idx != vq.used->idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + hw->vring[i].last_avail_idx = vq.avail->idx; + hw->vring[i].last_used_idx = vq.used->idx; + } + } + + ifcvf_hw_disable(hw); + + for (i = 0; i < hw->nr_vring; i++) + rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, + hw->vring[i].last_used_idx); + + if (internal->sw_lm) + return; + + rte_vhost_get_negotiated_features(vid, &features); + if (RTE_VHOST_NEED_LOG(features)) { + if (internal->device_type == IFCVF_NET) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, + &log_size); + rte_vfio_container_dma_unmap( + internal->vfio_container_fd, log_base, + IFCVF_LOG_BASE, log_size); + } /* * IFCVF marks dirty memory pages for only packet buffer, * SW helps to mark the used ring as dirty after device stops. @@ -661,15 +718,17 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /* NETWORK: Direct I/O for Tx queue, relay for Rx queue + * BLK: relay every queue + */ + if ((i & 1) && (internal->device_type == IFCVF_NET)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); return -1; } hw->vring[i].used = gpa; - } else { + } else if (internal->device_type == IFCVF_BLK) { hw->vring[i].used = m_vring_iova + (char *)internal->m_vring[i].used - (char *)internal->m_vring[i].desc; @@ -688,7 +747,10 @@ struct rte_vdpa_dev_info { } hw->nr_vring = nr_vring; - return ifcvf_start_hw(&internal->hw); + if (internal->device_type == IFCVF_NET) + return ifcvf_start_hw(&internal->hw); + else if (internal->device_type == IFCVF_BLK) + return ifcvf_hw_enable(&internal->hw); error: for (i = 0; i < nr_vring; i++) @@ -713,8 +775,10 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (((i & 1) == 0 && internal->device_type == IFCVF_NET) || + internal->device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -726,6 +790,8 @@ struct rte_vdpa_dev_info { (uint64_t)(uintptr_t)internal->m_vring[i].desc, m_vring_iova, size); + hw->vring[i].last_avail_idx = vq.used->idx; + hw->vring[i].last_used_idx = vq.used->idx; rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, hw->vring[i].last_used_idx); rte_free(internal->m_vring[i].desc); @@ -776,17 +842,36 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { - ev.events = EPOLLIN | EPOLLPRI; - /* leave a flag to mark it's for interrupt */ - ev.data.u64 = 1 | qid << 1 | - (uint64_t)internal->intr_fd[qid] << 32; - if (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev) - < 0) { - DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); - return NULL; + if (internal->device_type == IFCVF_NET) { + for (qid = 0; qid < q_num; qid += 2) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); + } + } else if (internal->device_type == IFCVF_BLK) { + for (qid = 0; qid < q_num; qid += 1) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); } - update_used_ring(internal, qid); } /* start relay with a first kick */ @@ -874,7 +959,10 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_NET) + vdpa_ifcvf_stop(internal); + else if (internal->device_type == IFCVF_BLK) + vdpa_ifcvf_blk_pause(internal); vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Sat Jan 29 03:03:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106688 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 011ECA00C4; Sat, 29 Jan 2022 04:10:06 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EC3F7427C4; Sat, 29 Jan 2022 04:09:54 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id F35C64279D for ; Sat, 29 Jan 2022 04:09:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425790; x=1674961790; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=sl1F7F+q6m/30C0lTa30qO79tfeAXta7cCMAWACG5rM=; b=SkIIY7rdEt0Rgb6KNUzLHoamvYzZXwCQRZhO+Jh/aovWI5xqBLO8wco9 XsR4VTID4dne5OhFGCo1kuY4mZKJIedAXfI27wG/QhT09c+h/eOUbfxEq d3u5Cl3lHF5CiQlfVJc3IXUlq2Ypiqc/AedV2lzPLyqCQPSP1gwW+n0xC WA+3cVqo1qd5s4y3fkgk3tpXAf9MyRlq+8tvM/Fx3J1xEruCdKeehF7MY 4SZEbEQ3EwkiW4OkZzyWg6nemiyhDPhjB0z5li5+2hmCAEQfyVGzNhW2z jWdcTwMr70RH2AowIIwZQ1SVnq1QNxHZTBaZ39S0B6ioyONGcFs64RfkJ w==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076620" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076620" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047812" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:47 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 06/15] example/vdpa:add vdpa blk support in example Date: Sat, 29 Jan 2022 11:03:28 +0800 Message-Id: <1643425417-215270-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to vdpa example. Signed-off-by: Andy Pei --- examples/vdpa/Makefile | 2 +- examples/vdpa/main.c | 8 ++ examples/vdpa/meson.build | 1 + examples/vdpa/vdpa_blk_compact.c | 150 +++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 117 ++++++++++++++++++++++++ examples/vdpa/vhost_user.h | 189 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 466 insertions(+), 1 deletion(-) create mode 100644 examples/vdpa/vdpa_blk_compact.c create mode 100644 examples/vdpa/vdpa_blk_compact.h create mode 100644 examples/vdpa/vhost_user.h diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile index d974db4..9d0479b 100644 --- a/examples/vdpa/Makefile +++ b/examples/vdpa/Makefile @@ -5,7 +5,7 @@ APP = vdpa # all source are stored in SRCS-y -SRCS-y := main.c +SRCS-y := main.c vdpa_blk_compact.c CFLAGS += -DALLOW_EXPERIMENTAL_API PKGCONF ?= pkg-config diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..924ad7b 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -156,6 +157,7 @@ struct vdpa_port { static const struct rte_vhost_device_ops vdpa_sample_devops = { .new_device = new_device, .destroy_device = destroy_device, + .new_connection = rte_vhost_blk_session_install_rte_compat_hooks, }; static int @@ -192,6 +194,12 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + if (vdpa_blk_device_set_features_and_protocol(socket_path, vport->dev) + < 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build index bd08605..f0d111c 100644 --- a/examples/vdpa/meson.build +++ b/examples/vdpa/meson.build @@ -15,4 +15,5 @@ deps += 'vhost' allow_experimental_apis = true sources = files( 'main.c', + 'vdpa_blk_compact.c', ) diff --git a/examples/vdpa/vdpa_blk_compact.c b/examples/vdpa/vdpa_blk_compact.c new file mode 100644 index 0000000..0c4d3ee --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.c @@ -0,0 +1,150 @@ +/* INTEL CONFIDENTIAL + * + * Copyright (c) Intel Corporation. + * All rights reserved. + * + * The source code contained or described herein and all documents related + * to the source code ("Material") are owned by Intel Corporation or its + * suppliers or licensors. Title to the Material remains with Intel + * Corporation or its suppliers and licensors. The Material contains trade + * secrets and proprietary and confidential information of Intel or its + * suppliers and licensors. The Material is protected by worldwide + * copyright and trade secret laws and treaty provisions. No part of the + * Material may be used, copied, reproduced, modified, published, uploaded, + * posted, transmitted, distributed, or disclosed in any way without Intel's + * prior express written permission. + * + * No license under any patent, copyright, trade secret or other + * intellectual property right is granted to or conferred upon you by + * disclosure or delivery of the Materials, either expressly, by + * implication, inducement, estoppel or otherwise. Any license under such + * intellectual property rights must be express and approved by Intel in + * writing. + */ + +/* @file + * + * Block device specific vhost lib + */ + +#include + +#include +#include +#include +#include "vdpa_blk_compact.h" +#include "vhost_user.h" + +#define VHOST_USER_GET_CONFIG 24 +#define VHOST_USER_SET_CONFIG 25 + +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + +/* + * Function to handle vhost user blk message + */ +static enum rte_vhost_msg_result +rte_vhost_blk_extern_vhost_pre_msg_handler(int vid, void *_msg) +{ + struct VhostUserMsg *msg = _msg; + struct rte_vdpa_device *vdev = NULL; + + vdev = rte_vhost_get_vdpa_device(vid); + if (vdev == NULL) + return RTE_VHOST_MSG_RESULT_ERR; + + fprintf(stderr, "msg is %d\n", msg->request.master); + switch (msg->request.master) { + case VHOST_USER_GET_CONFIG: { + int rc = 0; + + fprintf(stdout, "read message VHOST_USER_GET_CONFIG\n"); + + if (vdev->ops->get_config) { + fprintf(stdout, "get_config() function is valid!\n"); + rc = vdev->ops->get_config(vid, + msg->payload.cfg.region, + msg->payload.cfg.size); + if (rc != 0) { + msg->size = 0; + fprintf(stdout, "get_config() return error!\n"); + } + } else { + fprintf(stdout, "get_config() function is invalid!\n"); + } + + return RTE_VHOST_MSG_RESULT_REPLY; + } + case VHOST_USER_SET_CONFIG: { + int rc = 0; + + fprintf(stdout, + "read message VHOST_USER_SET_CONFIG\n"); + + if (vdev->ops->set_config) { + rc = vdev->ops->set_config(vid, + msg->payload.cfg.region, + msg->payload.cfg.offset, + msg->payload.cfg.size, + msg->payload.cfg.flags); + } + + return rc == 0 ? RTE_VHOST_MSG_RESULT_OK : RTE_VHOST_MSG_RESULT_ERR; + } + default: + break; + } + + return RTE_VHOST_MSG_RESULT_NOT_HANDLED; +} + +struct rte_vhost_user_extern_ops g_blk_extern_vhost_ops = { + .pre_msg_handle = rte_vhost_blk_extern_vhost_pre_msg_handler, + .post_msg_handle = NULL, +}; + +int +rte_vhost_blk_session_install_rte_compat_hooks(int vid) +{ + int rc; + + rc = rte_vhost_extern_callback_register(vid, + &g_blk_extern_vhost_ops, + NULL); + if (rc != 0) { + fprintf(stderr, "%s() failed for vid = %d\n", __func__, vid); + return -1; + } + fprintf(stdout, "register extern vhost ops on vid = %d\n", vid); + return 0; +} + + +int +vdpa_blk_device_set_features_and_protocol(const char *path, + struct rte_vdpa_device *vdev) +{ + uint64_t protocol_features = 0; + + if (!vdev) { + fprintf(stdout, "vdev is NULL.\n"); + return -EINVAL; + } + + /* vdpa net does not have the get_config */ + if (!vdev->ops->get_config) + return 0; + + rte_vhost_driver_set_features(path, SPDK_VHOST_BLK_FEATURES_BASE); + rte_vhost_driver_disable_features(path, + SPDK_VHOST_BLK_DISABLED_FEATURES); + + rte_vhost_driver_get_protocol_features(path, &protocol_features); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_CONFIG); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD); + rte_vhost_driver_set_protocol_features(path, protocol_features); + + return 0; +} diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..420d48e --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,117 @@ +/* INTEL CONFIDENTIAL + * + * Copyright (c) Intel Corporation. + * All rights reserved. + * + * The source code contained or described herein and all documents related + * to the source code ("Material") are owned by Intel Corporation or its + * suppliers or licensors. Title to the Material remains with Intel + * Corporation or its suppliers and licensors. The Material contains trade + * secrets and proprietary and confidential information of Intel or its + * suppliers and licensors. The Material is protected by worldwide + * copyright and trade secret laws and treaty provisions. No part of the + * Material may be used, copied, reproduced, modified, published, uploaded, + * posted, transmitted, distributed, or disclosed in any way without Intel's + * prior express written permission. + * + * No license under any patent, copyright, trade secret or other + * intellectual property right is granted to or conferred upon you by + * disclosure or delivery of the Materials, either expressly, by + * implication, inducement, estoppel or otherwise. Any license under such + * intellectual property rights must be express and approved by Intel in + * writing. + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** + * @file + * + * Device specific vhost lib + */ + +#include + +#include +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_RO 5 /* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ +#define VIRTIO_BLK_F_DISCARD 13 /* DISCARD is supported */ +#define VIRTIO_BLK_F_WRITE_ZEROES 14 /* WRITE ZEROES is supported */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_FLUSH 9 /* Flush command supported */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ + +/* Old (deprecated) name for VIRTIO_BLK_F_FLUSH. */ +#define VIRTIO_BLK_F_WCE VIRTIO_BLK_F_FLUSH +#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#endif + +#define SPDK_VHOST_FEATURES ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC)) + +#define SPDK_VHOST_DISABLED_FEATURES ((1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY)) + +#define SPDK_VHOST_BLK_FEATURES_BASE (SPDK_VHOST_FEATURES | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define SPDK_VHOST_BLK_DISABLED_FEATURES (SPDK_VHOST_DISABLED_FEATURES | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_BARRIER) | (1ULL << VIRTIO_BLK_F_SCSI)) + +/* Vhost-blk support protocol features */ +#define SPDK_VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_CONFIG) | \ + (1ULL << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * This function will set vhost user block + * + * @param path + * socket path + */ +int +vdpa_blk_device_set_features_and_protocol(const char *path, + struct rte_vdpa_device *vdev); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Install external hook to handle vhost user block message + * + * @param vid + * vhost device id + */ +int +rte_vhost_blk_session_install_rte_compat_hooks(int vid); + +#endif /* _VDPA_BLK_COMPACT_H_ */ diff --git a/examples/vdpa/vhost_user.h b/examples/vdpa/vhost_user.h new file mode 100644 index 0000000..8b747d0 --- /dev/null +++ b/examples/vdpa/vhost_user.h @@ -0,0 +1,189 @@ +/* INTEL CONFIDENTIAL + * + * Copyright (c) Intel Corporation. + * All rights reserved. + * + * The source code contained or described herein and all documents related + * to the source code ("Material") are owned by Intel Corporation or its + * suppliers or licensors. Title to the Material remains with Intel + * Corporation or its suppliers and licensors. The Material contains trade + * secrets and proprietary and confidential information of Intel or its + * suppliers and licensors. The Material is protected by worldwide + * copyright and trade secret laws and treaty provisions. No part of the + * Material may be used, copied, reproduced, modified, published, uploaded, + * posted, transmitted, distributed, or disclosed in any way without Intel's + * prior express written permission. + * + * No license under any patent, copyright, trade secret or other + * intellectual property right is granted to or conferred upon you by + * disclosure or delivery of the Materials, either expressly, by + * implication, inducement, estoppel or otherwise. Any license under such + * intellectual property rights must be express and approved by Intel in + * writing. + */ + +#ifndef _VHOST_NET_USER_H +#define _VHOST_NET_USER_H + +#include +#include + +#include "rte_vhost.h" + +/* refer to hw/virtio/vhost-user.c */ + +#define VHOST_MEMORY_MAX_NREGIONS 8 + +#ifndef VHOST_USER_MAX_CONFIG_SIZE +#define VHOST_USER_MAX_CONFIG_SIZE 256 +#endif + +#define VHOST_USER_PROTOCOL_FEATURES ((1ULL << VHOST_USER_PROTOCOL_F_MQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) |\ + (1ULL << VHOST_USER_PROTOCOL_F_RARP) | \ + (1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \ + (1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CRYPTO_SESSION) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER) | \ + (1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT)) + +typedef enum VhostUserRequest { + VHOST_USER_NONE = 0, + VHOST_USER_GET_FEATURES = 1, + VHOST_USER_SET_FEATURES = 2, + VHOST_USER_SET_OWNER = 3, + VHOST_USER_RESET_OWNER = 4, + VHOST_USER_SET_MEM_TABLE = 5, + VHOST_USER_SET_LOG_BASE = 6, + VHOST_USER_SET_LOG_FD = 7, + VHOST_USER_SET_VRING_NUM = 8, + VHOST_USER_SET_VRING_ADDR = 9, + VHOST_USER_SET_VRING_BASE = 10, + VHOST_USER_GET_VRING_BASE = 11, + VHOST_USER_SET_VRING_KICK = 12, + VHOST_USER_SET_VRING_CALL = 13, + VHOST_USER_SET_VRING_ERR = 14, + VHOST_USER_GET_PROTOCOL_FEATURES = 15, + VHOST_USER_SET_PROTOCOL_FEATURES = 16, + VHOST_USER_GET_QUEUE_NUM = 17, + VHOST_USER_SET_VRING_ENABLE = 18, + VHOST_USER_SEND_RARP = 19, + VHOST_USER_NET_SET_MTU = 20, + VHOST_USER_SET_SLAVE_REQ_FD = 21, + VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_CRYPTO_CREATE_SESS = 26, + VHOST_USER_CRYPTO_CLOSE_SESS = 27, + VHOST_USER_POSTCOPY_ADVISE = 28, + VHOST_USER_POSTCOPY_LISTEN = 29, + VHOST_USER_POSTCOPY_END = 30, + VHOST_USER_GET_INFLIGHT_FD = 31, + VHOST_USER_SET_INFLIGHT_FD = 32, + VHOST_USER_MAX = 33 +} VhostUserRequest; + +typedef enum VhostUserSlaveRequest { + VHOST_USER_SLAVE_NONE = 0, + VHOST_USER_SLAVE_IOTLB_MSG = 1, + VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG = 3, + VHOST_USER_SLAVE_MAX +} VhostUserSlaveRequest; + +typedef struct VhostUserMemoryRegion { + uint64_t guest_phys_addr; + uint64_t memory_size; + uint64_t userspace_addr; + uint64_t mmap_offset; +} VhostUserMemoryRegion; + +typedef struct VhostUserMemory { + uint32_t nregions; + uint32_t padding; + VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS]; +} VhostUserMemory; + +typedef struct VhostUserLog { + uint64_t mmap_size; + uint64_t mmap_offset; +} VhostUserLog; + +/* Comply with Cryptodev-Linux */ +#define VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH 512 +#define VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH 64 + +/* Same structure as vhost-user backend session info */ +typedef struct VhostUserCryptoSessionParam { + int64_t session_id; + uint32_t op_code; + uint32_t cipher_algo; + uint32_t cipher_key_len; + uint32_t hash_algo; + uint32_t digest_len; + uint32_t auth_key_len; + uint32_t aad_len; + uint8_t op_type; + uint8_t dir; + uint8_t hash_mode; + uint8_t chaining_dir; + uint8_t *ciphe_key; + uint8_t *auth_key; + uint8_t cipher_key_buf[VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH]; + uint8_t auth_key_buf[VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH]; +} VhostUserCryptoSessionParam; + +typedef struct VhostUserVringArea { + uint64_t u64; + uint64_t size; + uint64_t offset; +} VhostUserVringArea; + +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint16_t num_queues; + uint16_t queue_size; +} VhostUserInflight; + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + +typedef struct VhostUserMsg { + union { + uint32_t master; /* a VhostUserRequest value */ + uint32_t slave; /* a VhostUserSlaveRequest value*/ + } request; + +#define VHOST_USER_VERSION_MASK 0x3 +#define VHOST_USER_REPLY_MASK (0x1 << 2) +#define VHOST_USER_NEED_REPLY (0x1 << 3) + uint32_t flags; + uint32_t size; /* the following payload size */ + union { +#define VHOST_USER_VRING_IDX_MASK 0xff +#define VHOST_USER_VRING_NOFD_MASK (0x1<<8) + uint64_t u64; + struct vhost_vring_state state; + struct vhost_vring_addr addr; + VhostUserMemory memory; + VhostUserLog log; + struct vhost_iotlb_msg iotlb; + VhostUserCryptoSessionParam crypto_session; + VhostUserVringArea area; + VhostUserInflight inflight; + struct vhost_user_config cfg; + } payload; + int fds[VHOST_MEMORY_MAX_NREGIONS]; + int fd_num; +} __attribute((packed)) VhostUserMsg; + +#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) + +/* The version of the protocol we support */ +#define VHOST_USER_VERSION 0x1 +#endif From patchwork Sat Jan 29 03:03:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106689 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C7C64A00C4; Sat, 29 Jan 2022 04:10:13 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0145A42823; Sat, 29 Jan 2022 04:09:56 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 7BA49427B8 for ; Sat, 29 Jan 2022 04:09:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425792; x=1674961792; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Q4pWHhsiiMJon/PmG+i6zWiX4I7IA0g3rSbtXM7w4vE=; b=P9g9d6MRzYvf66m9ukOaheTgaHOB2zkkdrDepsHges4NBLvavmD1KMp9 wcAlwsRV2m3WhspGUyPLO3Bh8vk9f8rNrk5Qw1NNnOrwb7gWzJMjP5iZo fFORHFw8DN7IPmD5pgrwk5zoiH6808aN4utiPkmT402QldL9vBMrjJAWx PU55bQ9FLaNJahqdwPxFh3KGa+QLEFqof0QlTAQTJGuZKWe59lRw4cwFH /rPN6TmTSqDLMg0rgd79TIONDRAMRInyV7zlKEcBCw/MBEVOOtkQmaoV6 tgAaAIW/NfL5bz0bTmt1hK8YpDB7JDqRJ+020IOv5hkAVQivtRzMq8gHr Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076624" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076624" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047825" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:50 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 07/15] usertools: add support for virtio blk device Date: Sat, 29 Jan 2022 11:03:29 +0800 Message-Id: <1643425417-215270-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to devbind. Signed-off-by: Andy Pei --- usertools/dpdk-devbind.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..cbe336f 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -14,6 +14,8 @@ from os.path import join as path_join # The PCI base class for all devices +virtio_blk_class = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': '8086', 'SDevice': '0002'} network_class = {'Class': '02', 'Vendor': None, 'Device': None, 'SVendor': None, 'SDevice': None} acceleration_class = {'Class': '12', 'Vendor': None, 'Device': None, @@ -72,6 +74,7 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk_devices = [virtio_blk_class] network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -587,6 +590,9 @@ def show_status(): Displays to the user what devices are bound to the igb_uio driver, the kernel driver or to no driver''' + if status_dev in ["virtio_blk", "all"]: + show_device_status(virtio_blk_devices, "virtio_blk") + if status_dev in ["net", "all"]: show_device_status(network_devices, "Network", if_field=True) @@ -746,6 +752,7 @@ def do_arg_actions(): if b_flag is not None: clear_data() # refresh if we have changed anything + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) @@ -769,6 +776,7 @@ def main(): parse_args() check_modules() clear_data() + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) From patchwork Sat Jan 29 03:03:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106690 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 87C86A00C4; Sat, 29 Jan 2022 04:10:21 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 51E6A42853; Sat, 29 Jan 2022 04:09:57 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id D45B3427DC for ; Sat, 29 Jan 2022 04:09:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425796; x=1674961796; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bYxWLeAm+dqMzQdYLg5yoMMhddqGA0gRNZ00kQeIZH0=; b=VSSG7QZwGmQUKAaGEJJ3edyMLdRlS2J9XPVOnBQSEf/YIxwGTX2oLbWB RnOQhT4bqFwnnb8wZL8PwTixyzn2Nm+V4zfMiK+2yVMvbHz5PP811m/NR 5oa881Xi3AhOK3w7A1Adh2/HLIjsMpocTfGgmGHSQbEYwZxcIR3Tb+/cX 3dTN3WmHaftr7fHoxFH814ksXHYLpPLk/VauBDFES/IAVFoDI1Z339ptQ yOhXkgFHCSHpFJgMclQp5SZ0QkcFc6owhaO1pmk3jdPBngin0nF2qbT4D 5MqteF4awLHDVIK59DTtADn+aWLXbMlLXd5jQGyKNdF3BfQUTUDy5/yV/ Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076627" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076627" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047837" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:53 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 08/15] vdpa/ifc: set_vring_state op is mandatory, add set_vring_state for blk device Date: Sat, 29 Jan 2022 11:03:30 +0800 Message-Id: <1643425417-215270-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Set_vring_state op is mandatory, add set_vring_state for blk device. Currently set_vring_state for blk device is not implemented. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index a930825..24ae27b 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1373,6 +1373,16 @@ struct rte_vdpa_dev_info { } static int +ifcvf_blk_set_vring_state(int vid, int vring, int state) +{ + RTE_SET_USED(vid); + RTE_SET_USED(vring); + RTE_SET_USED(state); + + return 0; +} + +static int ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1390,7 +1400,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, From patchwork Sat Jan 29 03:03:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106691 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 78B43A00C4; Sat, 29 Jan 2022 04:10:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5AFAA42866; Sat, 29 Jan 2022 04:09:59 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 7C3E04285B for ; Sat, 29 Jan 2022 04:09:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425797; x=1674961797; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=+dW/ijgvYJMhOCfLVVcCwGFQXvWBTeUec6EcnHev+5I=; b=TOcLi7fxNETh0h6xeeIHjFuIuI2496EUf6pH0ibsZXk4UbF/rmxvv10L Q0CjWly6ue9deaOGJjvz9G6jPVRm1JS4pJrXVEpk0U9FmhKtdhoa3ijjE bNRSALAATqAhbvwy5WwM5pGKFrQpHCo9uWOigiWj5yiGaK7M1XGJHEtIX 5W0gCLU2tX2jCn4lw3NNUvmqGzoGMcYH+Ra+NOVrhlZZYu+f0uzne8X7A AYhAw+17ak9v5Jf1j2n/LEXK/KYAfbBKDl7h5uYCqCuSykdBxcgoR6ADN UlibZCSx+mwdYMKzT79XE0B5q4LahMfGr541lSj7RcnCYb4iNUWJvzQIB Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076631" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076631" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:09:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047848" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:55 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 09/15] vdpa/ifc: add some log at VDPA launch before qemu connect Date: Sat, 29 Jan 2022 11:03:31 +0800 Message-Id: <1643425417-215270-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add some log of virtio blk device config space information at VDPA launch before qemu connects. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 24ae27b..3c4e5f6 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1436,6 +1436,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + uint64_t capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1502,6 +1505,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /* cannot read 64-bit register in one attempt, + * so read byte by byte. + */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Sat Jan 29 03:03:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106692 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D2446A00C4; Sat, 29 Jan 2022 04:10:31 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4662642874; Sat, 29 Jan 2022 04:10:02 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id C17F142840 for ; Sat, 29 Jan 2022 04:10:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425800; x=1674961800; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=J0wMip7Z4h5J0il2JjVcXYRvHfh5llXFtrrz+9EOfG0=; b=Tjkrwl3HMYoiAiIGBZUkZTb8ev+mcN4MrdQ8L7QWTGhBqHnK1YGgkF9v RkrYPxP0QPkHLD8iXOcNQkGKTpBk9FK8fxwSpf1h4xW/OWowMm584JsBe HmJ8FHYZt0DCGFZ4rcm9t3E+xuaNlcOsq08KL8UUNBtRLOgkH0wKz7CYn z0p/5azrkViMrJp3IPKfmw7RNfImt4TeeIscx9/IeVSeBFuWzoNd7r6lj mPGKy3HCySV9n73nKhLTYpX8USZ7vhGE0rYHV4bUBSZgmUx7SkQYVBTB5 1b3YxLhtgtUeHZj1QHA8RNgbMuRgsOOqu3eTVfru74wq0GWrksIRggr3K w==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076638" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076638" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:10:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047869" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:09:59 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 10/15] vdpa/ifc: read virtio max_queues from hardware Date: Sat, 29 Jan 2022 11:03:32 +0800 Message-Id: <1643425417-215270-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Original code max_queues is set to IFCVF_MAX_QUEUES. New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3c4e5f6..86dd1c6 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1530,6 +1530,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(INFO, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + /* reset max_queue here, to minimum modification */ + internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Sat Jan 29 03:03:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106693 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1A574A00C4; Sat, 29 Jan 2022 04:10:37 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4D066427EC; Sat, 29 Jan 2022 04:10:05 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 887BC4287B for ; Sat, 29 Jan 2022 04:10:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425803; x=1674961803; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=i0R5Ub+qHnRWnY4rzHrKoJ24ilbJCUthpjNuYsiJHm0=; b=cxg2tKswt1ZHAa8WOkJcmDJwSXxBevmKVX8Y5vjxWJjV4j8biPmqe0/B iwzQBBtk+48D01FyyGHIcPqLFdXyBc95Y0PUu/Eu8ycMcvSD+bqzAPX3u YDN6SjGumoMfjnyxZZ/ahyDVqpuRYstk3mKbPwVq5uJ7gP4TgM0Ut5sqX atE7wbDpgqGRHeSGbAZxM90nED5anTlN1r9Om/QJZC6e+/vIIwMcHudcg 3oarWHJqrYsYKTbTOgCJviTH7tKTLZQdBei8XqeJoLTkJ3qKHzBM4gKFV SWjXDXHkWeIEOj/ecqJDwxrUrQyXau2uCRqmagj7vz9oiaI21d3tLo7uj g==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076643" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076643" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:10:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047891" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:10:01 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 11/15] vdpa: add config space change interrupt register and handle for virtio_blk Date: Sat, 29 Jan 2022 11:03:33 +0800 Message-Id: <1643425417-215270-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 111 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 86dd1c6..37fa45e 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for intr relay */ int epfd; + int csc_fd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -617,6 +619,106 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_fd, csc_val = 0; + + csc_fd = epoll_create(1); + if (csc_fd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_fd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + return NULL; + } + + internal->csc_fd = csc_fd; + + for (;;) { + csc_val = epoll_wait(csc_fd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail\n"); + return NULL; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || errno == EWOULDBLOCK) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + return NULL; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + int ret; + + ret = pthread_create(&internal->intr_tid, NULL, intr_relay, + (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static int +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_fd >= 0) + close(internal->csc_fd); + internal->csc_fd = -1; + + return 0; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -643,10 +745,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + ret = unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -963,6 +1071,9 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(internal); else if (internal->device_type == IFCVF_BLK) vdpa_ifcvf_blk_pause(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Sat Jan 29 03:03:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106694 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1738AA00C4; Sat, 29 Jan 2022 04:10:42 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3EC784287C; Sat, 29 Jan 2022 04:10:07 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 757664287C for ; Sat, 29 Jan 2022 04:10:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425805; x=1674961805; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=R+qNdOwaAl9rT7qqmMsCSuCALqb077qFtF0bAkVlzu8=; b=i3cbV07S7l1zIsWrvyR9xxW8jWMvsHEsoZBDGl059Bl+842EK71YOXVq JUMzodtiXp3tI8hQIj7Dc/fH2sdVOZCZuens0NTg57Rw4g70P1BPqEF3l 0RWmp2SzF4qzjfSTXMLJip95dGeRBnUoWV5dUqrcirmFo1doQ/DuHYOv6 MynauWgzIInQ4ENA+ZDmvRArUpxVgiP+I6Rb8er6QqE3dxOPWQPMqS8vQ CdHqvYHDQ4TRZ6rseBrdmQ/QnrviiK2A/9+D5wYqM6x34MCKfBzmhap4a 2yIACLrM15eprGmP+yyYIrRhwx8KCVGld7vQAbpOSNvFxuWlokFPqqad4 w==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076647" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076647" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:10:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047924" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:10:03 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 12/15] vdpa/ifc: add is_blk flag to ifcvf_hw, and init is_blk during probe Date: Sat, 29 Jan 2022 11:03:34 +0800 Message-Id: <1643425417-215270-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add is_blk flag to ifcvf_hw, and init is_blk during probe. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 6dd7925..8e602af 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -149,6 +149,7 @@ struct ifcvf_hw { u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + u8 is_blk; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 37fa45e..b65e3a3 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1608,11 +1608,13 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->device_type = IFCVF_NET; + internal->hw.is_blk = IFCVF_NET; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; } else if (device_id == VIRTIO_ID_BLOCK) { internal->device_type = IFCVF_BLK; + internal->hw.is_blk = IFCVF_BLK; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; From patchwork Sat Jan 29 03:03:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106695 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 708AFA00C4; Sat, 29 Jan 2022 04:10:46 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 458694288A; Sat, 29 Jan 2022 04:10:10 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 1D9F44288A for ; Sat, 29 Jan 2022 04:10:07 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425808; x=1674961808; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=OfMQKUusCbeZdl+8sJdrAlffNHrBeBS10u16yx50v50=; b=kIrQzHH5yO+4EsKOkzjaJZGxE9F1hVulKPXiuIvfRp4RK1RFpPiyeZna Te6cqdhTyK0xp8qo2AGDxgaWYDnrhJPmL+ww7Gm9NdjyOBVEIr5JNzprk fj7aCD00m2PjR76TaMoC/Mc5MvoP4IvQPf9touyFThGCLC5mC5vqdsAgv w5T+rz55WcEaCjO+8kd2OPbKKLWzw7OBLrgEtS8p38kbRI3xd76uKOlvz zy0pjPdwR5H1Dsde9nL/AEeZarRJ26PRjPSnJkIEyADJ5cMv1+at2n+Og l6BFCACGvcNIpz0x+tIypPR1T7bMdeg+bKB6hXJv+cl8jOit+QaETujlJ g==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076650" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076650" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:10:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047964" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:10:06 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 13/15] vdpa/ifc/base: for blk device, live migration register is different from net device Date: Sat, 29 Jan 2022 11:03:35 +0800 Message-Id: <1643425417-215270-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 1.last_avail_idx is lower 16 bit of the register. 2.address of ring_state register is different between net and blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 3a69e53..a8a4728 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -216,10 +216,18 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->is_blk == IFCVF_BLK) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } else if (hw->is_blk == IFCVF_NET) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -252,9 +260,23 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + } else if (hw->is_blk == IFCVF_NET) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + } + + if (hw->is_blk == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else if (hw->is_blk == IFCVF_NET) + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } } diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 8e602af..7367094 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 From patchwork Sat Jan 29 03:03:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106696 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DD0FEA00C4; Sat, 29 Jan 2022 04:10:50 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 657374289D; Sat, 29 Jan 2022 04:10:12 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id A406D40143 for ; Sat, 29 Jan 2022 04:10:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425810; x=1674961810; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=5087tad1YGUkw8YHGaOPminXBtE8KkMaP2tTS2ecQgQ=; b=danAoAhTH4Jx3NTUmafAWL/AX+5MACHrHnVL8Tos/pL4tBPsHmyy42b7 DeCm0z7VNdGZcphsv/GaJp62ixkET8Jl6F/1frTWnotQwIRWwhHjuaqQz KctCizeP+8BkTE6oVCol2IwPtjPA4H4948x+ueMXzE4wUOeOQ+183O7RT im6kb8GfU6WwAlqFOJkXUjXDl/uKyvPgDJWyTsUG9mm36STs9FOZomfgJ TWtf+ii/k3c0lgE9f7ccxN9GKGcogu9gtwlvm4pzo05Q8OPJE+eVs03ed 6nQ2VblHU+MkHLJiHRj4ZQF3yZ1UnitQfNQX4eFRrePJiud4UuTkC0jDO w==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076652" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076652" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:10:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047980" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:10:08 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 14/15] vdpa/ifc: make sure hardware last_avail_idx and last_used_idx are the same when blk device pause Date: Sat, 29 Jan 2022 11:03:36 +0800 Message-Id: <1643425417-215270-15-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 2 +- drivers/vdpa/ifc/base/ifcvf.h | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 32 +++++++++++++++++++++++--------- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index a8a4728..7018048 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -116,7 +116,7 @@ IFCVF_WRITE_REG8(status, &hw->common_cfg->device_status); } -STATIC void +void ifcvf_reset(struct ifcvf_hw *hw) { ifcvf_set_status(hw, 0); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 7367094..f22d18b 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -157,6 +157,9 @@ struct ifcvf_hw { int ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev); +void +ifcvf_reset(struct ifcvf_hw *hw); + u64 ifcvf_get_features(struct ifcvf_hw *hw); diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index b65e3a3..75dbe63 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -356,23 +356,32 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) { struct ifcvf_hw *hw = &internal->hw; - struct rte_vhost_vring vq; int i, vid; uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; if (internal->device_type == IFCVF_BLK) { for (i = 0; i < hw->nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vq); - while (vq.avail->idx != vq.used->idx) { - ifcvf_notify_queue(hw, i); - usleep(10); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } @@ -759,7 +768,12 @@ struct rte_vdpa_dev_info { if (ret) goto err; - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_BLK) { + vdpa_ifcvf_blk_pause(internal); + ifcvf_reset(&internal->hw); + } else { + vdpa_ifcvf_stop(internal); + } ret = vdpa_disable_vfio_intr(internal); if (ret) From patchwork Sat Jan 29 03:03:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 106697 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07255A00C4; Sat, 29 Jan 2022 04:10:57 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C4094428B6; Sat, 29 Jan 2022 04:10:14 +0100 (CET) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 15368428A6 for ; Sat, 29 Jan 2022 04:10:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643425813; x=1674961813; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=CYVxCE6tK347SJO9ema6bsNPo2bc/mSy5Uqe9fuByJc=; b=Hp6qmIfhL2I66LGUv9Bo0HOKo3y4N2efujT74NDDpJEXzMSVj9Rd1++u bZcFP7UNbYPPImVJR4Dw+YXRNMBwB0RqmC73p3Z4geS0E+AUaeTCE9XGi 8HDua3e1iHytg5Deq+0kOLeLM9WhgmcNh2W7AnXi5pdfZX656E8L2vJGD PpY8epndRZ36VogOC2/dZ3rynQmBxDxJHHRPNihQtZ0O6Ngm6uur5FvLy j75Y/GgKfSAEcmC3wxdquhAOeS+KvKXfbpskVQX+a9HywAx3lqk7BiYQ7 rNe20cZTweK1cA0uAsOQbSWWRKBk+yrxLxIb9lOdUaCe10n/36uYSUrFQ A==; X-IronPort-AV: E=McAfee;i="6200,9189,10241"; a="246076653" X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="246076653" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2022 19:10:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,325,1635231600"; d="scan'208";a="582047994" Received: from dpdk-dipei.sh.intel.com ([10.67.111.91]) by fmsmga008.fm.intel.com with ESMTP; 28 Jan 2022 19:10:11 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v3 15/15] vhost: make sure each queue callfd is configured Date: Sat, 29 Jan 2022 11:03:37 +0800 Message-Id: <1643425417-215270-16-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1643425417-215270-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1643425417-215270-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 5eb1dd6..b25b25f 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3137,12 +3137,26 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, if (!vdpa_dev) goto out; + if (request != VHOST_USER_SET_VRING_CALL) + goto out; + if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid)) VHOST_LOG_CONFIG(ERR, "Failed to configure vDPA device\n"); else dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED; + } else { + /* when VIRTIO_DEV_VDPA_CONFIGURED already configured + * close the device and config the device again, + * make sure the call fd of each queue is configured correctly. + */ + if (vdpa_dev->ops->dev_close(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to close vDPA device\n"); + if (vdpa_dev->ops->dev_conf(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to re-config vDPA device\n"); } out: