From patchwork Sun Mar 27 14:51:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108881 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A89DA034C; Sun, 27 Mar 2022 17:35:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1C24B42863; Sun, 27 Mar 2022 17:35:49 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 7400340151 for ; Sun, 27 Mar 2022 17:35:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395347; x=1679931347; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=zWAV2IbjItaFpvx6M55IDxoj7pCQRBoeuR8ntMToDes=; b=JaecjbeHPAonldUz0AaJPlTu5vq7JMu+gaYRuNXc0LmkcW0bOcbFKy2T 2jMnZfCe8tcoS91bnnB0UIj48f3HWiW/sU/BPqZawq1trFW6YTG4B6bwL 2KO7j6KvJOJgz4j3BM+IUSw6ERvtD5bRMP0L57ZQ7aWzZJztie9ysCJl5 U+1+v4DgLBGLD1o3Tz3mS8/Kme4wKAiRYehOpk3PocK9i7S1OtFgzW3kq rL9k3Ssd1ETOoCrGrfrRoO7cjdzUHYVC1pD+vhRefKmDwKyyLNW+x0gh5 H42FS+fU4GB0MtlW4grZ9zlSYQWav0DV7uNG3EHfPFdZkExezLaucxJJo A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045629" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045629" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421458" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:45 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 01/16] vdpa/ifc: add support for virtio blk device Date: Sun, 27 Mar 2022 22:51:24 +0800 Message-Id: <1648392699-60113-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 92 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..01522c6 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,8 +5,17 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 + #define IFCVF_VENDOR_ID 0x1AF4 #define IFCVF_DEVICE_ID 0x1041 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 @@ -57,7 +66,6 @@ #define IFCVF_32_BIT_MASK 0xffffffff - struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ @@ -126,7 +134,11 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9f05595..e3210a8 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -58,6 +58,7 @@ struct ifcvf_internal { struct rte_vdpa_device *vdev; uint16_t max_queues; uint64_t features; + int device_type; rte_atomic32_t started; rte_atomic32_t dev_attached; rte_atomic32_t running; @@ -75,6 +76,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1167,6 +1174,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1178,6 +1227,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1227,13 +1277,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1245,7 +1306,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1313,6 +1375,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Sun Mar 27 14:51:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108882 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8B042A034C; Sun, 27 Mar 2022 17:36:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E03E742868; Sun, 27 Mar 2022 17:35:50 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 22E2142865 for ; Sun, 27 Mar 2022 17:35:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395349; x=1679931349; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=dU1SiGIIQ4Zx0f9IfY0tRznDSHQsUgrPqE47DlycL08=; b=Ex70CfaCgvsq5sP1ea+oLTdM6MKmmryKj3zgu4A7C6NRbWoDlMlXz8XL wQyFw/ehp13OyrEum1VTWrgD5x9ZC3e3ZXGxnPngODsXXyb+YJSOF1XSx MMe/VzCfAPg+yOEU32RrOVjDAus0cVk4z5uEozXv3bmgcBeAZEDWeZ6dw 5vK3HS4Vc9Vd/qNEuKmCet9tpnq+8/HQyIRKG3qeMaEZtt2oCWvtfymzO TtqutbCzizo1EvkzEo83uJ6HZSnGaUWpAKh3dh2kppNzBMoBdFK0cxRxs CWSmeIVYLQ27icq8p8J4kM2PFMFyDMZ74vg3dX6pPr03r+B9tYDexBcP+ w==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045631" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045631" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421481" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:47 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 02/16] vhost: add vdpa ops for blk device Date: Sun, 27 Mar 2022 22:51:25 +0800 Message-Id: <1648392699-60113-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vdpa ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 88138be..e59a834 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t len); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Sun Mar 27 14:51:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108883 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 81622A034C; Sun, 27 Mar 2022 17:36:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BEEF24286E; Sun, 27 Mar 2022 17:35:52 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 1AF334286A for ; Sun, 27 Mar 2022 17:35:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395351; x=1679931351; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2LcymBiDZG8wq7nF89kd5xYYbE8C2LOh8O2oPjSKzng=; b=CaMaZDeCeP1sNHLMS13/gyU9ZNcKGBPhlDQI64W86gcUG/m9cC+GpKVW 3v4vxJKEUvKbjc7xNmaOv6gQ0wsqNGaKx0FqcPPLfpAKiNYjOMQRzxZuG xPoQxOzlPzmSG/8GmxG4zA7Ejr/D/vkO3XCIOzFG6PUH/ezDDv1JPgr3j 8qXF2zByO5dGXM5oOsLdVIWirJMiBK14+QEMDA75TP+kAaHooRz5ht/nU d1MZrojDj0UU4vZEyTsrTE1kntbvQqAWS6E1jSPGfz+GAKSFX2hwk7qqA cwrOkK+J8B/Mk8Sfy5ao+jNJrDA7M6L/UXsd3E8VQS/6FP1ritELrdkS2 g==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045635" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045635" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421500" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:48 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 03/16] vhost: add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG Date: Sun, 27 Mar 2022 22:51:26 +0800 Message-Id: <1648392699-60113-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 15 +++++++++++++++ 2 files changed, 65 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 1d39067..b11fafd 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -80,6 +80,8 @@ [VHOST_USER_NET_SET_MTU] = "VHOST_USER_NET_SET_MTU", [VHOST_USER_SET_SLAVE_REQ_FD] = "VHOST_USER_SET_SLAVE_REQ_FD", [VHOST_USER_IOTLB_MSG] = "VHOST_USER_IOTLB_MSG", + [VHOST_USER_GET_CONFIG] = "VHOST_USER_GET_CONFIG", + [VHOST_USER_SET_CONFIG] = "VHOST_USER_SET_CONFIG", [VHOST_USER_CRYPTO_CREATE_SESS] = "VHOST_USER_CRYPTO_CREATE_SESS", [VHOST_USER_CRYPTO_CLOSE_SESS] = "VHOST_USER_CRYPTO_CLOSE_SESS", [VHOST_USER_POSTCOPY_ADVISE] = "VHOST_USER_POSTCOPY_ADVISE", @@ -2542,6 +2544,52 @@ static int is_vring_iotlb(struct virtio_net *dev, } static int +vhost_user_get_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); + if (ret != 0) { + ctx->msg.size = 0; + VHOST_LOG_CONFIG(ERR, "get_config() return error!\n"); + } + } else { + VHOST_LOG_CONFIG(ERR, "get_config() not supportted!\n"); + } + + return RTE_VHOST_MSG_RESULT_REPLY; +} + +static int +vhost_user_set_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); + } else { + VHOST_LOG_CONFIG(ERR, "set_config() not supportted!\n"); + } + + return ret == 0 ? RTE_VHOST_MSG_RESULT_OK : RTE_VHOST_MSG_RESULT_ERR; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx, int main_fd __rte_unused) @@ -2782,6 +2830,8 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, [VHOST_USER_NET_SET_MTU] = vhost_user_net_set_mtu, [VHOST_USER_SET_SLAVE_REQ_FD] = vhost_user_set_req_fd, [VHOST_USER_IOTLB_MSG] = vhost_user_iotlb_msg, + [VHOST_USER_GET_CONFIG] = vhost_user_get_config, + [VHOST_USER_SET_CONFIG] = vhost_user_set_config, [VHOST_USER_POSTCOPY_ADVISE] = vhost_user_set_postcopy_advise, [VHOST_USER_POSTCOPY_LISTEN] = vhost_user_set_postcopy_listen, [VHOST_USER_POSTCOPY_END] = vhost_user_postcopy_end, diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index c946cc2..d3f014e 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -50,6 +50,8 @@ VHOST_USER_NET_SET_MTU = 20, VHOST_USER_SET_SLAVE_REQ_FD = 21, VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_GET_CONFIG = 24, + VHOST_USER_SET_CONFIG = 25, VHOST_USER_CRYPTO_CREATE_SESS = 26, VHOST_USER_CRYPTO_CLOSE_SESS = 27, VHOST_USER_POSTCOPY_ADVISE = 28, @@ -125,6 +127,18 @@ uint16_t queue_size; } VhostUserInflight; +#ifndef VHOST_USER_MAX_CONFIG_SIZE +#define VHOST_USER_MAX_CONFIG_SIZE 256 +#endif + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -148,6 +162,7 @@ VhostUserCryptoSessionParam crypto_session; VhostUserVringArea area; VhostUserInflight inflight; + struct vhost_user_config cfg; } payload; /* Nothing should be added after the payload */ } __rte_packed VhostUserMsg; From patchwork Sun Mar 27 14:51:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108884 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6FC2BA034C; Sun, 27 Mar 2022 17:36:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D88364287B; Sun, 27 Mar 2022 17:35:54 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 048D142871 for ; Sun, 27 Mar 2022 17:35:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395353; x=1679931353; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=/K8my91Bi3zBcNK1klWq65z4PAi11xo392zGrqZpMCM=; b=hQQzljwKXvwEGFy5QcKropAjL/7AdrO1ai8NMehvGfova/re9GjNPG11 YgqWL7WlZB0Pf5QppgYS2EcY+kAT6wB9wRSeWd7Z87SxUNaAS0HNl3fPd WeSZ9AyG7IFqfpVYCHqM3MUvrgTYgQMSlDzG8DedfFkQX8PI7t4z1OFgD qi5dLHsyGSfxn88mnqvSiyPZkk86P3IzF+SVAR5BHSzY7NGTAlKYuMaF5 M8c48qkC22tXF0eUaIKrUMG7iVXJ9BPHz3E0LRv4ge46NLtSffYw+IFY/ 0nW6zoVbhqLap/yRLzUu4lda2olTHdJGQUtquyUpWXF+E/eKLDG+usVRE A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045638" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045638" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421517" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:50 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 04/16] vdpa/ifc: add blk ops for ifc device Date: Sun, 27 Mar 2022 22:51:27 +0800 Message-Id: <1648392699-60113-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 85 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 88 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 01522c6..769c603 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -66,6 +66,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index e3210a8..8ee041f 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1088,6 +1088,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1200,6 +1204,85 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + uint64_t capacity = 0; + uint8_t *byte; + + if (len < sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %u", + len, (uint32_t)sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(INFO, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = NULL, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1212,7 +1295,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Sun Mar 27 14:51:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108885 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ECFF6A034C; Sun, 27 Mar 2022 17:36:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A64D74287E; Sun, 27 Mar 2022 17:35:55 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id AF18442878 for ; Sun, 27 Mar 2022 17:35:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395354; x=1679931354; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=35G+OEmAcn+Mf41lG6xR3Ct0ZEzvMmBr/cnDqzP82lo=; b=apkBIkB9WckefiQGUxx7VQUbxc+wT/XuxCck9Eo7QQqBY987Jv+QWv10 DEZXU1VnWavaxKbsBgTKpqIGDchAzEn1nF5IkFSKD/CE5kN0vDUumWVOX 44X0ys4R7woB4y+rUUxivnrGmRjwcEWZ6YulUrAfUrEXLpBnZWnfvIifP UHSEyLEmj91VT5Q5nfzYnAdL2CkRSnkUaEzel7FvQJ+qmS8FJBYrRMSKT 8r8O7OEYKeUrlQD83zXXhCQFrP+koAglyKtMuRqs54iEmhllHiO20UR6J mViKdNI+9Po+IGGrqZT+R6cuXD9L2XVKhregRwyiS1YjX/Y6RkWBky9LF g==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045641" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045641" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421525" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:52 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 05/16] vdpa/ifc: add vdpa interrupt for blk device Date: Sun, 27 Mar 2022 22:51:28 +0800 Message-Id: <1648392699-60113-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the block device type, we have to relay the commands on all queues. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 46 ++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 8ee041f..8d104b7 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -370,24 +370,48 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); for (i = 0; i < nr_vring; i++) internal->intr_fd[i] = -1; - for (i = 0; i < nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vring); - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { - fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); - if (fd < 0) { - DRV_LOG(ERR, "can't setup eventfd: %s", - strerror(errno)); - return -1; + if (internal->device_type == IFCVF_NET) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if ((i & 1) == 0 && m_rx == true) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + */ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; + } + } + } else if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vring); + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + if (m_rx == true) { + /* For the blk we need to relay all the read cmd + * of each queue + */ + fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); + if (fd < 0) { + DRV_LOG(ERR, "can't setup eventfd: %s", + strerror(errno)); + return -1; + } + internal->intr_fd[i] = fd; + fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } - internal->intr_fd[i] = fd; - fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd; } } From patchwork Sun Mar 27 14:51:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108886 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D79CEA034C; Sun, 27 Mar 2022 17:36:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8438242881; Sun, 27 Mar 2022 17:35:58 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 9F55042881 for ; Sun, 27 Mar 2022 17:35:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395356; x=1679931356; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=LVT+qXlVR4XjvNNS/0W0jV1fzKJ1YW3hp0aXF04YCAE=; b=iO0I5q/CJLq7EQ50VKz0zovUclNZn574cgBE8Okf5W7shQ5c13l4XpS2 9QXnZYZzPWUsCfHUgE+p21hm+aBh54HiZCYiDwN3K/GbH9RW+kAMQJbJk G2kmt6yfEtkxDQTLGMjXURuz0ZaNpn2SOIk9UiklJtBDQf/fo7xliu+td QFHgxbCnXsaFa5/SLbCWgYyzicU/6I+OqBsLJ5tOa3ax7OHqGZhMi9UIa mIkdzV5T//ZT1pORaGK0X4MRLtDsQMwHvLNq0OLst324yYaqI1BSYmz05 cSpkYyI0zRxQXecFJ4NdTTESVkveO+uvrI9IxCgPjy4Stof+Z6LLlIDFT Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045645" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045645" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421534" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:54 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 06/16] vdpa/ifc: add block device SW live-migration Date: Sun, 27 Mar 2022 22:51:29 +0800 Message-Id: <1648392699-60113-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add SW live-migration support to block device. Add dirty page logging to block device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 4 +- drivers/vdpa/ifc/base/ifcvf.h | 6 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 128 +++++++++++++++++++++++++++++++++++------- 3 files changed, 115 insertions(+), 23 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..e417c50 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -191,7 +191,7 @@ IFCVF_WRITE_REG32(val >> 32, hi); } -STATIC int +int ifcvf_hw_enable(struct ifcvf_hw *hw) { struct ifcvf_pci_common_cfg *cfg; @@ -240,7 +240,7 @@ return 0; } -STATIC void +void ifcvf_hw_disable(struct ifcvf_hw *hw) { u32 i; diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 769c603..6dd7925 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -179,4 +179,10 @@ struct ifcvf_hw { u64 ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid); +int +ifcvf_hw_enable(struct ifcvf_hw *hw); + +void +ifcvf_hw_disable(struct ifcvf_hw *hw); + #endif /* _IFCVF_H_ */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 8d104b7..a23dc2d 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -345,6 +345,56 @@ struct rte_vdpa_dev_info { } } +static void +vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) +{ + struct ifcvf_hw *hw = &internal->hw; + struct rte_vhost_vring vq; + int i, vid; + uint64_t features = 0; + uint64_t log_base = 0, log_size = 0; + uint64_t len; + + vid = internal->vid; + + if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vq); + while (vq.avail->idx != vq.used->idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + hw->vring[i].last_avail_idx = vq.avail->idx; + hw->vring[i].last_used_idx = vq.used->idx; + } + } + + ifcvf_hw_disable(hw); + + for (i = 0; i < hw->nr_vring; i++) + rte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx, + hw->vring[i].last_used_idx); + + if (internal->sw_lm) + return; + + rte_vhost_get_negotiated_features(vid, &features); + if (RTE_VHOST_NEED_LOG(features)) { + ifcvf_disable_logging(hw); + rte_vhost_get_log_base(internal->vid, &log_base, &log_size); + rte_vfio_container_dma_unmap(internal->vfio_container_fd, + log_base, IFCVF_LOG_BASE, log_size); + /* + * IFCVF marks dirty memory pages for only packet buffer, + * SW helps to mark the used ring as dirty after device stops. + */ + for (i = 0; i < hw->nr_vring; i++) { + len = IFCVF_USED_RING_LEN(hw->vring[i].size); + rte_vhost_log_used_vring(vid, i, 0, len); + } + } +} + #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \ sizeof(int) * (IFCVF_MAX_QUEUES * 2 + 1)) static int @@ -659,15 +709,22 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { - gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); - if (gpa == 0) { - DRV_LOG(ERR, "Fail to get GPA for used ring."); - return -1; + if (internal->device_type == IFCVF_NET) { + /* Direct I/O for Tx queue, relay for Rx queue */ + if (i & 1) { + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for used ring."); + return -1; + } + hw->vring[i].used = gpa; + } else { + hw->vring[i].used = m_vring_iova + + (char *)internal->m_vring[i].used - + (char *)internal->m_vring[i].desc; } - hw->vring[i].used = gpa; - } else { + } else if (internal->device_type == IFCVF_BLK) { + /* BLK: relay every queue */ hw->vring[i].used = m_vring_iova + (char *)internal->m_vring[i].used - (char *)internal->m_vring[i].desc; @@ -686,7 +743,10 @@ struct rte_vdpa_dev_info { } hw->nr_vring = nr_vring; - return ifcvf_start_hw(&internal->hw); + if (internal->device_type == IFCVF_NET) + return ifcvf_start_hw(&internal->hw); + else if (internal->device_type == IFCVF_BLK) + return ifcvf_hw_enable(&internal->hw); error: for (i = 0; i < nr_vring; i++) @@ -710,8 +770,12 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (internal->device_type == IFCVF_NET) { + if ((i & 1) == 0) + update_used_ring(internal, i); + } else if (internal->device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -773,17 +837,36 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { - ev.events = EPOLLIN | EPOLLPRI; - /* leave a flag to mark it's for interrupt */ - ev.data.u64 = 1 | qid << 1 | - (uint64_t)internal->intr_fd[qid] << 32; - if (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev) - < 0) { - DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); - return NULL; + if (internal->device_type == IFCVF_NET) { + for (qid = 0; qid < q_num; qid += 2) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); + } + } else if (internal->device_type == IFCVF_BLK) { + for (qid = 0; qid < q_num; qid += 1) { + ev.events = EPOLLIN | EPOLLPRI; + /* leave a flag to mark it's for interrupt */ + ev.data.u64 = 1 | qid << 1 | + (uint64_t)internal->intr_fd[qid] << 32; + if (epoll_ctl(epfd, EPOLL_CTL_ADD, + internal->intr_fd[qid], &ev) + < 0) { + DRV_LOG(ERR, "epoll add error: %s", + strerror(errno)); + return NULL; + } + update_used_ring(internal, qid); } - update_used_ring(internal, qid); } /* start relay with a first kick */ @@ -871,7 +954,10 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_NET) + vdpa_ifcvf_stop(internal); + else if (internal->device_type == IFCVF_BLK) + vdpa_ifcvf_blk_pause(internal); vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Sun Mar 27 14:51:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108887 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DED17A034C; Sun, 27 Mar 2022 17:36:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6F48742886; Sun, 27 Mar 2022 17:36:00 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id BA67742882 for ; Sun, 27 Mar 2022 17:35:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395358; x=1679931358; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=moWTACEIrrZx6lewXL9Rv/TQK/BstbW4Z53GJ/WDUr4=; b=kxueeWEx62nlqoS5UWKEdDvaS+3uwoNxMvTrr3bjjVShHHnMEbihgxHA MmK/oQV0r54H7lTb53gDWJfjoQ+2Msr6KWtEivpZFfHuMrHV2YWunzkBR CafcM4SE2NaRqcrxd5fyvr/fq61Vk3sPhhTbrpyxBWwDNBWOmOVepEv5E RqfIXO5sBqI+yYLXK9vPcL5M3vm6MJ+WTqCdVvOEKhjLjfExf8Nshid4D QBGBqmxsH4Dsbfk/c3ZLVouGh65Fg7M7owvml/LSeQ5D68PSfL5/mhVSd FiXnu84TuNGlgyaDKZ1CfeJxh2S4r2CU3gCwvDtJDKJLEW7fFt0dR34CS g==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045648" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045648" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:35:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421540" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:56 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 07/16] example/vdpa:add vdpa blk support in example Date: Sun, 27 Mar 2022 22:51:30 +0800 Message-Id: <1648392699-60113-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to vdpa example. Signed-off-by: Andy Pei --- examples/vdpa/main.c | 61 +++++++++++++- examples/vdpa/vdpa_blk_compact.h | 72 +++++++++++++++++ examples/vdpa/vhost_user.h | 169 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 301 insertions(+), 1 deletion(-) create mode 100644 examples/vdpa/vdpa_blk_compact.h create mode 100644 examples/vdpa/vhost_user.h diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..1c809ab 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -41,6 +42,7 @@ struct vdpa_port { static int devcnt; static int interactive; static int client_mode; +static int isblk; /* display usage */ static void @@ -49,7 +51,8 @@ struct vdpa_port { printf("Usage: %s [EAL options] -- " " --interactive|-i: run in interactive mode.\n" " --iface : specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n" - " --client: register a vhost-user socket as client mode.\n", + " --client: register a vhost-user socket as client mode.\n" + " --isblk: device is a block device, e.g. virtio_blk device.\n", prgname); } @@ -61,6 +64,7 @@ struct vdpa_port { {"iface", required_argument, NULL, 0}, {"interactive", no_argument, &interactive, 1}, {"client", no_argument, &client_mode, 1}, + {"isblk", no_argument, &isblk, 1}, {NULL, 0, 0, 0}, }; int opt, idx; @@ -159,6 +163,52 @@ struct vdpa_port { }; static int +vdpa_blk_device_set_features_and_protocol(const char *path) +{ + uint64_t protocol_features = 0; + int ret; + + ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES_BASE); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_disable_features(path, + VHOST_VDPA_BLK_DISABLED_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_disable_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_get_protocol_features(path, &protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_CONFIG); + protocol_features |= (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD); + + ret = rte_vhost_driver_set_protocol_features(path, protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + goto out; + } + +out: + return ret; +} + +static int start_vdpa(struct vdpa_port *vport) { int ret; @@ -192,6 +242,15 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + if (isblk) { + RTE_LOG(NOTICE, VDPA, "is a blk device\n"); + ret = vdpa_blk_device_set_features_and_protocol(socket_path); + if (ret != 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..e7c0f22 --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** + * @file + * + * Device specific vhost lib + */ + +#include + +#include +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_RO 5 /* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ +#define VIRTIO_BLK_F_DISCARD 13 /* DISCARD is supported */ +#define VIRTIO_BLK_F_WRITE_ZEROES 14 /* WRITE ZEROES is supported */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_FLUSH 9 /* Flush command supported */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ + +/* Old (deprecated) name for VIRTIO_BLK_F_FLUSH. */ +#define VIRTIO_BLK_F_WCE VIRTIO_BLK_F_FLUSH +#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#endif + +#define VHOST_BLK_FEATURES ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC)) + +#define VHOST_BLK_DISABLED_FEATURES ((1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY)) + +#define VHOST_BLK_FEATURES_BASE (VHOST_BLK_FEATURES | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_VDPA_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_BARRIER) | (1ULL << VIRTIO_BLK_F_SCSI)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_CONFIG) | \ + (1ULL << VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ diff --git a/examples/vdpa/vhost_user.h b/examples/vdpa/vhost_user.h new file mode 100644 index 0000000..137bbc2 --- /dev/null +++ b/examples/vdpa/vhost_user.h @@ -0,0 +1,169 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _VHOST_NET_USER_H +#define _VHOST_NET_USER_H + +#include +#include + +#include "rte_vhost.h" + +/* refer to hw/virtio/vhost-user.c */ + +#define VHOST_MEMORY_MAX_NREGIONS 8 + +#ifndef VHOST_USER_MAX_CONFIG_SIZE +#define VHOST_USER_MAX_CONFIG_SIZE 256 +#endif + +#define VHOST_USER_PROTOCOL_FEATURES ((1ULL << VHOST_USER_PROTOCOL_F_MQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) |\ + (1ULL << VHOST_USER_PROTOCOL_F_RARP) | \ + (1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \ + (1ULL << VHOST_USER_PROTOCOL_F_NET_MTU) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CRYPTO_SESSION) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER) | \ + (1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT)) + +typedef enum VhostUserRequest { + VHOST_USER_NONE = 0, + VHOST_USER_GET_FEATURES = 1, + VHOST_USER_SET_FEATURES = 2, + VHOST_USER_SET_OWNER = 3, + VHOST_USER_RESET_OWNER = 4, + VHOST_USER_SET_MEM_TABLE = 5, + VHOST_USER_SET_LOG_BASE = 6, + VHOST_USER_SET_LOG_FD = 7, + VHOST_USER_SET_VRING_NUM = 8, + VHOST_USER_SET_VRING_ADDR = 9, + VHOST_USER_SET_VRING_BASE = 10, + VHOST_USER_GET_VRING_BASE = 11, + VHOST_USER_SET_VRING_KICK = 12, + VHOST_USER_SET_VRING_CALL = 13, + VHOST_USER_SET_VRING_ERR = 14, + VHOST_USER_GET_PROTOCOL_FEATURES = 15, + VHOST_USER_SET_PROTOCOL_FEATURES = 16, + VHOST_USER_GET_QUEUE_NUM = 17, + VHOST_USER_SET_VRING_ENABLE = 18, + VHOST_USER_SEND_RARP = 19, + VHOST_USER_NET_SET_MTU = 20, + VHOST_USER_SET_SLAVE_REQ_FD = 21, + VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_CRYPTO_CREATE_SESS = 26, + VHOST_USER_CRYPTO_CLOSE_SESS = 27, + VHOST_USER_POSTCOPY_ADVISE = 28, + VHOST_USER_POSTCOPY_LISTEN = 29, + VHOST_USER_POSTCOPY_END = 30, + VHOST_USER_GET_INFLIGHT_FD = 31, + VHOST_USER_SET_INFLIGHT_FD = 32, + VHOST_USER_MAX = 33 +} VhostUserRequest; + +typedef enum VhostUserSlaveRequest { + VHOST_USER_SLAVE_NONE = 0, + VHOST_USER_SLAVE_IOTLB_MSG = 1, + VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG = 3, + VHOST_USER_SLAVE_MAX +} VhostUserSlaveRequest; + +typedef struct VhostUserMemoryRegion { + uint64_t guest_phys_addr; + uint64_t memory_size; + uint64_t userspace_addr; + uint64_t mmap_offset; +} VhostUserMemoryRegion; + +typedef struct VhostUserMemory { + uint32_t nregions; + uint32_t padding; + VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS]; +} VhostUserMemory; + +typedef struct VhostUserLog { + uint64_t mmap_size; + uint64_t mmap_offset; +} VhostUserLog; + +/* Comply with Cryptodev-Linux */ +#define VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH 512 +#define VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH 64 + +/* Same structure as vhost-user backend session info */ +typedef struct VhostUserCryptoSessionParam { + int64_t session_id; + uint32_t op_code; + uint32_t cipher_algo; + uint32_t cipher_key_len; + uint32_t hash_algo; + uint32_t digest_len; + uint32_t auth_key_len; + uint32_t aad_len; + uint8_t op_type; + uint8_t dir; + uint8_t hash_mode; + uint8_t chaining_dir; + uint8_t *ciphe_key; + uint8_t *auth_key; + uint8_t cipher_key_buf[VHOST_USER_CRYPTO_MAX_CIPHER_KEY_LENGTH]; + uint8_t auth_key_buf[VHOST_USER_CRYPTO_MAX_HMAC_KEY_LENGTH]; +} VhostUserCryptoSessionParam; + +typedef struct VhostUserVringArea { + uint64_t u64; + uint64_t size; + uint64_t offset; +} VhostUserVringArea; + +typedef struct VhostUserInflight { + uint64_t mmap_size; + uint64_t mmap_offset; + uint16_t num_queues; + uint16_t queue_size; +} VhostUserInflight; + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + +typedef struct VhostUserMsg { + union { + uint32_t master; /* a VhostUserRequest value */ + uint32_t slave; /* a VhostUserSlaveRequest value*/ + } request; + +#define VHOST_USER_VERSION_MASK 0x3 +#define VHOST_USER_REPLY_MASK (0x1 << 2) +#define VHOST_USER_NEED_REPLY (0x1 << 3) + uint32_t flags; + uint32_t size; /* the following payload size */ + union { +#define VHOST_USER_VRING_IDX_MASK 0xff +#define VHOST_USER_VRING_NOFD_MASK (0x1<<8) + uint64_t u64; + struct vhost_vring_state state; + struct vhost_vring_addr addr; + VhostUserMemory memory; + VhostUserLog log; + struct vhost_iotlb_msg iotlb; + VhostUserCryptoSessionParam crypto_session; + VhostUserVringArea area; + VhostUserInflight inflight; + struct vhost_user_config cfg; + } payload; + int fds[VHOST_MEMORY_MAX_NREGIONS]; + int fd_num; +} __attribute((packed)) VhostUserMsg; + +#define VHOST_USER_HDR_SIZE offsetof(VhostUserMsg, payload.u64) + +/* The version of the protocol we support */ +#define VHOST_USER_VERSION 0x1 +#endif From patchwork Sun Mar 27 14:51:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108888 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C0D06A034C; Sun, 27 Mar 2022 17:36:37 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 920064288F; Sun, 27 Mar 2022 17:36:02 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 6CBB642885 for ; Sun, 27 Mar 2022 17:36:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395360; x=1679931360; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=f9iIhO50rLyIGKCTBiUd876H7y7ZRQkY7IipgetRm0o=; b=SSRP4hu7SjPZv2rx12iMrF+8ngBkld0FgXeYlo6DX8IvFOb4gLxcb44C Ane8UQt2MaYp4Xo8xeGzdK7CazUhIel5Ue8XXwcEvea0m6Mz1GkAfuod4 UJ8FKdvK69ByZMcPQ8HxkNirggCfoy+wKk3y1r5Ic1py7U/hdWOR4N974 SyRisW/5E5Qc9b4nZZU0UqcwcwMeDbY/3HqBvJ8V2GXuiommank3JJSyE mnWl5XVLF0h4Bh4W9oXjlO9tsNPFGCWZU2jwgRxOgtkrrZJtJM1tlMAEI jd0Ioro6b1jG6Et3qu4xElkS1LjpHCe6vZ93ZUAjSirdSZ0/J7GJr/frb A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045651" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045651" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421549" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:35:58 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 08/16] usertools: add support for virtio blk device Date: Sun, 27 Mar 2022 22:51:31 +0800 Message-Id: <1648392699-60113-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- usertools/dpdk-devbind.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..cbe336f 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -14,6 +14,8 @@ from os.path import join as path_join # The PCI base class for all devices +virtio_blk_class = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': '8086', 'SDevice': '0002'} network_class = {'Class': '02', 'Vendor': None, 'Device': None, 'SVendor': None, 'SDevice': None} acceleration_class = {'Class': '12', 'Vendor': None, 'Device': None, @@ -72,6 +74,7 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk_devices = [virtio_blk_class] network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -587,6 +590,9 @@ def show_status(): Displays to the user what devices are bound to the igb_uio driver, the kernel driver or to no driver''' + if status_dev in ["virtio_blk", "all"]: + show_device_status(virtio_blk_devices, "virtio_blk") + if status_dev in ["net", "all"]: show_device_status(network_devices, "Network", if_field=True) @@ -746,6 +752,7 @@ def do_arg_actions(): if b_flag is not None: clear_data() # refresh if we have changed anything + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) @@ -769,6 +776,7 @@ def main(): parse_args() check_modules() clear_data() + get_device_details(virtio_blk_devices) get_device_details(network_devices) get_device_details(baseband_devices) get_device_details(crypto_devices) From patchwork Sun Mar 27 14:51:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108889 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ACDBBA034C; Sun, 27 Mar 2022 17:36:43 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 733C042894; Sun, 27 Mar 2022 17:36:03 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 2297E4288E for ; Sun, 27 Mar 2022 17:36:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395362; x=1679931362; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bo7WL2+URlaL7lMv/7Z5CoN7zy2GZiVZ2pmCi1TxoMI=; b=mqCpZewxvCHbAdqUcx8M8VB6P2FkAX7f3hEfttHcATtMlEBVYXoJdgZE 8zuTxRIG63xemZi4Cc+UeVayNro9lCvszJBtl0ccgbYPFyRGDGP1Un69t hWeVYrY32f4tyGBWGjz9Zvvyas0MngMpZ1MBYcqS+vWevz8plfRijAsYq /c9ANAZ2cF/VEmfn34E0+fLw1GGHkHk6MLiIgFwMMibvOTwu8SBEdN8im 8RuYPj39LB1Cq4OIwEHmSfjOAqZzUfZd00iI7i56K/GVY04LLMR+lj7Yw PsrHTVmze+eoU4G8BrC59MFzija77MxZQYCUoeQduecQB+XVC/ea8PKdL g==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045653" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045653" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421556" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:00 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 09/16] vdpa/ifc: set_vring_state op is mandatory, add set_vring_state for blk device Date: Sun, 27 Mar 2022 22:51:32 +0800 Message-Id: <1648392699-60113-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index a23dc2d..28191e4 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1368,6 +1368,16 @@ struct rte_vdpa_dev_info { } static int +ifcvf_blk_set_vring_state(int vid, int vring, int state) +{ + RTE_SET_USED(vid); + RTE_SET_USED(vring); + RTE_SET_USED(state); + + return 0; +} + +static int ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1385,7 +1395,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, From patchwork Sun Mar 27 14:51:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108890 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 74297A034C; Sun, 27 Mar 2022 17:36:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 53F1D4288A; Sun, 27 Mar 2022 17:36:05 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id F3D7F42896 for ; Sun, 27 Mar 2022 17:36:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395364; x=1679931364; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=4QV8Vtzr8Qw4ZXuXhZdnuMqpIxdI0oilBYOJX8Y8blM=; b=iMYfcANEyjq94UHEse/ZvazRM5LouYwYLKhS2KBsFnb9QgoupOmtl+Ot GJ1bEGX/3i9npsbjPQD6Y1YYnE1xIKbHnpZ8GjaWZ4200Gb0CO/6UHCeS 4kbkdEi4vFxiuk+nQBx0U+x0ZiJPdRIbZ4RnTepRfJc0jpJ+45gFmQ/ol rbqxsWIgxuQ6TO25VVV0XIiIqglJkYE3hnDgRaPHGg3jATtZ4Kjl+HCsz Bdh3JtSXEMLjbzcqfl2fIZG0RM9WrlnrzLlKcAIajpbkGpECbO1viqRJq zYTbE+tVsLuMlJ7a8KfhMP4GeLbrKlbaupwbdRXcV7nRGk4zCyMHwTSRY Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045654" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045654" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421568" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:01 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 10/16] vdpa/ifc: add some log at VDPA lauch before qemu connect Date: Sun, 27 Mar 2022 22:51:33 +0800 Message-Id: <1648392699-60113-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 28191e4..9bc2f47 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1431,6 +1431,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + __u64 capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1497,6 +1500,32 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /** + ** cannot read 64-bit register in one attempt, + ** so read byte by byte. + **/ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (__u64)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %quG", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Sun Mar 27 14:51:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108891 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3DB25A034C; Sun, 27 Mar 2022 17:36:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4AC4A42898; Sun, 27 Mar 2022 17:36:08 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id A499642898 for ; Sun, 27 Mar 2022 17:36:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395365; x=1679931365; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=iEgZuq2zqXREe7HTQr40FJztSq2gc9kt44R4q/xdFG8=; b=gI+YJNblVHptKlTX6BtlMfo3l4vJOz5OuqD8LMNCprp10aRgKDN3ygjx 0jxsRqcZ3GLElcFXTYbkMM34l5QCr6OdIR7MVuiakeUOVzxta5njACRDn b6FF8u/T/YlR1qatWfzDJkHAQIRjw3xGMz4RaVs/aZ9CTo34HVM7SYf8f YyayYHExWuOTCNYRRlQjizwzNz2UKbAF/mxRbxQSuqVVJCqGCiAn0O+yg tFK6rh+ocBWH+J5Wc7wX1wQ1rCTpAUdTsFRM5DigwQNzpV+/dxN1fNNt8 FNCuyzgC126Ux9otbMjLVtebtp18Ey5XRU+DAa+ih4SkajNDVAac3SGeE g==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045655" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045655" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421576" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:03 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 11/16] vdpa/ifc: read virtio max_queues from hardware Date: Sun, 27 Mar 2022 22:51:34 +0800 Message-Id: <1648392699-60113-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org original code max_queues is set to IFCVF_MAX_QUEUES. New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9bc2f47..20a0b01 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1526,6 +1526,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(INFO, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + /* reset max_queue here, to minimum modification */ + internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Sun Mar 27 14:51:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108892 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 393A5A034C; Sun, 27 Mar 2022 17:37:01 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1E0FB4289F; Sun, 27 Mar 2022 17:36:10 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 9D62A42898 for ; Sun, 27 Mar 2022 17:36:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395367; x=1679931367; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=RpRN47JtxeCldxQ2ZS37Pc1lqLNGEQSPMI9x2X5IdVY=; b=K1H8//Ehp/RBAzvCxIScL9A2KuvWJKHKO84rs5h+aihp5lGy2mg4O/On LOPqfo+sICyDqbIQ1qxilQbnkQiVP+zhkSah7tw7/6eBA24wNHFPxMC5X yrPzL+Jm3Ul3YouBOESfBVENrP7hL76rbAMN6Cgds3G5i3WrqZl7iasgu 88K1wIRGD+RbQvBWyWodGnaJOErJVnSkQ9xB/pCu5hzMud6SKf0hU3ili fqUIf/v5tyHH5eGjNqzBNi/2U3XDKrvgVL02YOWoCW+Yle03JeQTwzg0r vRIhAptCZrtODObcm6ZbJW/dKW7aAtca0Heb1o0lqGrcWWWrSf/iM7bS2 g==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045665" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045665" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421580" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:05 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 12/16] vdpa: add config space change interrupt register and handle for virtio_blk Date: Sun, 27 Mar 2022 22:51:35 +0800 Message-Id: <1648392699-60113-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 113 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 20a0b01..826b408 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for intr relay */ int epfd; + int csc_fd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -608,6 +610,108 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); + + return; +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_fd, csc_val = 0; + + csc_fd = epoll_create(1); + if (csc_fd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_fd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + return NULL; + } + + internal->csc_fd = csc_fd; + + for (;;) { + csc_val = epoll_wait(csc_fd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail\n"); + return NULL; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || errno == EWOULDBLOCK) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + return NULL; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + int ret; + + ret = pthread_create(&internal->intr_tid, NULL, intr_relay, + (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static int +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_fd >= 0) + close(internal->csc_fd); + internal->csc_fd = -1; + + return 0; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -634,10 +738,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + ret = unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -958,6 +1068,9 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(internal); else if (internal->device_type == IFCVF_BLK) vdpa_ifcvf_blk_pause(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Sun Mar 27 14:51:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108893 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 19796A034C; Sun, 27 Mar 2022 17:37:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DFF6F428A3; Sun, 27 Mar 2022 17:36:11 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 447284289E for ; Sun, 27 Mar 2022 17:36:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395369; x=1679931369; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=oE9cAlKTuL4vCQj83iT5JhajvRZHRPhjAkiVL1lYaw4=; b=C1IcrYCk0Li5SUCJeq2TIOp4lBAkCWOxJw39Sdd3SVl/AFvCjuSbEhQZ YobvAp2p8wv9D6ZDpT5pILxHWRfK4E4AkbTafQ3859ByDBgFwdVjtU+jM 3zCybSIpXq1IhBb9vNvJ309IzMZO1Ycv6yPRH7rbAXkLN36QY4lvEqyCa dJ6Ic5KQpQndFHf8A2kKvmhx2gH43bn25VCQXuVqkKXuLFtscQ+R01X/K GP/BvLq6qEJKOB0pYrByiyLT4B5yDn5A8+cJ2IGtBXLB9LZqi9ZBbwvD6 9Fj90jLxnYN2BfX8eo1p9QS/j+TO4HFKSUIe9gd9g+gxIHRrGiRQNmb6/ A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045667" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045667" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421583" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:07 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 13/16] vdpa/ifc: add is_blk flag to ifcvf_hw, and init is_blk during probe Date: Sun, 27 Mar 2022 22:51:36 +0800 Message-Id: <1648392699-60113-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 6dd7925..8e602af 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -149,6 +149,7 @@ struct ifcvf_hw { u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + u8 is_blk; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 826b408..95538c1 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1605,11 +1605,13 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->device_type = IFCVF_NET; + internal->hw.is_blk = IFCVF_NET; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; } else if (device_id == VIRTIO_ID_BLOCK) { internal->device_type = IFCVF_BLK; + internal->hw.is_blk = IFCVF_BLK; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; From patchwork Sun Mar 27 14:51:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108894 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 59AF9A034C; Sun, 27 Mar 2022 17:37:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AE2CE428A8; Sun, 27 Mar 2022 17:36:12 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 2B611428A2 for ; Sun, 27 Mar 2022 17:36:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395371; x=1679931371; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=O4O8JuyQ9suOa8sxqwfQTN2iaftJhSnkRuD1m+3P+dg=; b=CMOJGUyDLI3eKghUkKBRhiKJESEEvXis4IZEwaY1++1ISKFakSrKxFN0 5hF5bpo2IWBWXEHsOgLHqilbpleJqGzBTOT4BWfdPKgj/tsWNeRRuuNGo CP661QzdWERrcHdFFIfqU4jY5+aYK00rvd75/60nzzicel1Ii5dM/N+tK 6uedmrHWW3I1nwtWmrRlcINIzxhh6rMoWv/GX6DVWJD+zFyVr3+tPXZYC 2ai0N5eSARxzOxtO6jP9Xcblu/ODFDJcyrTcdzaSFAS5i9bs1R4dFIfo+ Xu+lDruWL3Ryr1shqQn3FTHS7U1XK6zGaR1WSvT2iysddLEHcp/wyiptC A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045672" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045672" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421600" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:09 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 14/16] vdpa/ifc/base: for blk device, live migration register is different from net device Date: Sun, 27 Mar 2022 22:51:37 +0800 Message-Id: <1648392699-60113-15-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 1.last_avail_idx is lower 16 bit of the register. 2.address of ring_state register is different between net and blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index e417c50..d923266 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -218,10 +218,18 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->is_blk == IFCVF_BLK) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } else if (hw->is_blk == IFCVF_NET) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +262,23 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + } else if (hw->is_blk == IFCVF_NET) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + } + + if (hw->is_blk == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else if (hw->is_blk == IFCVF_NET) + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } } diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 8e602af..7367094 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 From patchwork Sun Mar 27 14:51:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108895 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 54AC3A034C; Sun, 27 Mar 2022 17:37:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7F277428AD; Sun, 27 Mar 2022 17:36:14 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 1F6BD428AB for ; Sun, 27 Mar 2022 17:36:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395373; x=1679931373; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=shlvPVu73+lMRceBSh/EoFio4G+PlCZCQCcKi4FEywE=; b=YJZxbsBWR7/a5Hq4XUEUbotaskp/n1L5T/caIUNQY9Z3hq9U7WF8TLvl oHH25zx0pb4J3Pphw/ZtKUp27RydaUsR1ONnlvgTiIQQqgbWiLix5wyoT rLhgb/PNUPaVDvnUlgddm9orcHQZZXE/QlF0yCG/AQh/YGD1fu/ec38l1 ntDrj3zrLWFgSoMdlgWd6Kw/VeeYUKr5yj51anJDp3lnUeyA+cP03LoLC Y6lyu7uo/65PM7Sjp6i7vXAFS0qAWxkXhkn5yuxKKz9pJabZq+uu/BGN+ O/mzXbmaFwOdVgzzrAKd18F7ivJPt82+xQCHZ6YUxKygOBbbHPU1EOQvL A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045676" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045676" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421612" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:11 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 15/16] vdpa/ifc: make sure hardware last_avail_idx and last_used_idx is the same when blk device pause Date: Sun, 27 Mar 2022 22:51:38 +0800 Message-Id: <1648392699-60113-16-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 2 +- drivers/vdpa/ifc/base/ifcvf.h | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 32 +++++++++++++++++++++++--------- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index d923266..d89cb73 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -118,7 +118,7 @@ IFCVF_WRITE_REG8(status, &hw->common_cfg->device_status); } -STATIC void +void ifcvf_reset(struct ifcvf_hw *hw) { ifcvf_set_status(hw, 0); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 7367094..f22d18b 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -157,6 +157,9 @@ struct ifcvf_hw { int ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev); +void +ifcvf_reset(struct ifcvf_hw *hw); + u64 ifcvf_get_features(struct ifcvf_hw *hw); diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 95538c1..36fd850 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -351,23 +351,32 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_blk_pause(struct ifcvf_internal *internal) { struct ifcvf_hw *hw = &internal->hw; - struct rte_vhost_vring vq; int i, vid; uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; if (internal->device_type == IFCVF_BLK) { for (i = 0; i < hw->nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vq); - while (vq.avail->idx != vq.used->idx) { - ifcvf_notify_queue(hw, i); - usleep(10); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } @@ -752,7 +761,12 @@ struct rte_vdpa_dev_info { if (ret) goto err; - vdpa_ifcvf_stop(internal); + if (internal->device_type == IFCVF_BLK) { + vdpa_ifcvf_blk_pause(internal); + ifcvf_reset(&internal->hw); + } else { + vdpa_ifcvf_stop(internal); + } ret = vdpa_disable_vfio_intr(internal); if (ret) From patchwork Sun Mar 27 14:51:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 108896 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4D167A034C; Sun, 27 Mar 2022 17:37:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B465D428B7; Sun, 27 Mar 2022 17:36:16 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id C8654428AF for ; Sun, 27 Mar 2022 17:36:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1648395374; x=1679931374; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=S3MLMAqbl75k872rPBUBNw3QcbfpkBVyzIwi/QB+q8M=; b=WNKnvhV7Zz5cMzhoy93ypzuRBkg5L3mWFHiJuzqdAfmZl6dKZyJlGne8 QGjmW6HrdvRWylTHj1+XacxWP5F86n2kxtCxPe03lemCiqmK9MpP1fjmh 1Wl4heho7QcbIE6h0qa59lfC/lZxUx0hg8VU5u65BCBjBZWX/Ybz7qyMJ oayhxcZ2Bt4R8qziZcgM3dvrxaPHhl80KG56swQ6y+aKRGXQ4ELr3xd9j eiKFhCyXoxK9ag2LDTWFreAmF+Jz1Cv9zhhU881hD6g3odWmn3PPz1Vqx rT/yenzuDF/ARvAmzNG96lqAmmz2Wn//fVztTf5qynBNO9SYkfCK7Z+RB A==; X-IronPort-AV: E=McAfee;i="6200,9189,10299"; a="259045680" X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="259045680" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2022 08:36:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,215,1643702400"; d="scan'208";a="562421622" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Mar 2022 08:36:12 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v4 16/16] vhost: make sure each queue callfd is configured Date: Sun, 27 Mar 2022 22:51:39 +0800 Message-Id: <1648392699-60113-17-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1648392699-60113-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1648392699-60113-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index b11fafd..8c5904f 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3197,12 +3197,27 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, if (!vdpa_dev) goto out; + if (request != VHOST_USER_SET_VRING_CALL) + goto out; + if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid)) VHOST_LOG_CONFIG(ERR, "(%s) failed to configure vDPA device\n", dev->ifname); else dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED; + } else { + /** + ** VIRTIO_DEV_VDPA_CONFIGURED already configured + ** close the device and open the device again, + ** make sure the call fd of each queue is configed to haedware. + **/ + if (vdpa_dev->ops->dev_close(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to close vDPA device\n"); + if (vdpa_dev->ops->dev_conf(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to re-config vDPA device\n"); } out: