From patchwork Wed Apr 27 08:29:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110307 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D211A0507; Wed, 27 Apr 2022 11:16:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A92A41156; Wed, 27 Apr 2022 11:16:44 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id E5E0E40691 for ; Wed, 27 Apr 2022 11:16:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051003; x=1682587003; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=zWAV2IbjItaFpvx6M55IDxoj7pCQRBoeuR8ntMToDes=; b=lHJuIKFlPwDfAd1JdPv2xbYEMdRwjET8tjlvztJJT68nJV6QN4WNFKW6 wrlOeRex91pbI3s8+RDCa1PqGgECoVyExQJTlQx/CHQQw70jIXOfjomyd /jsN9ysp4r6RUbiZiYvN13nG63YqfP5TwH1Bb7z34KSV0HnZhhh5sfn+v ttFyx7b1ldljoXSdwvCy3PUzUaLA+K0jkR68/15dbDsq0BOEcivdA+Uov 7lJ01E1ezU0Ubz6MgG4p6vciQCGmDYpy6P7t8rgu0mYiDmioTCA8DK35t oYcqIbE74af6bjCy7hYQQxndPOxnNGUt56CFFQBy896SrG+TcN2a+S6UX A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308234" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308234" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488607" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:24 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 01/18] vdpa/ifc: add support for virtio blk device Date: Wed, 27 Apr 2022 16:29:49 +0800 Message-Id: <1651048206-282372-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 92 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 10 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..01522c6 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,8 +5,17 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 + #define IFCVF_VENDOR_ID 0x1AF4 #define IFCVF_DEVICE_ID 0x1041 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 @@ -57,7 +66,6 @@ #define IFCVF_32_BIT_MASK 0xffffffff - struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ @@ -126,7 +134,11 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9f05595..e3210a8 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -58,6 +58,7 @@ struct ifcvf_internal { struct rte_vdpa_device *vdev; uint16_t max_queues; uint64_t features; + int device_type; rte_atomic32_t started; rte_atomic32_t dev_attached; rte_atomic32_t running; @@ -75,6 +76,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1167,6 +1174,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1178,6 +1227,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1227,13 +1277,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1245,7 +1306,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1313,6 +1375,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Wed Apr 27 08:29:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110308 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DCB76A0507; Wed, 27 Apr 2022 11:16:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 70614427FD; Wed, 27 Apr 2022 11:16:47 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id DC7A440691 for ; Wed, 27 Apr 2022 11:16:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051004; x=1682587004; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=lW5isKc3/9enixxJrZo0+rHrLl9944fbMj0sPaJaFOE=; b=P59jDByfdAQTOcfx9qkKeCtnMaZls9Zg+qQTNMtZULIXniZfDt2f9ytL EQ04vyar5gl4mMqvojXtHaN9EbWOWkS+51gWVrIyesvSbkieWFN7H0r6j P2hYnur3RkRTyI71JYzDph98peUPNjhxhYCiTeLS3MzEg2pvWTqQZB7Gs G2Rf7yiaqgpAFTWs7YKoIYBgEAtY8YeBKswVlNHdYsJs9QsmcoWo9hDjm YUDlVH2ApJ9w/0VJqmbgH4/iEPpy9nNTJih9xtnHVJ8k6kp4omDKkyWZT PnlvI6wYiJgGbzgcH3YfoDg3XCtNP1ZbEost4Xe/p3eaHJvAAMLI8C5s4 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308241" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308241" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488624" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:26 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 02/18] vhost: add vDPA ops for blk device Date: Wed, 27 Apr 2022 16:29:50 +0800 Message-Id: <1651048206-282372-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 88138be..e59a834 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t len); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Wed Apr 27 08:29:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110309 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 06FA4A0507; Wed, 27 Apr 2022 11:17:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A1E7E42808; Wed, 27 Apr 2022 11:16:48 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 68115427F9 for ; Wed, 27 Apr 2022 11:16:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051006; x=1682587006; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=WXj2TYdLRXBVR9nK4QU2s/rzGVgHNLpfBlWTuxjs9ew=; b=M3xGQ/xd3WpfxsKNAAzqME82/QFv6QngAK+7Qt2HxmMY4PnZAoZTvfJ9 TBJZwq2XfH+9xr0aHb0EOjyQkKapCpI/AZV/LiHjqrHtOc/rtFkz6t6oG rr1tuIe/8qg/wysXU71FFRhIv7XNJLI06GES96eFDFoLwxEIKwACk5AUE IqWBjl8VhSCqCQKgb2VSFgU+NVpCWyoWRpWhHA/B8xZsSMbVTZNCRdKFc p/6DYCClmin8+7IG+cp9FRiFQDFjnZkgebTUSOmxpcKGT2OxjZQRPWmiR fW76lav62satHg4JADiua8ejfMGK9LDSW6sHcqfHV0YTHKZzFpqM9AMMW A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308247" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308247" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488645" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:28 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 03/18] vhost: add vhost msg support Date: Wed, 27 Apr 2022 16:29:51 +0800 Message-Id: <1651048206-282372-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 ++++++++ 2 files changed, 96 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 1d39067..e925428 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -80,6 +80,8 @@ [VHOST_USER_NET_SET_MTU] = "VHOST_USER_NET_SET_MTU", [VHOST_USER_SET_SLAVE_REQ_FD] = "VHOST_USER_SET_SLAVE_REQ_FD", [VHOST_USER_IOTLB_MSG] = "VHOST_USER_IOTLB_MSG", + [VHOST_USER_GET_CONFIG] = "VHOST_USER_GET_CONFIG", + [VHOST_USER_SET_CONFIG] = "VHOST_USER_SET_CONFIG", [VHOST_USER_CRYPTO_CREATE_SESS] = "VHOST_USER_CRYPTO_CREATE_SESS", [VHOST_USER_CRYPTO_CLOSE_SESS] = "VHOST_USER_CRYPTO_CLOSE_SESS", [VHOST_USER_POSTCOPY_ADVISE] = "VHOST_USER_POSTCOPY_ADVISE", @@ -2542,6 +2544,85 @@ static int is_vring_iotlb(struct virtio_net *dev, } static int +vhost_user_get_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); + if (ret != 0) { + ctx->msg.size = 0; + VHOST_LOG_CONFIG(ERR, + "(%s) get_config() return error!\n", + dev->ifname); + } + } else { + VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_REPLY; +} + +static int +vhost_user_set_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (ctx->msg.size != sizeof(struct vhost_user_config)) { + VHOST_LOG_CONFIG(ERR, + "(%s) invalid set config msg size: %"PRIu32" != %d\n", + dev->ifname, ctx->msg.size, + (int)sizeof(struct vhost_user_config)); + goto out; + } + + if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) { + VHOST_LOG_CONFIG(ERR, + "(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, + VHOST_USER_MAX_CONFIG_SIZE); + goto out; + } + + if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); + if (ret) + VHOST_LOG_CONFIG(ERR, + "(%s) set_config() return error!\n", + dev->ifname); + } else { + VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_OK; + +out: + return RTE_VHOST_MSG_RESULT_ERR; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx, int main_fd __rte_unused) @@ -2782,6 +2863,8 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, [VHOST_USER_NET_SET_MTU] = vhost_user_net_set_mtu, [VHOST_USER_SET_SLAVE_REQ_FD] = vhost_user_set_req_fd, [VHOST_USER_IOTLB_MSG] = vhost_user_iotlb_msg, + [VHOST_USER_GET_CONFIG] = vhost_user_get_config, + [VHOST_USER_SET_CONFIG] = vhost_user_set_config, [VHOST_USER_POSTCOPY_ADVISE] = vhost_user_set_postcopy_advise, [VHOST_USER_POSTCOPY_LISTEN] = vhost_user_set_postcopy_listen, [VHOST_USER_POSTCOPY_END] = vhost_user_postcopy_end, diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index c946cc2..97cfb2f 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -50,6 +50,8 @@ VHOST_USER_NET_SET_MTU = 20, VHOST_USER_SET_SLAVE_REQ_FD = 21, VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_GET_CONFIG = 24, + VHOST_USER_SET_CONFIG = 25, VHOST_USER_CRYPTO_CREATE_SESS = 26, VHOST_USER_CRYPTO_CLOSE_SESS = 27, VHOST_USER_POSTCOPY_ADVISE = 28, @@ -125,6 +127,16 @@ uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -148,6 +160,7 @@ VhostUserCryptoSessionParam crypto_session; VhostUserVringArea area; VhostUserInflight inflight; + struct vhost_user_config cfg; } payload; /* Nothing should be added after the payload */ } __rte_packed VhostUserMsg; From patchwork Wed Apr 27 08:29:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110310 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 91E87A0507; Wed, 27 Apr 2022 11:17:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 925484280E; Wed, 27 Apr 2022 11:16:49 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 067CC427FA for ; Wed, 27 Apr 2022 11:16:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051007; x=1682587007; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=/K8my91Bi3zBcNK1klWq65z4PAi11xo392zGrqZpMCM=; b=YFXt8HROU0moaRX8JuB/vfAwUszR2GBkARP3tzUHVCUbpIZkiPS1/qql mVcqWfPgVwzeZBh54Z/b9Q3AInPTSJacreKcSkfo4UKGaQ9UYs2iYssM9 R+DSVyCWZ0/i8G/k1ijcY0v66F9p5JJmW6XaK+OKTyXqpa0MrQC0gKMj5 52EcbOkxfGXatL7t5jZXJ5FA+COtsm8DRI9oPPMwnjE+XrXhCc7F1hWd+ 5htjy7R2zOdW12R8rZgCIwWCwUG45oETg/dXehb7VSkMXaZCOQ1QO1ktR fPbAMeM4gGWcWaohJ9uBiw7y8pxER2qLhhsXV0hRMWlPdxrHX8Cm7uS9a A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308249" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308249" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488663" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:29 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 04/18] vdpa/ifc: add blk ops for ifc device Date: Wed, 27 Apr 2022 16:29:52 +0800 Message-Id: <1651048206-282372-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 85 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 88 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 01522c6..769c603 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -66,6 +66,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index e3210a8..8ee041f 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1088,6 +1088,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1200,6 +1204,85 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + uint64_t capacity = 0; + uint8_t *byte; + + if (len < sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %u", + len, (uint32_t)sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(INFO, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = NULL, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1212,7 +1295,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Wed Apr 27 08:29:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110311 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 684B5A0507; Wed, 27 Apr 2022 11:17:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6C0344280C; Wed, 27 Apr 2022 11:16:50 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 4E02E427F9 for ; Wed, 27 Apr 2022 11:16:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051007; x=1682587007; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=nhDz1GDcUFS4P/yyd4KLPaZs9X8THWHjTZMM432Rau0=; b=EfsDNnkV7YodfU9PBJ2UbduQ6yLIW0sW1cmY+yBHz0vpPETDupPLg7wT ttBTisLLBo3PgV0+PMiv1hgeI9jYt+ig2kDh5mnhHC9Z5dybCjjpJI/sE k77ZgsjrQR3b/59xB+pcU637Xh7OmiuBoCkg0zqsgp352b+29Umm5R2HL 6fIvThFFwmOG9eborrxm2J+6gc44VwRhYuL5Pw2jqO4itTHmCsv+WXZfZ SZLAni6iiR5yuR7Hz6Lc1xzyxAQP3WcAE/IqTgQacPqFgqNwyhna8g609 Inkv3nO+Rdh1Lck8v1DsPMw2Sl44njGieV5Z9uhA51jPmHRvmG0oAydol A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308254" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308254" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488684" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:31 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 05/18] vdpa/ifc: add vDPA interrupt for blk device Date: Wed, 27 Apr 2022 16:29:53 +0800 Message-Id: <1651048206-282372-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the block device type, we use one queue to transfer both read and write requests, so we have to relay commands on all queues. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 8ee041f..07fc3ca 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -370,6 +370,7 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); @@ -379,7 +380,13 @@ struct rte_vdpa_dev_info { for (i = 0; i < nr_vring; i++) { rte_vhost_get_vhost_vring(internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { + if (m_rx == true && + ((i & 1) == 0 || internal->device_type == IFCVF_BLK)) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + * For the blk we need to relay all the read cmd + * of each queue + */ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); if (fd < 0) { DRV_LOG(ERR, "can't setup eventfd: %s", From patchwork Wed Apr 27 08:29:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110312 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EF3E8A0507; Wed, 27 Apr 2022 11:17:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3C99341145; Wed, 27 Apr 2022 11:16:51 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id B0F50427FA for ; Wed, 27 Apr 2022 11:16:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051007; x=1682587007; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=J5zxZdAN36tpYL85FaFgxFn+RS9HN6lHyRIBYvtkKlo=; b=DxMEKG61CGgQ+qHnlbTqdu8yotpMpkzwozZmnybQW4t9ToJvUf8JD1o1 J0OO76t82ZD2NxePlOs2eZOMvCxYN/0DQvUtD0sAW65bOVZvcc1UDnpQM xsAf+xOEqpQnmBMjGp3GtVOZIeDy0j/uRIuEgRc0JmQeL3SScO4A+1hrq dwbi1yH8oagFyQdxk8vWjK1kDIBpb6rAeNOMjewN+OVL+2pbXshYTP7qu CQpTNlYbKOMQNtHsT4YrqkEcDK75t8bwMOU8kY/TTgfhYs+e4csUFd6A6 875uxnJHncKUrzfQ6HLwvaI+utnWEhBWx91hWn+Lc/nyzAsEU6FKrHOy/ Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308269" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308269" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488694" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:33 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 06/18] vdpa/ifc: add block device SW live-migration Date: Wed, 27 Apr 2022 16:29:54 +0800 Message-Id: <1651048206-282372-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add SW live-migration support to block device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 07fc3ca..8a260b7 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -312,6 +312,7 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(struct ifcvf_internal *internal) { struct ifcvf_hw *hw = &internal->hw; + struct rte_vhost_vring vq; uint32_t i; int vid; uint64_t features = 0; @@ -319,6 +320,22 @@ struct rte_vdpa_dev_info { uint64_t len; vid = internal->vid; + + /* to make sure no packet is lost for blk device + * do not stop until last_avail_idx == last_used_idx + */ + if (internal->device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + rte_vhost_get_vhost_vring(internal->vid, i, &vq); + while (vq.avail->idx != vq.used->idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + hw->vring[i].last_avail_idx = vq.avail->idx; + hw->vring[i].last_used_idx = vq.used->idx; + } + } + ifcvf_stop_hw(hw); for (i = 0; i < hw->nr_vring; i++) @@ -642,8 +659,10 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /* NET: Direct I/O for Tx queue, relay for Rx queue + * BLK: relay every queue + */ + if ((internal->device_type == IFCVF_NET) && (i & 1)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); @@ -693,8 +712,12 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (internal->device_type == IFCVF_NET) { + if ((i & 1) == 0) + update_used_ring(internal, i); + } else if (internal->device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -756,7 +779,9 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { + for (qid = 0; qid < q_num; qid += 1) { + if ((internal->device_type == IFCVF_NET) && (qid & 1)) + continue; ev.events = EPOLLIN | EPOLLPRI; /* leave a flag to mark it's for interrupt */ ev.data.u64 = 1 | qid << 1 | From patchwork Wed Apr 27 08:29:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110313 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 38C20A0507; Wed, 27 Apr 2022 11:17:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 29BDD42816; Wed, 27 Apr 2022 11:16:52 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 155AC427F9 for ; Wed, 27 Apr 2022 11:16:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051008; x=1682587008; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=HdhV11tCE4ZigKupibeeOl4gcfO5a+AOPupfoh/v9E0=; b=F9IQxr+3F9w6f7ppNJYiJ66/lZnz3Fp+ClY0heT75vClqB9iiWAypVoB G5e4/+guFlMScwaN5AlcxzhH4N/Xa+rGgmDGRep0j4Wmt6m/jL18oMVli Ce4laqGOmZ7G0BKucNQfKKZLE4NFjvD32aXSspHjw5JCq7wV0VeKp8K5g zFR+Ur2ePgf70ZFdKlIqZYCYtuvf6vjbxqUiRlVFLBfUxdjRfWV3xEmix OdZPEk+UIxbKFEDXn3VIHoYTR9zWurkkD2CUS5L5Jjkaqb+/ahu5I24K4 ib3WpfCCFaJQGggOsbQ/Y/u14YRyB/vymBCJ1T3VVwQOyIeQCU6+QQgKO A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308286" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308286" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488708" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:35 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 07/18] vhost: add API to get vDPA device type Date: Wed, 27 Apr 2022 16:29:55 +0800 Message-Id: <1651048206-282372-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Vhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices. Signed-off-by: Andy Pei --- lib/vhost/rte_vhost.h | 17 +++++++++++++++++ lib/vhost/socket.c | 39 +++++++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 2 ++ 4 files changed, 61 insertions(+) diff --git a/lib/vhost/rte_vhost.h b/lib/vhost/rte_vhost.h index c733f85..c977a24 100644 --- a/lib/vhost/rte_vhost.h +++ b/lib/vhost/rte_vhost.h @@ -117,6 +117,9 @@ #define RTE_MAX_VHOST_DEVICE 1024 +#define VDPA_DEVICE_TYPE_NET 0 +#define VDPA_DEVICE_TYPE_BLK 1 + struct rte_vdpa_device; /** @@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path); /** + * Get the device type of the vdpa device. + * + * @param path + * The vhost-user socket file path + * @param type + * the device type of the vdpa device + * @return + * 0 on success, -1 on failure + */ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index b304339..7da90e8 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -619,6 +619,45 @@ struct rte_vdpa_device * } int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type) +{ + struct vhost_user_socket *vsocket; + struct rte_vdpa_device *vdpa_dev; + uint32_t vdpa_type = 0; + int ret = 0; + + pthread_mutex_lock(&vhost_user.mutex); + vsocket = find_vhost_user_socket(path); + if (!vsocket) { + VHOST_LOG_CONFIG(ERR, + "(%s) socket file is not registered yet.\n", + path); + ret = -1; + goto unlock_exit; + } + + vdpa_dev = vsocket->vdpa_dev; + if (!vdpa_dev) { + ret = -1; + goto unlock_exit; + } + + if (vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type) < 0) { + VHOST_LOG_CONFIG(ERR, + "(%s) failed to get vdpa dev type for socket file.\n", + path); + ret = -1; + goto unlock_exit; + } + + *type = vdpa_type; + +unlock_exit: + pthread_mutex_unlock(&vhost_user.mutex); + return ret; +} + +int rte_vhost_driver_disable_features(const char *path, uint64_t features) { struct vhost_user_socket *vsocket; diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index e59a834..9cbd7cd 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops { /** Set the device configuration space */ int (*set_config)(int vid, uint8_t *config, uint32_t offset, uint32_t size, uint32_t flags); + + /** get device type: net device, blk device... */ + int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type); }; /** diff --git a/lib/vhost/version.map b/lib/vhost/version.map index 0a66c58..fe4e8de 100644 --- a/lib/vhost/version.map +++ b/lib/vhost/version.map @@ -87,6 +87,8 @@ EXPERIMENTAL { # added in 22.03 rte_vhost_async_dma_configure; + + rte_vhost_driver_get_vdpa_dev_type; }; INTERNAL { From patchwork Wed Apr 27 08:29:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110314 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29681A0507; Wed, 27 Apr 2022 11:17:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ECB9F4281A; Wed, 27 Apr 2022 11:16:52 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 60E5F42803 for ; Wed, 27 Apr 2022 11:16:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051008; x=1682587008; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=c+t775mbY3VJIc948L5SIbpJ2hBbs/9fOuwGRCIZ/d4=; b=GXF/jZEGXo4wQeNlfi9RkmYNBAHOsl6FLU1fXA4WytnCE6IanBHXcks0 7nU/PVg804QdDd4FhtQZnimqlaHr6oEzUpuRIT5eSRtLyKKrP8KzbsySH yhG2ZrJ0Vj4aT9wsiZosBAw092cOUiV+OUoqqTTzB0W7inXC1aOm7C/Ur 1O2ctYaxe3pW2GRYhqS71+chj4SX/LLQJKtvT8cKtdL4EjcsjxBepzNqL fAEqkDqXkWZFQQlAgc2lKpxaWobia8IR993sPzoObwKJ4IWLQDpr7QV4E Y8WUzZ8Xvmot9al1sEQPSw+3Ou7cMQKXaxhd1OCZ4nTAqBzqF+1aVeYcA g==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308296" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308296" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488725" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:36 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 08/18] vdpa/ifc: add get device type ops to ifc driver Date: Wed, 27 Apr 2022 16:29:56 +0800 Message-Id: <1651048206-282372-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add get device type ops to ifc driver. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 8a260b7..99a6ab0 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1300,6 +1300,15 @@ struct rte_vdpa_dev_info { return 0; } +static int +ifcvf_blk_get_device_type(struct rte_vdpa_device *vdev, + uint32_t *type) +{ + RTE_SET_USED(vdev); + *type = VDPA_DEVICE_TYPE_BLK; + return 0; +} + static struct rte_vdpa_dev_ops ifcvf_blk_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, @@ -1313,6 +1322,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_blk_get_device_type, }; struct rte_vdpa_dev_info dev_info[] = { From patchwork Wed Apr 27 08:29:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110315 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 63740A0507; Wed, 27 Apr 2022 11:17:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CA61642820; Wed, 27 Apr 2022 11:16:53 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id DF1BC4280A for ; Wed, 27 Apr 2022 11:16:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051009; x=1682587009; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=lnXQUW8kzhUocwn8F7VAhBXpnhr24stKZ4dW3p9+m2M=; b=a9Hh2VyQQYMxXv8nvWYnjW1isAfv29y4/cUj/MWjU+Ty7xgPsyrYaKad 03nORT++tcgdiYbP0ZOLKlnzx8MjtK/SPbODz+XWPb8jjJtccg2Cy8KST Nms+bZFJ0PPUpmLscpdX7pdGq947kMl62KndtnEkOTMauqrZl74w0QcYs E3mnkfbyGZQ8wd9+Eimd6kOzQ6KJD/RbzS7PXPEuTvtTB91i9Z6Dc3UGX HCd1ti+egIYjYlZ+H9E6cbOXHWJeLsP+olN9zHOVltOaxVxrdIeg1bklq oiBmMpoArFqGpZs3/C42oOai/1RorU5y6/KBoGMf86E0ETBqEFrmY9dLV Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308306" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308306" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488742" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:38 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 09/18] examples/vdpa: add vDPA blk support in example Date: Wed, 27 Apr 2022 16:29:57 +0800 Message-Id: <1651048206-282372-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to vDPA example. Signed-off-by: Andy Pei --- examples/vdpa/main.c | 57 +++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 65 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..2544141 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -159,8 +160,54 @@ struct vdpa_port { }; static int +vdpa_blk_device_set_features_and_protocol(const char *path) +{ + uint64_t protocol_features = 0; + int ret; + + ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_disable_features(path, + VHOST_BLK_DISABLED_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_disable_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_get_protocol_features(path, &protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |= VHOST_BLK_PROTOCOL_FEATURES; + + ret = rte_vhost_driver_set_protocol_features(path, protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + goto out; + } + +out: + return ret; +} + +static int start_vdpa(struct vdpa_port *vport) { + uint32_t device_type = 0; int ret; char *socket_path = vport->ifname; @@ -192,6 +239,16 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type); + if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) { + RTE_LOG(NOTICE, VDPA, "is a blk device\n"); + ret = vdpa_blk_device_set_features_and_protocol(socket_path); + if (ret != 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..136c3f6 --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,65 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** + * @file + * + * Device specific vhost lib + */ + +#include + +#include +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ +#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#ifndef VHOST_USER_F_PROTOCOL_FEATURES +#define VHOST_USER_F_PROTOCOL_FEATURES 30 +#endif + +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ From patchwork Wed Apr 27 08:29:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110316 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6BC33A0507; Wed, 27 Apr 2022 11:17:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A50D642824; Wed, 27 Apr 2022 11:16:54 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 541634280C for ; Wed, 27 Apr 2022 11:16:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051009; x=1682587009; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=1jsRjHf2H1JRmMIdLdkv2C6iAm7xMFkU5+GRJK/A/RI=; b=YH0MbZPaeZP2vqIeyIQDoGgdYkBCu67smqD1fWjEqIQDODOe9rs6Fnl/ aGlnt+VPZZEwEuxMKkPRDtq+0UpWNKQLJIbYtj/cfJUNGaREXgeB3ik2L cFmnOblbeT0dZ6USeKcZV5It+fGgsrlhBPY5pvyv+/MHBzfp8VZu9eQ20 qulwMjjiGqUnTj030htQZMS+BdqDtmPO9pKqjGa+OT9Q4Hlz2c3PNEm6j cQpjGWJWlyElerpKF/bdCyM4VwXJKwsxfamMwbAOIfP1VA1DqFWU4VtKX /yXVdHAB6CvMig4t4ev17lH3OWOh3RJKpr5w6k6hIO7A5hr4uJkjlIH1Q g==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308316" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308316" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488760" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:40 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 10/18] usertools: add support for virtio blk device Date: Wed, 27 Apr 2022 16:29:58 +0800 Message-Id: <1651048206-282372-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to devbind. Signed-off-by: Andy Pei --- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..7231be4 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties From patchwork Wed Apr 27 08:29:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110317 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8DEE1A0507; Wed, 27 Apr 2022 11:18:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 937DF42829; Wed, 27 Apr 2022 11:16:55 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 0A53F42800 for ; Wed, 27 Apr 2022 11:16:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051010; x=1682587010; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=otbhiVuAWK+g9FwMjCHtQgaAwjIyvpGd2CwjlqprWlg=; b=d4lUkJhbkxOi7UaSZIVoi8oHmy3Lcxa7MRIzBxTowf9iyZFNq73Hf2Gl iNyKEiNdRNmVQnN4M+rezNjw8EKZ073g3MYZ1Jo1noA7PHZBXCS7XRCpo iNFr2FhyY0qjFv2MVM+yrBl1l4mBkWIo/sgNUXor27BWcQKubsWo4qHDu 1tqWm2VvnqaIGGm9SDawNL1WseCujVwtK3OqzDGCi9ofYdipOHJxXEXSH EltBMQJjxi7N8w8zflvGGIXbOhCmjOcqvv9obKRacqcNdLic6cjM4/VVO RjO6bOYPPpBVq+oNfSOTE1hLaM/wDWpUAw39Nt2e2VgoN++pnE4SaJkQc Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308323" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308323" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488774" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:41 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 11/18] vdpa/ifc: add set vring state for blk device Date: Wed, 27 Apr 2022 16:29:59 +0800 Message-Id: <1651048206-282372-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Set_vring_state op is mandatory, add set_vring_state for blk device. Currently set_vring_state for blk device is not implemented. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 99a6ab0..ca49bc3 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1290,6 +1290,16 @@ struct rte_vdpa_dev_info { } static int +ifcvf_blk_set_vring_state(int vid, int vring, int state) +{ + RTE_SET_USED(vid); + RTE_SET_USED(vring); + RTE_SET_USED(state); + + return 0; +} + +static int ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1316,7 +1326,7 @@ struct rte_vdpa_dev_info { .get_protocol_features = ifcvf_blk_get_protocol_features, .dev_conf = ifcvf_dev_config, .dev_close = ifcvf_dev_close, - .set_vring_state = NULL, + .set_vring_state = ifcvf_blk_set_vring_state, .migration_done = NULL, .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, From patchwork Wed Apr 27 08:30:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110318 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3AEC6A0507; Wed, 27 Apr 2022 11:18:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6B4A64282F; Wed, 27 Apr 2022 11:16:56 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 5D0724280A for ; Wed, 27 Apr 2022 11:16:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051010; x=1682587010; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=8qnlOI6xpBdt6LkE6zuqyztxBuMYBW188SC8PjV9NXs=; b=HrAncBfi3auIgHTlpYYtJKTJ6SBKa4FFStKnemvtAEZKjiduK5ZJIG8b 0EiOVt2ZaGxiMSXPtKrBzaB1tjUfqZEAY7QXOaObPn/ICdZeCIt2vYglg lHnbYinW/Q89uk9+frjEvJMZ+xBxwR1tSJdbtWskv3NJRfgkMabBVwZFl /HN1gjT41BYxXpP+/iXY4XJ/qPMTdI5oNLbrfYwI3J/i7Q3WXDWXPl/8p d45gxOkeIQVbp/GThgne8v6L2cZ7mloOexZ9TBl9kPUgduvwIQgcQ6YuS 7tKMqKHRq8geQVE7Pdp8EIRloBRegKscTzBix/r2p9xNj3dn0CtsAQrm+ A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308330" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308330" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488787" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:43 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 12/18] vdpa/ifc: add some log at vDPA launch before qemu connect Date: Wed, 27 Apr 2022 16:30:00 +0800 Message-Id: <1651048206-282372-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add some log of virtio blk device config space information at VDPA launch before qemu connects. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index ca49bc3..4060a44 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1363,6 +1363,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + uint64_t capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1429,6 +1432,31 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /* cannot read 64-bit register in one attempt, + * so read byte by byte. + */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + DRV_LOG(INFO, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(INFO, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(INFO, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(INFO, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(INFO, "geometry"); + DRV_LOG(INFO, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(INFO, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(INFO, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(INFO, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Wed Apr 27 08:30:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110319 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 15DACA0507; Wed, 27 Apr 2022 11:18:14 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5E75942831; Wed, 27 Apr 2022 11:16:57 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 858B442811 for ; Wed, 27 Apr 2022 11:16:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051010; x=1682587010; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=H7zC3g6QI5iG2cuqXa+Be/n5UmBaiKellT4IsshGLmQ=; b=T0L9aF1NyfXYSKSxDh6BmOkhp5JDkVvdTFnca0bjceYLQX1sG+h95koI 3igX7pFzXQjiipmqUroYIe5RC+ixWdZkrxp2WQzTJmQk5XT66rxa3pENN XNdOlD2LxWmZSBYZ43n4IdHiCyTtGv4ozmc7hRzuaWvxtJlyqHWrKDVAj vFgzGDR+245fxqTlWIEWmfVLJ8vYw1jDNQO7Jcx6DTNZzF99xikkF6nqM 3v6xxEzmW49a1G2DN2yBvNiEo+9eKd3S0UD3Q9DcgEO/djhhzLGMNAFRF dYrgQTdBZhtIo7mhKpobASdn5HWDrrMPCp9O0M4VYcGE7ej1H2gvNBtqf A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308336" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308336" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488798" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:45 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 13/18] vdpa/ifc: read virtio max queues from hardware Date: Wed, 27 Apr 2022 16:30:01 +0800 Message-Id: <1651048206-282372-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Original code max_queues is set to IFCVF_MAX_QUEUES. New code max_queues is the min of IFCVF_MAX_QUEUES and hardware num_queues. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 4060a44..5a8cf1c 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1457,6 +1457,10 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(INFO, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + /* reset max_queue here, to minimum modification */ + internal->max_queues = RTE_MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Wed Apr 27 08:30:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110320 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A30AA0507; Wed, 27 Apr 2022 11:18:20 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3118842836; Wed, 27 Apr 2022 11:16:58 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id AC70942800 for ; Wed, 27 Apr 2022 11:16:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051010; x=1682587010; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=xTx0/0F6PeWG+gwi6ssUGnTSiDLGQWBG/leuVPF8Qyk=; b=VFiUsfs3qZk6RZE7HJrMFtf8g1K1AfEEX8XA1J2JsufvDXjTAple/MEd d01L4ttz1qC9QN+aiFn/B9b2+T46LLWDG9L5L6unaxB3jBJ9AlvlFnXD3 XB6nEd+zoB+iDD5/a3Tb66+BoqUzS/D6LWaimbrjEGZWZY5VkZxI4Ok8C fWwNXwo0J6AVNzCCMbuJtpL5Su7cinPvg+xClqK/8ecIxhJqpz2izLkfM SIPHsc3/KD8dLyVKF8JqZpb0JhwSj+o/sbB4xJQ4gO86sm3KhDVRTiZ81 a3ZpSn/6BH43A+GP8Ytk4k+gFJZbf+d7oT6ivDr0D5GCjwJnCSUQol5JE Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308369" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308369" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488810" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:46 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 14/18] vdpa/ifc: add interrupt and handle for virtio blk Date: Wed, 27 Apr 2022 16:30:02 +0800 Message-Id: <1651048206-282372-15-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to info qemu. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 112 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 5a8cf1c..0e94e1f 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for intr relay */ int epfd; + int csc_fd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -558,6 +560,107 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_fd, csc_val = 0; + + csc_fd = epoll_create(1); + if (csc_fd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_fd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + return NULL; + } + + internal->csc_fd = csc_fd; + + for (;;) { + csc_val = epoll_wait(csc_fd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail\n"); + return NULL; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || errno == EWOULDBLOCK) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + return NULL; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + int ret; + + ret = pthread_create(&internal->intr_tid, NULL, intr_relay, + (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static int +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_fd >= 0) + close(internal->csc_fd); + internal->csc_fd = -1; + + return 0; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -584,10 +687,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + ret = unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -880,6 +989,9 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); vdpa_ifcvf_stop(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Wed Apr 27 08:30:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110321 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1F9C1A0507; Wed, 27 Apr 2022 11:18:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6756B4283B; Wed, 27 Apr 2022 11:16:59 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 013BF4280A for ; Wed, 27 Apr 2022 11:16:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051011; x=1682587011; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=ACfWn5tUZ7Z4JKJdkl3nvKs9VP43EVkZX0kAHnn4V4Q=; b=NilxzUd+RV/3QKavLcKPx/IOFWjXAiBZkRvF8++Z9Qk6AuHuSKvjHx8w taCseG5EAvSv8XaEPhYnwkfXHgjmwAnz122knJmeN7EKKaPryAKY0uFT/ 4G26YS1XExb2NzaOPhX4ydMf4l93vOwHXH6qF3Njh2Nl6IattO/366Sbz 4xwY+Jwe7npGChyTCeNvjuKmeITeosF2TOf2cZIpUxzYpDNmZ0tZYpSye /MbQZ2ndxtENYdn74DdC/VEzjCuy1EExsKZw/dHxPqSRSS+DmzUNGG2X4 HPdOear/FK1bCe3+6ba8WK3oIq/je2N7UVV7bnku/+sD8tqGbQ96lVIWF A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308378" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308378" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488822" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:48 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 15/18] vdpa/ifc: add is blk flag to ifcvf HW struct Date: Wed, 27 Apr 2022 16:30:03 +0800 Message-Id: <1651048206-282372-16-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add is_blk flag to ifcvf_hw, and init is_blk during probe. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 769c603..8591ef1 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -149,6 +149,7 @@ struct ifcvf_hw { u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + u8 is_blk; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 0e94e1f..4923bc1 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1536,11 +1536,13 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->device_type = IFCVF_NET; + internal->hw.is_blk = IFCVF_NET; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; } else if (device_id == VIRTIO_ID_BLOCK) { internal->device_type = IFCVF_BLK; + internal->hw.is_blk = IFCVF_BLK; internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; From patchwork Wed Apr 27 08:30:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110322 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 75983A0507; Wed, 27 Apr 2022 11:18:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5545342840; Wed, 27 Apr 2022 11:17:00 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 3257842817 for ; Wed, 27 Apr 2022 11:16:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051012; x=1682587012; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=kZQC3vJ8B6ytymPnXxwjq3Dl6wWYgPYKjxn8zJbZ820=; b=gxk49vDzpQ+LzY1BMNc7S3l+RN8rw45ZkkOTAvaCOX92ALZRC6mYPnPI OWISAAPtJmQp2SZ9HyHFjLmOHLHKeD5QMOXeHflhafOMgBaRNhPuIUxBz LTWCg/wqNodVLB/AX3s4SxbCWkLOeul0L/tkeg4f86vEmvkq2BpqDzNZr A5xR7CX4UJgnOvrQg+OfFFRAhzAe5cBJ5enFJiThnzcgUXQnUfMKSsM4N A6J0ft29MJwKwpdQVu4nlhPYsKDVELGEnUzuMvqRLtdcqk41CMaNEh9Hr sCu2gegL28RiMvJOMDRAGoyA2uRMre0flM6ynauG8jrYNPaILcA3zEJIX Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308389" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308389" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488835" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:50 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 16/18] vdpa/ifc/base: access correct register for blk device Date: Wed, 27 Apr 2022 16:30:04 +0800 Message-Id: <1651048206-282372-17-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 1.last_avail_idx is lower 16 bit of the register. 2.address of ring_state register is different between net and blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 36 +++++++++++++++++++++++++++++------- drivers/vdpa/ifc/base/ifcvf.h | 1 + 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..4d5881a 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -218,10 +218,18 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->is_blk == IFCVF_BLK) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } else if (hw->is_blk == IFCVF_NET) { + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +262,23 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->is_blk) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + } else if (hw->is_blk == IFCVF_NET) { + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + } + + if (hw->is_blk == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else if (hw->is_blk == IFCVF_NET) + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } } diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 8591ef1..ff11b12 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG #define VHOST_USER_PROTOCOL_F_CONFIG 9 From patchwork Wed Apr 27 08:30:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110323 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1F9C0A0507; Wed, 27 Apr 2022 11:18:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3393E42844; Wed, 27 Apr 2022 11:17:01 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id C4CB24281E for ; Wed, 27 Apr 2022 11:16:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051014; x=1682587014; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=1y6KqWowvtH1fe0AsCdYRP4lfrjY0EivF0010G/JchA=; b=CUzqvQkHxQvlQjyAshknIlP7ii7VKtaRTkKSWsKbjICc39y2NbmeJ0Xv 1e1SZOINTV3dQDCx+js5xkZ//d63cebUvBUi+NPgD459nj7NXd4y1NBtC onRxYAiCZTRoAkAaxr++n8yP/RQBvL+6U0phNbwuEuZnN8/v3Y0dHfE7x T1kgF00aNmELidCxoyha+uEXdDskPnrx5I9ANM8ptCJSjP8d/r9Mh3lTj rdDIrb/iExebeDhKEiW7dQX1qaghLEN/V73MoTg6qhCi0oKpbuh5UVZHx 4HTPpJzNaHctRltUn/QOK13W3l2ehF0Zyjadz/4hz+PpD5TErVqpwlB+z A==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308392" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308392" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488847" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:51 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 17/18] vdpa/ifc: blk device pause without no inflight IO Date: Wed, 27 Apr 2022 16:30:05 +0800 Message-Id: <1651048206-282372-18-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 4923bc1..def6adf 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -314,12 +314,12 @@ struct rte_vdpa_dev_info { vdpa_ifcvf_stop(struct ifcvf_internal *internal) { struct ifcvf_hw *hw = &internal->hw; - struct rte_vhost_vring vq; uint32_t i; int vid; uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; @@ -328,13 +328,22 @@ struct rte_vdpa_dev_info { */ if (internal->device_type == IFCVF_BLK) { for (i = 0; i < hw->nr_vring; i++) { - rte_vhost_get_vhost_vring(internal->vid, i, &vq); - while (vq.avail->idx != vq.used->idx) { - ifcvf_notify_queue(hw, i); - usleep(10); - } - hw->vring[i].last_avail_idx = vq.avail->idx; - hw->vring[i].last_used_idx = vq.used->idx; + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); } } From patchwork Wed Apr 27 08:30:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 110324 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 29C94A0507; Wed, 27 Apr 2022 11:18:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C50E642854; Wed, 27 Apr 2022 11:17:02 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id C05434282B for ; Wed, 27 Apr 2022 11:16:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651051016; x=1682587016; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=DAEqCRiQsRunsVdQAkYspDpyNW7f+GNlGE7tCFXt0RY=; b=lululJK5HbvvKkNPTGLh9V/gzsmYGpH/DBzQYW8cwqG8IbEW4CelJ6Bt Oo5vTXT3ktKLWV58YtstITt0PJWCEM+l7aWpxw9I3n0JoKUAH25/nzPpJ EyoqeTgsKNy83Iv8okhHjPOpvsjJuzogaM4LIul4MTM4wsU1srcrw0a9M AmzM68a2nfszYR5EOc/h1KZAsYA+nnJ2N9IO3tHwMqCZ9YBWMtFV/QoQX npOqjT3dFwVKAm1wIR5z3MnaKz1Lju0pk2PuQJx9UZv3F9h/14xPEGwDT WgUOQuLQIusLN1xk5PSb4V/jrYJCFd3IQunkjlCIVgHYnkUqS2rBMSbDt g==; X-IronPort-AV: E=McAfee;i="6400,9594,10329"; a="352308400" X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="352308400" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Apr 2022 02:16:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,292,1643702400"; d="scan'208";a="580488855" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga008.jf.intel.com with ESMTP; 27 Apr 2022 02:16:53 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com Subject: [PATCH v7 18/18] vhost: make sure each queue callfd is configured Date: Wed, 27 Apr 2022 16:30:06 +0800 Message-Id: <1651048206-282372-19-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1651048206-282372-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1651048206-282372-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org During the vhost data path building process, qemu will create a call fd at first, and create another call fd in the end. The final call fd will be used to relay notify. In the original code, after kick fd is set, dev_conf will set the first call fd. Even though the actual call fd will set, the data path will not work correctly. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index e925428..82122b6 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -3230,12 +3230,26 @@ typedef int (*vhost_message_handler_t)(struct virtio_net **pdev, if (!vdpa_dev) goto out; + if (request != VHOST_USER_SET_VRING_CALL) + goto out; + if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid)) VHOST_LOG_CONFIG(ERR, "(%s) failed to configure vDPA device\n", dev->ifname); else dev->flags |= VIRTIO_DEV_VDPA_CONFIGURED; + } else { + /* when VIRTIO_DEV_VDPA_CONFIGURED already configured + * close the device and config the device again, + * make sure the call fd of each queue is configured correctly. + */ + if (vdpa_dev->ops->dev_close(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to close vDPA device\n"); + if (vdpa_dev->ops->dev_conf(dev->vid)) + VHOST_LOG_CONFIG(ERR, + "Failed to re-config vDPA device\n"); } out: