From patchwork Wed May 18 12:13:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111309 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 70E4CA0503; Wed, 18 May 2022 15:00:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AF39E42684; Wed, 18 May 2022 15:00:32 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 1C52B42684 for ; Wed, 18 May 2022 15:00:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878830; x=1684414830; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=xxQ6X5gdopucqL9pfUEay3j+uOJN8HLwV1O6BL9XOU0=; b=n2TVIhC51SWFTuV377V7fSmMsFaNPexNBKcV2Si4K/bNjjAjZhNbAbG9 gMQFwj4wVEqPlaymlXvJvH88DGIKitmX2HJqv807K31s2iWLAqi7dbeYe LLCXmwvRYMFEBf/SWKClZduI44GJfi82NF+tr0Hzwsv+BGDsKSwGF5/ii R3k0LbumgNi6wdF+6t/YgNrH27jSc01SGIY31aSu6eAJAj2O1oJQJorIv 5UWBvEyKPvJ6HvX+KP2eIppfo8EPfNaR2DsMbV/3axfuz78Q0q7mHZDYy FFE/lFk+7COlt+3THGS2bioIB3mb54DCjtw7oIxj7+9z32g0oUDfWp36t g==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353537" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353537" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312093" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:27 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 01/13] vdpa/ifc: add support for virtio blk device Date: Wed, 18 May 2022 20:13:43 +0800 Message-Id: <1652876035-70513-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Re-use the vdpa/ifc code, distinguish blk and net device by pci_device_id. Blk and net device are implemented with proper feature and ops. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 16 +++++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 91 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 98 insertions(+), 9 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 573a35f..483d38b 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -5,8 +5,17 @@ #ifndef _IFCVF_H_ #define _IFCVF_H_ +#include #include "ifcvf_osdep.h" +#define IFCVF_NET 0 +#define IFCVF_BLK 1 + +/* for BLK */ +#define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 +#define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_BLK_DEVICE_ID 0x0002 + #define IFCVF_VENDOR_ID 0x1AF4 #define IFCVF_DEVICE_ID 0x1041 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 @@ -126,13 +135,18 @@ struct ifcvf_hw { u8 notify_region; u32 notify_off_multiplier; struct ifcvf_pci_common_cfg *common_cfg; - struct ifcvf_net_config *dev_cfg; + union { + struct ifcvf_net_config *net_cfg; + struct virtio_blk_config *blk_cfg; + void *dev_cfg; + }; u8 *isr; u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; u8 *lm_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; + int device_type; struct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE]; }; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9f05595..be0efd3 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -75,6 +75,12 @@ struct internal_list { struct ifcvf_internal *internal; }; +/* vdpa device info includes device features and devcic operation. */ +struct rte_vdpa_dev_info { + uint64_t features; + struct rte_vdpa_dev_ops *ops; +}; + TAILQ_HEAD(internal_list_head, internal_list); static struct internal_list_head internal_list = TAILQ_HEAD_INITIALIZER(internal_list); @@ -1167,6 +1173,48 @@ struct internal_list { return 0; } +static int16_t +ifcvf_pci_get_device_type(struct rte_pci_device *pci_dev) +{ + uint16_t pci_device_id = pci_dev->id.device_id; + uint16_t device_id; + + if (pci_device_id < 0x1000 || pci_device_id > 0x107f) { + DRV_LOG(ERR, "Probe device is not a virtio device\n"); + return -1; + } + + if (pci_device_id < 0x1040) { + /* Transitional devices: use the PCI subsystem device id as + * virtio device id, same as legacy driver always did. + */ + device_id = pci_dev->id.subsystem_device_id; + } else { + /* Modern devices: simply use PCI device id, + * but start from 0x1040. + */ + device_id = pci_device_id - 0x1040; + } + + return device_id; +} + +struct rte_vdpa_dev_info dev_info[] = { + { + .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | + (1ULL << VIRTIO_NET_F_CTRL_VQ) | + (1ULL << VIRTIO_NET_F_STATUS) | + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = &ifcvf_ops, + }, + { + .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | + (1ULL << VHOST_F_LOG_ALL), + .ops = NULL, + }, +}; + static int ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev) @@ -1178,6 +1226,7 @@ struct internal_list { int sw_fallback_lm = 0; struct rte_kvargs *kvlist = NULL; int ret = 0; + int16_t device_id; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1227,13 +1276,24 @@ struct internal_list { internal->configured = 0; internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); - internal->features = (features & - ~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) | - (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | - (1ULL << VIRTIO_NET_F_CTRL_VQ) | - (1ULL << VIRTIO_NET_F_STATUS) | - (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | - (1ULL << VHOST_F_LOG_ALL); + + device_id = ifcvf_pci_get_device_type(pci_dev); + if (device_id < 0) { + DRV_LOG(ERR, "failed to get device %s type", pci_dev->name); + goto error; + } + + if (device_id == VIRTIO_ID_NET) { + internal->hw.device_type = IFCVF_NET; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_NET].features; + } else if (device_id == VIRTIO_ID_BLOCK) { + internal->hw.device_type = IFCVF_BLK; + internal->features = features & + ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); + internal->features |= dev_info[IFCVF_BLK].features; + } list->internal = internal; @@ -1245,7 +1305,8 @@ struct internal_list { } internal->sw_lm = sw_fallback_lm; - internal->vdev = rte_vdpa_register_device(&pci_dev->device, &ifcvf_ops); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, + dev_info[internal->hw.device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); goto error; @@ -1313,6 +1374,20 @@ struct internal_list { .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_BLK_MODERN_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + }, + { .vendor_id = 0, /* sentinel */ }, }; From patchwork Wed May 18 12:13:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111310 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8BD9EA0503; Wed, 18 May 2022 15:00:44 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DED0D42B76; Wed, 18 May 2022 15:00:33 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 5118242684 for ; Wed, 18 May 2022 15:00:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878831; x=1684414831; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=lW5isKc3/9enixxJrZo0+rHrLl9944fbMj0sPaJaFOE=; b=YwrsWLVkyt4wAjUJGiPE4PnFM0D+Dz2s9az+ZlvLrYPJmCrdEysX7ftq UhV26+munEuTwA33IkkzNj0RhpQjyeFo0Ie7MNNn2Q7HP5/tfpAwWM0E1 MTIekisXEBFwFnttbM/vsLiXSMU8WJBpkzaeLOKWMXExEr7Njn1h8svYa HVwVWdWOmJ6YZ0asufbd6oRPXrXN9WJ7YyjgeS+l30Gbtc67WPIvzKgdA KCC2X4rTHlBBzMY/L55Dz7ioRPJs5ao7wp9BW6D6Chywfp2/DAfKQO4/N sYr2Gy0Y62VZJu66hYqCVZrx+2g0KGFB3mV5zhVcn5TwldDbUX8i0Sh+Y A==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353559" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353559" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312103" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:29 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 02/13] vhost: add vDPA ops for blk device Date: Wed, 18 May 2022 20:13:44 +0800 Message-Id: <1652876035-70513-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Get_config and set_config are necessary ops for blk device. Add get_config and set_config ops to vDPA ops. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vdpa_driver.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 88138be..e59a834 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -72,8 +72,12 @@ struct rte_vdpa_dev_ops { /** Reset statistics of the queue */ int (*reset_stats)(struct rte_vdpa_device *dev, int qid); - /** Reserved for future extension */ - void *reserved[2]; + /** Get the device configuration space */ + int (*get_config)(int vid, uint8_t *config, uint32_t len); + + /** Set the device configuration space */ + int (*set_config)(int vid, uint8_t *config, uint32_t offset, + uint32_t size, uint32_t flags); }; /** From patchwork Wed May 18 12:13:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111311 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6B02AA0503; Wed, 18 May 2022 15:00:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D71AA42B7C; Wed, 18 May 2022 15:00:36 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id E670642B7B for ; Wed, 18 May 2022 15:00:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878835; x=1684414835; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=06RayleQ0P7xyHqysiCVifyjN4TmhTcWe/KoT+KGIB8=; b=dBIYdKMwNLRxzmL0v6/15HOsR3/d3OEzErTb4gIjAqjUPQYO6TqMVs5B Nyb5s9iDJiR/BaSClFWaovsNcxxGcLDIy54viSAU8wAgiFQlpAiOd5t67 yId/OOjirBANoVxU6OS2BS3mlzxRmwZ/1IF8ODK071R2R3cRrObQW+s1X 4yhW7O/r58YMXE4RYfVcCGgB2Qm/2QpyIOwU0zHGyBTVTuxoz14nMi8J5 7Wg7ETjNDw/NDeTBreMgDbO1/gKBSTOa5JpypLFqfvbVtgbLO5m5mTuwP aa5UoE5bPWM5y4kpFNaAnHOe6EMBNZbL833bPSeh6w1S4PITlwPhMvtAu Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353582" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353582" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312128" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:31 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 03/13] vhost: add vhost msg support for get/set config Date: Wed, 18 May 2022 20:13:45 +0800 Message-Id: <1652876035-70513-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add support for VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG. VHOST_USER_GET_CONFIG and VHOST_USER_SET_CONFIG message is only supported by virtio blk VDPA device. Signed-off-by: Andy Pei --- lib/vhost/vhost_user.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/vhost/vhost_user.h | 13 +++++++++ 2 files changed, 86 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 850848c..1dd1e25f 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2468,6 +2468,77 @@ static int is_vring_iotlb(struct virtio_net *dev, } static int +vhost_user_get_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (vdpa_dev->ops->get_config) { + ret = vdpa_dev->ops->get_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.size); + if (ret != 0) { + ctx->msg.size = 0; + VHOST_LOG_CONFIG(ERR, + "(%s) get_config() return error!\n", + dev->ifname); + } + } else { + VHOST_LOG_CONFIG(ERR, "(%s) get_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_REPLY; +} + +static int +vhost_user_set_config(struct virtio_net **pdev, + struct vhu_msg_context *ctx, + int main_fd __rte_unused) +{ + struct virtio_net *dev = *pdev; + struct rte_vdpa_device *vdpa_dev = dev->vdpa_dev; + int ret = 0; + + if (validate_msg_fds(dev, ctx, 0) != 0) + return RTE_VHOST_MSG_RESULT_ERR; + + if (ctx->msg.payload.cfg.size > VHOST_USER_MAX_CONFIG_SIZE) { + VHOST_LOG_CONFIG(ERR, + "(%s) vhost_user_config size: %"PRIu32", should not be larger than %d\n", + dev->ifname, ctx->msg.payload.cfg.size, + VHOST_USER_MAX_CONFIG_SIZE); + goto out; + } + + if (vdpa_dev->ops->set_config) { + ret = vdpa_dev->ops->set_config(dev->vid, + ctx->msg.payload.cfg.region, + ctx->msg.payload.cfg.offset, + ctx->msg.payload.cfg.size, + ctx->msg.payload.cfg.flags); + if (ret) + VHOST_LOG_CONFIG(ERR, + "(%s) set_config() return error!\n", + dev->ifname); + } else { + VHOST_LOG_CONFIG(ERR, "(%s) set_config() not supported!\n", + dev->ifname); + } + + return RTE_VHOST_MSG_RESULT_OK; + +out: + return RTE_VHOST_MSG_RESULT_ERR; +} + +static int vhost_user_iotlb_msg(struct virtio_net **pdev, struct vhu_msg_context *ctx, int main_fd __rte_unused) @@ -2686,6 +2757,8 @@ static int is_vring_iotlb(struct virtio_net *dev, VHOST_MESSAGE_HANDLER(VHOST_USER_NET_SET_MTU, vhost_user_net_set_mtu, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_SET_SLAVE_REQ_FD, vhost_user_set_req_fd, true) \ VHOST_MESSAGE_HANDLER(VHOST_USER_IOTLB_MSG, vhost_user_iotlb_msg, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_GET_CONFIG, vhost_user_get_config, false) \ +VHOST_MESSAGE_HANDLER(VHOST_USER_SET_CONFIG, vhost_user_set_config, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_ADVISE, vhost_user_set_postcopy_advise, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_LISTEN, vhost_user_set_postcopy_listen, false) \ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END, vhost_user_postcopy_end, false) \ diff --git a/lib/vhost/vhost_user.h b/lib/vhost/vhost_user.h index ba1c5c7..c4d091e 100644 --- a/lib/vhost/vhost_user.h +++ b/lib/vhost/vhost_user.h @@ -50,6 +50,8 @@ VHOST_USER_NET_SET_MTU = 20, VHOST_USER_SET_SLAVE_REQ_FD = 21, VHOST_USER_IOTLB_MSG = 22, + VHOST_USER_GET_CONFIG = 24, + VHOST_USER_SET_CONFIG = 25, VHOST_USER_CRYPTO_CREATE_SESS = 26, VHOST_USER_CRYPTO_CLOSE_SESS = 27, VHOST_USER_POSTCOPY_ADVISE = 28, @@ -123,6 +125,16 @@ uint16_t queue_size; } VhostUserInflight; +#define VHOST_USER_MAX_CONFIG_SIZE 256 + +/** Get/set config msg payload */ +struct vhost_user_config { + uint32_t offset; + uint32_t size; + uint32_t flags; + uint8_t region[VHOST_USER_MAX_CONFIG_SIZE]; +}; + typedef struct VhostUserMsg { union { uint32_t master; /* a VhostUserRequest value */ @@ -146,6 +158,7 @@ VhostUserCryptoSessionParam crypto_session; VhostUserVringArea area; VhostUserInflight inflight; + struct vhost_user_config cfg; } payload; /* Nothing should be added after the payload */ } __rte_packed VhostUserMsg; From patchwork Wed May 18 12:13:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111312 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4F0FCA0503; Wed, 18 May 2022 15:01:01 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C8A8B42B81; Wed, 18 May 2022 15:00:37 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 375C042B70 for ; Wed, 18 May 2022 15:00:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878836; x=1684414836; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=PyGbHOi2ZLaPYd+E0RhqWbVaUMUWyRfxCfTFrX41iLk=; b=IoYyco9LaZux1+BwYS/0jbwbqltcNJ4cZbOJBKtUKrIVIZ4uqe/k9+F1 dyE128IDexuVvYBgO8PxHe1lbXUWEB8tWBDEO/qC0AkfNrdC4cv5ExhsC kDSTfVw/smgI+WE5Sb5gIrgUK1CAnegOKXtJHRMWC/FyoNHbVEgAdnSrs bsfm1lUX3Og2OhxSADThAg13z/AVhr7QliU9i696DVb6CsZV00gCGwnB5 hjLJEVGPaCiL0k5JcLnXSy0grXLS7mzs6qMxOJ6+XGtCUKpArqxrFa4SC bcMAD1t+r2e9B3Y5vIN7Y58A+OT/Wq9cZQ5OShnUZLPsrQrHJgjhrhp2U w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353603" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353603" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312171" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:33 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 04/13] vdpa/ifc: add blk ops for ifc device Date: Wed, 18 May 2022 20:13:46 +0800 Message-Id: <1652876035-70513-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For virtio blk device, re-use part of ifc driver ops. Implement ifcvf_blk_get_config for virtio blk device. Support VHOST_USER_PROTOCOL_F_CONFIG feature for virtio blk device. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 4 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 483d38b..244de46 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -67,6 +67,10 @@ #define IFCVF_32_BIT_MASK 0xffffffff +#ifndef VHOST_USER_PROTOCOL_F_CONFIG +#define VHOST_USER_PROTOCOL_F_CONFIG 9 +#endif + struct ifcvf_pci_cap { u8 cap_vndr; /* Generic PCI field: PCI_CAP_ID_VNDR */ u8 cap_next; /* Generic PCI field: next ptr. */ diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index be0efd3..350214a 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1087,6 +1087,10 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) + +#define VDPA_BLK_PROTOCOL_FEATURES \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG) + static int ifcvf_get_protocol_features(struct rte_vdpa_device *vdev, uint64_t *features) { @@ -1199,6 +1203,91 @@ struct rte_vdpa_dev_info { return device_id; } +static int +ifcvf_blk_get_config(int vid, uint8_t *config, uint32_t len) +{ + struct virtio_blk_config *dev_cfg; + struct ifcvf_internal *internal; + struct rte_vdpa_device *vdev; + struct internal_list *list; + uint32_t i; + uint64_t capacity = 0; + uint8_t *byte; + + if (len < sizeof(struct virtio_blk_config)) { + DRV_LOG(ERR, "Invalid len: %u, required: %u", + len, (uint32_t)sizeof(struct virtio_blk_config)); + return -1; + } + + vdev = rte_vhost_get_vdpa_device(vid); + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + for (i = 0; i < sizeof(struct virtio_blk_config); i++) + config[i] = *((u8 *)internal->hw.blk_cfg + i); + + dev_cfg = (struct virtio_blk_config *)internal->hw.blk_cfg; + + /* cannot read 64-bit register in one attempt, so read byte by byte. */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + /* The capacity is number of sectors in 512-byte. + * So right shift 1 bit we get in K, + * another right shift 10 bits we get in M, + * right shift 10 more bits, we get in G. + * To show capacity in G, we right shift 21 bits in total. + */ + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(DEBUG, "size_max : 0x%08x", dev_cfg->size_max); + DRV_LOG(DEBUG, "seg_max : 0x%08x", dev_cfg->seg_max); + DRV_LOG(DEBUG, "blk_size : 0x%08x", dev_cfg->blk_size); + DRV_LOG(DEBUG, "geometry"); + DRV_LOG(DEBUG, " cylinders: %u", dev_cfg->geometry.cylinders); + DRV_LOG(DEBUG, " heads : %u", dev_cfg->geometry.heads); + DRV_LOG(DEBUG, " sectors : %u", dev_cfg->geometry.sectors); + DRV_LOG(DEBUG, "num_queues: 0x%08x", dev_cfg->num_queues); + + DRV_LOG(DEBUG, "config: [%x] [%x] [%x] [%x] [%x] [%x] [%x] [%x]\n", + config[0], config[1], config[2], config[3], config[4], + config[5], config[6], config[7]); + return 0; +} + +static int +ifcvf_blk_get_protocol_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + RTE_SET_USED(vdev); + + *features = VDPA_SUPPORTED_PROTOCOL_FEATURES; + *features |= VDPA_BLK_PROTOCOL_FEATURES; + return 0; +} + +static struct rte_vdpa_dev_ops ifcvf_blk_ops = { + .get_queue_num = ifcvf_get_queue_num, + .get_features = ifcvf_get_vdpa_features, + .set_features = ifcvf_set_features, + .get_protocol_features = ifcvf_blk_get_protocol_features, + .dev_conf = ifcvf_dev_config, + .dev_close = ifcvf_dev_close, + .set_vring_state = ifcvf_set_vring_state, + .migration_done = NULL, + .get_vfio_group_fd = ifcvf_get_vfio_group_fd, + .get_vfio_device_fd = ifcvf_get_vfio_device_fd, + .get_notify_area = ifcvf_get_notify_area, + .get_config = ifcvf_blk_get_config, +}; + struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | @@ -1211,7 +1300,7 @@ struct rte_vdpa_dev_info dev_info[] = { { .features = (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | (1ULL << VHOST_F_LOG_ALL), - .ops = NULL, + .ops = &ifcvf_blk_ops, }, }; From patchwork Wed May 18 12:13:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111313 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2BC19A0503; Wed, 18 May 2022 15:01:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A40F542B6F; Wed, 18 May 2022 15:00:39 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 5FDD142B6E for ; Wed, 18 May 2022 15:00:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878838; x=1684414838; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2psPMGbtrAxXB7QXkGsn33hx0gaygu6MzUlp4kIFvyQ=; b=HcdT79SPl1OiLuBRIz0YYIHjQQuKxZVfnEy+lFlQhaI3Xrbx5zQvB4Ew 7KVUiTEFXynkuIR36iKPk87Tdi9hHhRaxJrXfX7v5pwGzo9ey6tjqWcBQ sV69kUjV/zI9abhbj8O6HQ/wKFk1j1iId0PI2t/OzzzQ5Hk2J8u2abSZH 0uYsi6M4841hSjexjDEDR9SjnDmZPuGp8rkvkwXz8A3r9rxuJJei8+9ak Y/5Y7a9sJFTBfDd8aDYBeVsvjblm5QthktE0aKVJIpqAtu+n3khv4zTsJ J1XkjLpTM67/Xe9Kk2xsPNOkp3K44+W8bw7Nrmegq56ZqhlfJ2QcDc59K w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353633" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353633" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312228" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:35 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 05/13] vdpa/ifc: add vDPA interrupt relay for blk device Date: Wed, 18 May 2022 20:13:47 +0800 Message-Id: <1652876035-70513-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org For the net device type, only interrupt of rxq needed to be relayed. But for block, since all the queues are used for both read and write requests. Interrupt of all queues needed to be relayed. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 350214a..509a1ed 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -369,6 +369,7 @@ struct rte_vdpa_dev_info { irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; irq_set->start = 0; fd_ptr = (int *)&irq_set->data; + /* The first interrupt is for the configure space change notification */ fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = rte_intr_fd_get(internal->pdev->intr_handle); @@ -378,7 +379,13 @@ struct rte_vdpa_dev_info { for (i = 0; i < nr_vring; i++) { rte_vhost_get_vhost_vring(internal->vid, i, &vring); fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; - if ((i & 1) == 0 && m_rx == true) { + if (m_rx == true && + ((i & 1) == 0 || internal->hw.device_type == IFCVF_BLK)) { + /* For the net we only need to relay rx queue, + * which will change the mem of VM. + * For the blk we need to relay all the read cmd + * of each queue + */ fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC); if (fd < 0) { DRV_LOG(ERR, "can't setup eventfd: %s", From patchwork Wed May 18 12:13:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111314 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B6E86A0503; Wed, 18 May 2022 15:01:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A06BF42B8E; Wed, 18 May 2022 15:00:42 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 7D9C042B8E for ; Wed, 18 May 2022 15:00:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878840; x=1684414840; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=sz9onOMaUbJvEFJ/7VpSN4b2i/5Xag2+m6sgSvsvpxc=; b=MrjPTEm8Su7rBO06DQWqy9xWxrK8FPnlMynUQptQCpBdU8fubyKtgVNa 1rvS8UjxT4GKMp0gnLpOtK+n34Tx0lNC+oxT4ZC9dbj05+i8pthKkY7xP iiA8/vHNM8scFp6XS9veYtSI99SW2+O9zTgQ5xBnnRjNmCMb/fLm+Y8+g C0lKI0RQIQOX9Sg6s4Z5SoZVuQ/n2AAULwHIzzflJ3jb3TUzd6n93FbII 6dfVVWkfd962Lu+1M0tZ4oZTU8NdTZ2wWl4Z8DHeGA/TSsc3B90qZ0GIL m1orDczQBbujqeKSAF037wGNlHDxJdtNGfLq5gqiGp2PHL+080SqWx1Va w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353664" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353664" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312273" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:37 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 06/13] vdpa/ifc: add block device SW live-migration Date: Wed, 18 May 2022 20:13:48 +0800 Message-Id: <1652876035-70513-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add SW live-migration support to block device. For block device, it is critical that no packet should be dropped. So when virtio blk device is pause, make sure hardware last_avail_idx and last_used_idx are the same. This indicates all requests have recieved acks, and no inflight IO. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 244de46..4fb1736 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -65,6 +65,7 @@ #define IFCVF_MEDIATED_VRING 0x200000000000 #define IFCVF_32_BIT_MASK 0xffffffff +#define IFCVF_16_BIT_MASK 0xffff #ifndef VHOST_USER_PROTOCOL_F_CONFIG diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 509a1ed..3e78c7d 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -316,8 +316,34 @@ struct rte_vdpa_dev_info { uint64_t features = 0; uint64_t log_base = 0, log_size = 0; uint64_t len; + u32 ring_state = 0; vid = internal->vid; + + /* to make sure no packet is lost for blk device + * do not stop until last_avail_idx == last_used_idx + */ + if (internal->hw.device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + if (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx) { + ifcvf_notify_queue(hw, i); + usleep(10); + } + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); + } + } + ifcvf_stop_hw(hw); for (i = 0; i < hw->nr_vring; i++) @@ -641,8 +667,10 @@ struct rte_vdpa_dev_info { } hw->vring[i].avail = gpa; - /* Direct I/O for Tx queue, relay for Rx queue */ - if (i & 1) { + /* NET: Direct I/O for Tx queue, relay for Rx queue + * BLK: relay every queue + */ + if ((internal->hw.device_type == IFCVF_NET) && (i & 1)) { gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); if (gpa == 0) { DRV_LOG(ERR, "Fail to get GPA for used ring."); @@ -692,8 +720,12 @@ struct rte_vdpa_dev_info { for (i = 0; i < hw->nr_vring; i++) { /* synchronize remaining new used entries if any */ - if ((i & 1) == 0) + if (internal->hw.device_type == IFCVF_NET) { + if ((i & 1) == 0) + update_used_ring(internal, i); + } else if (internal->hw.device_type == IFCVF_BLK) { update_used_ring(internal, i); + } rte_vhost_get_vhost_vring(vid, i, &vq); len = IFCVF_USED_RING_LEN(vq.size); @@ -755,7 +787,9 @@ struct rte_vdpa_dev_info { } } - for (qid = 0; qid < q_num; qid += 2) { + for (qid = 0; qid < q_num; qid += 1) { + if ((internal->hw.device_type == IFCVF_NET) && (qid & 1)) + continue; ev.events = EPOLLIN | EPOLLPRI; /* leave a flag to mark it's for interrupt */ ev.data.u64 = 1 | qid << 1 | From patchwork Wed May 18 12:13:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111315 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10082A0503; Wed, 18 May 2022 15:01:28 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8A97C42B97; Wed, 18 May 2022 15:00:44 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id B8E9942B91 for ; Wed, 18 May 2022 15:00:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878843; x=1684414843; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QFPbsJBbq2v2K0YawsaFWr8vP8AKfJYtSK8Insoxb/8=; b=GQVtgtCM4DlOXRFdlyaxsPKUgty3FQQN/lk+QUSLMglnTl7/WmI20D0P pcLo/hmoe0QOFd34biGjHKKXN48Wc352m7b3UbCaU6QerZH0h5p2cXks2 AvrbGEG+Pma7r+PXbC9/0KvPQbT0Bef7rmTyh7yek++fBd0Wi89Rz/lch g8mPC6w7Zm5FyIxvL881OcINuCyhOMxVkTpgf21miL2jWsmafjG/qQwGc j2mUwm97LlNa8M+R8WTEUXmX4EczmFl92zP2MavRBJNiBeQpK8IocC45/ Az/QPYZuT0D+UayAiHgYzdrI2BiTkhii2mobj4C8pzz56wJw7t5VuQeAo Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353699" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353699" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312312" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:40 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 07/13] vhost: add API to get vDPA device type Date: Wed, 18 May 2022 20:13:49 +0800 Message-Id: <1652876035-70513-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Vhost backend of different devices have different features. Add a API to get vDPA device type, net device or blk device currently, so users can set different features for different kinds of devices. Signed-off-by: Andy Pei --- doc/guides/prog_guide/vhost_lib.rst | 5 ++++ doc/guides/rel_notes/release_22_07.rst | 4 ++++ lib/vhost/rte_vhost.h | 17 +++++++++++++ lib/vhost/socket.c | 44 ++++++++++++++++++++++++++++++++++ lib/vhost/vdpa_driver.h | 3 +++ lib/vhost/version.map | 1 + 6 files changed, 74 insertions(+) diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst index f287b76..0337b38 100644 --- a/doc/guides/prog_guide/vhost_lib.rst +++ b/doc/guides/prog_guide/vhost_lib.rst @@ -282,6 +282,11 @@ The following is an overview of some key Vhost API functions: Clear inflight packets which are submitted to DMA engine in vhost async data path. Completed packets are returned to applications through ``pkts``. +* ``rte_vhost_driver_get_vdpa_dev_type(path, type)`` + + Get device type of vDPA device, such as VDPA_DEVICE_TYPE_NET, + VDPA_DEVICE_TYPE_BLK. + Vhost-user Implementations -------------------------- diff --git a/doc/guides/rel_notes/release_22_07.rst b/doc/guides/rel_notes/release_22_07.rst index e49cace..9550977 100644 --- a/doc/guides/rel_notes/release_22_07.rst +++ b/doc/guides/rel_notes/release_22_07.rst @@ -104,6 +104,10 @@ New Features * ``RTE_EVENT_QUEUE_ATTR_WEIGHT`` * ``RTE_EVENT_QUEUE_ATTR_AFFINITY`` +* **Added vhost API to get the device type of a vDPA device.** + + Added an API which can get the device type of vDPA device. + Removed Items ------------- diff --git a/lib/vhost/rte_vhost.h b/lib/vhost/rte_vhost.h index c733f85..c977a24 100644 --- a/lib/vhost/rte_vhost.h +++ b/lib/vhost/rte_vhost.h @@ -117,6 +117,9 @@ #define RTE_MAX_VHOST_DEVICE 1024 +#define VDPA_DEVICE_TYPE_NET 0 +#define VDPA_DEVICE_TYPE_BLK 1 + struct rte_vdpa_device; /** @@ -486,6 +489,20 @@ struct rte_vdpa_device * rte_vhost_driver_get_vdpa_device(const char *path); /** + * Get the device type of the vdpa device. + * + * @param path + * The vhost-user socket file path + * @param type + * the device type of the vdpa device + * @return + * 0 on success, -1 on failure + */ +__rte_experimental +int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type); + +/** * Set the feature bits the vhost-user driver supports. * * @param path diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index b304339..ef0f401 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -619,6 +619,50 @@ struct rte_vdpa_device * } int +rte_vhost_driver_get_vdpa_dev_type(const char *path, uint32_t *type) +{ + struct vhost_user_socket *vsocket; + struct rte_vdpa_device *vdpa_dev; + uint32_t vdpa_type = 0; + int ret = 0; + + pthread_mutex_lock(&vhost_user.mutex); + vsocket = find_vhost_user_socket(path); + if (!vsocket) { + VHOST_LOG_CONFIG(ERR, + "(%s) socket file is not registered yet.\n", + path); + ret = -1; + goto unlock_exit; + } + + vdpa_dev = vsocket->vdpa_dev; + if (!vdpa_dev) { + ret = -1; + goto unlock_exit; + } + + if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); + if (ret) { + VHOST_LOG_CONFIG(ERR, + "(%s) failed to get vdpa dev type for socket file.\n", + path); + ret = -1; + goto unlock_exit; + } + } else { + vdpa_type = VDPA_DEVICE_TYPE_NET; + } + + *type = vdpa_type; + +unlock_exit: + pthread_mutex_unlock(&vhost_user.mutex); + return ret; +} + +int rte_vhost_driver_disable_features(const char *path, uint64_t features) { struct vhost_user_socket *vsocket; diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index e59a834..9cbd7cd 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -78,6 +78,9 @@ struct rte_vdpa_dev_ops { /** Set the device configuration space */ int (*set_config)(int vid, uint8_t *config, uint32_t offset, uint32_t size, uint32_t flags); + + /** get device type: net device, blk device... */ + int (*get_dev_type)(struct rte_vdpa_device *dev, uint32_t *type); }; /** diff --git a/lib/vhost/version.map b/lib/vhost/version.map index 5841315..583b4f3 100644 --- a/lib/vhost/version.map +++ b/lib/vhost/version.map @@ -90,6 +90,7 @@ EXPERIMENTAL { # added in 22.07 rte_vhost_async_get_inflight_thread_unsafe; + rte_vhost_driver_get_vdpa_dev_type; }; From patchwork Wed May 18 12:13:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111316 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 04A2DA0503; Wed, 18 May 2022 15:01:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DC4EE42BA0; Wed, 18 May 2022 15:00:47 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id C4C9942BA0 for ; Wed, 18 May 2022 15:00:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878846; x=1684414846; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=E9kVVt2fL4gabkH9b4/OD50ieIC+2mM5FYMd0NvIPNg=; b=dm8cjBZbevw7Woo6tLSQ7CCrlXeaGXHim5qrbLZVh0UUjIZIEEl8RfzW kr6H3x2od986c9zvDl9fPxsYXo6KFJ4X3iwyYN9qCNeqjS8aibiqhGVni JRIrMgued2g2gDs07Z673hNBzACni57nnHIyLOBQ9EaSELYqUlTM32KHZ T+9abWe5HUs0NpfellJD1b92tvF9jcvuQOIQx7xnU6sBuq7SeO9kyx20R hmDocND6j8r1SoqNjTMzEUvM88H/OuSK+rLCi1AhQLR8H5PLLM48JN1iz DQ7HTd9elCfd3tad9gUzN9aiSGwVtJvnqY7ZoYlXzqWLBCdTHG16gPYpX Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353738" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353738" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312336" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:42 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 08/13] vdpa/ifc: add get device type ops to ifc driver Date: Wed, 18 May 2022 20:13:50 +0800 Message-Id: <1652876035-70513-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add get device type ops to ifc driver. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3e78c7d..0acfa8c 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1189,6 +1189,29 @@ struct rte_vdpa_dev_info { return 0; } +static int +ifcvf_get_device_type(struct rte_vdpa_device *vdev, + uint32_t *type) +{ + struct ifcvf_internal *internal; + struct internal_list *list; + + list = find_internal_resource_by_vdev(vdev); + if (list == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -1; + } + + internal = list->internal; + + if (internal->hw.device_type == IFCVF_BLK) + *type = VDPA_DEVICE_TYPE_BLK; + else + *type = VDPA_DEVICE_TYPE_NET; + + return 0; +} + static struct rte_vdpa_dev_ops ifcvf_ops = { .get_queue_num = ifcvf_get_queue_num, .get_features = ifcvf_get_vdpa_features, @@ -1201,6 +1224,7 @@ struct rte_vdpa_dev_info { .get_vfio_group_fd = ifcvf_get_vfio_group_fd, .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, + .get_dev_type = ifcvf_get_device_type, }; static inline int @@ -1327,6 +1351,7 @@ struct rte_vdpa_dev_info { .get_vfio_device_fd = ifcvf_get_vfio_device_fd, .get_notify_area = ifcvf_get_notify_area, .get_config = ifcvf_blk_get_config, + .get_dev_type = ifcvf_get_device_type, }; struct rte_vdpa_dev_info dev_info[] = { From patchwork Wed May 18 12:13:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111317 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E435DA0503; Wed, 18 May 2022 15:01:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C31DE42BA3; Wed, 18 May 2022 15:00:48 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 839F742BA0 for ; Wed, 18 May 2022 15:00:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878846; x=1684414846; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=JMOfNZgUeh8g1UXiE38Dak6G+bAhVR2J68cpl3JFRjU=; b=PxK8ItFzrMycC0gM4HIRes1wZXPizaBR2vg8a3uSu5Qr6nnfiCBGh1Hx s4hPwqewHp2erUFukfBWMRayBeYSD0r2A6x6NsOZr8BUN6nBfTwOGrYeb OuwsH5vWPOD+tApim1C/MXfAA2ZHtopqHbj25HP1+crC2qw1JoTWWOqfG fmCDSyDAiX8w1ejj9gHnvcpwkWfsqz4N8JtfUA7qJRcDaUyHf+65pKo7c eJVXm6TMJKQlf/hCyJBrBu3SY9DZkD8Uouena6NN5lVXUGdXcKvPKjVKE ZAMOzt5fIPZm2XLtFISWnBtyKqSr+M74MnMN/BwfM95yPoQNBC24Ozd8D w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353757" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353757" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312390" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:44 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 09/13] examples/vdpa: add add virtio blk support Date: Wed, 18 May 2022 20:13:51 +0800 Message-Id: <1652876035-70513-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to vDPA example. Signed-off-by: Andy Pei --- examples/vdpa/main.c | 56 ++++++++++++++++++++++++++++++++++++++ examples/vdpa/vdpa_blk_compact.h | 58 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 examples/vdpa/vdpa_blk_compact.h diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c index 5ab0765..62b6660 100644 --- a/examples/vdpa/main.c +++ b/examples/vdpa/main.c @@ -20,6 +20,7 @@ #include #include #include +#include "vdpa_blk_compact.h" #define MAX_PATH_LEN 128 #define MAX_VDPA_SAMPLE_PORTS 1024 @@ -159,8 +160,53 @@ struct vdpa_port { }; static int +vdpa_blk_device_set_features_and_protocol(const char *path) +{ + uint64_t protocol_features = 0; + int ret; + + ret = rte_vhost_driver_set_features(path, VHOST_BLK_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_disable_features(path, + VHOST_BLK_DISABLED_FEATURES); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_disable_features for %s failed.\n", + path); + goto out; + } + + ret = rte_vhost_driver_get_protocol_features(path, &protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_get_protocol_features for %s failed.\n", + path); + goto out; + } + + protocol_features |= VHOST_BLK_PROTOCOL_FEATURES; + + ret = rte_vhost_driver_set_protocol_features(path, protocol_features); + if (ret != 0) { + RTE_LOG(ERR, VDPA, + "rte_vhost_driver_set_protocol_features for %s failed.\n", + path); + } + +out: + return ret; +} + +static int start_vdpa(struct vdpa_port *vport) { + uint32_t device_type = 0; int ret; char *socket_path = vport->ifname; @@ -192,6 +238,16 @@ struct vdpa_port { "attach vdpa device failed: %s\n", socket_path); + ret = rte_vhost_driver_get_vdpa_dev_type(socket_path, &device_type); + if (ret == 0 && device_type == VDPA_DEVICE_TYPE_BLK) { + RTE_LOG(NOTICE, VDPA, "%s is a blk device\n", socket_path); + ret = vdpa_blk_device_set_features_and_protocol(socket_path); + if (ret != 0) + rte_exit(EXIT_FAILURE, + "set vhost blk driver features and protocol features failed: %s\n", + socket_path); + } + if (rte_vhost_driver_start(socket_path) < 0) rte_exit(EXIT_FAILURE, "start vhost driver failed: %s\n", diff --git a/examples/vdpa/vdpa_blk_compact.h b/examples/vdpa/vdpa_blk_compact.h new file mode 100644 index 0000000..4193561 --- /dev/null +++ b/examples/vdpa/vdpa_blk_compact.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2022 Intel Corporation + */ + +#ifndef _VDPA_BLK_COMPACT_H_ +#define _VDPA_BLK_COMPACT_H_ + +/** + * @file + * + * Device specific vhost lib + */ + +#include + +/* Feature bits */ +#define VIRTIO_BLK_F_SIZE_MAX 1 /* Indicates maximum segment size */ +#define VIRTIO_BLK_F_SEG_MAX 2 /* Indicates maximum # of segments */ +#define VIRTIO_BLK_F_GEOMETRY 4 /* Legacy geometry available */ +#define VIRTIO_BLK_F_BLK_SIZE 6 /* Block size of disk is available */ +#define VIRTIO_BLK_F_TOPOLOGY 10 /* Topology information is available */ +#define VIRTIO_BLK_F_MQ 12 /* support more than one vq */ + +/* Legacy feature bits */ +#ifndef VIRTIO_BLK_NO_LEGACY +#define VIRTIO_BLK_F_BARRIER 0 /* Does host support barriers? */ +#define VIRTIO_BLK_F_SCSI 7 /* Supports scsi command passthru */ +#define VIRTIO_BLK_F_CONFIG_WCE 11 /* Writeback mode available in config */ +#endif /* !VIRTIO_BLK_NO_LEGACY */ + +#define VHOST_BLK_FEATURES_BASE ((1ULL << VHOST_F_LOG_ALL) | \ + (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_INDIRECT_DESC) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX) | \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \ + (1ULL << VIRTIO_F_VERSION_1)) + +#define VHOST_BLK_DISABLED_FEATURES_BASE ((1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) | \ + (1ULL << VIRTIO_RING_F_EVENT_IDX)) + +#define VHOST_BLK_FEATURES (VHOST_BLK_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_SIZE_MAX) | (1ULL << VIRTIO_BLK_F_SEG_MAX) | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \ + (1ULL << VIRTIO_BLK_F_TOPOLOGY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE) | \ + (1ULL << VIRTIO_BLK_F_MQ)) + +/* Not supported features */ +#define VHOST_BLK_DISABLED_FEATURES (VHOST_BLK_DISABLED_FEATURES_BASE | \ + (1ULL << VIRTIO_BLK_F_GEOMETRY) | (1ULL << VIRTIO_BLK_F_BARRIER) | \ + (1ULL << VIRTIO_BLK_F_SCSI) | (1ULL << VIRTIO_BLK_F_CONFIG_WCE)) + +/* Vhost-blk support protocol features */ +#define VHOST_BLK_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_CONFIG)) + +#endif /* _VDPA_BLK_COMPACT_H_ */ From patchwork Wed May 18 12:13:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111318 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1D4E8A0503; Wed, 18 May 2022 15:01:57 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD4A440221; Wed, 18 May 2022 15:01:16 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 0A1EF4021E for ; Wed, 18 May 2022 15:01:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878875; x=1684414875; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=1jsRjHf2H1JRmMIdLdkv2C6iAm7xMFkU5+GRJK/A/RI=; b=Nh/Uk9VCALZMKdrZCb8VRBxNFt7uVM2+PLwRoG+VhBhjW/rhJlwQbpDK uE9XbL6uH9hahnG1UsIaWhGwIK/EySrTVHU9LhTCmevV+ds9xnoqNDDGh pvRZ/c+jpKIGo3TWn44lmumDlgVMPaaf+K6f0/V87MFtTg905zG8ZFZJ1 +CtqDP1LcTl18D1U7zg331JlhqZerwk5bNsYnBEiPlVTqUWNmlzfxTeH4 rY/i/a+UVpsX8Ajt63ot4sD4kO8S3wos3edCLm1tmkXo+lZhMj2kfcnr7 nh3AucO4k4YeUlfiKikWnXTbCumGxrgneJfhK8WBoWvIdXuu2rMoZVvSU w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353784" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353784" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312426" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:46 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 10/13] usertools: add support for virtio blk device Date: Wed, 18 May 2022 20:13:52 +0800 Message-Id: <1652876035-70513-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add virtio blk device support to devbind. Signed-off-by: Andy Pei --- usertools/dpdk-devbind.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index ace4627..7231be4 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -72,6 +72,9 @@ cn9k_ree = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f4', 'SVendor': None, 'SDevice': None} +virtio_blk = {'Class': '01', 'Vendor': "1af4", 'Device': '1001', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] baseband_devices = [acceleration_class] crypto_devices = [encryption_class, intel_processor_class] @@ -82,7 +85,7 @@ compress_devices = [cavium_zip] regex_devices = [cn9k_ree] misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, - intel_ntb_skx, intel_ntb_icx] + intel_ntb_skx, intel_ntb_icx, virtio_blk] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties From patchwork Wed May 18 12:13:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111319 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D3DD1A0503; Wed, 18 May 2022 15:02:06 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B9C1242B7B; Wed, 18 May 2022 15:01:17 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 4E9E04021E for ; Wed, 18 May 2022 15:01:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878876; x=1684414876; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rB3n49Qm4f8/2gsmZ18d+vTRRsZqXrOrCAGzJmgp2Ww=; b=hmgkLMp2ucfitlyfYhwAvxBZq7CeitlDzTcx95rrSzXKNIEJQZZBMBI/ r4GZxIK4wTFIGqcCmno6IKjjyloobvBtFDk4Y0lu487WVeywwu0eitDwL uclDxf0pbrSyOHqcx4U0zDgd5yQPxWe4nfDTjg6JuZ4V0KGhVuW+vwZU8 Qs1NxYJhLyH4cy6SZ51MC5LPo6eq5kLnZkwWvXLxzHI/JQCBBxtW/QPg4 XBww9FZ7QuyIbqkHPNSOEHSHkihrMZggdkRJRtyuKrkO4sUCCpGKbby0H WlW3ZgrdSVxI4GGO8fYQb0hvqjQYWn0/WRqs26KPOWPC1E3Ge8OtTKIzV A==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353800" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353800" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312459" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:48 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 11/13] vdpa/ifc: add log for config space of virtio blk Date: Wed, 18 May 2022 20:13:53 +0800 Message-Id: <1652876035-70513-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add some log of virtio blk device config space information at VDPA launch before qemu connects. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 0acfa8c..376a1af 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1382,6 +1382,9 @@ struct rte_vdpa_dev_info dev_info[] = { struct rte_kvargs *kvlist = NULL; int ret = 0; int16_t device_id; + uint64_t capacity = 0; + uint8_t *byte; + uint32_t i; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1448,6 +1451,37 @@ struct rte_vdpa_dev_info dev_info[] = { internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_BLK].features; + + /* cannot read 64-bit register in one attempt, + * so read byte by byte. + */ + for (i = 0; i < sizeof(internal->hw.blk_cfg->capacity); i++) { + byte = (uint8_t *)&internal->hw.blk_cfg->capacity + i; + capacity |= (uint64_t)*byte << (i * 8); + } + /* The capacity is number of sectors in 512-byte. + * So right shift 1 bit we get in K, + * another right shift 10 bits we get in M, + * right shift 10 more bits, we get in G. + * To show capacity in G, we right shift 21 bits in total. + */ + DRV_LOG(DEBUG, "capacity : %"PRIu64"G", capacity >> 21); + + DRV_LOG(DEBUG, "size_max : 0x%08x", + internal->hw.blk_cfg->size_max); + DRV_LOG(DEBUG, "seg_max : 0x%08x", + internal->hw.blk_cfg->seg_max); + DRV_LOG(DEBUG, "blk_size : 0x%08x", + internal->hw.blk_cfg->blk_size); + DRV_LOG(DEBUG, "geometry"); + DRV_LOG(DEBUG, " cylinders: %u", + internal->hw.blk_cfg->geometry.cylinders); + DRV_LOG(DEBUG, " heads : %u", + internal->hw.blk_cfg->geometry.heads); + DRV_LOG(DEBUG, " sectors : %u", + internal->hw.blk_cfg->geometry.sectors); + DRV_LOG(DEBUG, "num_queues: 0x%08x", + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Wed May 18 12:13:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111320 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C3837A0505; Wed, 18 May 2022 15:02:15 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A8A8042BA2; Wed, 18 May 2022 15:01:18 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 0C54E42B71 for ; Wed, 18 May 2022 15:01:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878877; x=1684414877; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=2NDzS7VlmMZPuY5ypD5NFD349cUg+gGOXuWlwDiofQE=; b=V+hANQmKNim3T+UAA1kGY7K+pN3zvKzO/3G5VGAOnSQCWnRT3i7pqlrb LgZfkCZQASeCE3ZGhI5humhAR0hfmmRYK2jk69/Ty7i6rYvZzYBZVSm9N fvmhI44ZfI+GJTFxj6o1Uz/Y2tnRY2JzItm9OFpl6Am233MkamBI8NSBI rlbJhCoQvmJYtZIXntYiT09BcHQXXCHSnZX2K6V92BOttMi6Ov7nsVL+l Asz/LvrxqGPcEZrArfa1/7uS/yG1efSYE+vgtUC+I8A/EnUjp+EYghxCr FmPW4IL0M5ETsl4Zlv01ITde6e/k1vlrrBb7ncjyO1F9XOg5t/w6/FjYC w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353814" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353814" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312496" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:50 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config space Date: Wed, 18 May 2022 20:13:54 +0800 Message-Id: <1652876035-70513-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Create a thread to poll and relay config space change interrupt. Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU. Signed-off-by: Andy Pei --- drivers/vdpa/ifc/ifcvf_vdpa.c | 118 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 117 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 376a1af..8a49622 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -53,7 +53,9 @@ struct ifcvf_internal { int vfio_group_fd; int vfio_dev_fd; pthread_t tid; /* thread for notify relay */ + pthread_t intr_tid; /* thread for config space change interrupt relay */ int epfd; + int csc_epfd; int vid; struct rte_vdpa_device *vdev; uint16_t max_queues; @@ -566,6 +568,111 @@ struct rte_vdpa_dev_info { return 0; } +static void +virtio_interrupt_handler(struct ifcvf_internal *internal) +{ + int vid = internal->vid; + int ret; + + ret = rte_vhost_slave_config_change(vid, 1); + if (ret) + DRV_LOG(ERR, "failed to notify the guest about configuration space change."); +} + +static void * +intr_relay(void *arg) +{ + struct ifcvf_internal *internal = (struct ifcvf_internal *)arg; + struct epoll_event csc_event; + struct epoll_event ev; + uint64_t buf; + int nbytes; + int csc_epfd, csc_val = 0; + + csc_epfd = epoll_create(1); + if (csc_epfd < 0) { + DRV_LOG(ERR, "failed to create epoll for config space change."); + return NULL; + } + + ev.events = EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; + ev.data.fd = rte_intr_fd_get(internal->pdev->intr_handle); + if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD, + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); + return NULL; + } + + internal->csc_epfd = csc_epfd; + + for (;;) { + csc_val = epoll_wait(csc_epfd, &csc_event, 1, -1); + if (csc_val < 0) { + if (errno == EINTR) + continue; + DRV_LOG(ERR, "epoll_wait return fail."); + return NULL; + } else if (csc_val == 0) { + continue; + } else { + /* csc_val > 0 */ + nbytes = read(csc_event.data.fd, &buf, 8); + if (nbytes < 0) { + if (errno == EINTR || + errno == EWOULDBLOCK || + errno == EAGAIN) + continue; + DRV_LOG(ERR, "Error reading from file descriptor %d: %s\n", + csc_event.data.fd, + strerror(errno)); + return NULL; + } else if (nbytes == 0) { + DRV_LOG(ERR, "Read nothing from file descriptor %d\n", + csc_event.data.fd); + continue; + } else { + virtio_interrupt_handler(internal); + } + } + } + + return NULL; +} + +static int +setup_intr_relay(struct ifcvf_internal *internal) +{ + char name[THREAD_NAME_LEN]; + int ret; + + snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid); + ret = rte_ctrl_thread_create(&internal->intr_tid, name, NULL, + intr_relay, (void *)internal); + if (ret) { + DRV_LOG(ERR, "failed to create notify relay pthread."); + return -1; + } + return 0; +} + +static int +unset_intr_relay(struct ifcvf_internal *internal) +{ + void *status; + + if (internal->intr_tid) { + pthread_cancel(internal->intr_tid); + pthread_join(internal->intr_tid, &status); + } + internal->intr_tid = 0; + + if (internal->csc_epfd >= 0) + close(internal->csc_epfd); + internal->csc_epfd = -1; + + return 0; +} + static int update_datapath(struct ifcvf_internal *internal) { @@ -592,10 +699,16 @@ struct rte_vdpa_dev_info { if (ret) goto err; + ret = setup_intr_relay(internal); + if (ret) + goto err; + rte_atomic32_set(&internal->running, 1); } else if (rte_atomic32_read(&internal->running) && (!rte_atomic32_read(&internal->started) || !rte_atomic32_read(&internal->dev_attached))) { + ret = unset_intr_relay(internal); + ret = unset_notify_relay(internal); if (ret) goto err; @@ -812,7 +925,7 @@ struct rte_vdpa_dev_info { if (nfds < 0) { if (errno == EINTR) continue; - DRV_LOG(ERR, "epoll_wait return fail\n"); + DRV_LOG(ERR, "epoll_wait return fail."); return NULL; } @@ -888,6 +1001,9 @@ struct rte_vdpa_dev_info { /* stop the direct IO data path */ unset_notify_relay(internal); vdpa_ifcvf_stop(internal); + + unset_intr_relay(internal); + vdpa_disable_vfio_intr(internal); ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); From patchwork Wed May 18 12:13:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 111321 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 82509A0503; Wed, 18 May 2022 15:02:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A1FA142BAF; Wed, 18 May 2022 15:01:20 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 302D342BA2 for ; Wed, 18 May 2022 15:01:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652878878; x=1684414878; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=XSAhxIcTbia7rVOu01I5W/7tHy4l8+w4585hTSZVFsw=; b=GzG+eEdoTN7iGyEmY6ALGRfOod81Q8PQxoH8YgAFmFSGZWszOvn+hM9T dQMSwklzLh8jXeYhw4i3wf21EJpJQIXY8p+UFkdOmVRcNBzwBTVYB2A2M Iinakt7LwGNPK59BYCeaCE1maV7nR6AAlNwPHi5B0oMs9aEiV6vve/Vs7 qPC28v59xBCLltm3NK8tF16+ScvgxFk80LiS5Ir1spHm2eXZdJI14T7UA RLjaB/J/yZbOzkD4raQO+sZH3JE+6fm0GB0rRNjEMU7NxG3HIhjbd4lcQ A3WFYh0kwloYYKgAEZt4ivtHf2HZBTBfVN2vrBPUo1tc8/pibNcYoT5n2 w==; X-IronPort-AV: E=McAfee;i="6400,9594,10350"; a="271353830" X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="271353830" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 May 2022 06:00:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,234,1647327600"; d="scan'208";a="556312523" Received: from dpdk-dipei.sh.intel.com ([10.67.110.238]) by orsmga002.jf.intel.com with ESMTP; 18 May 2022 06:00:52 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, gang.cao@intel.com, changpeng.liu@intel.com, rosen.xu@intel.com, qimaix.xiao@intel.com Subject: [PATCH v8 13/13] vdpa/ifc/base: access correct register for blk device Date: Wed, 18 May 2022 20:13:55 +0800 Message-Id: <1652876035-70513-14-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1652876035-70513-1-git-send-email-andy.pei@intel.com> References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Register address is different between net and blk device. We are re-use most of the code, when register address is different, we have to check net and blk device go through different code. Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index d10c1fd..dd475a7 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -218,10 +218,17 @@ &cfg->queue_used_hi); IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); - *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) = - (u32)hw->vring[i].last_avail_idx | - ((u32)hw->vring[i].last_used_idx << 16); + if (hw->device_type == IFCVF_BLK) + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + else + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); if (IFCVF_READ_REG16(&cfg->queue_msix_vector) == @@ -254,9 +261,22 @@ IFCVF_WRITE_REG16(i, &cfg->queue_select); IFCVF_WRITE_REG16(0, &cfg->queue_enable); IFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector); - ring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET + - (i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4); - hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + + if (hw->device_type == IFCVF_BLK) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + else + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + + if (hw->device_type == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); hw->vring[i].last_used_idx = (u16)(ring_state >> 16); } }