From patchwork Thu Sep 8 05:54:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116073 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2386A0548; Thu, 8 Sep 2022 07:53:25 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 442F9427F9; Thu, 8 Sep 2022 07:53:20 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id E828740DDC for ; Thu, 8 Sep 2022 07:53:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616399; x=1694152399; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=OSxntA4BTV8/dX1BCMaoUhNiAfGhgvAYpHMkgnNiExc=; b=EWDGQUPH2RhYyx5wJK+cSW5cTqOLwA/2S6WNCwsvSEEnq0DSg2f472/Z CpeuvyKMWA9rwgBa7Hq25ICgjPlS4fphqlTwnjg6LOf0i8oe1D7ksVlZu TRJbASxtd7TBLDLushFSTXSp33FH0Nx68bQ/DjKri7t0LHES8FAleb/jl gY2jxnVfRyq4FiFRUz6JfGE+ctfQLFkDQkDw1ZHduMqksCd99/CWJ0bGc ZBrc0VqzAZDeK7aml1V4PbPdnOb4fE5LVTjidj2M29w9Lgs4xpFkZroiE bJKRuXzq/SeNF209hu4PCgHM0TOeGPXhyMpM3p5f94smHISBTdq+ymHIt A==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464143" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464143" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531766" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:16 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 1/8] vdpa/ifc: add new device ID Date: Thu, 8 Sep 2022 13:54:11 +0800 Message-Id: <1662616458-164613-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Huang Wei Add new device id to support IFCVF_NET_TRANSITIONAL_DEVICE_ID (0x1000). Signed-off-by: Huang Wei Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.h | 4 +++- drivers/vdpa/ifc/ifcvf_vdpa.c | 9 ++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 9d95aac..7ede738 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -12,11 +12,13 @@ #define IFCVF_BLK 1 #define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_NET_MODERN_DEVICE_ID 0x1041 #define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_NET_TRANSITIONAL_DEVICE_ID 0x1000 #define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 #define IFCVF_SUBSYS_DEVICE_ID 0x001A +#define IFCVF_NET_DEVICE_ID 0x0001 #define IFCVF_BLK_DEVICE_ID 0x0002 #define IFCVF_MAX_QUEUES 1 diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index ac42de9..61d0250 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1684,13 +1684,20 @@ struct rte_vdpa_dev_info dev_info[] = { static const struct rte_pci_id pci_id_ifcvf_map[] = { { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, - .device_id = IFCVF_NET_DEVICE_ID, + .device_id = IFCVF_NET_MODERN_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_NET_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_NET_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, .subsystem_device_id = IFCVF_BLK_DEVICE_ID, From patchwork Thu Sep 8 05:54:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116074 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 07D9DA0548; Thu, 8 Sep 2022 07:53:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 105444281B; Thu, 8 Sep 2022 07:53:22 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 6C6C342802 for ; Thu, 8 Sep 2022 07:53:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616401; x=1694152401; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Zot4Am17jNHQAQS8iST1qki7PQWNspdi4CIszg6n2Bo=; b=ZRM6f+v4XHIYGYDQfA42ES71jHgI/lugQ6Y+aMKnRtbk+oYOZZbco6q7 6igVhcbIQBRCIzTEjaSi7qsC6o7qrwlwwZg/VqJMDiBuL8SmpOE4nKmwM mxTfAcSuEXKpdyHv4VWZuQ+MkjoRL5bH+stxZdkwpMXrmWbciT/kl6mkF fYk5FOXNdmXo1Dgnv36Az5rlGAtY5kKtjygsz31yBX/rPA1d/AoEEfM5U jS5fcwS8HcEqA0Q2Lzdgxji51vUgZ4twUkANnpCJ9wIxRpbC/3ZWGBKhc Z5eV18Ct8oeUnxsOKJVeTh1xGw7+h+pt2nOJzkdI3b+BQLV/Eb5Qo7FIp A==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464156" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464156" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531785" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:19 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 2/8] vdpa/ifc: add multi queue support Date: Thu, 8 Sep 2022 13:54:12 +0800 Message-Id: <1662616458-164613-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable VHOST_USER_PROTOCOL_F_MQ feature. ExposeIFCVF_MQ_OFFSET register to enable multi queue. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.c | 5 +++++ drivers/vdpa/ifc/base/ifcvf.h | 2 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 1 + 3 files changed, 8 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index f1e1474..34c8226 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -90,6 +90,11 @@ if (!hw->lm_cfg) WARNINGOUT("HW support live migration not support!\n"); + if (hw->mem_resource[4].addr) + hw->mq_cfg = hw->mem_resource[4].addr + IFCVF_MQ_OFFSET; + else + hw->mq_cfg = NULL; + if (hw->common_cfg == NULL || hw->notify_base == NULL || hw->isr == NULL || hw->dev_cfg == NULL) { DEBUGOUT("capability incomplete\n"); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 7ede738..ad505f1 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -50,6 +50,7 @@ #define IFCVF_LM_CFG_SIZE 0x40 #define IFCVF_LM_RING_STATE_OFFSET 0x20 +#define IFCVF_MQ_OFFSET 0x28 #define IFCVF_LM_LOGGING_CTRL 0x0 @@ -149,6 +150,7 @@ struct ifcvf_hw { u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; u8 *lm_cfg; + u8 *mq_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; int device_type; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 61d0250..2d165c0 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1248,6 +1248,7 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD | \ 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ + 1ULL << VHOST_USER_PROTOCOL_F_MQ | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) #define VDPA_BLK_PROTOCOL_FEATURES \ From patchwork Thu Sep 8 05:54:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116075 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 89543A0548; Thu, 8 Sep 2022 07:53:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EFE5D40DDC; Thu, 8 Sep 2022 07:53:26 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 45ED741155 for ; Thu, 8 Sep 2022 07:53:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616404; x=1694152404; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Ms+q2ncWfd04w0caB0rPcGsZvIaJNY+UXnRqq/lupSs=; b=IkDTvAlFhhHySZUPvRxrpcLK/u6TVWfOmSrBBK4IqnOFBJDJ//FtlZFW MunrtPzyHAbPf1RwdagExSzamqGgZgrBu8Dg4WEigd0Ld3Lrkypi23RhL xKAHjoAJ0cuuGJ4BY81jPDHR9cSGhOVAZzdXo9usdA5UisBi7OJ7K3DC2 zkx1kQm6BQshJIWEg1h0EZZ3lsmPlbfYG/1w4KOtu3dDpFaORBmIAxSbH K0vZVX9BnxY4q7IEvnjICmgLJfo+1OI1Py9cBw5EWdOFY5SBD5JooQgsz 15hpXP/55CFjCNHwkI9/ird9VboBoq5t5LKunzYLxgm58rC5hG5Il2k9P A==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464162" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464162" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531807" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:22 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 3/8] vdpa/ifc: set max queues according to HW spec Date: Thu, 8 Sep 2022 13:54:13 +0800 Message-Id: <1662616458-164613-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Set max_queues according to virtio HW spec. For virtio BLK device, set max_queues to the value of "num_queues". "num_queues" is element of struct virtio_blk_config. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.h | 2 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 21 ++++++++++++++++++++- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index ad505f1..c17bf2a 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -21,7 +21,7 @@ #define IFCVF_NET_DEVICE_ID 0x0001 #define IFCVF_BLK_DEVICE_ID 0x0002 -#define IFCVF_MAX_QUEUES 1 +#define IFCVF_MAX_QUEUES 32 #ifndef VIRTIO_F_IOMMU_PLATFORM #define VIRTIO_F_IOMMU_PLATFORM 33 diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 2d165c0..2b42850 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -26,6 +26,18 @@ #include "base/ifcvf.h" +/* + * RTE_MAX() and RTE_MIN() cannot be used since braced-group within + * expression allowed only inside a function, but MAX() is used as + * a number of elements in array. + */ +#ifndef MAX +#define MAX(v1, v2) ((v1) > (v2) ? (v1) : (v2)) +#endif +#ifndef MIN +#define MIN(v1, v2) ((v1) < (v2) ? (v1) : (v2)) +#endif + RTE_LOG_REGISTER(ifcvf_vdpa_logtype, pmd.vdpa.ifcvf, NOTICE); #define DRV_LOG(level, fmt, args...) \ rte_log(RTE_LOG_ ## level, ifcvf_vdpa_logtype, \ @@ -1512,6 +1524,7 @@ struct rte_vdpa_dev_info dev_info[] = { uint64_t capacity = 0; uint8_t *byte; uint32_t i; + uint16_t queue_pairs; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1559,7 +1572,6 @@ struct rte_vdpa_dev_info dev_info[] = { } internal->configured = 0; - internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); device_id = ifcvf_pci_get_device_type(pci_dev); @@ -1570,6 +1582,10 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->hw.device_type = IFCVF_NET; + queue_pairs = (internal->hw.common_cfg->num_queues - 1) / 2; + DRV_LOG(INFO, "%s support %u queue pairs", pci_dev->name, + queue_pairs); + internal->max_queues = MIN(IFCVF_MAX_QUEUES, queue_pairs); internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; @@ -1609,6 +1625,9 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(DEBUG, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + internal->max_queues = MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Thu Sep 8 05:54:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116078 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E37C7A0548; Thu, 8 Sep 2022 07:53:42 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CB2164282D; Thu, 8 Sep 2022 07:53:28 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 15E7D42802 for ; Thu, 8 Sep 2022 07:53:26 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616407; x=1694152407; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=C1qcvZr+v+7XM9BeQOmJS3S3cvFJdhMF6r7Tecu9Mq4=; b=EtapcCDORBmE2W1ioMpJUevuLEfufCkCyG+Kmt0/Nff0rpGe9u4MGE2K AAVxqVZyoSyKcslp2HVLAldHf5J4djTkqoNV6mdmDGpCrx7aernbywn5D SrziyLS9MXxUwu0D90K0GWmCnyHBUF/lrYZH0WvcXbE416ToqDOGnL0vJ rAlWFVmbSD3WhF2QKlKnzm2sGZX/fL+mL+C4UxCZQzX+6XkDoGqiw00De UkIigxwRqdueRqSJptfnoyONv3OqigBJT9la+0UE5UUmkm9hsmx3LyVG7 QGpejTKjbY/xBeeyuXSlpiIFiEtZH/91mlJHGFUIfHsJbwdStAOZzrcej Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464167" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464167" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531818" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:25 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 4/8] vdpa/ifc: write queue count to MQ register Date: Thu, 8 Sep 2022 13:54:14 +0800 Message-Id: <1662616458-164613-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Write queue count to IFCVF_MQ_OFFSET register to enable multi queue feature. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 34c8226..0444d74 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -198,6 +198,38 @@ IFCVF_WRITE_REG32(val >> 32, hi); } +STATIC void +ifcvf_enable_multiqueue(struct ifcvf_hw *hw) +{ + u8 *mq_cfg; + int qid; + int nr_queue_pair = 0; + + if (hw->device_type == IFCVF_NET) { + for (qid = 0; qid < hw->nr_vring; qid++) { + if (!hw->vring[qid].enable) + continue; + nr_queue_pair++; + } + + if (nr_queue_pair == 0) { + WARNINGOUT("no enabled vring\n"); + return; + } + + nr_queue_pair = (nr_queue_pair + 1) / 2; + } else if (hw->device_type == IFCVF_BLK) { + nr_queue_pair = hw->nr_vring; + } + + mq_cfg = hw->mq_cfg; + if (mq_cfg) { + *(u32 *)mq_cfg = nr_queue_pair; + RTE_LOG(INFO, PMD, "%d queue pairs are enabled\n", + nr_queue_pair); + } +} + STATIC int ifcvf_hw_enable(struct ifcvf_hw *hw) { @@ -215,6 +247,7 @@ return -1; } + ifcvf_enable_multiqueue(hw); for (i = 0; i < hw->nr_vring; i++) { IFCVF_WRITE_REG16(i, &cfg->queue_select); io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, From patchwork Thu Sep 8 05:54:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116079 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9EACEA0548; Thu, 8 Sep 2022 07:53:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B932C42B6D; Thu, 8 Sep 2022 07:53:31 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id E670A4284D for ; Thu, 8 Sep 2022 07:53:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616410; x=1694152410; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=zikk6/T+EE2i7U1XmhFV8tFXCWLtaacfTsjpSu5qt3M=; b=fPgqFt1eeFZR7FJ+M6SnyV38OZyVCByojjtwoVHXsgyH3pHQMJ58VIpn HIDsSNbYWXeEiuK4LZWwf0Ovvs1hh3Hz753ILlGHTSu2wgtK22gJbpBLT 7bzPevhFex6c3MerGFrq0nyNc1Y5KEY0UD9HCs8HUsPouacW8lIl0NmOV LfEbCkDI72bx0hdkkLhZ/KP3qe7Xrq/U31fM6Xx2A3GG7GIrlnKr8rzlQ M/UI/EECWMiXZbGmVIMxKbeSr5mT0Wtn9dbxqiIviq2JFXCwdyUqZyWo9 pasKFZJphm8T5Lft5O8vdpNdF8YOxVVkVaJx6ZDdvlfEspgoUl1J0tNnu A==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464174" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464174" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531843" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:27 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 5/8] vdpa/ifc: only configure enabled queue Date: Thu, 8 Sep 2022 13:54:15 +0800 Message-Id: <1662616458-164613-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org when configure the hardware queue, we only configure queues which have been enabled by vhost. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.c | 6 +++++- drivers/vdpa/ifc/ifcvf_vdpa.c | 16 ++++++++++++++-- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 0444d74..4875ea1 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -249,6 +249,9 @@ ifcvf_enable_multiqueue(hw); for (i = 0; i < hw->nr_vring; i++) { + if (!hw->vring[i].enable) + continue; + IFCVF_WRITE_REG16(i, &cfg->queue_select); io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, &cfg->queue_desc_hi); @@ -283,7 +286,8 @@ notify_off = IFCVF_READ_REG16(&cfg->queue_notify_off); hw->notify_addr[i] = (void *)((u8 *)hw->notify_base + notify_off * hw->notify_off_multiplier); - IFCVF_WRITE_REG16(1, &cfg->queue_enable); + if (hw->vring[i].enable) + IFCVF_WRITE_REG16(1, &cfg->queue_enable); } return 0; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 2b42850..48f1a89 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -290,6 +290,8 @@ struct rte_vdpa_dev_info { rte_vhost_get_negotiated_features(vid, &hw->req_features); for (i = 0; i < nr_vring; i++) { + if (!hw->vring[i].enable) + continue; rte_vhost_get_vhost_vring(vid, i, &vq); gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc); if (gpa == 0) { @@ -505,6 +507,8 @@ struct rte_vdpa_dev_info { vring.kickfd = -1; for (qid = 0; qid < q_num; qid++) { + if (!hw->vring[qid].enable) + continue; ev.events = EPOLLIN | EPOLLPRI; rte_vhost_get_vhost_vring(internal->vid, qid, &vring); ev.data.u64 = qid | (uint64_t)vring.kickfd << 32; @@ -1064,6 +1068,8 @@ struct rte_vdpa_dev_info { struct rte_vdpa_device *vdev; struct internal_list *list; struct ifcvf_internal *internal; + struct ifcvf_hw *hw; + uint16_t i; vdev = rte_vhost_get_vdpa_device(vid); list = find_internal_resource_by_vdev(vdev); @@ -1077,11 +1083,17 @@ struct rte_vdpa_dev_info { rte_atomic32_set(&internal->dev_attached, 1); update_datapath(internal); - if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) - DRV_LOG(NOTICE, "vDPA (%s): software relay is used.", + hw = &internal->hw; + for (i = 0; i < hw->nr_vring; i++) { + if (!hw->vring[i].enable) + continue; + if (rte_vhost_host_notifier_ctrl(vid, i, true) != 0) + DRV_LOG(NOTICE, "vDPA (%s): software relay is used.", vdev->device->name); + } internal->configured = 1; + DRV_LOG(INFO, "vDPA device %s is configured", vdev->device->name); return 0; } From patchwork Thu Sep 8 05:54:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116080 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 171CCA0548; Thu, 8 Sep 2022 07:53:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B119642B6F; Thu, 8 Sep 2022 07:53:34 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id C17C54284D for ; Thu, 8 Sep 2022 07:53:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616413; x=1694152413; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=qmYPrbvzT0qVwJZI86FfxXWf1tpnVsi/b0TzNiP5QB4=; b=cKoyeARolynpCxCG8g0veclEKWyqmPaFuqLkz0wO5HTzJpRoYyboQ573 qapTYUC+mCpEB12WxSOlpwlFVrZEmap5hjZxP3oNjUKUA8DzqkM8L8dAC e3LkpIbifPyylf/i3F8BblI8HzFOUM5ZlldkiUv3Pa6/f8gDc3MT9RM/P xNEbx028LriaH0tyzJwlxeB9KwKijt+Kg5tFEvi44bl1p9JEtSpE3gFIu ngPRaoIlhhh6HRuEBPA8sIuT1dAyH3gGrz1hOD+kV/ZXLl4owUuxUVtUX GgPETOYpmg5oykIbB2juasFqQYUyvtoCDGuq0f3AsDg/Etbw1SQZ6igcI Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464185" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464185" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531863" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:30 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 6/8] vdpa/ifc: support dynamic enable/disable queue Date: Thu, 8 Sep 2022 13:54:16 +0800 Message-Id: <1662616458-164613-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Huang Wei Support dynamic enable or disable queue. For front end, like QEMU, user can use ethtool to configurate queue. For example, "ethtool -L eth0 combined 3" to enable 3 queues pairs. Signed-off-by: Huang Wei Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 101 ++++++++++++++++++++++++++++++++++++ drivers/vdpa/ifc/base/ifcvf.h | 6 +++ drivers/vdpa/ifc/base/ifcvf_osdep.h | 1 + drivers/vdpa/ifc/ifcvf_vdpa.c | 93 +++++++++++++++++++++++++++------ 4 files changed, 186 insertions(+), 15 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 4875ea1..34f32f8 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -230,6 +230,107 @@ } } +int +ifcvf_enable_vring_hw(struct ifcvf_hw *hw, int i) +{ + struct ifcvf_pci_common_cfg *cfg; + u8 *lm_cfg; + u16 notify_off; + int msix_vector; + + if (!hw || (i >= (int)hw->nr_vring)) + return -1; + + cfg = hw->common_cfg; + if (!cfg) { + ERROUT("common_cfg in HW is NULL.\n"); + return -1; + } + + ifcvf_enable_multiqueue(hw); + + IFCVF_WRITE_REG16(i, &cfg->queue_select); + msix_vector = IFCVF_READ_REG16(&cfg->queue_msix_vector); + if (msix_vector != (i + 1)) { + IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); + msix_vector = IFCVF_READ_REG16(&cfg->queue_msix_vector); + if (msix_vector == IFCVF_MSI_NO_VECTOR) { + ERROUT("queue %u, msix vec alloc failed\n", i); + return -1; + } + } + + io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, + &cfg->queue_desc_hi); + io_write64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo, + &cfg->queue_avail_hi); + io_write64_twopart(hw->vring[i].used, &cfg->queue_used_lo, + &cfg->queue_used_hi); + IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); + + lm_cfg = hw->lm_cfg; + if (lm_cfg) { + if (hw->device_type == IFCVF_BLK) + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + else + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } + + notify_off = IFCVF_READ_REG16(&cfg->queue_notify_off); + hw->notify_addr[i] = (void *)((u8 *)hw->notify_base + + notify_off * hw->notify_off_multiplier); + IFCVF_WRITE_REG16(1, &cfg->queue_enable); + + return 0; +} + +void +ifcvf_disable_vring_hw(struct ifcvf_hw *hw, int i) +{ + struct ifcvf_pci_common_cfg *cfg; + u32 ring_state; + u8 *lm_cfg; + + if (!hw || (i >= (int)hw->nr_vring)) + return; + + cfg = hw->common_cfg; + if (!cfg) { + ERROUT("common_cfg in HW is NULL.\n"); + return; + } + + IFCVF_WRITE_REG16(i, &cfg->queue_select); + IFCVF_WRITE_REG16(0, &cfg->queue_enable); + + lm_cfg = hw->lm_cfg; + if (lm_cfg) { + if (hw->device_type == IFCVF_BLK) + ring_state = *(u32 *)(lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + else + ring_state = *(u32 *)(lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + + if (hw->device_type == IFCVF_BLK) + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + else + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + hw->vring[i].last_used_idx = (u16)(ring_state >> 16); + } +} + STATIC int ifcvf_hw_enable(struct ifcvf_hw *hw) { diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index c17bf2a..e67d4e8 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -164,6 +164,12 @@ struct ifcvf_hw { ifcvf_get_features(struct ifcvf_hw *hw); int +ifcvf_enable_vring_hw(struct ifcvf_hw *hw, int i); + +void +ifcvf_disable_vring_hw(struct ifcvf_hw *hw, int i); + +int ifcvf_start_hw(struct ifcvf_hw *hw); void diff --git a/drivers/vdpa/ifc/base/ifcvf_osdep.h b/drivers/vdpa/ifc/base/ifcvf_osdep.h index 3d56769..4a1bfec 100644 --- a/drivers/vdpa/ifc/base/ifcvf_osdep.h +++ b/drivers/vdpa/ifc/base/ifcvf_osdep.h @@ -16,6 +16,7 @@ #define WARNINGOUT(S, args...) RTE_LOG(WARNING, PMD, S, ##args) #define DEBUGOUT(S, args...) RTE_LOG(DEBUG, PMD, S, ##args) +#define ERROUT(S, args...) RTE_LOG(ERR, PMD, S, ##args) #define STATIC static #define msec_delay(x) rte_delay_us_sleep(1000 * (x)) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 48f1a89..16fd0fd 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1288,13 +1288,59 @@ struct rte_vdpa_dev_info { } static int +ifcvf_config_vring(struct ifcvf_internal *internal, int vring) +{ + struct ifcvf_hw *hw = &internal->hw; + int vid = internal->vid; + struct rte_vhost_vring vq; + uint64_t gpa; + + if (hw->vring[vring].enable) { + rte_vhost_get_vhost_vring(vid, vring, &vq); + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for descriptor ring."); + return -1; + } + hw->vring[vring].desc = gpa; + + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for available ring."); + return -1; + } + hw->vring[vring].avail = gpa; + + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for used ring."); + return -1; + } + hw->vring[vring].used = gpa; + + hw->vring[vring].size = vq.size; + rte_vhost_get_vring_base(vid, vring, + &hw->vring[vring].last_avail_idx, + &hw->vring[vring].last_used_idx); + ifcvf_enable_vring_hw(&internal->hw, vring); + } else { + ifcvf_disable_vring_hw(&internal->hw, vring); + rte_vhost_set_vring_base(vid, vring, + hw->vring[vring].last_avail_idx, + hw->vring[vring].last_used_idx); + } + + return 0; +} + +static int ifcvf_set_vring_state(int vid, int vring, int state) { struct rte_vdpa_device *vdev; struct internal_list *list; struct ifcvf_internal *internal; struct ifcvf_hw *hw; - struct ifcvf_pci_common_cfg *cfg; + bool enable = !!state; int ret = 0; vdev = rte_vhost_get_vdpa_device(vid); @@ -1304,6 +1350,9 @@ struct rte_vdpa_dev_info { return -1; } + DRV_LOG(INFO, "%s queue %d of vDPA device %s", + enable ? "enable" : "disable", vring, vdev->device->name); + internal = list->internal; if (vring < 0 || vring >= internal->max_queues * 2) { DRV_LOG(ERR, "Vring index %d not correct", vring); @@ -1311,27 +1360,41 @@ struct rte_vdpa_dev_info { } hw = &internal->hw; + hw->vring[vring].enable = enable; + if (!internal->configured) - goto exit; + return 0; - cfg = hw->common_cfg; - IFCVF_WRITE_REG16(vring, &cfg->queue_select); - IFCVF_WRITE_REG16(!!state, &cfg->queue_enable); + unset_notify_relay(internal); - if (!state && hw->vring[vring].enable) { - ret = vdpa_disable_vfio_intr(internal); - if (ret) - return ret; + ret = vdpa_enable_vfio_intr(internal, false); + if (ret) { + DRV_LOG(ERR, "failed to set vfio interrupt of vDPA device %s", + vdev->device->name); + return ret; } - if (state && !hw->vring[vring].enable) { - ret = vdpa_enable_vfio_intr(internal, false); - if (ret) - return ret; + ret = ifcvf_config_vring(internal, vring); + if (ret) { + DRV_LOG(ERR, "failed to configure queue %d of vDPA device %s", + vring, vdev->device->name); + return ret; + } + + ret = setup_notify_relay(internal); + if (ret) { + DRV_LOG(ERR, "failed to setup notify relay of vDPA device %s", + vdev->device->name); + return ret; + } + + ret = rte_vhost_host_notifier_ctrl(vid, vring, enable); + if (ret) { + DRV_LOG(ERR, "vDPA device %s queue %d host notifier ctrl fail", + vdev->device->name, vring); + return ret; } -exit: - hw->vring[vring].enable = !!state; return 0; } From patchwork Thu Sep 8 05:54:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116081 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9CB9BA0548; Thu, 8 Sep 2022 07:54:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0436F42B73; Thu, 8 Sep 2022 07:53:38 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id A801042B74 for ; Thu, 8 Sep 2022 07:53:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616415; x=1694152415; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=mQBnveExC4aXZ9sRhnpJgI7f3HetDsIo3esXazcvIy8=; b=SNFON0x64/qRiALP2Lv1D6IBg/HIxouu8F6GIQn/fpZcnq04RH3I4FLv qZgoazKFdfHSiPHNUKrrKIJzc/6ABDYQxXRWLWLK4clpKbZY92rgB30ik vcNgYSD9G49fVH8iRB9gwMfj0NWj1udeW5RK2CoBzvdcvQz5cdtBcsMTT ZyOaEKD4tQmQezNuLKdQTU8qPKLIuQu69XnTLFgtodXXXfYfoVyb/ZJ4x i13mOseI5qFQJ598WOUueXD6wLN9garK8w0d63ePVSueZ0TmQtRj13Y8r C7aEX82OOnsbbXDzPbRb6S9OvflzIbHvll1mhTv7ZGnXY+ggBC/Rf+I02 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464198" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464198" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531872" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:33 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 7/8] vhost: configure device when any queue is ready for BLK device Date: Thu, 8 Sep 2022 13:54:17 +0800 Message-Id: <1662616458-164613-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When boot from virtio blk device, seabois in QEMU only enables one queue. To work in this scenario, vDPA BLK device back-end conf_dev when any queue is ready. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- lib/vhost/vhost_user.c | 56 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 44 insertions(+), 12 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 4ad28ba..b65fba3 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -1451,6 +1451,25 @@ #define VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY 2u static int +virtio_has_queue_ready(struct virtio_net *dev) +{ + struct vhost_virtqueue *vq; + uint32_t i, nr_vring = dev->nr_vring; + + if (!dev->nr_vring) + return 0; + + for (i = 0; i < nr_vring; i++) { + vq = dev->virtqueue[i]; + + if (vq_is_ready(dev, vq)) + return 1; + } + + return 0; +} + +static int virtio_is_ready(struct virtio_net *dev) { struct vhost_virtqueue *vq; @@ -3167,9 +3186,33 @@ static int is_vring_iotlb(struct virtio_net *dev, if (unlock_required) vhost_user_unlock_all_queue_pairs(dev); - if (ret != 0 || !virtio_is_ready(dev)) + if (ret != 0) goto out; + vdpa_dev = dev->vdpa_dev; + if (vdpa_dev) { + if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); + if (ret) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "failed to get vdpa dev type.\n"); + ret = -1; + goto out; + } + } else { + vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + } + } + + if (!virtio_is_ready(dev)) { + if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + if (!virtio_has_queue_ready(dev)) + goto out; + } else { + goto out; + } + } + /* * Virtio is now ready. If not done already, it is time * to notify the application it can process the rings and @@ -3181,20 +3224,9 @@ static int is_vring_iotlb(struct virtio_net *dev, dev->flags |= VIRTIO_DEV_RUNNING; } - vdpa_dev = dev->vdpa_dev; if (!vdpa_dev) goto out; - if (vdpa_dev->ops->get_dev_type) { - ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); - if (ret) { - VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to get vdpa dev type.\n"); - ret = -1; - goto out; - } - } else { - vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; - } if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK && request != VHOST_USER_SET_VRING_CALL) goto out; From patchwork Thu Sep 8 05:54:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116082 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DCDE0A0548; Thu, 8 Sep 2022 07:54:06 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2626042B80; Thu, 8 Sep 2022 07:53:40 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 8F1F74284D for ; Thu, 8 Sep 2022 07:53:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662616418; x=1694152418; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=7qG3Y51GWlAz7AHesqlycodc+Bd00jZmFjOa76lS9LY=; b=SJssE6KRqZIt9YBEJ/QFNl3GAOPACvgpUv+LZe0oexlT5G2VZbIPeC1E XEAx+Dgnwsz+9QKcKBxk5nbGyfe3QhxwVXpIRgUI22/d6Mq5dDOuPEUZg P05jq+a2sunTZHN9wGJN/1S581iZ/tDqg2cGNVoU8eiJFTTJQdOGHG0qi IrlFiphbhIDdM6WAx4uYBftbxg2uPWEln7JF178Bc6JHpFH42irDhbGLc qJ8kSgzNrkseXf6/QvAV1Z5ZQ2OHF8vNUJxwwUXq4VzTzL4eMwyBKb0PV MNTQlZq3zmF7nfl9kWcjeq2FRMTI+RXR2GoQeF7o8O1w9cb4LKjZsrEF6 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10463"; a="277464223" X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="277464223" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Sep 2022 22:53:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,299,1654585200"; d="scan'208";a="676531885" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga008.fm.intel.com with ESMTP; 07 Sep 2022 22:53:36 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com, Huang Wei Subject: [PATCH v2 8/8] vhost: vDPA BLK devices configure device when all queue callfds are set Date: Thu, 8 Sep 2022 13:54:18 +0800 Message-Id: <1662616458-164613-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1662616458-164613-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1662616458-164613-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In the virtio blk vDPA live migration use case, before the live migration process, QEMU will set call fd to vDPA back-end. QEMU and vDPA back-end stand by until live migration starts. During live migration process, QEMU sets kick fd and a new call fd. However, after the kick fd is set to the vDPA back-end, the vDPA back-end configures device and data path starts. The new call fd will cause some kind of "re-configuration", this kind of "re-configuration" cause IO drop. After this patch, vDPA back-end configures device after kick fd and call fd are well set and make sure no IO drops. This patch only impact virtio blk vDPA device and does not impact net device. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- lib/vhost/vhost_user.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index b65fba3..568030a 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2994,6 +2994,7 @@ static int is_vring_iotlb(struct virtio_net *dev, uint32_t vdpa_type = 0; uint32_t request; uint32_t i; + uint16_t blk_call_fd; dev = get_device(vid); if (dev == NULL) @@ -3227,9 +3228,15 @@ static int is_vring_iotlb(struct virtio_net *dev, if (!vdpa_dev) goto out; - if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK - && request != VHOST_USER_SET_VRING_CALL) - goto out; + if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + if (request == VHOST_USER_SET_VRING_CALL) { + blk_call_fd = ctx.msg.payload.u64 & VHOST_USER_VRING_IDX_MASK; + if (blk_call_fd != dev->nr_vring - 1) + goto out; + } else { + goto out; + } + } if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid))