From patchwork Wed Dec 14 07:47:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 120877 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 15905A0543; Wed, 14 Dec 2022 09:39:49 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F3A67400D6; Wed, 14 Dec 2022 09:39:48 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 2C2CF4003F for ; Wed, 14 Dec 2022 09:39:47 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1671007187; x=1702543187; h=from:to:cc:subject:date:message-id; bh=T69brXRRI/rt4+rBfHreF1ihUedLJDTRvwkeqJBnZQU=; b=NnNzlro0skwcWwnlMqRuPy1LLXj/GuUip/mmJF6e5Od8eGySt7UVnm0k iMC2TjflT8It/1IRBnIIysWBxb8PAznszq3EK1l7fLli5VBlT6Yp16rUz UXJdcq7T4iFxSiVPXMX8/bxC7s6RvYROuHcmUoa9nhU9+O4miErVMy7j9 rul4JSFqXQmyFknWSCay9QobztjA7kqpoZUhZjpGtY+6K4601ue7NlmvV gceUvtCDdadrQpARQbPYnTgCoImfKEGz5pVSasTIyQgD4ZrV68J6Y1WoC XLl8cnjYFpY72cvDiMTFNCGpyf/4iQ5owC8cBaygIsP+m8WkBaNopnFeR w==; X-IronPort-AV: E=McAfee;i="6500,9779,10560"; a="320208775" X-IronPort-AV: E=Sophos;i="5.96,243,1665471600"; d="scan'208";a="320208775" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Dec 2022 00:39:46 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10560"; a="648928856" X-IronPort-AV: E=Sophos;i="5.96,243,1665471600"; d="scan'208";a="648928856" Received: from dpdk-dipei.sh.intel.com ([10.67.110.229]) by orsmga002.jf.intel.com with ESMTP; 14 Dec 2022 00:39:39 -0800 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, maxime.coquelin@redhat.com, xiao.w.wang@intel.com Subject: [PATCH] vdpa/ifc: add live migration for block device Date: Wed, 14 Dec 2022 15:47:40 +0800 Message-Id: <1671004060-62997-1-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When we use "sw-live-migration=1" in arguments, we run in SW assisted live migration mode. In SW assisted live migration mode, driver will stop the device and setup a mediated virtio ring to relay the communication between the virtio driver and the VDPA device. For block devices, we have to make sure that each IO is completed. When we terminate the above mediated virtio ring, we stop notifying queue about new IOs, wait for all the in-flight IOs to be completed. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/ifcvf_vdpa.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 49d68ad..17838ac 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -865,8 +865,30 @@ struct rte_vdpa_dev_info { struct ifcvf_hw *hw = &internal->hw; uint64_t m_vring_iova = IFCVF_MEDIATED_VRING; uint64_t size, len; + u32 ring_state = 0; vid = internal->vid; + + /* to make sure no packet is lost for blk device + * do not stop until last_avail_idx == last_used_idx + */ + if (internal->hw.device_type == IFCVF_BLK) { + for (i = 0; i < hw->nr_vring; i++) { + do { + if (hw->lm_cfg != NULL) + ring_state = *(u32 *)(hw->lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + hw->vring[i].last_used_idx = + (u16)(ring_state >> 16); + usleep(10); + } while (hw->vring[i].last_avail_idx != + hw->vring[i].last_used_idx); + } + } + ifcvf_stop_hw(hw); for (i = 0; i < hw->nr_vring; i++) {