From patchwork Sun Nov 17 15:12:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 63065 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1A0E6A04B4; Sun, 17 Nov 2019 16:13:23 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1EC851B948; Sun, 17 Nov 2019 16:13:09 +0100 (CET) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) by dpdk.org (Postfix) with ESMTP id 5C2FA5B3E for ; Sun, 17 Nov 2019 16:13:07 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574003586; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DE3+/ZqHDSqHRqoaQu2vO7+cwb95SbnjWzoH/XLcFWE=; b=T1aFY3PRxtzdAx1EujqZ4bSp1gY+xamd41MwYt7rkGlpHRUwP9d4MmPriFj65NdY8OFfjl hXRgh6SzwFwKSGTm6+y/zs9FRWS0hlalk2yICO4vS/EIsmcBuHQSVm9Gd3t5KItZspOyG3 ogbbThhvWYGyFAr7d4VMsBppS9iB0V8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-3-iOM5ZaVONiCChd2bIMJfqg-1; Sun, 17 Nov 2019 10:13:03 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 63C1A801E5D; Sun, 17 Nov 2019 15:13:01 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.205.71]) by smtp.corp.redhat.com (Postfix) with ESMTP id 23F7C60171; Sun, 17 Nov 2019 15:12:58 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: thomas@monjalon.net, kirankumark@marvell.com, olivier.matz@6wind.com, ferruh.yigit@intel.com, anatoly.burakov@intel.com, arybchenko@solarflare.com, stephen@networkplumber.org, vattunuru@marvell.com Date: Sun, 17 Nov 2019 16:12:44 +0100 Message-Id: <20191117151244.3854-3-david.marchand@redhat.com> In-Reply-To: <20191117151244.3854-1-david.marchand@redhat.com> References: <20191117151244.3854-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: iOM5ZaVONiCChd2bIMJfqg-1 X-Mimecast-Spam-Score: 0 Subject: [dpdk-dev] [PATCH v15 2/2] eal/linux: remove KNI restriction on IOVA X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vamsi Attunuru Now that KNI supports VA (with kernel versions starting 4.6.0), we can accept IOVA as VA, but KNI must be configured for this. Pass iova_mode when creating KNI netdevs. So far, IOVA detection policy forced IOVA as PA when KNI is loaded, whatever the buses IOVA requirements were. We can now use IOVA as VA, but this comes with a cost in KNI. When no constraint is expressed by the buses, keep the current behavior of choosing PA. Note: this change supposes that dpdk is built on the same kernel than the target system kernel; no objection has been expressed on this topic. Signed-off-by: Vamsi Attunuru Signed-off-by: Kiran Kumar K Signed-off-by: David Marchand Reviewed-by: Jerin Jacob --- Changelog since v14: - reworded commitlog, - added note on kernel version check, - updated EAL documentation, - fixed broken LTO link in release note update, - s/eal/EAL/g, - inverted kernel version check in KNI, --- doc/guides/prog_guide/env_abstraction_layer.rst | 3 +++ doc/guides/prog_guide/kernel_nic_interface.rst | 14 ++++++++++++++ doc/guides/rel_notes/release_19_11.rst | 11 +++++++++++ lib/librte_eal/linux/eal/eal.c | 11 +++++++++-- lib/librte_kni/rte_kni.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-) diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst index cd8e3003e..6e7c2080a 100644 --- a/doc/guides/prog_guide/env_abstraction_layer.rst +++ b/doc/guides/prog_guide/env_abstraction_layer.rst @@ -475,6 +475,9 @@ devices would fail anyway. ``RTE_PCI_DRV_NEED_IOVA_AS_VA`` flag is used to dictate that this PCI driver can only work in RTE_IOVA_VA mode. + When the KNI kernel module is detected, RTE_IOVA_PA mode is preferred as a + performance penalty is expected in RTE_IOVA_VA mode. + IOVA Mode Configuration ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst index e12634ddc..c4479ffbf 100644 --- a/doc/guides/prog_guide/kernel_nic_interface.rst +++ b/doc/guides/prog_guide/kernel_nic_interface.rst @@ -300,6 +300,20 @@ The sk_buff is then freed and the mbuf sent in the tx_q FIFO. The DPDK TX thread dequeues the mbuf and sends it to the PMD via ``rte_eth_tx_burst()``. It then puts the mbuf back in the cache. +IOVA = VA: Support +------------------ + +KNI operates in IOVA_VA scheme when + +- LINUX_VERSION_CODE >= KERNEL_VERSION(4, 6, 0) and +- EAL option `iova-mode=va` is passed or bus IOVA scheme in the DPDK is selected + as RTE_IOVA_VA. + +Due to IOVA to KVA address translations, based on the KNI use case there +can be a performance impact. For mitigation, forcing IOVA to PA via EAL +"--iova-mode=pa" option can be used, IOVA_DC bus iommu scheme can also +result in IOVA as PA. + Ethtool ------- diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst index c0045a91f..21be600ab 100644 --- a/doc/guides/rel_notes/release_19_11.rst +++ b/doc/guides/rel_notes/release_19_11.rst @@ -294,6 +294,17 @@ New Features See :doc:`../prog_guide/lto` for more information: +* **Added IOVA as VA support for KNI.** + + * Added IOVA = VA support for KNI, KNI can operate in IOVA = VA mode when + `iova-mode=va` EAL option is passed to the application or when bus IOVA + scheme is selected as RTE_IOVA_VA. This mode only works on Linux Kernel + versions 4.6.0 and above. + + * Due to IOVA to KVA address translations, based on the KNI use case there + can be a performance impact. For mitigation, forcing IOVA to PA via EAL + "--iova-mode=pa" option can be used, IOVA_DC bus iommu scheme can also + result in IOVA as PA. Removed Items diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c index 9e2d50cfb..b5b71500c 100644 --- a/lib/librte_eal/linux/eal/eal.c +++ b/lib/librte_eal/linux/eal/eal.c @@ -1073,6 +1073,11 @@ rte_eal_init(int argc, char **argv) */ iova_mode = RTE_IOVA_VA; RTE_LOG(DEBUG, EAL, "Physical addresses are unavailable, selecting IOVA as VA mode.\n"); +#if defined(RTE_LIBRTE_KNI) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 6, 0) + } else if (rte_eal_check_module("rte_kni") == 1) { + iova_mode = RTE_IOVA_PA; + RTE_LOG(DEBUG, EAL, "KNI is loaded, selecting IOVA as PA mode for better KNI perfomance.\n"); +#endif } else if (is_iommu_enabled()) { /* we have an IOMMU, pick IOVA as VA mode */ iova_mode = RTE_IOVA_VA; @@ -1085,8 +1090,10 @@ rte_eal_init(int argc, char **argv) RTE_LOG(DEBUG, EAL, "IOMMU is not available, selecting IOVA as PA mode.\n"); } } -#ifdef RTE_LIBRTE_KNI - /* Workaround for KNI which requires physical address to work */ +#if defined(RTE_LIBRTE_KNI) && LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0) + /* Workaround for KNI which requires physical address to work + * in kernels < 4.6 + */ if (iova_mode == RTE_IOVA_VA && rte_eal_check_module("rte_kni") == 1) { if (phys_addrs) { diff --git a/lib/librte_kni/rte_kni.c b/lib/librte_kni/rte_kni.c index 7fbcf2201..86995fc81 100644 --- a/lib/librte_kni/rte_kni.c +++ b/lib/librte_kni/rte_kni.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -97,10 +98,12 @@ static volatile int kni_fd = -1; int rte_kni_init(unsigned int max_kni_ifaces __rte_unused) { +#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0) if (rte_eal_iova_mode() != RTE_IOVA_PA) { RTE_LOG(ERR, KNI, "KNI requires IOVA as PA\n"); return -1; } +#endif /* Check FD and open */ if (kni_fd < 0) { @@ -302,6 +305,8 @@ rte_kni_alloc(struct rte_mempool *pktmbuf_pool, kni->group_id = conf->group_id; kni->mbuf_size = conf->mbuf_size; + dev_info.iova_mode = (rte_eal_iova_mode() == RTE_IOVA_VA) ? 1 : 0; + ret = ioctl(kni_fd, RTE_KNI_IOCTL_CREATE, &dev_info); if (ret < 0) goto ioctl_fail;