[v15,2/2] eal/linux: remove KNI restriction on IOVA
Checks
Commit Message
From: Vamsi Attunuru <vattunuru@marvell.com>
Now that KNI supports VA (with kernel versions starting 4.6.0), we can
accept IOVA as VA, but KNI must be configured for this.
Pass iova_mode when creating KNI netdevs.
So far, IOVA detection policy forced IOVA as PA when KNI is loaded,
whatever the buses IOVA requirements were.
We can now use IOVA as VA, but this comes with a cost in KNI.
When no constraint is expressed by the buses, keep the current behavior
of choosing PA.
Note: this change supposes that dpdk is built on the same kernel than
the target system kernel; no objection has been expressed on this topic.
Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changelog since v14:
- reworded commitlog,
- added note on kernel version check,
- updated EAL documentation,
- fixed broken LTO link in release note update,
- s/eal/EAL/g,
- inverted kernel version check in KNI,
---
doc/guides/prog_guide/env_abstraction_layer.rst | 3 +++
doc/guides/prog_guide/kernel_nic_interface.rst | 14 ++++++++++++++
doc/guides/rel_notes/release_19_11.rst | 11 +++++++++++
lib/librte_eal/linux/eal/eal.c | 11 +++++++++--
lib/librte_kni/rte_kni.c | 5 +++++
5 files changed, 42 insertions(+), 2 deletions(-)
Comments
On Sun, Nov 17, 2019 at 8:43 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> From: Vamsi Attunuru <vattunuru@marvell.com>
>
> Now that KNI supports VA (with kernel versions starting 4.6.0), we can
> accept IOVA as VA, but KNI must be configured for this.
> Pass iova_mode when creating KNI netdevs.
>
> So far, IOVA detection policy forced IOVA as PA when KNI is loaded,
> whatever the buses IOVA requirements were.
>
> We can now use IOVA as VA, but this comes with a cost in KNI.
> When no constraint is expressed by the buses, keep the current behavior
> of choosing PA.
>
> Note: this change supposes that dpdk is built on the same kernel than
> the target system kernel; no objection has been expressed on this topic.
>
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Jerin Jacob <jerinj@marvell.com>
@@ -475,6 +475,9 @@ devices would fail anyway.
``RTE_PCI_DRV_NEED_IOVA_AS_VA`` flag is used to dictate that this PCI
driver can only work in RTE_IOVA_VA mode.
+ When the KNI kernel module is detected, RTE_IOVA_PA mode is preferred as a
+ performance penalty is expected in RTE_IOVA_VA mode.
+
IOVA Mode Configuration
~~~~~~~~~~~~~~~~~~~~~~~
@@ -300,6 +300,20 @@ The sk_buff is then freed and the mbuf sent in the tx_q FIFO.
The DPDK TX thread dequeues the mbuf and sends it to the PMD via ``rte_eth_tx_burst()``.
It then puts the mbuf back in the cache.
+IOVA = VA: Support
+------------------
+
+KNI operates in IOVA_VA scheme when
+
+- LINUX_VERSION_CODE >= KERNEL_VERSION(4, 6, 0) and
+- EAL option `iova-mode=va` is passed or bus IOVA scheme in the DPDK is selected
+ as RTE_IOVA_VA.
+
+Due to IOVA to KVA address translations, based on the KNI use case there
+can be a performance impact. For mitigation, forcing IOVA to PA via EAL
+"--iova-mode=pa" option can be used, IOVA_DC bus iommu scheme can also
+result in IOVA as PA.
+
Ethtool
-------
@@ -294,6 +294,17 @@ New Features
See :doc:`../prog_guide/lto` for more information:
+* **Added IOVA as VA support for KNI.**
+
+ * Added IOVA = VA support for KNI, KNI can operate in IOVA = VA mode when
+ `iova-mode=va` EAL option is passed to the application or when bus IOVA
+ scheme is selected as RTE_IOVA_VA. This mode only works on Linux Kernel
+ versions 4.6.0 and above.
+
+ * Due to IOVA to KVA address translations, based on the KNI use case there
+ can be a performance impact. For mitigation, forcing IOVA to PA via EAL
+ "--iova-mode=pa" option can be used, IOVA_DC bus iommu scheme can also
+ result in IOVA as PA.
Removed Items
@@ -1073,6 +1073,11 @@ rte_eal_init(int argc, char **argv)
*/
iova_mode = RTE_IOVA_VA;
RTE_LOG(DEBUG, EAL, "Physical addresses are unavailable, selecting IOVA as VA mode.\n");
+#if defined(RTE_LIBRTE_KNI) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 6, 0)
+ } else if (rte_eal_check_module("rte_kni") == 1) {
+ iova_mode = RTE_IOVA_PA;
+ RTE_LOG(DEBUG, EAL, "KNI is loaded, selecting IOVA as PA mode for better KNI perfomance.\n");
+#endif
} else if (is_iommu_enabled()) {
/* we have an IOMMU, pick IOVA as VA mode */
iova_mode = RTE_IOVA_VA;
@@ -1085,8 +1090,10 @@ rte_eal_init(int argc, char **argv)
RTE_LOG(DEBUG, EAL, "IOMMU is not available, selecting IOVA as PA mode.\n");
}
}
-#ifdef RTE_LIBRTE_KNI
- /* Workaround for KNI which requires physical address to work */
+#if defined(RTE_LIBRTE_KNI) && LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
+ /* Workaround for KNI which requires physical address to work
+ * in kernels < 4.6
+ */
if (iova_mode == RTE_IOVA_VA &&
rte_eal_check_module("rte_kni") == 1) {
if (phys_addrs) {
@@ -10,6 +10,7 @@
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
+#include <linux/version.h>
#include <rte_spinlock.h>
#include <rte_string_fns.h>
@@ -97,10 +98,12 @@ static volatile int kni_fd = -1;
int
rte_kni_init(unsigned int max_kni_ifaces __rte_unused)
{
+#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
if (rte_eal_iova_mode() != RTE_IOVA_PA) {
RTE_LOG(ERR, KNI, "KNI requires IOVA as PA\n");
return -1;
}
+#endif
/* Check FD and open */
if (kni_fd < 0) {
@@ -302,6 +305,8 @@ rte_kni_alloc(struct rte_mempool *pktmbuf_pool,
kni->group_id = conf->group_id;
kni->mbuf_size = conf->mbuf_size;
+ dev_info.iova_mode = (rte_eal_iova_mode() == RTE_IOVA_VA) ? 1 : 0;
+
ret = ioctl(kni_fd, RTE_KNI_IOCTL_CREATE, &dev_info);
if (ret < 0)
goto ioctl_fail;