[dpdk-dev,v10,6/9] eal: auto detect iova mode
Checks
Commit Message
iova autodetection depends on rte_bus_scan result. Result of bus scan will
have updated device_list and each device in that list has its '.kdev' state
updated. That kdrv state used to detect iova mapping mode for that device.
_device_parse() has dependency on rt_bus_scan so,
Below calls moved up in the eal initialization order:
- eal_option_device_parse
- rte_bus_scan
And based on the result of rte_bus_scan_iommu_class - select iova
mapping mode.
Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
Tested-by: Hemant Agrawal <hemant.agrawal@nxp.com>
---
lib/librte_eal/bsdapp/eal/eal.c | 27 ++++++++++++++++-----------
lib/librte_eal/linuxapp/eal/eal.c | 27 ++++++++++++++++-----------
2 files changed, 32 insertions(+), 22 deletions(-)
Comments
Hi Santosh,
On 10/06/2017 01:03 PM, Santosh Shukla wrote:
> iova autodetection depends on rte_bus_scan result. Result of bus scan will
> have updated device_list and each device in that list has its '.kdev' state
> updated. That kdrv state used to detect iova mapping mode for that device.
>
> _device_parse() has dependency on rt_bus_scan so,
> Below calls moved up in the eal initialization order:
> - eal_option_device_parse
> - rte_bus_scan
>
> And based on the result of rte_bus_scan_iommu_class - select iova
> mapping mode.
>
> Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> ---
> lib/librte_eal/bsdapp/eal/eal.c | 27 ++++++++++++++++-----------
> lib/librte_eal/linuxapp/eal/eal.c | 27 ++++++++++++++++-----------
> 2 files changed, 32 insertions(+), 22 deletions(-)
We noticed a regression on current master, which prevents to use Vhost
PMD with CONFIG_RTE_BUILD_SHARED_LIB=y:
# ./install/bin/testpmd --file-prefix=src -l 0,2 -n 4 --vdev
'net_vhost0,iface=/tmp/vhost-user2' -d ./install/lib/librte_pmd_vhost.so
-- --portmask=1 --disable-hw-vlan -i --rxq=1 --txq=1 --nb-cores=1
--eth-peer=0,52:54:00:11:22:12
EAL: Detected 4 lcore(s)
ERROR: failed to parse device "net_vhost0"
EAL: Unable to parse device 'net_vhost0,iface=/tmp/vhost-user2'
PANIC in main():
Cannot init EAL
5: [./install/bin/testpmd(_start+0x2a) [0x41e91a]]
4: [/lib64/libc.so.6(__libc_start_main+0xea) [0x7f551882550a]]
3: [./install/bin/testpmd(main+0x68e) [0x41e77e]]
2:
[/home/max/projects/src/mainline/dpdk/x86_64-native-linuxapp-gcc/lib/librte_eal.so.5.1(__rte_panic+0xba)
[0x7f551982c05a]]
1:
[/home/max/projects/src/mainline/dpdk/x86_64-native-linuxapp-gcc/lib/librte_eal.so.5.1(rte_dump_stack+0x1b)
[0x7f551983645b]]
Aborted (core dumped)
Git bisect seems to point to this patch:
$ git bisect log
git bisect start
# bad: [5518fc95427891e8bcf72f461cdaa38604226442] mempool/dpaa2: improve
error handling
git bisect bad 5518fc95427891e8bcf72f461cdaa38604226442
# good: [02657b4adcb8af773e26ec061b01cd7abdd3f0b6] version: 17.08.0
git bisect good 02657b4adcb8af773e26ec061b01cd7abdd3f0b6
# good: [4fa5e0bbc5730887a4a15b915bb15deb5ef1f607] net/dpaa: support
hashed RSS
git bisect good 4fa5e0bbc5730887a4a15b915bb15deb5ef1f607
# bad: [381acec2b1bd838c4a494b82c692db35573554da] eventdev: ease
single-link queue config requirements
git bisect bad 381acec2b1bd838c4a494b82c692db35573554da
# bad: [f1810113590373b157ebba555d6b51f38c8ca10f] config: enable igb_uio
on arm64
git bisect bad f1810113590373b157ebba555d6b51f38c8ca10f
# good: [69293c7762a0dbb3c28f5e93be00aaa49b52cb48] bus/fslmc: remove
unused funcs and align names in QBMAN
git bisect good 69293c7762a0dbb3c28f5e93be00aaa49b52cb48
# good: [f8244c6399d9fae6afab6770ae367aef38742ea5] ethdev: increase port
id range
git bisect good f8244c6399d9fae6afab6770ae367aef38742ea5
# bad: [680f6c12600f5d341c5968a1daeef7c5a055451b] mem: honor IOVA mode
in virt2phy
git bisect bad 680f6c12600f5d341c5968a1daeef7c5a055451b
# good: [a4f0a2dbe5abc2cadf0300fb4d5767b66254035d] pci: get IOMMU class
git bisect good a4f0a2dbe5abc2cadf0300fb4d5767b66254035d
# good: [93878cf0255e9dc21322ed99ad535adc048fa44f] eal: introduce helper
API for IOVA mode
git bisect good 93878cf0255e9dc21322ed99ad535adc048fa44f
# bad: [e85a919286d2543500bc384df206740845e85362] vfio: honor IOVA mode
before mapping
git bisect bad e85a919286d2543500bc384df206740845e85362
# bad: [cf408c22476c9f866deacac634dd17591e07a5c5] eal: auto detect IOVA mode
git bisect bad cf408c22476c9f866deacac634dd17591e07a5c5
# first bad commit: [cf408c22476c9f866deacac634dd17591e07a5c5] eal: auto
detect IOVA mode
This is the build commands I used to run the bisection:
sed -i 's/CONFIG_RTE_BUILD_SHARED_LIB=n/CONFIG_RTE_BUILD_SHARED_LIB=y/g'
config/common_base
make -j4 install T=x86_64-native-linuxapp-gcc DESTDIR=install
EXTRA_CFLAGS='-g'
sed -i 's/CONFIG_RTE_BUILD_SHARED_LIB=y/CONFIG_RTE_BUILD_SHARED_LIB=n/g'
config/common_base
Regards,
Maxime
13/10/2017 10:48, Maxime Coquelin:
> Hi Santosh,
>
> On 10/06/2017 01:03 PM, Santosh Shukla wrote:
> > iova autodetection depends on rte_bus_scan result. Result of bus scan will
> > have updated device_list and each device in that list has its '.kdev' state
> > updated. That kdrv state used to detect iova mapping mode for that device.
> >
> > _device_parse() has dependency on rt_bus_scan so,
> > Below calls moved up in the eal initialization order:
> > - eal_option_device_parse
> > - rte_bus_scan
> >
> > And based on the result of rte_bus_scan_iommu_class - select iova
> > mapping mode.
> >
> > Signed-off-by: Santosh Shukla <santosh.shukla@caviumnetworks.com>
> > Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> > Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > Tested-by: Hemant Agrawal <hemant.agrawal@nxp.com>
> > ---
> > lib/librte_eal/bsdapp/eal/eal.c | 27 ++++++++++++++++-----------
> > lib/librte_eal/linuxapp/eal/eal.c | 27 ++++++++++++++++-----------
> > 2 files changed, 32 insertions(+), 22 deletions(-)
>
> We noticed a regression on current master, which prevents to use Vhost
> PMD with CONFIG_RTE_BUILD_SHARED_LIB=y:
It was my guess during review:
http://dpdk.org/ml/archives/dev/2017-October/077863.html
I really don't understand how it can work,
because the bus scan is moved before shared libraries (plugins)
are loaded.
It will be even worst when PCI and vdev buses will be some
shared libraries.
Is it a chicken/egg issue?
If we cannot find a good solution, we may have to revert for RC1.
@@ -541,6 +541,22 @@ rte_eal_init(int argc, char **argv)
return -1;
}
+ if (eal_option_device_parse()) {
+ rte_errno = ENODEV;
+ rte_atomic32_clear(&run_once);
+ return -1;
+ }
+
+ if (rte_bus_scan()) {
+ rte_eal_init_alert("Cannot scan the buses for devices\n");
+ rte_errno = ENODEV;
+ rte_atomic32_clear(&run_once);
+ return -1;
+ }
+
+ /* autodetect the iova mapping mode (default is iova_pa) */
+ rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+
if (internal_config.no_hugetlbfs == 0 &&
internal_config.process_type != RTE_PROC_SECONDARY &&
eal_hugepage_info_init() < 0) {
@@ -620,17 +636,6 @@ rte_eal_init(int argc, char **argv)
rte_config.master_lcore, thread_id, cpuset,
ret == 0 ? "" : "...");
- if (eal_option_device_parse()) {
- rte_errno = ENODEV;
- return -1;
- }
-
- if (rte_bus_scan()) {
- rte_eal_init_alert("Cannot scan the buses for devices\n");
- rte_errno = ENODEV;
- return -1;
- }
-
RTE_LCORE_FOREACH_SLAVE(i) {
/*
@@ -798,6 +798,22 @@ rte_eal_init(int argc, char **argv)
return -1;
}
+ if (eal_option_device_parse()) {
+ rte_errno = ENODEV;
+ rte_atomic32_clear(&run_once);
+ return -1;
+ }
+
+ if (rte_bus_scan()) {
+ rte_eal_init_alert("Cannot scan the buses for devices\n");
+ rte_errno = ENODEV;
+ rte_atomic32_clear(&run_once);
+ return -1;
+ }
+
+ /* autodetect the iova mapping mode (default is iova_pa) */
+ rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+
if (internal_config.no_hugetlbfs == 0 &&
internal_config.process_type != RTE_PROC_SECONDARY &&
internal_config.xen_dom0_support == 0 &&
@@ -895,17 +911,6 @@ rte_eal_init(int argc, char **argv)
return -1;
}
- if (eal_option_device_parse()) {
- rte_errno = ENODEV;
- return -1;
- }
-
- if (rte_bus_scan()) {
- rte_eal_init_alert("Cannot scan the buses for devices\n");
- rte_errno = ENODEV;
- return -1;
- }
-
RTE_LCORE_FOREACH_SLAVE(i) {
/*