[dpdk-dev,v2,00/41] Memory Hotplug for DPDK
Checks
Commit Message
Hi Anatoly,
I am trying to verify this patchset and have encountered few issues.
Few -Werror=maybe-uninitialized errors in eal_memalloc.c/eal_memory.c/
eal_common_memzone.c files.
@Hemanth
Also, this patchset breaks dpaa/dpaa2 bus drivers (they rely on
`rte_eal_get_physmem_layout` that is depricated
http://dpdk.org/dev/patchwork/patch/34002/)
So, generic arm64 linuxapp build is broken.
Regards,
Pavan.
On Wed, Mar 07, 2018 at 04:56:28PM +0000, Anatoly Burakov wrote:
> This patchset introduces dynamic memory allocation for DPDK (aka memory
> hotplug). Based upon RFC submitted in December [1].
>
> Dependencies (to be applied in specified order):
> - IPC bugfixes patchset [2]
> - IPC improvements patchset [3]
> - IPC asynchronous request API patch [4]
> - Function to return number of sockets [5]
>
<snip>
--
> 2.7.4
Comments
On 08-Mar-18 10:18 AM, Pavan Nikhilesh wrote:
> Hi Anatoly,
>
> I am trying to verify this patchset and have encountered few issues.
>
> Few -Werror=maybe-uninitialized errors in eal_memalloc.c/eal_memory.c/
> eal_common_memzone.c files.
Thanks for the heads up, i'll fix those in the next revision. Out of
curiousity, which compiler version are you using?
>
> diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
> index a7cfdaf03..ad4413507 100644
> --- a/lib/librte_eal/common/eal_common_memzone.c
> +++ b/lib/librte_eal/common/eal_common_memzone.c
> @@ -321,7 +321,7 @@ rte_memzone_free(const struct rte_memzone *mz)
> struct rte_fbarray *arr;
> struct rte_memzone *found_mz;
> int ret = 0;
> - void *addr;
> + void *addr = NULL;
> unsigned idx;
>
> if (mz == NULL)
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memalloc.c b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> index 1008faed6..32b0d5133 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> @@ -570,7 +570,7 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n,
> unsigned int msl_idx;
> int cur_idx, start_idx, end_idx, i, j, ret = -1;
> #ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES
> - bool have_numa;
> + bool have_numa = false;
> int oldpolicy;
> struct bitmask *oldmask = numa_allocate_nodemask();
> #endif
> diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> index f74291fb6..d37b4a59b 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> @@ -1386,9 +1386,9 @@ eal_legacy_hugepage_attach(void)
> struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
> struct hugepage_file *hp = NULL;
> unsigned int num_hp = 0;
> - unsigned int i;
> + unsigned int i = 0;
> int ms_idx, msl_idx;
> - unsigned int cur_seg, max_seg;
> + unsigned int cur_seg, max_seg = 0;
> off_t size = 0;
> int fd, fd_hugepage = -1;
>
>
>
> @Hemanth
> Also, this patchset breaks dpaa/dpaa2 bus drivers (they rely on
> `rte_eal_get_physmem_layout` that is depricated
> http://dpdk.org/dev/patchwork/patch/34002/)
> So, generic arm64 linuxapp build is broken.
Should the deprecation notice have been accompanied with marking that
function as __rte_deprecated?
>
> Regards,
> Pavan.
>
> On Wed, Mar 07, 2018 at 04:56:28PM +0000, Anatoly Burakov wrote:
>> This patchset introduces dynamic memory allocation for DPDK (aka memory
>> hotplug). Based upon RFC submitted in December [1].
>>
>> Dependencies (to be applied in specified order):
>> - IPC bugfixes patchset [2]
>> - IPC improvements patchset [3]
>> - IPC asynchronous request API patch [4]
>> - Function to return number of sockets [5]
>>
> <snip>
> --
>> 2.7.4
>
On Thu, Mar 08, 2018 at 10:46:46AM +0000, Burakov, Anatoly wrote:
> On 08-Mar-18 10:18 AM, Pavan Nikhilesh wrote:
> > Hi Anatoly,
> >
> > I am trying to verify this patchset and have encountered few issues.
> >
> > Few -Werror=maybe-uninitialized errors in eal_memalloc.c/eal_memory.c/
> > eal_common_memzone.c files.
>
> Thanks for the heads up, i'll fix those in the next revision. Out of
> curiousity, which compiler version are you using?
I'm using gcc 5.3.0.
>
> >
> > diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
> > index a7cfdaf03..ad4413507 100644
> > --- a/lib/librte_eal/common/eal_common_memzone.c
> > +++ b/lib/librte_eal/common/eal_common_memzone.c
> > @@ -321,7 +321,7 @@ rte_memzone_free(const struct rte_memzone *mz)
> > struct rte_fbarray *arr;
> > struct rte_memzone *found_mz;
> > int ret = 0;
> > - void *addr;
> > + void *addr = NULL;
> > unsigned idx;
> >
> > if (mz == NULL)
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_memalloc.c b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > index 1008faed6..32b0d5133 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > @@ -570,7 +570,7 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n,
> > unsigned int msl_idx;
> > int cur_idx, start_idx, end_idx, i, j, ret = -1;
> > #ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES
> > - bool have_numa;
> > + bool have_numa = false;
> > int oldpolicy;
> > struct bitmask *oldmask = numa_allocate_nodemask();
> > #endif
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > index f74291fb6..d37b4a59b 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > @@ -1386,9 +1386,9 @@ eal_legacy_hugepage_attach(void)
> > struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
> > struct hugepage_file *hp = NULL;
> > unsigned int num_hp = 0;
> > - unsigned int i;
> > + unsigned int i = 0;
> > int ms_idx, msl_idx;
> > - unsigned int cur_seg, max_seg;
> > + unsigned int cur_seg, max_seg = 0;
> > off_t size = 0;
> > int fd, fd_hugepage = -1;
> >
> >
> >
> > @Hemanth
> > Also, this patchset breaks dpaa/dpaa2 bus drivers (they rely on
> > `rte_eal_get_physmem_layout` that is depricated
> > http://dpdk.org/dev/patchwork/patch/34002/)
> > So, generic arm64 linuxapp build is broken.
>
> Should the deprecation notice have been accompanied with marking that
> function as __rte_deprecated?
Yup that's the general sequence.
>
> >
> > Regards,
> > Pavan.
> >
> > On Wed, Mar 07, 2018 at 04:56:28PM +0000, Anatoly Burakov wrote:
> > > This patchset introduces dynamic memory allocation for DPDK (aka memory
> > > hotplug). Based upon RFC submitted in December [1].
> > >
> > > Dependencies (to be applied in specified order):
> > > - IPC bugfixes patchset [2]
> > > - IPC improvements patchset [3]
> > > - IPC asynchronous request API patch [4]
> > > - Function to return number of sockets [5]
> > >
> > <snip>
> > --
> > > 2.7.4
> >
>
>
> --
> Thanks,
> Anatoly
Hi Anatoly,
We are currently facing issues with running testpmd on thunderx platform.
The issue seems to be with vfio
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: No free hugepages reported in hugepages-2048kB
EAL: Multi-process socket /var/run/.rte_unix
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: VFIO support not initialized
<snip>
EAL: probe driver: 177d:a053 octeontx_fpavf
EAL: PCI device 0001:01:00.1 on NUMA socket 0
EAL: probe driver: 177d:a034 net_thunderx
EAL: using IOMMU type 1 (Type 1)
EAL: cannot set up DMA remapping, error 22 (Invalid argument)
EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
EAL: Requested device 0001:01:00.1 cannot be used
EAL: PCI device 0001:01:00.2 on NUMA socket 0
<snip>
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: VFIO support not initialized
EAL: VFIO support not initialized
EAL: VFIO support not initialized
Done
This is because rte_service_init() calls rte_calloc() before
rte_bus_probe() and vfio_dma_mem_map fails because iommu type is not set.
Call stack:
gdb) bt
#0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152, len=536870912, do_map=1) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
#1 0x00000000004fd974 in rte_vfio_dma_map (vaddr=281439006359552, iova=11274289152, len=536870912) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
#2 0x00000000004fbe78 in vfio_mem_event_callback (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
#3 0x00000000005070ac in eal_memalloc_notify (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912) at /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
#4 0x0000000000515c98 in try_expand_heap_primary (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
#5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
#6 0x00000000005163a0 in alloc_more_mem_on_socket (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
#7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90 "rte_services", size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
#8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90 "rte_services", size=8192, socket_arg=-1, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
#9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90 "rte_services", size=8192, align=128, socket_arg=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
#10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90 "rte_services", size=8192, align=128, socket=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
#11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90 "rte_services", size=8192, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
#12 0x0000000000513c90 in rte_calloc (type=0x85bf90 "rte_services", num=64, size=128, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
#13 0x0000000000518cec in rte_service_init () at /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
#14 0x00000000004f55f4 in rte_eal_init (argc=3, argv=0xfffffffff488) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
#15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at /root/clean/dpdk/app/test-pmd/testpmd.c:2483
Also, I have tried running with --legacy-mem but I'm stuck in
`pci_find_max_end_va` loop because `rte_fbarray_find_next_used` always return
0.
HugePages_Total: 15
HugePages_Free: 11
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 524288 kB
Call Stack:
(gdb) bt
#0 find_next (arr=0xffffb7fb009c, start=0, used=true) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
#1 0x00000000005132a8 in rte_fbarray_find_next_used (arr=0xffffb7fb009c, start=0) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
#2 0x000000000052d030 in pci_find_max_end_va () at /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
#3 0x0000000000530ab8 in pci_vfio_map_resource_primary (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
#4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
#5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
#6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20 <rte_nicvf_pmd>, dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
#7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
#8 0x0000000000531f68 in rte_pci_probe () at /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
#9 0x000000000050a140 in rte_bus_probe () at /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
#10 0x00000000004f55f4 in rte_eal_init (argc=1, argv=0xfffffffff498) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
#11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at /root/clean/dpdk/app/test-pmd/testpmd.c:2483
Am I missing something here?
Thanks,
Pavan.
On Thu, Mar 08, 2018 at 04:43:38PM +0530, Pavan Nikhilesh wrote:
> On Thu, Mar 08, 2018 at 10:46:46AM +0000, Burakov, Anatoly wrote:
> > On 08-Mar-18 10:18 AM, Pavan Nikhilesh wrote:
> > > Hi Anatoly,
> > >
> > > I am trying to verify this patchset and have encountered few issues.
> > >
> > > Few -Werror=maybe-uninitialized errors in eal_memalloc.c/eal_memory.c/
> > > eal_common_memzone.c files.
> >
> > Thanks for the heads up, i'll fix those in the next revision. Out of
> > curiousity, which compiler version are you using?
>
> I'm using gcc 5.3.0.
>
> >
> > >
> > > diff --git a/lib/librte_eal/common/eal_common_memzone.c b/lib/librte_eal/common/eal_common_memzone.c
> > > index a7cfdaf03..ad4413507 100644
> > > --- a/lib/librte_eal/common/eal_common_memzone.c
> > > +++ b/lib/librte_eal/common/eal_common_memzone.c
> > > @@ -321,7 +321,7 @@ rte_memzone_free(const struct rte_memzone *mz)
> > > struct rte_fbarray *arr;
> > > struct rte_memzone *found_mz;
> > > int ret = 0;
> > > - void *addr;
> > > + void *addr = NULL;
> > > unsigned idx;
> > >
> > > if (mz == NULL)
> > > diff --git a/lib/librte_eal/linuxapp/eal/eal_memalloc.c b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > > index 1008faed6..32b0d5133 100644
> > > --- a/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > > +++ b/lib/librte_eal/linuxapp/eal/eal_memalloc.c
> > > @@ -570,7 +570,7 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n,
> > > unsigned int msl_idx;
> > > int cur_idx, start_idx, end_idx, i, j, ret = -1;
> > > #ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES
> > > - bool have_numa;
> > > + bool have_numa = false;
> > > int oldpolicy;
> > > struct bitmask *oldmask = numa_allocate_nodemask();
> > > #endif
> > > diff --git a/lib/librte_eal/linuxapp/eal/eal_memory.c b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > > index f74291fb6..d37b4a59b 100644
> > > --- a/lib/librte_eal/linuxapp/eal/eal_memory.c
> > > +++ b/lib/librte_eal/linuxapp/eal/eal_memory.c
> > > @@ -1386,9 +1386,9 @@ eal_legacy_hugepage_attach(void)
> > > struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
> > > struct hugepage_file *hp = NULL;
> > > unsigned int num_hp = 0;
> > > - unsigned int i;
> > > + unsigned int i = 0;
> > > int ms_idx, msl_idx;
> > > - unsigned int cur_seg, max_seg;
> > > + unsigned int cur_seg, max_seg = 0;
> > > off_t size = 0;
> > > int fd, fd_hugepage = -1;
> > >
> > >
> > >
> > > @Hemanth
> > > Also, this patchset breaks dpaa/dpaa2 bus drivers (they rely on
> > > `rte_eal_get_physmem_layout` that is depricated
> > > http://dpdk.org/dev/patchwork/patch/34002/)
> > > So, generic arm64 linuxapp build is broken.
> >
> > Should the deprecation notice have been accompanied with marking that
> > function as __rte_deprecated?
>
> Yup that's the general sequence.
>
> >
> > >
> > > Regards,
> > > Pavan.
> > >
> > > On Wed, Mar 07, 2018 at 04:56:28PM +0000, Anatoly Burakov wrote:
> > > > This patchset introduces dynamic memory allocation for DPDK (aka memory
> > > > hotplug). Based upon RFC submitted in December [1].
> > > >
> > > > Dependencies (to be applied in specified order):
> > > > - IPC bugfixes patchset [2]
> > > > - IPC improvements patchset [3]
> > > > - IPC asynchronous request API patch [4]
> > > > - Function to return number of sockets [5]
> > > >
> > > <snip>
> > > --
> > > > 2.7.4
> > >
> >
> >
> > --
> > Thanks,
> > Anatoly
On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
> Hi Anatoly,
>
> We are currently facing issues with running testpmd on thunderx platform.
> The issue seems to be with vfio
>
> EAL: Detected 24 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: No free hugepages reported in hugepages-2048kB
> EAL: Multi-process socket /var/run/.rte_unix
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: VFIO support not initialized
>
> <snip>
>
> EAL: probe driver: 177d:a053 octeontx_fpavf
> EAL: PCI device 0001:01:00.1 on NUMA socket 0
> EAL: probe driver: 177d:a034 net_thunderx
> EAL: using IOMMU type 1 (Type 1)
> EAL: cannot set up DMA remapping, error 22 (Invalid argument)
> EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
> EAL: Requested device 0001:01:00.1 cannot be used
> EAL: PCI device 0001:01:00.2 on NUMA socket 0
> <snip>
> testpmd: No probed ethernet devices
> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> EAL: VFIO support not initialized
> EAL: VFIO support not initialized
> EAL: VFIO support not initialized
> Done
>
>
> This is because rte_service_init() calls rte_calloc() before
> rte_bus_probe() and vfio_dma_mem_map fails because iommu type is not set.
>
> Call stack:
> gdb) bt
> #0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152, len=536870912, do_map=1) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
> #1 0x00000000004fd974 in rte_vfio_dma_map (vaddr=281439006359552, iova=11274289152, len=536870912) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
> #2 0x00000000004fbe78 in vfio_mem_event_callback (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
> #3 0x00000000005070ac in eal_memalloc_notify (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912) at /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
> #4 0x0000000000515c98 in try_expand_heap_primary (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
> #5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
> #6 0x00000000005163a0 in alloc_more_mem_on_socket (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
> #7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90 "rte_services", size=8192, socket=0, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
> #8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90 "rte_services", size=8192, socket_arg=-1, flags=0, align=128, bound=0, contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
> #9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90 "rte_services", size=8192, align=128, socket_arg=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
> #10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90 "rte_services", size=8192, align=128, socket=-1) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
> #11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90 "rte_services", size=8192, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
> #12 0x0000000000513c90 in rte_calloc (type=0x85bf90 "rte_services", num=64, size=128, align=128) at /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
> #13 0x0000000000518cec in rte_service_init () at /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
> #14 0x00000000004f55f4 in rte_eal_init (argc=3, argv=0xfffffffff488) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
> #15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>
>
> Also, I have tried running with --legacy-mem but I'm stuck in
> `pci_find_max_end_va` loop because `rte_fbarray_find_next_used` always return
> 0. >
> HugePages_Total: 15
> HugePages_Free: 11
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 524288 kB
>
> Call Stack:
> (gdb) bt
> #0 find_next (arr=0xffffb7fb009c, start=0, used=true) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
> #1 0x00000000005132a8 in rte_fbarray_find_next_used (arr=0xffffb7fb009c, start=0) at /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
> #2 0x000000000052d030 in pci_find_max_end_va () at /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
> #3 0x0000000000530ab8 in pci_vfio_map_resource_primary (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
> #4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
> #5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
> #6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20 <rte_nicvf_pmd>, dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
> #7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
> #8 0x0000000000531f68 in rte_pci_probe () at /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
> #9 0x000000000050a140 in rte_bus_probe () at /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
> #10 0x00000000004f55f4 in rte_eal_init (argc=1, argv=0xfffffffff498) at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
> #11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>
> Am I missing something here?
I'll look into those, thanks!
Btw, i've now set up a github repo with the patchset applied:
https://github.com/anatolyburakov/dpdk
I will be pushing quick fixes there before spinning new revisions, so we
can discover and fix bugs more rapidly. I'll fix compile issues reported
earlier, then i'll take a look at your issues. The latter one seems like
a typo, the former is probably a matter of moving things around a bit.
(also, pull requests welcome if you find it easier to fix things
yourself and submit patches against my tree!)
Thanks for testing.
On 08-Mar-18 2:36 PM, Burakov, Anatoly wrote:
> On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
>> Hi Anatoly,
>>
>> We are currently facing issues with running testpmd on thunderx platform.
>> The issue seems to be with vfio
>>
>> EAL: Detected 24 lcore(s)
>> EAL: Detected 1 NUMA nodes
>> EAL: No free hugepages reported in hugepages-2048kB
>> EAL: Multi-process socket /var/run/.rte_unix
>> EAL: Probing VFIO support...
>> EAL: VFIO support initialized
>> EAL: VFIO support not initialized
>>
>> <snip>
>>
>> EAL: probe driver: 177d:a053 octeontx_fpavf
>> EAL: PCI device 0001:01:00.1 on NUMA socket 0
>> EAL: probe driver: 177d:a034 net_thunderx
>> EAL: using IOMMU type 1 (Type 1)
>> EAL: cannot set up DMA remapping, error 22 (Invalid argument)
>> EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
>> EAL: Requested device 0001:01:00.1 cannot be used
>> EAL: PCI device 0001:01:00.2 on NUMA socket 0
>> <snip>
>> testpmd: No probed ethernet devices
>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456,
>> size=2176, socket=0
>> testpmd: preferred mempool ops selected: ring_mp_mc
>> EAL: VFIO support not initialized
>> EAL: VFIO support not initialized
>> EAL: VFIO support not initialized
>> Done
>>
>>
>> This is because rte_service_init() calls rte_calloc() before
>> rte_bus_probe() and vfio_dma_mem_map fails because iommu type is not set.
>>
>> Call stack:
>> gdb) bt
>> #0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152,
>> len=536870912, do_map=1) at
>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
>> #1 0x00000000004fd974 in rte_vfio_dma_map (vaddr=281439006359552,
>> iova=11274289152, len=536870912) at
>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
>> #2 0x00000000004fbe78 in vfio_mem_event_callback
>> (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912) at
>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
>> #3 0x00000000005070ac in eal_memalloc_notify
>> (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912) at
>> /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
>> #4 0x0000000000515c98 in try_expand_heap_primary
>> (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0,
>> flags=0, align=128, bound=0, contig=false) at
>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
>> #5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c,
>> pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128, bound=0,
>> contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
>> #6 0x00000000005163a0 in alloc_more_mem_on_socket
>> (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128,
>> bound=0, contig=false) at
>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
>> #7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90
>> "rte_services", size=8192, socket=0, flags=0, align=128, bound=0,
>> contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
>> #8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90
>> "rte_services", size=8192, socket_arg=-1, flags=0, align=128, bound=0,
>> contig=false) at /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
>> #9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90
>> "rte_services", size=8192, align=128, socket_arg=-1) at
>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
>> #10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90
>> "rte_services", size=8192, align=128, socket=-1) at
>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
>> #11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90 "rte_services",
>> size=8192, align=128) at
>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
>> #12 0x0000000000513c90 in rte_calloc (type=0x85bf90 "rte_services",
>> num=64, size=128, align=128) at
>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
>> #13 0x0000000000518cec in rte_service_init () at
>> /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
>> #14 0x00000000004f55f4 in rte_eal_init (argc=3, argv=0xfffffffff488)
>> at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
>> #15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at
>> /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>>
>>
>> Also, I have tried running with --legacy-mem but I'm stuck in
>> `pci_find_max_end_va` loop because `rte_fbarray_find_next_used`
>> always return
>> 0. >
>> HugePages_Total: 15
>> HugePages_Free: 11
>> HugePages_Rsvd: 0
>> HugePages_Surp: 0
>> Hugepagesize: 524288 kB
>>
>> Call Stack:
>> (gdb) bt
>> #0 find_next (arr=0xffffb7fb009c, start=0, used=true) at
>> /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
>> #1 0x00000000005132a8 in rte_fbarray_find_next_used
>> (arr=0xffffb7fb009c, start=0) at
>> /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
>> #2 0x000000000052d030 in pci_find_max_end_va () at
>> /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
>> #3 0x0000000000530ab8 in pci_vfio_map_resource_primary (dev=0xeae700)
>> at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
>> #4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700) at
>> /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
>> #5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at
>> /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
>> #6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20
>> <rte_nicvf_pmd>, dev=0xeae700) at
>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
>> #7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700) at
>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
>> #8 0x0000000000531f68 in rte_pci_probe () at
>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
>> #9 0x000000000050a140 in rte_bus_probe () at
>> /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
>> #10 0x00000000004f55f4 in rte_eal_init (argc=1, argv=0xfffffffff498)
>> at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
>> #11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at
>> /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>>
>> Am I missing something here?
>
> I'll look into those, thanks!
>
> Btw, i've now set up a github repo with the patchset applied:
>
> https://github.com/anatolyburakov/dpdk
>
> I will be pushing quick fixes there before spinning new revisions, so we
> can discover and fix bugs more rapidly. I'll fix compile issues reported
> earlier, then i'll take a look at your issues. The latter one seems like
> a typo, the former is probably a matter of moving things around a bit.
>
> (also, pull requests welcome if you find it easier to fix things
> yourself and submit patches against my tree!)
>
> Thanks for testing.
>
I've looked into the failures.
The VFIO one is not actually a failure. It only prints out errors
because rte_malloc is called before VFIO is initialized. However, once
VFIO *is* initialized, all of that memory would be added to VFIO, so
these error messages are harmless. Regardless, i've added a check to see
if init is finished before printing out those errors, so they won't be
printed out any more.
Second one is a typo on my part that got lost in one of the rebases.
I've pushed fixes for both into the github repo.
On 08-Mar-18 8:11 PM, Burakov, Anatoly wrote:
> On 08-Mar-18 2:36 PM, Burakov, Anatoly wrote:
>> On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
>>> Hi Anatoly,
>>>
>>> We are currently facing issues with running testpmd on thunderx
>>> platform.
>>> The issue seems to be with vfio
>>>
>>> EAL: Detected 24 lcore(s)
>>> EAL: Detected 1 NUMA nodes
>>> EAL: No free hugepages reported in hugepages-2048kB
>>> EAL: Multi-process socket /var/run/.rte_unix
>>> EAL: Probing VFIO support...
>>> EAL: VFIO support initialized
>>> EAL: VFIO support not initialized
>>>
>>> <snip>
>>>
>>> EAL: probe driver: 177d:a053 octeontx_fpavf
>>> EAL: PCI device 0001:01:00.1 on NUMA socket 0
>>> EAL: probe driver: 177d:a034 net_thunderx
>>> EAL: using IOMMU type 1 (Type 1)
>>> EAL: cannot set up DMA remapping, error 22 (Invalid argument)
>>> EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
>>> EAL: Requested device 0001:01:00.1 cannot be used
>>> EAL: PCI device 0001:01:00.2 on NUMA socket 0
>>> <snip>
>>> testpmd: No probed ethernet devices
>>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456,
>>> size=2176, socket=0
>>> testpmd: preferred mempool ops selected: ring_mp_mc
>>> EAL: VFIO support not initialized
>>> EAL: VFIO support not initialized
>>> EAL: VFIO support not initialized
>>> Done
>>>
>>>
>>> This is because rte_service_init() calls rte_calloc() before
>>> rte_bus_probe() and vfio_dma_mem_map fails because iommu type is not
>>> set.
>>>
>>> Call stack:
>>> gdb) bt
>>> #0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152,
>>> len=536870912, do_map=1) at
>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
>>> #1 0x00000000004fd974 in rte_vfio_dma_map (vaddr=281439006359552,
>>> iova=11274289152, len=536870912) at
>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
>>> #2 0x00000000004fbe78 in vfio_mem_event_callback
>>> (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912) at
>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
>>> #3 0x00000000005070ac in eal_memalloc_notify
>>> (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912) at
>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
>>> #4 0x0000000000515c98 in try_expand_heap_primary
>>> (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0,
>>> flags=0, align=128, bound=0, contig=false) at
>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
>>> #5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c,
>>> pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128,
>>> bound=0, contig=false) at
>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
>>> #6 0x00000000005163a0 in alloc_more_mem_on_socket
>>> (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128,
>>> bound=0, contig=false) at
>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
>>> #7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90
>>> "rte_services", size=8192, socket=0, flags=0, align=128, bound=0,
>>> contig=false) at
>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
>>> #8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90
>>> "rte_services", size=8192, socket_arg=-1, flags=0, align=128,
>>> bound=0, contig=false) at
>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
>>> #9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90
>>> "rte_services", size=8192, align=128, socket_arg=-1) at
>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
>>> #10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90
>>> "rte_services", size=8192, align=128, socket=-1) at
>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
>>> #11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90 "rte_services",
>>> size=8192, align=128) at
>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
>>> #12 0x0000000000513c90 in rte_calloc (type=0x85bf90 "rte_services",
>>> num=64, size=128, align=128) at
>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
>>> #13 0x0000000000518cec in rte_service_init () at
>>> /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
>>> #14 0x00000000004f55f4 in rte_eal_init (argc=3, argv=0xfffffffff488)
>>> at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
>>> #15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at
>>> /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>>>
>>>
>>> Also, I have tried running with --legacy-mem but I'm stuck in
>>> `pci_find_max_end_va` loop because `rte_fbarray_find_next_used`
>>> always return
>>> 0. >
>>> HugePages_Total: 15
>>> HugePages_Free: 11
>>> HugePages_Rsvd: 0
>>> HugePages_Surp: 0
>>> Hugepagesize: 524288 kB
>>>
>>> Call Stack:
>>> (gdb) bt
>>> #0 find_next (arr=0xffffb7fb009c, start=0, used=true) at
>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
>>> #1 0x00000000005132a8 in rte_fbarray_find_next_used
>>> (arr=0xffffb7fb009c, start=0) at
>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
>>> #2 0x000000000052d030 in pci_find_max_end_va () at
>>> /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
>>> #3 0x0000000000530ab8 in pci_vfio_map_resource_primary
>>> (dev=0xeae700) at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
>>> #4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700) at
>>> /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
>>> #5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at
>>> /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
>>> #6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20
>>> <rte_nicvf_pmd>, dev=0xeae700) at
>>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
>>> #7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700) at
>>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
>>> #8 0x0000000000531f68 in rte_pci_probe () at
>>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
>>> #9 0x000000000050a140 in rte_bus_probe () at
>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
>>> #10 0x00000000004f55f4 in rte_eal_init (argc=1, argv=0xfffffffff498)
>>> at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
>>> #11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at
>>> /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>>>
>>> Am I missing something here?
>>
>> I'll look into those, thanks!
>>
>> Btw, i've now set up a github repo with the patchset applied:
>>
>> https://github.com/anatolyburakov/dpdk
>>
>> I will be pushing quick fixes there before spinning new revisions, so
>> we can discover and fix bugs more rapidly. I'll fix compile issues
>> reported earlier, then i'll take a look at your issues. The latter one
>> seems like a typo, the former is probably a matter of moving things
>> around a bit.
>>
>> (also, pull requests welcome if you find it easier to fix things
>> yourself and submit patches against my tree!)
>>
>> Thanks for testing.
>>
>
> I've looked into the failures.
>
> The VFIO one is not actually a failure. It only prints out errors
> because rte_malloc is called before VFIO is initialized. However, once
> VFIO *is* initialized, all of that memory would be added to VFIO, so
> these error messages are harmless. Regardless, i've added a check to see
> if init is finished before printing out those errors, so they won't be
> printed out any more.
>
> Second one is a typo on my part that got lost in one of the rebases.
>
> I've pushed fixes for both into the github repo.
>
Although i do wonder where do the DMA remapping errors come from. The
error message says "invalid argument", so that doesn't come from
rte_service or anything to do with rte_malloc - this is us not providing
valid arguments to VFIO. I'm not seeing these errors on my system. I'll
check on others to be sure.
On Thu, Mar 08, 2018 at 08:33:21PM +0000, Burakov, Anatoly wrote:
> On 08-Mar-18 8:11 PM, Burakov, Anatoly wrote:
> > On 08-Mar-18 2:36 PM, Burakov, Anatoly wrote:
> > > On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
> > > > Hi Anatoly,
> > > >
> > > > We are currently facing issues with running testpmd on thunderx
> > > > platform.
> > > > The issue seems to be with vfio
> > > >
> > > > EAL: Detected 24 lcore(s)
> > > > EAL: Detected 1 NUMA nodes
> > > > EAL: No free hugepages reported in hugepages-2048kB
> > > > EAL: Multi-process socket /var/run/.rte_unix
> > > > EAL: Probing VFIO support...
> > > > EAL: VFIO support initialized
> > > > EAL: VFIO support not initialized
> > > >
> > > > <snip>
> > > >
> > > > EAL: probe driver: 177d:a053 octeontx_fpavf
> > > > EAL: PCI device 0001:01:00.1 on NUMA socket 0
> > > > EAL: probe driver: 177d:a034 net_thunderx
> > > > EAL: using IOMMU type 1 (Type 1)
> > > > EAL: cannot set up DMA remapping, error 22 (Invalid argument)
> > > > EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
> > > > EAL: Requested device 0001:01:00.1 cannot be used
> > > > EAL: PCI device 0001:01:00.2 on NUMA socket 0
> > > > <snip>
> > > > testpmd: No probed ethernet devices
> > > > testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456,
> > > > size=2176, socket=0
> > > > testpmd: preferred mempool ops selected: ring_mp_mc
> > > > EAL: VFIO support not initialized
> > > > EAL: VFIO support not initialized
> > > > EAL: VFIO support not initialized
> > > > Done
> > > >
> > > >
> > > > This is because rte_service_init() calls rte_calloc() before
> > > > rte_bus_probe() and vfio_dma_mem_map fails because iommu type is
> > > > not set.
> > > >
> > > > Call stack:
> > > > gdb) bt
> > > > #0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152,
> > > > len=536870912, do_map=1) at
> > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
> > > > #1 0x00000000004fd974 in rte_vfio_dma_map
> > > > (vaddr=281439006359552, iova=11274289152, len=536870912) at
> > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
> > > > #2 0x00000000004fbe78 in vfio_mem_event_callback
> > > > (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912)
> > > > at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
> > > > #3 0x00000000005070ac in eal_memalloc_notify
> > > > (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912)
> > > > at
> > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
> > > > #4 0x0000000000515c98 in try_expand_heap_primary
> > > > (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0,
> > > > flags=0, align=128, bound=0, contig=false) at
> > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
> > > > #5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c,
> > > > pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128,
> > > > bound=0, contig=false) at
> > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
> > > > #6 0x00000000005163a0 in alloc_more_mem_on_socket
> > > > (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128,
> > > > bound=0, contig=false) at
> > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
> > > > #7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90
> > > > "rte_services", size=8192, socket=0, flags=0, align=128,
> > > > bound=0, contig=false) at
> > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
> > > > #8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90
> > > > "rte_services", size=8192, socket_arg=-1, flags=0, align=128,
> > > > bound=0, contig=false) at
> > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
> > > > #9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90
> > > > "rte_services", size=8192, align=128, socket_arg=-1) at
> > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
> > > > #10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90
> > > > "rte_services", size=8192, align=128, socket=-1) at
> > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
> > > > #11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90
> > > > "rte_services", size=8192, align=128) at
> > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
> > > > #12 0x0000000000513c90 in rte_calloc (type=0x85bf90
> > > > "rte_services", num=64, size=128, align=128) at
> > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
> > > > #13 0x0000000000518cec in rte_service_init () at
> > > > /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
> > > > #14 0x00000000004f55f4 in rte_eal_init (argc=3,
> > > > argv=0xfffffffff488) at
> > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
> > > > #15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at
> > > > /root/clean/dpdk/app/test-pmd/testpmd.c:2483
> > > >
> > > >
> > > > Also, I have tried running with --legacy-mem but I'm stuck in
> > > > `pci_find_max_end_va` loop because `rte_fbarray_find_next_used`
> > > > always return
> > > > 0. >
> > > > HugePages_Total: 15
> > > > HugePages_Free: 11
> > > > HugePages_Rsvd: 0
> > > > HugePages_Surp: 0
> > > > Hugepagesize: 524288 kB
> > > >
> > > > Call Stack:
> > > > (gdb) bt
> > > > #0 find_next (arr=0xffffb7fb009c, start=0, used=true) at
> > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
> > > > #1 0x00000000005132a8 in rte_fbarray_find_next_used
> > > > (arr=0xffffb7fb009c, start=0) at
> > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
> > > > #2 0x000000000052d030 in pci_find_max_end_va () at
> > > > /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
> > > > #3 0x0000000000530ab8 in pci_vfio_map_resource_primary
> > > > (dev=0xeae700) at
> > > > /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
> > > > #4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700)
> > > > at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
> > > > #5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at
> > > > /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
> > > > #6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20
> > > > <rte_nicvf_pmd>, dev=0xeae700) at
> > > > /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
> > > > #7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700)
> > > > at /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
> > > > #8 0x0000000000531f68 in rte_pci_probe () at
> > > > /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
> > > > #9 0x000000000050a140 in rte_bus_probe () at
> > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
> > > > #10 0x00000000004f55f4 in rte_eal_init (argc=1,
> > > > argv=0xfffffffff498) at
> > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
> > > > #11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at
> > > > /root/clean/dpdk/app/test-pmd/testpmd.c:2483
> > > >
> > > > Am I missing something here?
> > >
> > > I'll look into those, thanks!
> > >
> > > Btw, i've now set up a github repo with the patchset applied:
> > >
> > > https://github.com/anatolyburakov/dpdk
> > >
> > > I will be pushing quick fixes there before spinning new revisions,
> > > so we can discover and fix bugs more rapidly. I'll fix compile
> > > issues reported earlier, then i'll take a look at your issues. The
> > > latter one seems like a typo, the former is probably a matter of
> > > moving things around a bit.
> > >
> > > (also, pull requests welcome if you find it easier to fix things
> > > yourself and submit patches against my tree!)
> > >
> > > Thanks for testing.
> > >
> >
> > I've looked into the failures.
> >
> > The VFIO one is not actually a failure. It only prints out errors
> > because rte_malloc is called before VFIO is initialized. However, once
> > VFIO *is* initialized, all of that memory would be added to VFIO, so
> > these error messages are harmless. Regardless, i've added a check to see
> > if init is finished before printing out those errors, so they won't be
> > printed out any more.
> >
> > Second one is a typo on my part that got lost in one of the rebases.
> >
> > I've pushed fixes for both into the github repo.
> >
>
> Although i do wonder where do the DMA remapping errors come from. The error
> message says "invalid argument", so that doesn't come from rte_service or
> anything to do with rte_malloc - this is us not providing valid arguments to
> VFIO. I'm not seeing these errors on my system. I'll check on others to be
> sure.
I have taken a look at the github tree the issues with VFIO are gone, Although
compilation issues with dpaa/dpaa2 are still present due to their dependency on
`rte_eal_get_physmem_layout`.
>
> --
> Thanks,
> Anatoly
Thanks,
Pavan
On 09-Mar-18 9:15 AM, Pavan Nikhilesh wrote:
> On Thu, Mar 08, 2018 at 08:33:21PM +0000, Burakov, Anatoly wrote:
>> On 08-Mar-18 8:11 PM, Burakov, Anatoly wrote:
>>> On 08-Mar-18 2:36 PM, Burakov, Anatoly wrote:
>>>> On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
>>>>> Hi Anatoly,
>>>>>
>>>>> We are currently facing issues with running testpmd on thunderx
>>>>> platform.
>>>>> The issue seems to be with vfio
>>>>>
>>>>> EAL: Detected 24 lcore(s)
>>>>> EAL: Detected 1 NUMA nodes
>>>>> EAL: No free hugepages reported in hugepages-2048kB
>>>>> EAL: Multi-process socket /var/run/.rte_unix
>>>>> EAL: Probing VFIO support...
>>>>> EAL: VFIO support initialized
>>>>> EAL: VFIO support not initialized
>>>>>
>>>>> <snip>
>>>>>
>>>>> EAL: probe driver: 177d:a053 octeontx_fpavf
>>>>> EAL: PCI device 0001:01:00.1 on NUMA socket 0
>>>>> EAL: probe driver: 177d:a034 net_thunderx
>>>>> EAL: using IOMMU type 1 (Type 1)
>>>>> EAL: cannot set up DMA remapping, error 22 (Invalid argument)
>>>>> EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
>>>>> EAL: Requested device 0001:01:00.1 cannot be used
>>>>> EAL: PCI device 0001:01:00.2 on NUMA socket 0
>>>>> <snip>
>>>>> testpmd: No probed ethernet devices
>>>>> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456,
>>>>> size=2176, socket=0
>>>>> testpmd: preferred mempool ops selected: ring_mp_mc
>>>>> EAL: VFIO support not initialized
>>>>> EAL: VFIO support not initialized
>>>>> EAL: VFIO support not initialized
>>>>> Done
>>>>>
>>>>>
>>>>> This is because rte_service_init() calls rte_calloc() before
>>>>> rte_bus_probe() and vfio_dma_mem_map fails because iommu type is
>>>>> not set.
>>>>>
>>>>> Call stack:
>>>>> gdb) bt
>>>>> #0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152,
>>>>> len=536870912, do_map=1) at
>>>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
>>>>> #1 0x00000000004fd974 in rte_vfio_dma_map
>>>>> (vaddr=281439006359552, iova=11274289152, len=536870912) at
>>>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
>>>>> #2 0x00000000004fbe78 in vfio_mem_event_callback
>>>>> (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912)
>>>>> at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
>>>>> #3 0x00000000005070ac in eal_memalloc_notify
>>>>> (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912)
>>>>> at
>>>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
>>>>> #4 0x0000000000515c98 in try_expand_heap_primary
>>>>> (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0,
>>>>> flags=0, align=128, bound=0, contig=false) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
>>>>> #5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c,
>>>>> pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128,
>>>>> bound=0, contig=false) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
>>>>> #6 0x00000000005163a0 in alloc_more_mem_on_socket
>>>>> (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128,
>>>>> bound=0, contig=false) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
>>>>> #7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90
>>>>> "rte_services", size=8192, socket=0, flags=0, align=128,
>>>>> bound=0, contig=false) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
>>>>> #8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90
>>>>> "rte_services", size=8192, socket_arg=-1, flags=0, align=128,
>>>>> bound=0, contig=false) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
>>>>> #9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90
>>>>> "rte_services", size=8192, align=128, socket_arg=-1) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
>>>>> #10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90
>>>>> "rte_services", size=8192, align=128, socket=-1) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
>>>>> #11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90
>>>>> "rte_services", size=8192, align=128) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
>>>>> #12 0x0000000000513c90 in rte_calloc (type=0x85bf90
>>>>> "rte_services", num=64, size=128, align=128) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
>>>>> #13 0x0000000000518cec in rte_service_init () at
>>>>> /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
>>>>> #14 0x00000000004f55f4 in rte_eal_init (argc=3,
>>>>> argv=0xfffffffff488) at
>>>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
>>>>> #15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at
>>>>> /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>>>>>
>>>>>
>>>>> Also, I have tried running with --legacy-mem but I'm stuck in
>>>>> `pci_find_max_end_va` loop because `rte_fbarray_find_next_used`
>>>>> always return
>>>>> 0. >
>>>>> HugePages_Total: 15
>>>>> HugePages_Free: 11
>>>>> HugePages_Rsvd: 0
>>>>> HugePages_Surp: 0
>>>>> Hugepagesize: 524288 kB
>>>>>
>>>>> Call Stack:
>>>>> (gdb) bt
>>>>> #0 find_next (arr=0xffffb7fb009c, start=0, used=true) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
>>>>> #1 0x00000000005132a8 in rte_fbarray_find_next_used
>>>>> (arr=0xffffb7fb009c, start=0) at
>>>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
>>>>> #2 0x000000000052d030 in pci_find_max_end_va () at
>>>>> /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
>>>>> #3 0x0000000000530ab8 in pci_vfio_map_resource_primary
>>>>> (dev=0xeae700) at
>>>>> /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
>>>>> #4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700)
>>>>> at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
>>>>> #5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at
>>>>> /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
>>>>> #6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20
>>>>> <rte_nicvf_pmd>, dev=0xeae700) at
>>>>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
>>>>> #7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700)
>>>>> at /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
>>>>> #8 0x0000000000531f68 in rte_pci_probe () at
>>>>> /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
>>>>> #9 0x000000000050a140 in rte_bus_probe () at
>>>>> /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
>>>>> #10 0x00000000004f55f4 in rte_eal_init (argc=1,
>>>>> argv=0xfffffffff498) at
>>>>> /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
>>>>> #11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at
>>>>> /root/clean/dpdk/app/test-pmd/testpmd.c:2483
>>>>>
>>>>> Am I missing something here?
>>>>
>>>> I'll look into those, thanks!
>>>>
>>>> Btw, i've now set up a github repo with the patchset applied:
>>>>
>>>> https://github.com/anatolyburakov/dpdk
>>>>
>>>> I will be pushing quick fixes there before spinning new revisions,
>>>> so we can discover and fix bugs more rapidly. I'll fix compile
>>>> issues reported earlier, then i'll take a look at your issues. The
>>>> latter one seems like a typo, the former is probably a matter of
>>>> moving things around a bit.
>>>>
>>>> (also, pull requests welcome if you find it easier to fix things
>>>> yourself and submit patches against my tree!)
>>>>
>>>> Thanks for testing.
>>>>
>>>
>>> I've looked into the failures.
>>>
>>> The VFIO one is not actually a failure. It only prints out errors
>>> because rte_malloc is called before VFIO is initialized. However, once
>>> VFIO *is* initialized, all of that memory would be added to VFIO, so
>>> these error messages are harmless. Regardless, i've added a check to see
>>> if init is finished before printing out those errors, so they won't be
>>> printed out any more.
>>>
>>> Second one is a typo on my part that got lost in one of the rebases.
>>>
>>> I've pushed fixes for both into the github repo.
>>>
>>
>> Although i do wonder where do the DMA remapping errors come from. The error
>> message says "invalid argument", so that doesn't come from rte_service or
>> anything to do with rte_malloc - this is us not providing valid arguments to
>> VFIO. I'm not seeing these errors on my system. I'll check on others to be
>> sure.
>
> I have taken a look at the github tree the issues with VFIO are gone, Although
> compilation issues with dpaa/dpaa2 are still present due to their dependency on
> `rte_eal_get_physmem_layout`.
I've fixed the dpaa compile issue and pushed it to github. I've tried to
keep the semantics the same as before, but i can't compile-test (let
alone test-test) them as i don't have access to a system with dpaa bus.
Also, you might want to know that dpaa bus driver references
RTE_LIBRTE_DPAA_MAX_CRYPTODEV which is only found in
config/common_armv8a_linuxapp but is not present in base config. Not
sure if that's an issue.
>
>>
>> --
>> Thanks,
>> Anatoly
>
> Thanks,
> Pavan
>
On Fri, Mar 09, 2018 at 10:42:03AM +0000, Burakov, Anatoly wrote:
> On 09-Mar-18 9:15 AM, Pavan Nikhilesh wrote:
> > On Thu, Mar 08, 2018 at 08:33:21PM +0000, Burakov, Anatoly wrote:
> > > On 08-Mar-18 8:11 PM, Burakov, Anatoly wrote:
> > > > On 08-Mar-18 2:36 PM, Burakov, Anatoly wrote:
> > > > > On 08-Mar-18 1:36 PM, Pavan Nikhilesh wrote:
> > > > > > Hi Anatoly,
> > > > > >
> > > > > > We are currently facing issues with running testpmd on thunderx
> > > > > > platform.
> > > > > > The issue seems to be with vfio
> > > > > >
> > > > > > EAL: Detected 24 lcore(s)
> > > > > > EAL: Detected 1 NUMA nodes
> > > > > > EAL: No free hugepages reported in hugepages-2048kB
> > > > > > EAL: Multi-process socket /var/run/.rte_unix
> > > > > > EAL: Probing VFIO support...
> > > > > > EAL: VFIO support initialized
> > > > > > EAL: VFIO support not initialized
> > > > > >
> > > > > > <snip>
> > > > > >
> > > > > > EAL: probe driver: 177d:a053 octeontx_fpavf
> > > > > > EAL: PCI device 0001:01:00.1 on NUMA socket 0
> > > > > > EAL: probe driver: 177d:a034 net_thunderx
> > > > > > EAL: using IOMMU type 1 (Type 1)
> > > > > > EAL: cannot set up DMA remapping, error 22 (Invalid argument)
> > > > > > EAL: 0001:01:00.1 DMA remapping failed, error 22 (Invalid argument)
> > > > > > EAL: Requested device 0001:01:00.1 cannot be used
> > > > > > EAL: PCI device 0001:01:00.2 on NUMA socket 0
> > > > > > <snip>
> > > > > > testpmd: No probed ethernet devices
> > > > > > testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=251456,
> > > > > > size=2176, socket=0
> > > > > > testpmd: preferred mempool ops selected: ring_mp_mc
> > > > > > EAL: VFIO support not initialized
> > > > > > EAL: VFIO support not initialized
> > > > > > EAL: VFIO support not initialized
> > > > > > Done
> > > > > >
> > > > > >
> > > > > > This is because rte_service_init() calls rte_calloc() before
> > > > > > rte_bus_probe() and vfio_dma_mem_map fails because iommu type is
> > > > > > not set.
> > > > > >
> > > > > > Call stack:
> > > > > > gdb) bt
> > > > > > #0 vfio_dma_mem_map (vaddr=281439006359552, iova=11274289152,
> > > > > > len=536870912, do_map=1) at
> > > > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:967
> > > > > > #1 0x00000000004fd974 in rte_vfio_dma_map
> > > > > > (vaddr=281439006359552, iova=11274289152, len=536870912) at
> > > > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:988
> > > > > > #2 0x00000000004fbe78 in vfio_mem_event_callback
> > > > > > (type=RTE_MEM_EVENT_ALLOC, addr=0xfff7a0000000, len=536870912)
> > > > > > at /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c:240
> > > > > > #3 0x00000000005070ac in eal_memalloc_notify
> > > > > > (event=RTE_MEM_EVENT_ALLOC, start=0xfff7a0000000, len=536870912)
> > > > > > at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_memalloc.c:177
> > > > > > #4 0x0000000000515c98 in try_expand_heap_primary
> > > > > > (heap=0xffffb7fb167c, pg_sz=536870912, elt_size=8192, socket=0,
> > > > > > flags=0, align=128, bound=0, contig=false) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:247
> > > > > > #5 0x0000000000515e94 in try_expand_heap (heap=0xffffb7fb167c,
> > > > > > pg_sz=536870912, elt_size=8192, socket=0, flags=0, align=128,
> > > > > > bound=0, contig=false) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:327
> > > > > > #6 0x00000000005163a0 in alloc_more_mem_on_socket
> > > > > > (heap=0xffffb7fb167c, size=8192, socket=0, flags=0, align=128,
> > > > > > bound=0, contig=false) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:455
> > > > > > #7 0x0000000000516514 in heap_alloc_on_socket (type=0x85bf90
> > > > > > "rte_services", size=8192, socket=0, flags=0, align=128,
> > > > > > bound=0, contig=false) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:491
> > > > > > #8 0x0000000000516664 in malloc_heap_alloc (type=0x85bf90
> > > > > > "rte_services", size=8192, socket_arg=-1, flags=0, align=128,
> > > > > > bound=0, contig=false) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/malloc_heap.c:527
> > > > > > #9 0x0000000000513b54 in rte_malloc_socket (type=0x85bf90
> > > > > > "rte_services", size=8192, align=128, socket_arg=-1) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:54
> > > > > > #10 0x0000000000513bc8 in rte_zmalloc_socket (type=0x85bf90
> > > > > > "rte_services", size=8192, align=128, socket=-1) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:72
> > > > > > #11 0x0000000000513c00 in rte_zmalloc (type=0x85bf90
> > > > > > "rte_services", size=8192, align=128) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:81
> > > > > > #12 0x0000000000513c90 in rte_calloc (type=0x85bf90
> > > > > > "rte_services", num=64, size=128, align=128) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/rte_malloc.c:99
> > > > > > #13 0x0000000000518cec in rte_service_init () at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/rte_service.c:81
> > > > > > #14 0x00000000004f55f4 in rte_eal_init (argc=3,
> > > > > > argv=0xfffffffff488) at
> > > > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:959
> > > > > > #15 0x000000000045af5c in main (argc=3, argv=0xfffffffff488) at
> > > > > > /root/clean/dpdk/app/test-pmd/testpmd.c:2483
> > > > > >
> > > > > >
> > > > > > Also, I have tried running with --legacy-mem but I'm stuck in
> > > > > > `pci_find_max_end_va` loop because `rte_fbarray_find_next_used`
> > > > > > always return
> > > > > > 0. >
> > > > > > HugePages_Total: 15
> > > > > > HugePages_Free: 11
> > > > > > HugePages_Rsvd: 0
> > > > > > HugePages_Surp: 0
> > > > > > Hugepagesize: 524288 kB
> > > > > >
> > > > > > Call Stack:
> > > > > > (gdb) bt
> > > > > > #0 find_next (arr=0xffffb7fb009c, start=0, used=true) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:248
> > > > > > #1 0x00000000005132a8 in rte_fbarray_find_next_used
> > > > > > (arr=0xffffb7fb009c, start=0) at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_fbarray.c:700
> > > > > > #2 0x000000000052d030 in pci_find_max_end_va () at
> > > > > > /root/clean/dpdk/drivers/bus/pci/linux/pci.c:138
> > > > > > #3 0x0000000000530ab8 in pci_vfio_map_resource_primary
> > > > > > (dev=0xeae700) at
> > > > > > /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:499
> > > > > > #4 0x0000000000530ffc in pci_vfio_map_resource (dev=0xeae700)
> > > > > > at /root/clean/dpdk/drivers/bus/pci/linux/pci_vfio.c:601
> > > > > > #5 0x000000000052ce90 in rte_pci_map_device (dev=0xeae700) at
> > > > > > /root/clean/dpdk/drivers/bus/pci/linux/pci.c:75
> > > > > > #6 0x0000000000531a20 in rte_pci_probe_one_driver (dr=0x997e20
> > > > > > <rte_nicvf_pmd>, dev=0xeae700) at
> > > > > > /root/clean/dpdk/drivers/bus/pci/pci_common.c:164
> > > > > > #7 0x0000000000531c68 in pci_probe_all_drivers (dev=0xeae700)
> > > > > > at /root/clean/dpdk/drivers/bus/pci/pci_common.c:249
> > > > > > #8 0x0000000000531f68 in rte_pci_probe () at
> > > > > > /root/clean/dpdk/drivers/bus/pci/pci_common.c:359
> > > > > > #9 0x000000000050a140 in rte_bus_probe () at
> > > > > > /root/clean/dpdk/lib/librte_eal/common/eal_common_bus.c:98
> > > > > > #10 0x00000000004f55f4 in rte_eal_init (argc=1,
> > > > > > argv=0xfffffffff498) at
> > > > > > /root/clean/dpdk/lib/librte_eal/linuxapp/eal/eal.c:967
> > > > > > #11 0x000000000045af5c in main (argc=1, argv=0xfffffffff498) at
> > > > > > /root/clean/dpdk/app/test-pmd/testpmd.c:2483
> > > > > >
> > > > > > Am I missing something here?
> > > > >
> > > > > I'll look into those, thanks!
> > > > >
> > > > > Btw, i've now set up a github repo with the patchset applied:
> > > > >
> > > > > https://github.com/anatolyburakov/dpdk
> > > > >
> > > > > I will be pushing quick fixes there before spinning new revisions,
> > > > > so we can discover and fix bugs more rapidly. I'll fix compile
> > > > > issues reported earlier, then i'll take a look at your issues. The
> > > > > latter one seems like a typo, the former is probably a matter of
> > > > > moving things around a bit.
> > > > >
> > > > > (also, pull requests welcome if you find it easier to fix things
> > > > > yourself and submit patches against my tree!)
> > > > >
> > > > > Thanks for testing.
> > > > >
> > > >
> > > > I've looked into the failures.
> > > >
> > > > The VFIO one is not actually a failure. It only prints out errors
> > > > because rte_malloc is called before VFIO is initialized. However, once
> > > > VFIO *is* initialized, all of that memory would be added to VFIO, so
> > > > these error messages are harmless. Regardless, i've added a check to see
> > > > if init is finished before printing out those errors, so they won't be
> > > > printed out any more.
> > > >
> > > > Second one is a typo on my part that got lost in one of the rebases.
> > > >
> > > > I've pushed fixes for both into the github repo.
> > > >
> > >
> > > Although i do wonder where do the DMA remapping errors come from. The error
> > > message says "invalid argument", so that doesn't come from rte_service or
> > > anything to do with rte_malloc - this is us not providing valid arguments to
> > > VFIO. I'm not seeing these errors on my system. I'll check on others to be
> > > sure.
> >
> > I have taken a look at the github tree the issues with VFIO are gone, Although
> > compilation issues with dpaa/dpaa2 are still present due to their dependency on
> > `rte_eal_get_physmem_layout`.
>
> I've fixed the dpaa compile issue and pushed it to github. I've tried to
> keep the semantics the same as before, but i can't compile-test (let alone
> test-test) them as i don't have access to a system with dpaa bus.
>
> Also, you might want to know that dpaa bus driver references
> RTE_LIBRTE_DPAA_MAX_CRYPTODEV which is only found in
> config/common_armv8a_linuxapp but is not present in base config. Not sure if
> that's an issue.
Hi anatoly,
I've checkout your branch on github
commit 61e6876fd1d2 ("remove panic from memalloc")
I am compiling it with clang:
clang version 3.8.1-24 (tags/RELEASE_381/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
There are a lot of compilation errors not present on DPDK branch.
They need to be fixed.
Thanks,
Hello Anatoly,
On Fri, Mar 9, 2018 at 4:12 PM, Burakov, Anatoly
<anatoly.burakov@intel.com> wrote:
> On 09-Mar-18 9:15 AM, Pavan Nikhilesh wrote:
[...]
>>
>>
>> I have taken a look at the github tree the issues with VFIO are gone,
>> Although
>> compilation issues with dpaa/dpaa2 are still present due to their
>> dependency on
>> `rte_eal_get_physmem_layout`.
>
>
> I've fixed the dpaa compile issue and pushed it to github. I've tried to
> keep the semantics the same as before, but i can't compile-test (let alone
> test-test) them as i don't have access to a system with dpaa bus.
Thanks. I will have a look at this.
>
> Also, you might want to know that dpaa bus driver references
> RTE_LIBRTE_DPAA_MAX_CRYPTODEV which is only found in
> config/common_armv8a_linuxapp but is not present in base config. Not sure if
> that's an issue.
>
This might be an issue as very recently some patches updated the base
config. I will cross check this as well.
-
Shreyansh
Hello Anatoly,
On Tue, Mar 13, 2018 at 10:47 AM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
> Hello Anatoly,
>
> On Fri, Mar 9, 2018 at 4:12 PM, Burakov, Anatoly
> <anatoly.burakov@intel.com> wrote:
>> On 09-Mar-18 9:15 AM, Pavan Nikhilesh wrote:
>
> [...]
>
>>>
>>>
>>> I have taken a look at the github tree the issues with VFIO are gone,
>>> Although
>>> compilation issues with dpaa/dpaa2 are still present due to their
>>> dependency on
>>> `rte_eal_get_physmem_layout`.
>>
>>
>> I've fixed the dpaa compile issue and pushed it to github. I've tried to
>> keep the semantics the same as before, but i can't compile-test (let alone
>> test-test) them as i don't have access to a system with dpaa bus.
>
> Thanks. I will have a look at this.
Just a heads-up, DPAA2 is broken on top-of-tree (github:
784e041f6b520) as of now:
--->8---
root@ls2088ardb:~/shreyansh/07_dpdk_memory#
./arm64-dpaa2-linuxapp-gcc/app/testpmd -c 0xE -n 1 --log-level=eal,8
--log-level=mem,8 -- -i --portmask=0x3
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 1 on socket 0
EAL: Detected lcore 4 as core 0 on socket 0
EAL: Detected lcore 5 as core 1 on socket 0
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Support maximum 16 logical core(s) by configuration.
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: VFIO PCI modules not loaded
EAL: DPAA Bus not present. Skipping.
EAL: Container: dprc.2 has VFIO iommu group id = 4
EAL: fslmc: Bus scan completed
EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
EAL: Multi-process socket /var/run/.rte_unix
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 7 (sPAPR) is not supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Mem event callback 'vfio_mem_event_clb' registered
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0xffff86cae000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0xffff8873f000 (size = 0x1000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0xfff780000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0xffff8873e000 (size = 0x1000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0xffef40000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0xffff8873d000 (size = 0x1000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0xffe700000000 (size = 0x800000000)
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0xffff8873c000 (size = 0x1000)
EAL: Memseg list allocated: 0x100000kB at socket 0
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0xffdec0000000 (size = 0x800000000)
EAL: TSC frequency is ~25000 KHz
EAL: Master lcore 1 is ready (tid=88742110;cpuset=[1])
EAL: lcore 3 is ready (tid=85cab910;cpuset=[3])
EAL: lcore 2 is ready (tid=864ab910;cpuset=[2])
EAL: eal_memalloc_alloc_page_bulk(): couldn't find suitable memseg_list
error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
PANIC in main():
Cannot init EAL
1: [./arm64-dpaa2-linuxapp-gcc/app/testpmd(rte_dump_stack+0x38) [0x4f37a8]]
Aborted
--->8--
Above is an initial output - still investigating. I will keep you posted.
>
>>
>> Also, you might want to know that dpaa bus driver references
>> RTE_LIBRTE_DPAA_MAX_CRYPTODEV which is only found in
>> config/common_armv8a_linuxapp but is not present in base config. Not sure if
>> that's an issue.
A recent patch from Hemant has fixed this (yet to be merged in master).
[...]
-
Shreyansh
Hello Anatoly,
This is not necessarily right chain to reply to, but reusing this
email for another issue in DPAA2 so that all issues can be at a single
place.
On Thu, Mar 15, 2018 at 7:31 PM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
> Hello Anatoly,
>
> On Tue, Mar 13, 2018 at 10:47 AM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
>> Hello Anatoly,
>>
>> On Fri, Mar 9, 2018 at 4:12 PM, Burakov, Anatoly
>> <anatoly.burakov@intel.com> wrote:
>>> On 09-Mar-18 9:15 AM, Pavan Nikhilesh wrote:
>>
>> [...]
>>
>>>>
>>>>
>>>> I have taken a look at the github tree the issues with VFIO are gone,
>>>> Although
>>>> compilation issues with dpaa/dpaa2 are still present due to their
>>>> dependency on
>>>> `rte_eal_get_physmem_layout`.
>>>
>>>
>>> I've fixed the dpaa compile issue and pushed it to github. I've tried to
>>> keep the semantics the same as before, but i can't compile-test (let alone
>>> test-test) them as i don't have access to a system with dpaa bus.
>>
>> Thanks. I will have a look at this.
>
> Just a heads-up, DPAA2 is broken on top-of-tree (github:
> 784e041f6b520) as of now:
>
> --->8---
> root@ls2088ardb:~/shreyansh/07_dpdk_memory#
> ./arm64-dpaa2-linuxapp-gcc/app/testpmd -c 0xE -n 1 --log-level=eal,8
> --log-level=mem,8 -- -i --portmask=0x3
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 0 on socket 0
> EAL: Detected lcore 3 as core 1 on socket 0
> EAL: Detected lcore 4 as core 0 on socket 0
> EAL: Detected lcore 5 as core 1 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 0
> EAL: Detected lcore 7 as core 1 on socket 0
> EAL: Support maximum 16 logical core(s) by configuration.
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: VFIO PCI modules not loaded
> EAL: DPAA Bus not present. Skipping.
> EAL: Container: dprc.2 has VFIO iommu group id = 4
> EAL: fslmc: Bus scan completed
> EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
> EAL: Multi-process socket /var/run/.rte_unix
> EAL: Probing VFIO support...
> EAL: IOMMU type 1 (Type 1) is supported
> EAL: IOMMU type 7 (sPAPR) is not supported
> EAL: IOMMU type 8 (No-IOMMU) is not supported
> EAL: VFIO support initialized
> EAL: Mem event callback 'vfio_mem_event_clb' registered
> EAL: Ask a virtual area of 0x2e000 bytes
> EAL: Virtual area found at 0xffff86cae000 (size = 0x2e000)
> EAL: Setting up physically contiguous memory...
> EAL: Ask a virtual area of 0x1000 bytes
> EAL: Virtual area found at 0xffff8873f000 (size = 0x1000)
> EAL: Memseg list allocated: 0x100000kB at socket 0
> EAL: Ask a virtual area of 0x800000000 bytes
> EAL: Virtual area found at 0xfff780000000 (size = 0x800000000)
> EAL: Ask a virtual area of 0x1000 bytes
> EAL: Virtual area found at 0xffff8873e000 (size = 0x1000)
> EAL: Memseg list allocated: 0x100000kB at socket 0
> EAL: Ask a virtual area of 0x800000000 bytes
> EAL: Virtual area found at 0xffef40000000 (size = 0x800000000)
> EAL: Ask a virtual area of 0x1000 bytes
> EAL: Virtual area found at 0xffff8873d000 (size = 0x1000)
> EAL: Memseg list allocated: 0x100000kB at socket 0
> EAL: Ask a virtual area of 0x800000000 bytes
> EAL: Virtual area found at 0xffe700000000 (size = 0x800000000)
> EAL: Ask a virtual area of 0x1000 bytes
> EAL: Virtual area found at 0xffff8873c000 (size = 0x1000)
> EAL: Memseg list allocated: 0x100000kB at socket 0
> EAL: Ask a virtual area of 0x800000000 bytes
> EAL: Virtual area found at 0xffdec0000000 (size = 0x800000000)
> EAL: TSC frequency is ~25000 KHz
> EAL: Master lcore 1 is ready (tid=88742110;cpuset=[1])
> EAL: lcore 3 is ready (tid=85cab910;cpuset=[3])
> EAL: lcore 2 is ready (tid=864ab910;cpuset=[2])
> EAL: eal_memalloc_alloc_page_bulk(): couldn't find suitable memseg_list
> error allocating rte services array
> EAL: FATAL: rte_service_init() failed
>
> EAL: rte_service_init() failed
>
> PANIC in main():
> Cannot init EAL
> 1: [./arm64-dpaa2-linuxapp-gcc/app/testpmd(rte_dump_stack+0x38) [0x4f37a8]]
> Aborted
> --->8--
>
> Above is an initial output - still investigating. I will keep you posted.
>
While working on issue reported in [1], I have found another issue
which I might need you help.
[1] http://dpdk.org/ml/archives/dev/2018-March/093202.html
For [1], I bypassed by changing the mempool_add_elem code for time
being - it now allows non-contiguous (not explicitly demanded
contiguous) allocations to go through rte_mempool_populate_iova. With
that, I was able to get DPAA2 working.
Problem is:
1. When I am working with 1GB pages, I/O is working fine.
2. When using 2MB pages (1024 num), the initialization somewhere after
VFIO layer fails.
All with IOVA=VA mode.
Some logs:
This is the output of the virtual memory layout demanded by DPDK:
--->8---
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0xffffb6561000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x59000 bytes
EAL: Virtual area found at 0xffffb6508000 (size = 0x59000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0xfffbb6400000 (size = 0x400000000)
EAL: Ask a virtual area of 0x59000 bytes
EAL: Virtual area found at 0xfffbb62af000 (size = 0x59000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0xfff7b6200000 (size = 0x400000000)
EAL: Ask a virtual area of 0x59000 bytes
EAL: Virtual area found at 0xfff7b6056000 (size = 0x59000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0xfff3b6000000 (size = 0x400000000)
EAL: Ask a virtual area of 0x59000 bytes
EAL: Virtual area found at 0xfff3b5dfd000 (size = 0x59000)
EAL: Memseg list allocated: 0x800kB at socket 0
EAL: Ask a virtual area of 0x400000000 bytes
EAL: Virtual area found at 0xffefb5c00000 (size = 0x400000000)
--->8---
Then, somehow VFIO mapping is able to find only a single page to map
--->8---
EAL: Device (dpci.1) abstracted from VFIO
EAL: -->Initial SHM Virtual ADDR FFFBB6400000
EAL: -----> DMA size 0x200000
EAL: Total 1 segments found.
--->8---
Then, these logs appear probably when DPAA2 code requests for memory.
I am not sure why it repeats the same '...expanded by 10MB'.
--->8---
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 2MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 10MB
LPM or EM none selected, default LPM on
Initializing port 0 ...
--->8---
l3fwd is stuck at this point. What I observe is that DPAA2 driver has
gone ahead to register the queues (queue_setup) with hardware and the
memory has either overrun (smaller than requested size mapped) or the
addresses are corrupt (that is, not dma-able). (I get SMMU faults,
indicating one of these cases)
There is some change from you in the fslmc/fslmc_vfio.c file
(rte_fslmc_vfio_dmamap()). Ideally, that code should have walked over
all the available pages for mapping but that didn't happen and only a
single virtual area got dma-mapped.
--->8---
EAL: Device (dpci.1) abstracted from VFIO
EAL: -->Initial SHM Virtual ADDR FFFBB6400000
EAL: -----> DMA size 0x200000
EAL: Total 1 segments found.
--->8---
I am looking into this but if there is some hint which come to your
mind, it might help.
Regards,
Shreyansh
On 21-Mar-18 1:45 PM, Shreyansh Jain wrote:
> Hello Anatoly,
>
> This is not necessarily right chain to reply to, but reusing this
> email for another issue in DPAA2 so that all issues can be at a single
> place.
>
> On Thu, Mar 15, 2018 at 7:31 PM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
>> Hello Anatoly,
>>
>> On Tue, Mar 13, 2018 at 10:47 AM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
>>> Hello Anatoly,
>>>
>>> On Fri, Mar 9, 2018 at 4:12 PM, Burakov, Anatoly
>>> <anatoly.burakov@intel.com> wrote:
>>>> On 09-Mar-18 9:15 AM, Pavan Nikhilesh wrote:
>>>
>>> [...]
>>>
>>>>>
>>>>>
>>>>> I have taken a look at the github tree the issues with VFIO are gone,
>>>>> Although
>>>>> compilation issues with dpaa/dpaa2 are still present due to their
>>>>> dependency on
>>>>> `rte_eal_get_physmem_layout`.
>>>>
>>>>
>>>> I've fixed the dpaa compile issue and pushed it to github. I've tried to
>>>> keep the semantics the same as before, but i can't compile-test (let alone
>>>> test-test) them as i don't have access to a system with dpaa bus.
>>>
>>> Thanks. I will have a look at this.
>>
>> Just a heads-up, DPAA2 is broken on top-of-tree (github:
>> 784e041f6b520) as of now:
>>
>> --->8---
>> root@ls2088ardb:~/shreyansh/07_dpdk_memory#
>> ./arm64-dpaa2-linuxapp-gcc/app/testpmd -c 0xE -n 1 --log-level=eal,8
>> --log-level=mem,8 -- -i --portmask=0x3
>> EAL: Detected lcore 0 as core 0 on socket 0
>> EAL: Detected lcore 1 as core 1 on socket 0
>> EAL: Detected lcore 2 as core 0 on socket 0
>> EAL: Detected lcore 3 as core 1 on socket 0
>> EAL: Detected lcore 4 as core 0 on socket 0
>> EAL: Detected lcore 5 as core 1 on socket 0
>> EAL: Detected lcore 6 as core 0 on socket 0
>> EAL: Detected lcore 7 as core 1 on socket 0
>> EAL: Support maximum 16 logical core(s) by configuration.
>> EAL: Detected 8 lcore(s)
>> EAL: Detected 1 NUMA nodes
>> EAL: VFIO PCI modules not loaded
>> EAL: DPAA Bus not present. Skipping.
>> EAL: Container: dprc.2 has VFIO iommu group id = 4
>> EAL: fslmc: Bus scan completed
>> EAL: Module /sys/module/rte_kni not found! error 2 (No such file or directory)
>> EAL: Multi-process socket /var/run/.rte_unix
>> EAL: Probing VFIO support...
>> EAL: IOMMU type 1 (Type 1) is supported
>> EAL: IOMMU type 7 (sPAPR) is not supported
>> EAL: IOMMU type 8 (No-IOMMU) is not supported
>> EAL: VFIO support initialized
>> EAL: Mem event callback 'vfio_mem_event_clb' registered
>> EAL: Ask a virtual area of 0x2e000 bytes
>> EAL: Virtual area found at 0xffff86cae000 (size = 0x2e000)
>> EAL: Setting up physically contiguous memory...
>> EAL: Ask a virtual area of 0x1000 bytes
>> EAL: Virtual area found at 0xffff8873f000 (size = 0x1000)
>> EAL: Memseg list allocated: 0x100000kB at socket 0
>> EAL: Ask a virtual area of 0x800000000 bytes
>> EAL: Virtual area found at 0xfff780000000 (size = 0x800000000)
>> EAL: Ask a virtual area of 0x1000 bytes
>> EAL: Virtual area found at 0xffff8873e000 (size = 0x1000)
>> EAL: Memseg list allocated: 0x100000kB at socket 0
>> EAL: Ask a virtual area of 0x800000000 bytes
>> EAL: Virtual area found at 0xffef40000000 (size = 0x800000000)
>> EAL: Ask a virtual area of 0x1000 bytes
>> EAL: Virtual area found at 0xffff8873d000 (size = 0x1000)
>> EAL: Memseg list allocated: 0x100000kB at socket 0
>> EAL: Ask a virtual area of 0x800000000 bytes
>> EAL: Virtual area found at 0xffe700000000 (size = 0x800000000)
>> EAL: Ask a virtual area of 0x1000 bytes
>> EAL: Virtual area found at 0xffff8873c000 (size = 0x1000)
>> EAL: Memseg list allocated: 0x100000kB at socket 0
>> EAL: Ask a virtual area of 0x800000000 bytes
>> EAL: Virtual area found at 0xffdec0000000 (size = 0x800000000)
>> EAL: TSC frequency is ~25000 KHz
>> EAL: Master lcore 1 is ready (tid=88742110;cpuset=[1])
>> EAL: lcore 3 is ready (tid=85cab910;cpuset=[3])
>> EAL: lcore 2 is ready (tid=864ab910;cpuset=[2])
>> EAL: eal_memalloc_alloc_page_bulk(): couldn't find suitable memseg_list
>> error allocating rte services array
>> EAL: FATAL: rte_service_init() failed
>>
>> EAL: rte_service_init() failed
>>
>> PANIC in main():
>> Cannot init EAL
>> 1: [./arm64-dpaa2-linuxapp-gcc/app/testpmd(rte_dump_stack+0x38) [0x4f37a8]]
>> Aborted
>> --->8--
>>
>> Above is an initial output - still investigating. I will keep you posted.
>>
>
> While working on issue reported in [1], I have found another issue
> which I might need you help.
>
> [1] http://dpdk.org/ml/archives/dev/2018-March/093202.html
>
> For [1], I bypassed by changing the mempool_add_elem code for time
> being - it now allows non-contiguous (not explicitly demanded
> contiguous) allocations to go through rte_mempool_populate_iova. With
> that, I was able to get DPAA2 working.
>
> Problem is:
> 1. When I am working with 1GB pages, I/O is working fine.
> 2. When using 2MB pages (1024 num), the initialization somewhere after
> VFIO layer fails.
>
> All with IOVA=VA mode.
>
> Some logs:
>
> This is the output of the virtual memory layout demanded by DPDK:
>
> --->8---
> EAL: Ask a virtual area of 0x2e000 bytes
> EAL: Virtual area found at 0xffffb6561000 (size = 0x2e000)
> EAL: Setting up physically contiguous memory...
> EAL: Ask a virtual area of 0x59000 bytes
> EAL: Virtual area found at 0xffffb6508000 (size = 0x59000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0xfffbb6400000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x59000 bytes
> EAL: Virtual area found at 0xfffbb62af000 (size = 0x59000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0xfff7b6200000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x59000 bytes
> EAL: Virtual area found at 0xfff7b6056000 (size = 0x59000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0xfff3b6000000 (size = 0x400000000)
> EAL: Ask a virtual area of 0x59000 bytes
> EAL: Virtual area found at 0xfff3b5dfd000 (size = 0x59000)
> EAL: Memseg list allocated: 0x800kB at socket 0
> EAL: Ask a virtual area of 0x400000000 bytes
> EAL: Virtual area found at 0xffefb5c00000 (size = 0x400000000)
> --->8---
>
> Then, somehow VFIO mapping is able to find only a single page to map
>
> --->8---
> EAL: Device (dpci.1) abstracted from VFIO
> EAL: -->Initial SHM Virtual ADDR FFFBB6400000
> EAL: -----> DMA size 0x200000
> EAL: Total 1 segments found.
> --->8---
>
> Then, these logs appear probably when DPAA2 code requests for memory.
> I am not sure why it repeats the same '...expanded by 10MB'.
>
> --->8---
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 2MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> EAL: Calling mem event callback vfio_mem_event_clbEAL: request: mp_malloc_sync
> EAL: Heap on socket 0 was expanded by 10MB
> LPM or EM none selected, default LPM on
> Initializing port 0 ...
> --->8---
>
> l3fwd is stuck at this point. What I observe is that DPAA2 driver has
> gone ahead to register the queues (queue_setup) with hardware and the
> memory has either overrun (smaller than requested size mapped) or the
> addresses are corrupt (that is, not dma-able). (I get SMMU faults,
> indicating one of these cases)
>
> There is some change from you in the fslmc/fslmc_vfio.c file
> (rte_fslmc_vfio_dmamap()). Ideally, that code should have walked over
> all the available pages for mapping but that didn't happen and only a
> single virtual area got dma-mapped.
>
> --->8---
> EAL: Device (dpci.1) abstracted from VFIO
> EAL: -->Initial SHM Virtual ADDR FFFBB6400000
> EAL: -----> DMA size 0x200000
> EAL: Total 1 segments found.
> --->8---
>
> I am looking into this but if there is some hint which come to your
> mind, it might help.
>
> Regards,
> Shreyansh
>
Hi Shreyansh,
Thanks for the feedback.
The "heap on socket 0 was expanded by 10MB" has to do with
synchronization requests in primary/secondary processes. I can see
you're allocating LPM tables - that's most likely what these allocations
are about (it's hotplugging memory).
I think i might have an idea what is going on. I am assuming that you
are starting up your DPDK application without any -m or --socket-mem
flags, which means you are starting with empty heap.
During initialization, certain DPDK features (such as service cores,
PMD's) allocate memory. Most likely you have essentially started up with
1 2M page, which is what you see in fslmc logs: this page gets mapped
for VFIO.
Then, you allocate a bunch of LPM tables, which trigger more memory
allocation, and trigger memory allocation callbacks registered through
rte_mem_event_register_callback(). One of these callbacks is a VFIO
callback, which is registered in eal_vfio.c:rte_vfio_enable(). However,
since fslmc bus has its own VFIO implementation that is independent of
what happens in EAL VFIO code, what probably happens is that the fslmc
bus misses the necessary messages from the memory hotplug to map
additional resources for DMA.
Try adding a rte_mem_event_register_callback() somewhere in fslmc init
so that it calls necessary map function.
eal_vfio.c:vfio_mem_event_callback() should provide a good template on
how to approach creating such a callback. Let me know if this works!
(as a side note, how can we extend VFIO to move this stuff back into EAL
and expose it as an API?)
Hello Anatoly,
> -----Original Message-----
> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
> Sent: Wednesday, March 21, 2018 8:18 PM
> To: Shreyansh Jain <shreyansh.jain@nxp.com>
> Cc: dev@dpdk.org; Hemant Agrawal <hemant.agrawal@nxp.com>
> Subject: Re: [dpdk-dev] [PATCH v2 00/41] Memory Hotplug for DPDK
>
[...]
> >>
> >
> > While working on issue reported in [1], I have found another issue
> > which I might need you help.
> >
> > [1]
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpdk.o
> rg%2Fml%2Farchives%2Fdev%2F2018-
> March%2F093202.html&data=02%7C01%7Cshreyansh.jain%40nxp.com%7C5faee716e6
> fc4908bdb608d58f3ad1e5%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6365
> 72405182868376&sdata=WohDdktHHAuNDnss1atuixSa%2FqC7HRMSDVCtFC9Vnto%3D&re
> served=0
> >
> > For [1], I bypassed by changing the mempool_add_elem code for time
> > being - it now allows non-contiguous (not explicitly demanded
> > contiguous) allocations to go through rte_mempool_populate_iova. With
> > that, I was able to get DPAA2 working.
> >
> > Problem is:
> > 1. When I am working with 1GB pages, I/O is working fine.
> > 2. When using 2MB pages (1024 num), the initialization somewhere after
> > VFIO layer fails.
> >
> > All with IOVA=VA mode.
> >
> > Some logs:
> >
> > This is the output of the virtual memory layout demanded by DPDK:
> >
> > --->8---
> > EAL: Ask a virtual area of 0x2e000 bytes
> > EAL: Virtual area found at 0xffffb6561000 (size = 0x2e000)
> > EAL: Setting up physically contiguous memory...
> > EAL: Ask a virtual area of 0x59000 bytes
> > EAL: Virtual area found at 0xffffb6508000 (size = 0x59000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0xfffbb6400000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x59000 bytes
> > EAL: Virtual area found at 0xfffbb62af000 (size = 0x59000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0xfff7b6200000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x59000 bytes
> > EAL: Virtual area found at 0xfff7b6056000 (size = 0x59000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0xfff3b6000000 (size = 0x400000000)
> > EAL: Ask a virtual area of 0x59000 bytes
> > EAL: Virtual area found at 0xfff3b5dfd000 (size = 0x59000)
> > EAL: Memseg list allocated: 0x800kB at socket 0
> > EAL: Ask a virtual area of 0x400000000 bytes
> > EAL: Virtual area found at 0xffefb5c00000 (size = 0x400000000)
> > --->8---
> >
> > Then, somehow VFIO mapping is able to find only a single page to map
> >
> > --->8---
> > EAL: Device (dpci.1) abstracted from VFIO
> > EAL: -->Initial SHM Virtual ADDR FFFBB6400000
> > EAL: -----> DMA size 0x200000
> > EAL: Total 1 segments found.
> > --->8---
> >
> > Then, these logs appear probably when DPAA2 code requests for memory.
> > I am not sure why it repeats the same '...expanded by 10MB'.
> >
> > --->8---
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 2MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
> mp_malloc_sync
> > EAL: Heap on socket 0 was expanded by 10MB
> > LPM or EM none selected, default LPM on
> > Initializing port 0 ...
> > --->8---
> >
> > l3fwd is stuck at this point. What I observe is that DPAA2 driver has
> > gone ahead to register the queues (queue_setup) with hardware and the
> > memory has either overrun (smaller than requested size mapped) or the
> > addresses are corrupt (that is, not dma-able). (I get SMMU faults,
> > indicating one of these cases)
> >
> > There is some change from you in the fslmc/fslmc_vfio.c file
> > (rte_fslmc_vfio_dmamap()). Ideally, that code should have walked over
> > all the available pages for mapping but that didn't happen and only a
> > single virtual area got dma-mapped.
> >
> > --->8---
> > EAL: Device (dpci.1) abstracted from VFIO
> > EAL: -->Initial SHM Virtual ADDR FFFBB6400000
> > EAL: -----> DMA size 0x200000
> > EAL: Total 1 segments found.
> > --->8---
> >
> > I am looking into this but if there is some hint which come to your
> > mind, it might help.
> >
> > Regards,
> > Shreyansh
> >
>
> Hi Shreyansh,
>
> Thanks for the feedback.
>
> The "heap on socket 0 was expanded by 10MB" has to do with
> synchronization requests in primary/secondary processes. I can see
> you're allocating LPM tables - that's most likely what these allocations
> are about (it's hotplugging memory).
I get that but why same message multiple times without any change in the expansion. Further, I don't have multiple process - in fact, I'm working with a single datapath thread.
Anyways, I will look through the code for this.
>
> I think i might have an idea what is going on. I am assuming that you
> are starting up your DPDK application without any -m or --socket-mem
> flags, which means you are starting with empty heap.
Yes, no specific --socket-mem passed as argument.
>
> During initialization, certain DPDK features (such as service cores,
> PMD's) allocate memory. Most likely you have essentially started up with
> 1 2M page, which is what you see in fslmc logs: this page gets mapped
> for VFIO.
Agree.
>
> Then, you allocate a bunch of LPM tables, which trigger more memory
> allocation, and trigger memory allocation callbacks registered through
> rte_mem_event_register_callback(). One of these callbacks is a VFIO
> callback, which is registered in eal_vfio.c:rte_vfio_enable(). However,
> since fslmc bus has its own VFIO implementation that is independent of
> what happens in EAL VFIO code, what probably happens is that the fslmc
> bus misses the necessary messages from the memory hotplug to map
> additional resources for DMA.
Makes sense
>
> Try adding a rte_mem_event_register_callback() somewhere in fslmc init
> so that it calls necessary map function.
> eal_vfio.c:vfio_mem_event_callback() should provide a good template on
> how to approach creating such a callback. Let me know if this works!
OK. I will give this a try and update you.
>
> (as a side note, how can we extend VFIO to move this stuff back into EAL
> and expose it as an API?)
The problem is that FSLMC VFIO driver is slightly different from generic VFIO layer in the sense that device in a VFIO container is actually another level of container. Anyways, I will have a look how much generalization is possible. Or else, I will work with the vfio_mem_event_callback() as suggested above.
Thanks for suggestions.
>
> --
> Thanks,
> Anatoly
On 22-Mar-18 5:09 AM, Shreyansh Jain wrote:
> Hello Anatoly,
>
>> -----Original Message-----
>> From: Burakov, Anatoly [mailto:anatoly.burakov@intel.com]
>> Sent: Wednesday, March 21, 2018 8:18 PM
>> To: Shreyansh Jain <shreyansh.jain@nxp.com>
>> Cc: dev@dpdk.org; Hemant Agrawal <hemant.agrawal@nxp.com>
>> Subject: Re: [dpdk-dev] [PATCH v2 00/41] Memory Hotplug for DPDK
>>
>
> [...]
>
>>>>
>>>
>>> While working on issue reported in [1], I have found another issue
>>> which I might need you help.
>>>
>>> [1]
>> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdpdk.o
>> rg%2Fml%2Farchives%2Fdev%2F2018-
>> March%2F093202.html&data=02%7C01%7Cshreyansh.jain%40nxp.com%7C5faee716e6
>> fc4908bdb608d58f3ad1e5%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6365
>> 72405182868376&sdata=WohDdktHHAuNDnss1atuixSa%2FqC7HRMSDVCtFC9Vnto%3D&re
>> served=0
>>>
>>> For [1], I bypassed by changing the mempool_add_elem code for time
>>> being - it now allows non-contiguous (not explicitly demanded
>>> contiguous) allocations to go through rte_mempool_populate_iova. With
>>> that, I was able to get DPAA2 working.
>>>
>>> Problem is:
>>> 1. When I am working with 1GB pages, I/O is working fine.
>>> 2. When using 2MB pages (1024 num), the initialization somewhere after
>>> VFIO layer fails.
>>>
>>> All with IOVA=VA mode.
>>>
>>> Some logs:
>>>
>>> This is the output of the virtual memory layout demanded by DPDK:
>>>
>>> --->8---
>>> EAL: Ask a virtual area of 0x2e000 bytes
>>> EAL: Virtual area found at 0xffffb6561000 (size = 0x2e000)
>>> EAL: Setting up physically contiguous memory...
>>> EAL: Ask a virtual area of 0x59000 bytes
>>> EAL: Virtual area found at 0xffffb6508000 (size = 0x59000)
>>> EAL: Memseg list allocated: 0x800kB at socket 0
>>> EAL: Ask a virtual area of 0x400000000 bytes
>>> EAL: Virtual area found at 0xfffbb6400000 (size = 0x400000000)
>>> EAL: Ask a virtual area of 0x59000 bytes
>>> EAL: Virtual area found at 0xfffbb62af000 (size = 0x59000)
>>> EAL: Memseg list allocated: 0x800kB at socket 0
>>> EAL: Ask a virtual area of 0x400000000 bytes
>>> EAL: Virtual area found at 0xfff7b6200000 (size = 0x400000000)
>>> EAL: Ask a virtual area of 0x59000 bytes
>>> EAL: Virtual area found at 0xfff7b6056000 (size = 0x59000)
>>> EAL: Memseg list allocated: 0x800kB at socket 0
>>> EAL: Ask a virtual area of 0x400000000 bytes
>>> EAL: Virtual area found at 0xfff3b6000000 (size = 0x400000000)
>>> EAL: Ask a virtual area of 0x59000 bytes
>>> EAL: Virtual area found at 0xfff3b5dfd000 (size = 0x59000)
>>> EAL: Memseg list allocated: 0x800kB at socket 0
>>> EAL: Ask a virtual area of 0x400000000 bytes
>>> EAL: Virtual area found at 0xffefb5c00000 (size = 0x400000000)
>>> --->8---
>>>
>>> Then, somehow VFIO mapping is able to find only a single page to map
>>>
>>> --->8---
>>> EAL: Device (dpci.1) abstracted from VFIO
>>> EAL: -->Initial SHM Virtual ADDR FFFBB6400000
>>> EAL: -----> DMA size 0x200000
>>> EAL: Total 1 segments found.
>>> --->8---
>>>
>>> Then, these logs appear probably when DPAA2 code requests for memory.
>>> I am not sure why it repeats the same '...expanded by 10MB'.
>>>
>>> --->8---
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 2MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> EAL: Calling mem event callback vfio_mem_event_clbEAL: request:
>> mp_malloc_sync
>>> EAL: Heap on socket 0 was expanded by 10MB
>>> LPM or EM none selected, default LPM on
>>> Initializing port 0 ...
>>> --->8---
>>>
>>> l3fwd is stuck at this point. What I observe is that DPAA2 driver has
>>> gone ahead to register the queues (queue_setup) with hardware and the
>>> memory has either overrun (smaller than requested size mapped) or the
>>> addresses are corrupt (that is, not dma-able). (I get SMMU faults,
>>> indicating one of these cases)
>>>
>>> There is some change from you in the fslmc/fslmc_vfio.c file
>>> (rte_fslmc_vfio_dmamap()). Ideally, that code should have walked over
>>> all the available pages for mapping but that didn't happen and only a
>>> single virtual area got dma-mapped.
>>>
>>> --->8---
>>> EAL: Device (dpci.1) abstracted from VFIO
>>> EAL: -->Initial SHM Virtual ADDR FFFBB6400000
>>> EAL: -----> DMA size 0x200000
>>> EAL: Total 1 segments found.
>>> --->8---
>>>
>>> I am looking into this but if there is some hint which come to your
>>> mind, it might help.
>>>
>>> Regards,
>>> Shreyansh
>>>
>>
>> Hi Shreyansh,
>>
>> Thanks for the feedback.
>>
>> The "heap on socket 0 was expanded by 10MB" has to do with
>> synchronization requests in primary/secondary processes. I can see
>> you're allocating LPM tables - that's most likely what these allocations
>> are about (it's hotplugging memory).
>
> I get that but why same message multiple times without any change in the expansion. Further, I don't have multiple process - in fact, I'm working with a single datapath thread.
> Anyways, I will look through the code for this.
>
Hi Shreyansh,
I've misspoke - this has nothing to do with multiprocess. The "request:
mp_malloc_sync" does, but it's an attempt to notify other processes of
the allocation - if there are no processes, nothing happens.
However, multiple heap expansions do correspond to multiple allocations.
If you allocate an LPM table that takes up 10M of hugepage memory - you
expand heap by 10M. If you do it multiple times (e.g. per-NIC?), you do
multiple heap expansions. This message will be triggered on every heap
expansion.
>>
>> I think i might have an idea what is going on. I am assuming that you
>> are starting up your DPDK application without any -m or --socket-mem
>> flags, which means you are starting with empty heap.
>
> Yes, no specific --socket-mem passed as argument.
>
>>
>> During initialization, certain DPDK features (such as service cores,
>> PMD's) allocate memory. Most likely you have essentially started up with
>> 1 2M page, which is what you see in fslmc logs: this page gets mapped
>> for VFIO.
>
> Agree.
>
>>
>> Then, you allocate a bunch of LPM tables, which trigger more memory
>> allocation, and trigger memory allocation callbacks registered through
>> rte_mem_event_register_callback(). One of these callbacks is a VFIO
>> callback, which is registered in eal_vfio.c:rte_vfio_enable(). However,
>> since fslmc bus has its own VFIO implementation that is independent of
>> what happens in EAL VFIO code, what probably happens is that the fslmc
>> bus misses the necessary messages from the memory hotplug to map
>> additional resources for DMA.
>
> Makes sense
>
>>
>> Try adding a rte_mem_event_register_callback() somewhere in fslmc init
>> so that it calls necessary map function.
>> eal_vfio.c:vfio_mem_event_callback() should provide a good template on
>> how to approach creating such a callback. Let me know if this works!
>
> OK. I will give this a try and update you.
>
>>
>> (as a side note, how can we extend VFIO to move this stuff back into EAL
>> and expose it as an API?)
>
> The problem is that FSLMC VFIO driver is slightly different from generic VFIO layer in the sense that device in a VFIO container is actually another level of container. Anyways, I will have a look how much generalization is possible. Or else, I will work with the vfio_mem_event_callback() as suggested above.
This can wait :) The callback is probably the proper way to do it right now.
>
> Thanks for suggestions.
>
>>
>> --
>> Thanks,
>> Anatoly
@@ -321,7 +321,7 @@ rte_memzone_free(const struct rte_memzone *mz)
struct rte_fbarray *arr;
struct rte_memzone *found_mz;
int ret = 0;
- void *addr;
+ void *addr = NULL;
unsigned idx;
if (mz == NULL)
@@ -570,7 +570,7 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n,
unsigned int msl_idx;
int cur_idx, start_idx, end_idx, i, j, ret = -1;
#ifdef RTE_EAL_NUMA_AWARE_HUGEPAGES
- bool have_numa;
+ bool have_numa = false;
int oldpolicy;
struct bitmask *oldmask = numa_allocate_nodemask();
#endif
@@ -1386,9 +1386,9 @@ eal_legacy_hugepage_attach(void)
struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
struct hugepage_file *hp = NULL;
unsigned int num_hp = 0;
- unsigned int i;
+ unsigned int i = 0;
int ms_idx, msl_idx;
- unsigned int cur_seg, max_seg;
+ unsigned int cur_seg, max_seg = 0;
off_t size = 0;
int fd, fd_hugepage = -1;