[v3] eal/mem: preallocate VA space in no-huge mode
Checks
Commit Message
When --no-huge mode is used, the memory is currently allocated with
mmap(NULL, ...). This is fine in most cases, but can fail in cases
where DPDK is run on a machine with an IOMMU that is of more limited
address width than that of a VA, because we're not specifying the
address hint for mmap() call.
Fix it by preallocating VA space before mapping it.
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
Notes:
v3:
- Fix mmap flags used in place of offset
- Fix using internal_config.memory in place of mem_sz
- Add additional address sanity check
v2:
- Add unmap on unsuccessful mmap
I couldn't figure out which specific commit has introduced
the issue, so there's no fix tag. The most likely candidate
is one that introduced the DMA mask thing in the first place
but i'm not sure.
lib/librte_eal/linux/eal/eal_memory.c | 24 ++++++++++++++++++++----
1 file changed, 20 insertions(+), 4 deletions(-)
Comments
On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
<anatoly.burakov@intel.com> wrote:
>
> When --no-huge mode is used, the memory is currently allocated with
> mmap(NULL, ...). This is fine in most cases, but can fail in cases
> where DPDK is run on a machine with an IOMMU that is of more limited
> address width than that of a VA, because we're not specifying the
> address hint for mmap() call.
>
> Fix it by preallocating VA space before mapping it.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Reproduced issue reported by Thomas on v2.
Works fine with v3.
Tested-by: David Marchand <david.marchand@redhat.com>
Does this issue affect FreeBSD too?
On 25-Mar-20 2:39 PM, David Marchand wrote:
> On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
> <anatoly.burakov@intel.com> wrote:
>>
>> When --no-huge mode is used, the memory is currently allocated with
>> mmap(NULL, ...). This is fine in most cases, but can fail in cases
>> where DPDK is run on a machine with an IOMMU that is of more limited
>> address width than that of a VA, because we're not specifying the
>> address hint for mmap() call.
>>
>> Fix it by preallocating VA space before mapping it.
>>
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
>
> Reproduced issue reported by Thomas on v2.
> Works fine with v3.
>
> Tested-by: David Marchand <david.marchand@redhat.com>
>
>
> Does this issue affect FreeBSD too?
>
I don't think we have support for IOMMU on FreeBSD so my guess is no :)
Tested-by: Zhou, JunX W <junx.w.zhou@intel.com>
-----Original Message-----
From: Jiang, YuX
Sent: Thursday, March 26, 2020 8:24 PM
To: David Marchand <david.marchand@redhat.com>; Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: dev <dev@dpdk.org>; dpdk stable <stable@dpdk.org>; Zhou, JunX W <junx.w.zhou@intel.com>
Subject: RE: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
+ Zhou, JunX W <junx.w.zhou@intel.com>
-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of David Marchand
Sent: Wednesday, March 25, 2020 10:40 PM
To: Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: dev <dev@dpdk.org>; dpdk stable <stable@dpdk.org>
Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v3] eal/mem: preallocate VA space in no-huge mode
On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov <anatoly.burakov@intel.com> wrote:
>
> When --no-huge mode is used, the memory is currently allocated with
> mmap(NULL, ...). This is fine in most cases, but can fail in cases
> where DPDK is run on a machine with an IOMMU that is of more limited
> address width than that of a VA, because we're not specifying the
> address hint for mmap() call.
>
> Fix it by preallocating VA space before mapping it.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
Reproduced issue reported by Thomas on v2.
Works fine with v3.
Tested-by: David Marchand <david.marchand@redhat.com>
Does this issue affect FreeBSD too?
--
David Marchand
On Thu, Mar 26, 2020 at 6:07 PM Burakov, Anatoly
<anatoly.burakov@intel.com> wrote:
>
> On 25-Mar-20 2:39 PM, David Marchand wrote:
> > On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
> > <anatoly.burakov@intel.com> wrote:
> >>
> >> When --no-huge mode is used, the memory is currently allocated with
> >> mmap(NULL, ...). This is fine in most cases, but can fail in cases
> >> where DPDK is run on a machine with an IOMMU that is of more limited
> >> address width than that of a VA, because we're not specifying the
> >> address hint for mmap() call.
> >>
> >> Fix it by preallocating VA space before mapping it.
> >>
> >> Cc: stable@dpdk.org
> >>
> >> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> >
> > Reproduced issue reported by Thomas on v2.
> > Works fine with v3.
> >
> > Tested-by: David Marchand <david.marchand@redhat.com>
> >
> >
> > Does this issue affect FreeBSD too?
> >
>
> I don't think we have support for IOMMU on FreeBSD so my guess is no :)
Fair enough, I will take it today.
Thanks.
--
David Marchand
On Wed, Mar 25, 2020 at 3:39 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Fri, Feb 7, 2020 at 12:11 PM Anatoly Burakov
> <anatoly.burakov@intel.com> wrote:
> >
> > When --no-huge mode is used, the memory is currently allocated with
> > mmap(NULL, ...). This is fine in most cases, but can fail in cases
> > where DPDK is run on a machine with an IOMMU that is of more limited
> > address width than that of a VA, because we're not specifying the
> > address hint for mmap() call.
> >
> > Fix it by preallocating VA space before mapping it.
> >
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> Tested-by: David Marchand <david.marchand@redhat.com>
Tested-by: Jun W Zhou <junx.w.zhou@intel.com>
Applied, thanks.
@@ -1340,6 +1340,8 @@ eal_legacy_hugepage_init(void)
/* hugetlbfs can be disabled */
if (internal_config.no_hugetlbfs) {
+ void *prealloc_addr;
+ size_t mem_sz;
struct rte_memseg_list *msl;
int n_segs, cur_seg, fd, flags;
#ifdef MEMFD_SUPPORTED
@@ -1395,17 +1397,31 @@ eal_legacy_hugepage_init(void)
}
}
#endif
- addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE,
- flags, fd, 0);
- if (addr == MAP_FAILED) {
+ /* preallocate address space for the memory, so that it can be
+ * fit into the DMA mask.
+ */
+ mem_sz = internal_config.memory;
+ prealloc_addr = eal_get_virtual_area(
+ NULL, &mem_sz, page_sz, 0, 0);
+ if (prealloc_addr == NULL) {
+ RTE_LOG(ERR, EAL,
+ "%s: reserving memory area failed: "
+ "%s\n",
+ __func__, strerror(errno));
+ return -1;
+ }
+ addr = mmap(prealloc_addr, mem_sz, PROT_READ | PROT_WRITE,
+ flags | MAP_FIXED, fd, 0);
+ if (addr == MAP_FAILED || addr != prealloc_addr) {
RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__,
strerror(errno));
+ munmap(prealloc_addr, mem_sz);
return -1;
}
msl->base_va = addr;
msl->page_sz = page_sz;
msl->socket_id = 0;
- msl->len = internal_config.memory;
+ msl->len = mem_sz;
msl->heap = 1;
/* we're in single-file segments mode, so only the segment list