[2/2] doc/linux_gsg: update information on using hugepages
Checks
Commit Message
Current information regarding hugepage usage is a little out of date.
Update it to include information on in-memory mode, as well as on
default mountpoints provided by systemd.
Cc: stable@dpdk.org
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
doc/guides/linux_gsg/sys_reqs.rst | 39 +++++++++++++++++++------------
1 file changed, 24 insertions(+), 15 deletions(-)
Comments
On Mon, Aug 24, 2020 at 04:45:01PM +0100, Anatoly Burakov wrote:
> Current information regarding hugepage usage is a little out of date.
> Update it to include information on in-memory mode, as well as on
> default mountpoints provided by systemd.
>
> Cc: stable@dpdk.org
>
> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---
> doc/guides/linux_gsg/sys_reqs.rst | 39 +++++++++++++++++++------------
> 1 file changed, 24 insertions(+), 15 deletions(-)
>
> diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
> index a124656bcb..2ddd7ed667 100644
> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> @@ -155,8 +155,12 @@ Without hugepages, high TLB miss rates would occur with the standard 4k page siz
> Reserving Hugepages for DPDK Use
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> -The allocation of hugepages should be done at boot time or as soon as possible after system boot
> -to prevent memory from being fragmented in physical memory.
> +The allocation of hugepages can be performed either at run time or at boot time.
> +In the general case, reserving hugepages at run time is perfectly fine, but in
> +use cases where having lots of physically contiguous memory is required, it is
> +preferable to reserve hugepages at boot time, as that will help in preventing
> +physical memory from becoming heavily fragmented.
> +
Although we are removing the note about 1G pages requiring to be reserved
at boot time, I think we should still mention here that some older kernel
versions do not allow 1G reservations post-boot.
> To reserve hugepages at boot time, a parameter is passed to the Linux kernel on the kernel command line.
>
> For 2 MB pages, just pass the hugepages option to the kernel. For example, to reserve 1024 pages of 2 MB, use::
> @@ -187,9 +191,9 @@ See the Documentation/admin-guide/kernel-parameters.txt file in your Linux sourc
>
> **Alternative:**
>
> -For 2 MB pages, there is also the option of allocating hugepages after the system has booted.
> +There is also the option of allocating hugepages after the system has booted.
> This is done by echoing the number of hugepages required to a nr_hugepages file in the ``/sys/devices/`` directory.
> -For a single-node system, the command to use is as follows (assuming that 1024 pages are required)::
> +For a single-node system, the command to use is as follows (assuming that 1024 of 2MB pages are required)::
>
> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>
> @@ -198,22 +202,27 @@ On a NUMA machine, pages should be allocated explicitly on separate nodes::
> echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
> echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
>
> -.. note::
> -
> - For 1G pages, it is not possible to reserve the hugepage memory after the system has booted.
> -
> Using Hugepages with the DPDK
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> -Once the hugepage memory is reserved, to make the memory available for DPDK use, perform the following steps::
> +If secondary process support is not required, DPDK is able to use hugepages
> +without any configuration by using "in-memory" mode. Please see
> +:ref:`linux_eal_parameters` for more details.
> +
> +If secondary process support is required, mount points for hugepages need to be
> +created. On modern Linux distributions, a default mount point for hugepages is provided
> +by the system and is located at ``/dev/hugepages``. This mount point will use the
> +default hugepage size set by the kernel parameters as described above.
> +
> +However, in order to use multiple hugepage sizes, it is necessary to manually
Rather than multiple hugepage sizes, I'd suggest changing this to hugepage
sizes other than the default.
Do we also want to add a line somewhere explaining that the default size
can be set a boot using a kernel parameter?
> +create mount points for hugepage sizes that are not provided by the system
> +(e.g. 1GB pages).
> +
> +To make the hugepages of size 1GB available for DPDK use, perform the following steps::
>
> mkdir /mnt/huge
> - mount -t hugetlbfs nodev /mnt/huge
> + mount -t hugetlbfs pagesize=1GB /mnt/huge
>
> The mount point can be made permanent across reboots, by adding the following line to the ``/etc/fstab`` file::
>
> - nodev /mnt/huge hugetlbfs defaults 0 0
> -
> -For 1GB pages, the page size must be specified as a mount option::
> -
> - nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0
> + nodev /mnt/huge hugetlbfs pagesize=1GB 0 0
> --
> 2.17.1
On 24-Aug-20 6:13 PM, Bruce Richardson wrote:
> On Mon, Aug 24, 2020 at 04:45:01PM +0100, Anatoly Burakov wrote:
>> Current information regarding hugepage usage is a little out of date.
>> Update it to include information on in-memory mode, as well as on
>> default mountpoints provided by systemd.
>>
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
>> ---
>> doc/guides/linux_gsg/sys_reqs.rst | 39 +++++++++++++++++++------------
>> 1 file changed, 24 insertions(+), 15 deletions(-)
>>
>> diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
>> index a124656bcb..2ddd7ed667 100644
>> --- a/doc/guides/linux_gsg/sys_reqs.rst
>> +++ b/doc/guides/linux_gsg/sys_reqs.rst
>> @@ -155,8 +155,12 @@ Without hugepages, high TLB miss rates would occur with the standard 4k page siz
>> Reserving Hugepages for DPDK Use
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> -The allocation of hugepages should be done at boot time or as soon as possible after system boot
>> -to prevent memory from being fragmented in physical memory.
>> +The allocation of hugepages can be performed either at run time or at boot time.
>> +In the general case, reserving hugepages at run time is perfectly fine, but in
>> +use cases where having lots of physically contiguous memory is required, it is
>> +preferable to reserve hugepages at boot time, as that will help in preventing
>> +physical memory from becoming heavily fragmented.
>> +
>
> Although we are removing the note about 1G pages requiring to be reserved
> at boot time, I think we should still mention here that some older kernel
> versions do not allow 1G reservations post-boot.
Agreed, will fix in v2.
>
>> To reserve hugepages at boot time, a parameter is passed to the Linux kernel on the kernel command line.
>>
>> For 2 MB pages, just pass the hugepages option to the kernel. For example, to reserve 1024 pages of 2 MB, use::
>> @@ -187,9 +191,9 @@ See the Documentation/admin-guide/kernel-parameters.txt file in your Linux sourc
>>
>> **Alternative:**
>>
>> -For 2 MB pages, there is also the option of allocating hugepages after the system has booted.
>> +There is also the option of allocating hugepages after the system has booted.
>> This is done by echoing the number of hugepages required to a nr_hugepages file in the ``/sys/devices/`` directory.
>> -For a single-node system, the command to use is as follows (assuming that 1024 pages are required)::
>> +For a single-node system, the command to use is as follows (assuming that 1024 of 2MB pages are required)::
>>
>> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>>
>> @@ -198,22 +202,27 @@ On a NUMA machine, pages should be allocated explicitly on separate nodes::
>> echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>> echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
>>
>> -.. note::
>> -
>> - For 1G pages, it is not possible to reserve the hugepage memory after the system has booted.
>> -
>> Using Hugepages with the DPDK
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> -Once the hugepage memory is reserved, to make the memory available for DPDK use, perform the following steps::
>> +If secondary process support is not required, DPDK is able to use hugepages
>> +without any configuration by using "in-memory" mode. Please see
>> +:ref:`linux_eal_parameters` for more details.
>> +
>> +If secondary process support is required, mount points for hugepages need to be
>> +created. On modern Linux distributions, a default mount point for hugepages is provided
>> +by the system and is located at ``/dev/hugepages``. This mount point will use the
>> +default hugepage size set by the kernel parameters as described above.
>> +
>> +However, in order to use multiple hugepage sizes, it is necessary to manually
>
> Rather than multiple hugepage sizes, I'd suggest changing this to hugepage
> sizes other than the default.
OK, will fix.
>
> Do we also want to add a line somewhere explaining that the default size
> can be set a boot using a kernel parameter?
It's already there, right above this :)
>
>> +create mount points for hugepage sizes that are not provided by the system
>> +(e.g. 1GB pages).
>> +
>> +To make the hugepages of size 1GB available for DPDK use, perform the following steps::
>>
>> mkdir /mnt/huge
>> - mount -t hugetlbfs nodev /mnt/huge
>> + mount -t hugetlbfs pagesize=1GB /mnt/huge
>>
>> The mount point can be made permanent across reboots, by adding the following line to the ``/etc/fstab`` file::
>>
>> - nodev /mnt/huge hugetlbfs defaults 0 0
>> -
>> -For 1GB pages, the page size must be specified as a mount option::
>> -
>> - nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0
>> + nodev /mnt/huge hugetlbfs pagesize=1GB 0 0
>> --
>> 2.17.1
@@ -155,8 +155,12 @@ Without hugepages, high TLB miss rates would occur with the standard 4k page siz
Reserving Hugepages for DPDK Use
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The allocation of hugepages should be done at boot time or as soon as possible after system boot
-to prevent memory from being fragmented in physical memory.
+The allocation of hugepages can be performed either at run time or at boot time.
+In the general case, reserving hugepages at run time is perfectly fine, but in
+use cases where having lots of physically contiguous memory is required, it is
+preferable to reserve hugepages at boot time, as that will help in preventing
+physical memory from becoming heavily fragmented.
+
To reserve hugepages at boot time, a parameter is passed to the Linux kernel on the kernel command line.
For 2 MB pages, just pass the hugepages option to the kernel. For example, to reserve 1024 pages of 2 MB, use::
@@ -187,9 +191,9 @@ See the Documentation/admin-guide/kernel-parameters.txt file in your Linux sourc
**Alternative:**
-For 2 MB pages, there is also the option of allocating hugepages after the system has booted.
+There is also the option of allocating hugepages after the system has booted.
This is done by echoing the number of hugepages required to a nr_hugepages file in the ``/sys/devices/`` directory.
-For a single-node system, the command to use is as follows (assuming that 1024 pages are required)::
+For a single-node system, the command to use is as follows (assuming that 1024 of 2MB pages are required)::
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
@@ -198,22 +202,27 @@ On a NUMA machine, pages should be allocated explicitly on separate nodes::
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
-.. note::
-
- For 1G pages, it is not possible to reserve the hugepage memory after the system has booted.
-
Using Hugepages with the DPDK
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Once the hugepage memory is reserved, to make the memory available for DPDK use, perform the following steps::
+If secondary process support is not required, DPDK is able to use hugepages
+without any configuration by using "in-memory" mode. Please see
+:ref:`linux_eal_parameters` for more details.
+
+If secondary process support is required, mount points for hugepages need to be
+created. On modern Linux distributions, a default mount point for hugepages is provided
+by the system and is located at ``/dev/hugepages``. This mount point will use the
+default hugepage size set by the kernel parameters as described above.
+
+However, in order to use multiple hugepage sizes, it is necessary to manually
+create mount points for hugepage sizes that are not provided by the system
+(e.g. 1GB pages).
+
+To make the hugepages of size 1GB available for DPDK use, perform the following steps::
mkdir /mnt/huge
- mount -t hugetlbfs nodev /mnt/huge
+ mount -t hugetlbfs pagesize=1GB /mnt/huge
The mount point can be made permanent across reboots, by adding the following line to the ``/etc/fstab`` file::
- nodev /mnt/huge hugetlbfs defaults 0 0
-
-For 1GB pages, the page size must be specified as a mount option::
-
- nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0
+ nodev /mnt/huge hugetlbfs pagesize=1GB 0 0