110 lines
3.8 KiB
ReStructuredText
110 lines
3.8 KiB
ReStructuredText
|
.. _dynamic_hugetlb:
|
||
|
|
|
||
|
|
===============
|
||
|
|
Dynamic Hugetlb
|
||
|
|
===============
|
||
|
|
|
||
|
|
Overview
|
||
|
|
========
|
||
|
|
|
||
|
|
Dynamic hugetlb is a self-developed feature based on the hugetlb and memcontrol.
|
||
|
|
It supports to split huge page dynamically in a memory cgroup. There is a new structure
|
||
|
|
dhugetlb_pool in every mem_cgroup to manage the pages configured to the mem_cgroup.
|
||
|
|
For the mem_cgroup configured with dhugetlb_pool, processes in the mem_cgroup will
|
||
|
|
preferentially use the pages in dhugetlb_pool.
|
||
|
|
|
||
|
|
Dynamic hugetlb supports three types of pages, including 1G/2M huge pages and 4K pages.
|
||
|
|
For the mem_cgroup configured with dhugetlb_pool, processes will be limited to alloc
|
||
|
|
1G/2M huge pages only from dhugetlb_pool. But there is no such constraint for 4K pages.
|
||
|
|
If there are insufficient 4K pages in the dhugetlb_pool, pages can also be allocated from
|
||
|
|
buddy system. So before using dynamic hugetlb, user must know how many huge pages they
|
||
|
|
need.
|
||
|
|
|
||
|
|
Conflict
|
||
|
|
========
|
||
|
|
|
||
|
|
1. Conflict with THP
|
||
|
|
--------------------
|
||
|
|
|
||
|
|
When THP is enabled, the allocation of a page(order=0) may be converted to
|
||
|
|
an allocation of pages(order>0). In this case, the allocation will skip the
|
||
|
|
dhugetlb_pool. When we want to use dynamic hugetlb feature, we have to
|
||
|
|
disable THP for now.
|
||
|
|
|
||
|
|
2. Conflict with hugetlb_vmemmap
|
||
|
|
--------------------------------
|
||
|
|
|
||
|
|
The dynamic_hugetlb feature need to split and merge pages frequently.
|
||
|
|
hugetlb_vmemmap will affects the perforemance of page split and merge.
|
||
|
|
If want to use dynamic hugetlb, please disable hugetlb_vmemmap.
|
||
|
|
|
||
|
|
Usage
|
||
|
|
=====
|
||
|
|
|
||
|
|
1) Add 'dynamic_hugetlb=on' in cmdline to enable dynamic hugetlb feature.
|
||
|
|
|
||
|
|
2) Prealloc some 1G hugepages through hugetlb.
|
||
|
|
|
||
|
|
3) Create a mem_cgroup and configure dhugetlb_pool to mem_cgroup.
|
||
|
|
|
||
|
|
4) Configure the count of 1G/2M hugepages, and the remaining pages in dhugetlb_pool will
|
||
|
|
be used as basic pages.
|
||
|
|
|
||
|
|
5) Bound the process to mem_cgroup. then the memory for it will be allocated from dhugetlb_pool.
|
||
|
|
|
||
|
|
User control
|
||
|
|
============
|
||
|
|
|
||
|
|
1. dynamic_hugetlb=
|
||
|
|
-------------------
|
||
|
|
|
||
|
|
Add ``dynamic_hugtlb=on`` in cmdline to enable dynamic hugetlb feature.
|
||
|
|
By default, the feature si disabled.
|
||
|
|
|
||
|
|
2. dhugetlb.nr_pages
|
||
|
|
--------------------
|
||
|
|
|
||
|
|
In each memory cgroup, there is a ``dhugetlb.nr_pages`` interface used to create and configure dynamic
|
||
|
|
hugetlb. If this interface is not configured, the original functions are not affected. If configured,
|
||
|
|
then the memory used by processes in this memory cgroup will be allocated from corresponding hpool.
|
||
|
|
|
||
|
|
Usage:
|
||
|
|
``echo <nid> <nr_pages> > /sys/fs/cgroup/memory/<memory cgroup>/dhugetlb.nr_pages``:
|
||
|
|
|
||
|
|
Create a dynamic hugetlb pool and add <nr_pages> 1G hugepages from numa node <nid> to the pool.
|
||
|
|
|
||
|
|
``cat /sys/fs/cgroup/memory/<memory cgroup>/dhugetlb.nr_pages``:
|
||
|
|
|
||
|
|
Reads the memory information in the hpool, include the free amount and used amount of huge pages and
|
||
|
|
normal pages.
|
||
|
|
|
||
|
|
3. dhugetlb.1G.reserved_pages
|
||
|
|
-----------------------------
|
||
|
|
|
||
|
|
In each memory cgroup, there is a ``dhugetlb.nr_pages`` interface used to reserved 1G huge pages.
|
||
|
|
By default, all memory configured to a dynamic hugetlb pool can be used only as normal pages, if want to use
|
||
|
|
it as 1G huge pages, need to configure the number of 1G huge pages by this interface firstly.
|
||
|
|
|
||
|
|
Usage:
|
||
|
|
``echo <nr_pages> > /sys/fs/cgroup/memory/<memory cgroup>/dhugetlb.1G.reserved_pages``
|
||
|
|
|
||
|
|
4. dhugetlb.2M.reserved_pages
|
||
|
|
-----------------------------
|
||
|
|
|
||
|
|
Similar to the previous interface, this is used to configure the number of 2M huge pages.
|
||
|
|
|
||
|
|
Usage:
|
||
|
|
``echo <nr_pages> > /sys/fs/cgroup/memory/<memory cgroup>/dhugetlb.2M.reserved_pages``
|
||
|
|
|
||
|
|
5. dhugetlb.normal_pages_disabled
|
||
|
|
---------------------------------
|
||
|
|
|
||
|
|
The dynamic hugetlb pool can be used as just a huge pages pool. This interface is used to disable allocation
|
||
|
|
of normal pages from the dynamic hugetlb pool.
|
||
|
|
|
||
|
|
Usage:
|
||
|
|
``echo 1 > /sys/fs/cgroup/memory/<memory cgroup>/dhugetlb.normal_pages_disabled``
|
||
|
|
|
||
|
|
---
|
||
|
|
Liu Shixin, Jan 2022
|