Conversation
…ction under lock
In Linux 5.18 and later kernels, commit 9becb6889130 ("kvmalloc: use
vmalloc_huge for vmalloc allocations") changed the behavior of kvmalloc():
when the requested allocation size is greater than or equal to PMD_SIZE
(2MB on x86_64), it automatically adds the VM_ALLOW_HUGE_VMAP flag.
When registering 512MB or more memory, the page table array size pt_size
reaches 2MB (PMD_SIZE). In nvos_create_alloc(), this array was previously
allocated using kvzalloc(pt_size, NV_GFP_KERNEL).
Because the size reaches PMD_SIZE, kvzalloc() attempts to allocate 2MB
huge pages. When physical memory is heavily fragmented, the kernel enters
synchronous direct compaction to assemble 2MB contiguous physical memory.
This process involves page migration and cross-core TLB flushes, which can
take a very long time.
Crucially, this allocation is performed while holding the global write lock
g_RmApiLock. Holding this write lock exclusively during memory compaction
causes other processes in the system that need GPU access to be blocked in
uninterruptible sleep (D state). In severe cases, the blocking time exceeds
the hung_task_timeout_secs threshold.
In practice, the page_table array only needs to be virtually contiguous for
driver addressing and indexing. It does not require physically contiguous
memory, nor does it require 2MB huge page mappings.
To fix this: when pt_size >= PMD_SIZE, use nv_vmalloc() instead of kvzalloc().
nv_vmalloc() calls __vmalloc() without huge page flags, allocating only
standard 4KB pages and completely avoiding the direct compaction overhead of
2MB huge pages.
Since PMD_SIZE is much smaller than INT_MAX, this change also covers and
preserves the original protection logic for oversized page tables (> INT_MAX).
Signed-off-by: Jialin Wang <wjl.linux@gmail.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Avoid blocking other processes due to direct compaction while holding g_RmApiLock
Overview
When allocating the page table in
nvos_create_alloc()andpt_size >= PMD_SIZE(2MB on x86_64), usenv_vmalloc()instead ofkvzalloc(). This prevents the driver from triggering 2MB huge page allocation and time-consuming direct compaction while holding theg_RmApiLockwrite lock, fixing the issue where other GPU-dependent processes are blocked for a long time.Problem
In Linux 5.18 and later kernels, commit 9becb6889130 ("kvmalloc: use vmalloc_huge for vmalloc allocations") modified
kvmalloc()behavior: when the requested allocation size is greater than or equal toPMD_SIZE, it automatically passes theVM_ALLOW_HUGE_VMAPflag.When registering 512MB or more memory, the page table array size
pt_sizereaches 2MB (i.e.PMD_SIZE). Innvos_create_alloc(), this array was previously allocated withkvzalloc(pt_size, NV_GFP_KERNEL).This allocation happens while holding the
g_RmApiLockwrite lock. Because the size reachesPMD_SIZE,kvzalloc()attempts to allocate 2MB huge pages. When physical memory is heavily fragmented (for example, when too much pinned memory prevents page migration, making compaction difficult), the kernel enters the slow path and runs synchronous direct compaction, performing page migration and cross-core TLB flushes.The main problem is that during memory compaction, this thread keeps holding the
g_RmApiLockglobal write lock. As a result, all other processes in the system that need GPU access are severely blocked in uninterruptible sleep (D state), halting workload progress. Under severe fragmentation, the blocking duration can even exceed the system-configuredhung_task_timeout_secsthreshold (configured as 28 seconds on our machines), causing the kernel to dump hung task call traces.Call Traces
g_RmApiLockwrite lock:Solution
The
page_tablearray only needs to be contiguous in kernel virtual address space for driver access. It does not require physically contiguous memory, nor does it require 2MB huge page mappings.We adjust the threshold in
nvos_create_alloc()andnvos_free_alloc()fromINT_MAXtoPMD_SIZE:pt_size >= PMD_SIZE, usenv_vmalloc(). It calls__vmalloc()withoutVM_ALLOW_HUGE_VMAP, allocating only standard 4KB pages and avoiding synchronous direct compaction.pt_size < PMD_SIZE, continue usingkvzalloc(), preserving fastkmalloc()allocations for small sizes.PMD_SIZEis much smaller thanINT_MAX, the original protection logic for oversized page tables (> INT_MAX) is fully preserved.Reproduction Environment
Verification
After applying this change, bpftrace showed no huge page allocations on this call path, and the issue could no longer be reproduced.
sudo bpftrace -e 'rawtracepoint:mm_page_alloc /arg1 >= 9/ { @[comm, arg0 != 0 ? "SUCCESS" : "FAILED", kstack] = count(); } interval:s:10 { time("%H:%M:%S\n"); print(@); }'