From 2c4cce44595497148cd12c446a9a888ece0de01a Mon Sep 17 00:00:00 2001 From: Jialin Wang Date: Tue, 22 Sep 2026 13:14:53 +0800 Subject: [PATCH] nvidia: use nv_vmalloc when pt_size >= PMD_SIZE to avoid direct compaction under lock In Linux 5.18 and later kernels, commit 9becb6889130 ("kvmalloc: use vmalloc_huge for vmalloc allocations") changed the behavior of kvmalloc(): when the requested allocation size is greater than or equal to PMD_SIZE (2MB on x86_64), it automatically adds the VM_ALLOW_HUGE_VMAP flag. When registering 512MB or more memory, the page table array size pt_size reaches 2MB (PMD_SIZE). In nvos_create_alloc(), this array was previously allocated using kvzalloc(pt_size, NV_GFP_KERNEL). Because the size reaches PMD_SIZE, kvzalloc() attempts to allocate 2MB huge pages. When physical memory is heavily fragmented, the kernel enters synchronous direct compaction to assemble 2MB contiguous physical memory. This process involves page migration and cross-core TLB flushes, which can take a very long time. Crucially, this allocation is performed while holding the global write lock g_RmApiLock. Holding this write lock exclusively during memory compaction causes other processes in the system that need GPU access to be blocked in uninterruptible sleep (D state). In severe cases, the blocking time exceeds the hung_task_timeout_secs threshold. In practice, the page_table array only needs to be virtually contiguous for driver addressing and indexing. It does not require physically contiguous memory, nor does it require 2MB huge page mappings. To fix this: when pt_size >= PMD_SIZE, use nv_vmalloc() instead of kvzalloc(). nv_vmalloc() calls __vmalloc() without huge page flags, allocating only standard 4KB pages and completely avoiding the direct compaction overhead of 2MB huge pages. Since PMD_SIZE is much smaller than INT_MAX, this change also covers and preserves the original protection logic for oversized page tables (> INT_MAX). Signed-off-by: Jialin Wang --- kernel-open/nvidia/nv.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/kernel-open/nvidia/nv.c b/kernel-open/nvidia/nv.c index df5a5e0e5..d1f845a8a 100644 --- a/kernel-open/nvidia/nv.c +++ b/kernel-open/nvidia/nv.c @@ -433,8 +433,12 @@ nv_alloc_t *nvos_create_alloc( return NULL; } - /* kvzalloc() rejects sizes > INT_MAX; use vmalloc() for oversized tables. */ - if (pt_size > (NvU64)INT_MAX) + /* + * Use nv_vmalloc() for pt_size >= PMD_SIZE to avoid huge page allocation + * and direct compaction while holding locks. This also covers oversized + * tables (> INT_MAX), which kvzalloc() rejects. + */ + if (pt_size >= (NvU64)PMD_SIZE) at->page_table = nv_vmalloc(pt_size, NV_GFP_KERNEL | __GFP_ZERO); else at->page_table = kvzalloc(pt_size, NV_GFP_KERNEL); @@ -468,7 +472,7 @@ int nvos_free_alloc( pt_size = (NvU64)at->num_pages * sizeof(nvidia_pte_t); - if (pt_size > (NvU64)INT_MAX) + if (pt_size >= (NvU64)PMD_SIZE) nv_vfree(at->page_table, pt_size); else kvfree(at->page_table);