[GB202 / RTX 5090] Uninterruptible D-state deadlock in RemoteDpcdReadMessage / MessageManager::send on DisplayPort modeset with MST branch device

[cshabsin]

NVIDIA Open GPU Kernel Modules Version

610.57.04 (and 595.84) - linux-modules-nvidia-610-open-generic / nvidia-driver-610-open

Please confirm this issue does not happen with the proprietary driver

• I confirm that this does not happen with the proprietary driver package.
  (Note: RTX 50 series / GB202 Blackwell only supports the open kernel modules; the legacy proprietary closed driver cannot be used).

Operating System and Version

Ubuntu 26.04 LTS

Kernel Release

Linux medina 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug 1 04:26:38 UTC 2026 x86_64 GNU/Linux

Please confirm you are running a stable release kernel

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-96639ba1-6655-f088-8ffb-e025ec620129, Device ID: 2B85)

Describe the bug

During an atomic modeset commit under GNOME Wayland (Mutter), specifically when entering screen blanking or renegotiating DisplayPort links with an MST-enabled display, the driver enters an uninterruptible sleep (D state) semaphore deadlock.

The kernel begins reporting hung tasks after 122 seconds:
INFO: task nvidia-modeset/:1508 blocked on a semaphore likely last held by task KMS thread:16987

Once this deadlock occurs:

1. nvidia-modeset/kthread_q is trapped in D state and cannot be killed.
2. KMS thread (gnome-shell) is trapped in D state inside RemoteDpcdReadMessage / MessageManager::send.
3. The kernel module nvidia_modeset cannot be unloaded.
4. All subsequent atomic commits fail with Page flip failed: drmModeAtomicCommit: Invalid argument.
5. DisplayPort monitor fails to wake; only a hard system reboot can clear the deadlock.

Stack Trace - Thread 1 (nvidia-modeset/kthread_q, PID 1508):

task:nvidia-modeset/ state:D stack:0 pid:1508
Call Trace:
 <TASK>
 __schedule+0x2bb/0x650
 schedule+0x27/0xb0
 schedule_timeout+0xcf/0x110
 ___down_common+0x108/0x1e0
 __down_common+0x5a/0x180
 __down+0x1d/0x30
 down+0x5e/0x80
 nvkms_kthread_q_callback+0xb4/0x190 [nvidia_modeset]
 _main_loop+0x99/0x160 [nvidia_modeset]
 kthread+0xf7/0x130
 ret_from_fork+0x195/0x2a0
 ret_from_fork_asm+0x1a/0x30
 </TASK>

Stack Trace - Thread 2 (KMS thread, PID 16987, holding semaphore and waiting on DP AUX):

task:KMS thread state:D stack:0 pid:16987 tgid:16972
Call Trace:
 <TASK>
 __schedule+0x2bb/0x650
 schedule+0x27/0xb0
 schedule_timeout+0x88/0x110
 schedule_timeout_uninterruptible+0x29/0x50
 msleep+0x1b/0x30
 nvkms_usleep+0x4f/0x60 [nvidia_modeset]
 _ZN11DisplayPort14MessageManager4sendEPNS0_7MessageERNS_7NakDataE+0xf7/0x2f0 [nvidia_modeset]
 _ZN11DisplayPort21RemoteDpcdReadMessage3setERKNS_7AddressEjjj+0xca/0x150 [nvidia_modeset]
 _ZN11DisplayPort10DeviceImpl11transactionENS_6AuxBus6ActionENS1_4TypeEiPhjPjS5_hh+0xc21/0x12f0 [nvidia_modeset]
 _ZN11DisplayPort10DeviceImpl11getDpcdDataEjPhjPjS2_+0xb3/0x170 [nvidia_modeset]
 _ZN11DisplayPort10DeviceImpl18setIgnoreMSAEnableEb+0x95/0x130 [nvidia_modeset]
 nvDPLibSetAdaptiveSync+0x66/0x90 [nvidia_modeset]
 nvDisableVrr+0x298/0x350 [nvidia_modeset]
 nvSetDispModeEvo+0x2e7a/0x40a0 [nvidia_modeset]
 nvKmsIoctl+0xf2/0x240 [nvidia_modeset]
 nvkms_ioctl_from_kapi_try_pmlock+0x6b/0xc0 [nvidia_modeset]
 ApplyModeSetConfig+0x382/0xc70 [nvidia_modeset]
 nv_drm_atomic_apply_modeset_config+0x2c9/0xb40 [nvidia_drm]
 nv_drm_atomic_commit+0x25c/0x5d0 [nvidia_drm]
 drm_atomic_commit+0xad/0xf0
 drm_mode_atomic_ioctl+0x7d1/0x910
 </TASK>

To Reproduce

1. Connect a display with an internal MST hub via DisplayPort (Dell U3225QE) alongside a secondary display (Dell U2719DC on HDMI) to an RTX 5090.
2. Run GNOME Wayland session.
3. Have CUDA compute workload resident in background (e.g. Ollama with ~24 GB allocated in VRAM).
4. Trigger screen-blank modeset (e.g. idle timeout idle-delay 300 or DP link renegotiation).
5. The display fails to wake from blank; nvidia-modeset enters uninterruptible sleep D state with hung task reports repeating every 122 seconds.

Bug Incidence

Sometimes (reproduced on multiple days under both driver 595.84 and 610.57.04).

nvidia-bug-report.log.gz

(Please find nvidia-bug-report.log.gz attached to this issue, captured via sudo nvidia-bug-report.sh --safe-mode during the active deadlock).

More Info

Notice that nvSetDispModeEvo invokes setIgnoreMSAEnable, which issues RemoteDpcdReadMessage to the MST branch device. The modeset semaphore is held across the wait inside MessageManager::send (nvkms_usleep / schedule_timeout_uninterruptible). If the DP AUX reply is lost or delayed, the timeout mechanism appears unable to release the lock, permanently hanging both threads and preventing any future modesets until a hard reboot.

[cshabsin]

nvidia-bug-report.log.gz

[cshabsin]

Additional Findings & Hardware Reproduction Context

Following further diagnosis on the system, here are additional details that may help isolate the root cause in nvidia-modeset:

1. Reproducibility Across Driver Branches

• This deadlock is not isolated to 610.57.04. The exact same behavior occurred under 595.91.07 prior to upgrading, confirming the bug is present in the common MST/DPCD state machine across both driver series.

2. Specific Display Hardware & Internal MST Architecture

• The display triggering this is a Dell U3225QE (32" 4K 120Hz IPS Black with built-in Thunderbolt 4 / MST hub, connected directly via DisplayPort).
• The U3225QE internally routes its DisplayPort stream through an integrated MST branch controller (Microchip / Realtek ASIC hub).
• When the display enters low-power sleep or power-save, its internal MST controller shuts down or resets its AUX channel communication.
• We observed in kernel logs that power/mode transitions on this display trigger a complete reset and re-enumeration of its internal hub devices (Microchip USB4206 Smart Hub), during which DPCD/EDID over the DisplayPort AUX channel temporarily drops out entirely.

3. Root Cause of the Hang: Missing Timeout on DPCD AUX in MessageManager::send

• The deadlock consistently triggers during a DPMS power-down transition (e.g., lock-screen or idle blanking) when GNOME/Mutter (KMS thread) issues an atomic modeset commit at the same time the monitor's internal MST controller begins entering power standby.
• In nvidia-modeset, RemoteDpcdReadMessage / MessageManager::send appears to assume the MST branch device will always respond to DPCD requests. When the monitor drops off the AUX bus mid-transition without an ACK, MessageManager::send hangs indefinitely in uninterruptible D state while holding the driver's global modeset semaphore.
• As a result:
  1. nvidia-modeset/kthread_q hangs waiting on that semaphore.
  2. All other GPU operations (CUDA compute, nvidia-smi, DRM ioctls) block indefinitely behind it.
  3. The system cannot be safely shut down or recovered without a hard reset.

4. Workarounds Under Investigation

• We are currently testing disabling OS-level DPMS blanking (idle-delay 0 in GNOME) to prevent the desktop environment from automatically initiating DisplayPort power-down transitions.
• In parallel, we are experimenting with managing monitor standby directly via DDC/CI (VCP 0xd6 0x04 / 0x01), allowing the physical panel to sleep without requesting a GPU-level DPMS modeset.
• These workarounds have only been active for a short testing window, but early observations suggest that avoiding OS-level DPMS state changes bypasses the codepath where the driver gets stuck.

[cshabsin]

I hope it's OK I used AI to help diagnose this and author the text here.

[cshabsin]

Update: Direct Trigger Identified (nvDisableVrr / nvDPLibSetAdaptiveSync)

The system encountered the exact same D-state deadlock again today, and the latest kernel trace captured the specific callchain initiating the hang.

Trigger Scenario

1. The system was running in "Work Mode": the Dell U3225QE was switched away from DisplayPort to its secondary HDMI input (connected to a work laptop).
2. With the DisplayPort input inactive, the monitor's DisplayPort MST receiver entered low-power/unresponsive standby.
3. While the monitor was displaying HDMI, GNOME Wayland performed an atomic modeset commit (drm_mode_atomic_ioctl) that invoked nvDisableVrr / nvDPLibSetAdaptiveSync on the inactive DisplayPort connector.
4. The driver attempted to send DPCD AUX messages to update Adaptive Sync (_ZN11DisplayPort10DeviceImpl18setIgnoreMSAEnableEb), but because the monitor was switched away, the DPCD read received no response.
5. MessageManager::send waited indefinitely holding the modeset semaphore, deadlocking nvidia-modeset/kthread_q and freezing all subsequent graphics and CUDA calls.

Full Kernel Call Trace:

Call Trace:
 <TASK>
 ? _ZN11DisplayPort8AuxRetry5writeEiPhjj+0x116/0x180 [nvidia_modeset]
 ? _ZN11DisplayPort26OutgoingTransactionManager13writeToWindowEb+0x36/0x180 [nvidia_modeset]
 ? _ZN11DisplayPort5Timer4fireEb+0xc7/0xd0 [nvidia_modeset]
 ? _ZN11DisplayPort5Timer5_pumpEjb+0x34/0x60 [nvidia_modeset]
 ? nvkms_get_usec+0x32/0xa0 [nvidia_modeset]
 ? _ZN11DisplayPort14MessageManager4sendEPNS0_7MessageERNS_7NakDataE+0xf7/0x2f0 [nvidia_modeset]
 ? _ZN11DisplayPort21RemoteDpcdReadMessage3setERKNS_7AddressEjjj+0xca/0x150 [nvidia_modeset]
 ? _ZN11DisplayPort10DeviceImpl11transactionENS_6AuxBus6ActionENS1_4TypeEiPhjPjS5_hh+0xc21/0x12f0 [nvidia_modeset]
 ? portSyncSpinlockAcquire+0x18/0x30 [nvidia]
 ? os_release_spinlock+0x1a/0x30 [nvidia]
 ? _ZN11DisplayPort10DeviceImpl11getDpcdDataEjPhjPjS2_+0xb3/0x170 [nvidia_modeset]
 ? _ZN11DisplayPort10DeviceImpl18setIgnoreMSAEnableEb+0x95/0x130 [nvidia_modeset]
 ? nvDPLibSetAdaptiveSync+0x66/0x90 [nvidia_modeset]
 ? nvDisableVrr+0x298/0x350 [nvidia_modeset]
 ? nvSetDispModeEvo+0x2e7a/0x40a0 [nvidia_modeset]
 ? Flip+0xf0/0xf0 [nvidia_modeset]
 ? nvKmsIoctl+0xf2/0x240 [nvidia_modeset]
 ? nvkms_ioctl_from_kapi_try_pmlock+0x6b/0xc0 [nvidia_modeset]
 ? ApplyModeSetConfig+0x382/0xc70 [nvidia_modeset]
 ? nv_drm_atomic_apply_modeset_config+0x2c9/0xb40 [nvidia_drm]
 ? nv_drm_atomic_commit+0x25c/0x5d0 [nvidia_drm]
 ? nv_drm_atomic_check+0x12d/0x150 [nvidia_drm]
 ? drm_atomic_commit+0xad/0xf0
 ? drm_mode_atomic_ioctl+0x7d1/0x910
 ? nv_drm_ioctl+0x38/0x420 [nvidia_drm]
 </TASK>

Significance

This demonstrates that Variable Refresh Rate (VRR / Adaptive Sync / G-Sync Compatible) state changes over DPCD are the direct trigger. When the display is switched to another input or is in standby, any modeset that disables or modifies VRR hangs in MessageManager::send.