[Bug]: Windows 10 native recording can hang on stop and fail to save MP4 in 1.9.4-rc.3

[slipperpeng]

Search existing issues

• I have searched the existing issues.

Related: #252, #336, #346 (and the earlier #34 / #115 stop-path reports).

Describe the bug

On the packaged OpenScreen 1.9.4-rc.3 app on Windows 10, ending a native Windows screen recording can fail to save the recording. The user sees a save/recording error after clicking Stop, the editor does not open with a usable recording, and the MP4 may never be finalized.

I reproduced and debugged this from the openscreen-release-v1.9.4 source archive on the affected machine. There were three distinct problems in the native helper path:

1. Windows 10 startup crash: session_.IsBorderRequired(false) was called through the projected GraphicsCaptureSession object without first querying the optional IGraphicsCaptureSession3 interface. On Windows 10 19045 this could terminate wgc-capture.exe with 0xC0000005; the surrounding C++ exception handler cannot catch a call through a missing ABI interface.
2. Stop/save hang: the DXGI input path created a second D3D11 encoding device and copied every frame through a shared texture guarded by IDXGIKeyedMutex::AcquireSync. Under the full recording load on this machine, the display driver could fail to return from that cross-device synchronization. At the same time, application frame-state locking covered GPU copy/readback work. WGC callbacks then could not quiesce, the video thread could not join, and IMFSinkWriter::Finalize was never reached, leaving the MP4 unsaved.
3. Packaged-app DPI compatibility: the installed 1.9.4-rc.3 main process supplied the selected display as Electron DIP bounds (2560x1440 at 150%), while the rebuilt per-monitor-v2-aware helper enumerated the physical monitor as 3840x2160. Strict matching rejected the only monitor and the app fell back from native capture. This is closely related to wgc-capture.exe is DPI-unaware, and its monitor lookup silently depends on it #346.

The virtual display adapter and the discrete GPU may make the D3D synchronization failure more likely, but the blocking cross-device bridge and lock ownership make a driver stall fatal regardless of the trigger.

Expected behavior

Clicking Stop should quiesce WGC callbacks, join the writer threads, finalize the fragmented MP4, emit recording-stopped, and open the editor promptly. Optional Windows capture interfaces must not crash older Windows versions, and a single-monitor scaled setup must resolve the selected display without weakening multi-monitor correctness.

To Reproduce

1. Run packaged OpenScreen 1.9.4-rc.3 on Windows 10 22H2.
2. Select the 3840x2160 display at 150% Windows scaling.
3. Enable system audio, microphone, webcam, and editable cursor capture.
4. Record for several seconds.
5. Click Stop.
6. Observe that native capture can fail to stop/finalize and the application reports that the recording could not be saved.

The native-helper failure is below the React UI. Chromium MediaRecorder/WebM fallback saved correctly, but that only avoids the affected WGC helper path.

Root cause details

The relevant code is under electron/native/wgc-capture/src/.

• wgc_session.cpp: direct use of IsBorderRequired assumes IGraphicsCaptureSession3 is present.
• mf_encoder.cpp / .h: the DXGI pipeline owns a second D3D11 device plus a shared keyed-mutex bridge between capture and encoding devices.
• main.cpp: the WGC callback and video writer hold the frame-state mutex across D3D CopyResource, conversion, or staging readback work. A driver call that does not return prevents shutdown from draining the producer and joining the writer.
• monitor_utils.cpp: physical bounds are correct for the current helper, but an already packaged 1.9.4-rc.3 main process can still provide legacy DIP bounds.

The finite keyed-mutex timeout was not a sufficient safeguard on this machine: the driver-level synchronization could still fail to return. Because the writer was stranded before finalization, the user-visible result was a save failure rather than a recoverable dropped frame.

Fix implemented and tested locally

I rebuilt wgc-capture.exe with the following changes:

1. Query IGraphicsCaptureSession3 with try_as before setting IsBorderRequired; skip the cosmetic border setting when unavailable.
2. Remove the second D3D11 encoding device, the shared bridge textures, and both IDXGIKeyedMutex handshakes.
3. Reuse the WGC capture device and its multithread-protected immediate context for Media Foundation, BGRA-to-NV12 video processing, sample allocation, and hardware H.264 input.
4. Create one stable BGRA frame texture before callbacks start. On the DXGI path it has D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET.
5. Keep the frame-state mutex limited to timestamps and small state updates. Do CopyResource, GPU conversion, CPU readback, and synchronous encoder submission outside that application mutex. D3D's multithread protection serializes the shared immediate context.
6. Prefer the repaired dxgi-nv12 path by default; retain OPENSCREEN_WGC_ENABLE_DXGI_INPUT=0 as a diagnostic CPU fallback.
7. Preserve strict physical-bounds matching on multi-monitor systems. If and only if Windows enumerates exactly one monitor, accept legacy mismatched DIP bounds and select that sole monitor. This allows the rebuilt helper to work with the already packaged 1.9.4-rc.3 main process without reintroducing wgc-capture.exe is DPI-unaware, and its monitor lookup silently depends on it #346's wrong-monitor guessing.

Test results

All tests below were performed on the affected machine with the rebuilt helper.

Native helper, default configuration, display + system audio + microphone

• videoInput: dxgi-nv12
• 351 video frames
• wgc-quiesce drained=true
• recording-stopped emitted
• Media Foundation finalize: approximately 169 ms
• Total stop: approximately 215 ms
• Exit code: 0

Repository full Windows helper test

Command: node scripts/test-windows-wgc-helper.mjs --webcam --system-audio --microphone

• Screen: H.264, 8.033 s, 8,014,121 bytes
• Audio: AAC, 7.872 s
• Webcam: H.264, 1920x1080, 8.033 s, 8,329,975 bytes
• Decoded first-frame mean brightness: 92.52 (not black)
• Stop through all outputs saved: approximately 971 ms
• Fragmented MP4 prefix readability check passed
• Exit code: 0

End-to-end development app test, forced native helper

• Log contains [native-wgc] starting Windows capture and [native-wgc] capture started
• Encoder selection: default, videoInput: dxgi-nv12, container: fragmented-mp4
• Stop to editor: approximately 1.32 s
• Screen MP4: 3,818,300 bytes
• Webcam MP4: 8,020,193 bytes
• No Chromium/WebM fallback

End-to-end test of the actually installed packaged 1.9.4-rc.3 app

• Used the helper from the installed app resources, with display + system audio + microphone + webcam + editable cursor overlay
• Legacy 2560x1440 DIP bounds safely resolved to the sole physical 3840x2160 monitor
• Encoder selection: default, videoInput: dxgi-nv12, container: fragmented-mp4
• Stop to editor: 1,369 ms
• Screen file: H.264 3840x2160 + AAC 48 kHz stereo, 8.0668 s, 6,947,583 bytes
• Separate webcam file: H.264 1280x720, 8.0667 s, 7,860,646 bytes
• Both files were non-empty and readable by ffprobe
• No native-capture fallback and no save error

npm run build-vite also passed. The native helper and cursor sampler rebuilt successfully with Visual Studio 2022 Build Tools.

Screenshots

The original user-facing error was displayed immediately after ending the recording and stated that the recording could not be saved. I can provide the screenshot separately if useful; it contains no additional native stack trace.

OS

Windows

OS Version

Windows 10 Pro 22H2, version 10.0.19045, build 19045, 64-bit

Browser

Other

Browser Version

Packaged Electron app (Electron 41.2.1)

Device Type

Desktop

Additional context

• OpenScreen: 1.9.4-rc.3 (1.9.4.0 product version)
• CPU: AMD Ryzen 7 5700X, 8 cores / 16 logical processors
• Memory: 31.9 GiB
• Physical GPU: NVIDIA GeForce RTX 4060 Ti
• NVIDIA driver: 31.0.15.5152
• Additional display adapter: OrayIddDriver Device, driver 17.50.19.949
• Display: 3840x2160
• Windows scaling: 150% (AppliedDPI=144)
• Rebuilt helper SHA-256: 1234FA69D894E6AEFB805BFD0CC4E4D9AE042806FF95DE3784DB94AE226893C2
• Toolchain: Visual Studio 2022 Build Tools 17.10.5, Node 22.22.1, npm 10.9.4

No username, machine name, local paths, browser media device IDs, or recording content are included in this report.

[EtienneLescot]

Thanks for the detailed trace.

Two of your three findings are already addressed on release/v1.9.5 (rc.2):

• Stop hang. fe794ff1 made the DXGI input path opt-in, and it is not in
  1.9.4-rc.3 — which is why you got videoInput: dxgi-nv12 by default. rc.2
  defaults to cpu-rgb32, so the second D3D11 device and the keyed-mutex bridge
  are off the default path. Verified on a build here.
• DPI mismatch. 60bb6d7c converts Electron DIPs to physical pixels before
  the rect reaches the helper. Your single-monitor shim was needed because a new
  helper met the old rc.3 main process; in rc.2 both halves agree.

IsBorderRequired: which Windows SDK did you build against? With C++/WinRT
2.0.250303.5 the projected setter expands to
try_as_with_reason<IGraphicsCaptureSession3> + check_hresult — it throws
hresult_no_interface rather than making an unchecked vtable call, and the
existing catch (...) handles it. Older C++/WinRT emitted the unchecked form, so
this could be real against your SDK, but then it is a local-rebuild issue rather
than one in the shipped helper.

Can you run rc.2 as shipped, without your patch? Hang gone → your diagnosis is
confirmed. Hang persists → the DXGI bridge is ruled out and the search narrows.
That is more useful right now than merging a patch.

We won't restore dxgi-nv12 as the default: it is off because it cost the
reporter of #336 their recording. Worth revisiting in 1.10 with your rewrite
reviewed.

[slipperpeng]

Thanks. I tested the current release/v1.9.5 head without my patch. The result is the second branch of your diagnostic: the hang persists with cpu-rgb32, so the DXGI bridge is ruled out on this machine.

Exact build tested

There is no public rc.2 installer/tag yet, so I used the repository's own Windows diagnostic artifact rather than another local rebuild:

• branch/head: release/v1.9.5 at b1b81de582e5b801a58b1b3d3fdd33b7f86239b9
• successful official workflow: Diagnostic artifact run 31784641764
• artifact: openscreen-diagnostic-windows-x64
• wgc-capture.exe SHA-256: 11800C8859F559641E76052E2852B4CFAAD368B32D31076034EA85F2E22CCC2E
• no OPENSCREEN_WGC_ENABLE_DXGI_INPUT variable was set

The helper reported:

{"video":"default","videoInput":"cpu-rgb32","container":"fragmented-mp4","preferSoftwareEncoder":false}

SDK answer / IsBorderRequired

My local rebuild used:

• Windows SDK 10.0.22621.0
• C++/WinRT 2.0.220110.5
• Visual Studio Build Tools 17.10.5 / MSVC 19.40.33813

You were correct about the old projection. An unmodified local build of the same branch exits with 0xC0000005 immediately after the cursor-capture event, before recording starts or stop is sent. The official CI helper above does not crash there. I therefore consider IsBorderRequired a local old-SDK rebuild issue, separate from the shipped-helper stop hang.

Default rc.2 result

The repository helper test failed 3/3 times with the official CI helper (5 s display capture, no audio). All three runs received stop, then followed the same sequence:

command-received       0 ms
wgc-quiesce            ~5000 ms, drained=false
microphone             ~5000 ms
loopback               ~5000 ms
webcam                 ~5000 ms
audio-mixer            ~5000 ms
video-writer-join      ~13000 ms, phase=abandoned
exit code              3

The raw diagnostic contains:

WARNING: A WGC frame callback did not finish; leaving the device alive
[stop-timing] step=video-writer-join elapsed_ms=13030 phase=abandoned encode_stage=idle audio_stage=idle

I also ran the bundled diagnostic tool for 10 seconds:

Inputs	WGC drain	Abandoned step	Stop elapsed	Exit
display, no audio	5003 ms, false	video-writer-join at 13030 ms	13066 ms	3
display + system audio + default mic	5009 ms, false	video-writer-join at 13600 ms	13635 ms	3

The selected microphone in the second run was the real default endpoint. Audio changes the later timings slightly but is not required to reproduce the hang.

Two more controls fail in the same place:

• window capture instead of display capture
• display capture with the Microsoft software H.264 encoder forced

That rules out the display-source type and the hardware H.264 encoder as necessary triggers. It does not yet isolate texture size.

CPU-path A/B

As a control, I ran my backed-up repaired helper with OPENSCREEN_WGC_ENABLE_DXGI_INPUT=0, so it also reported videoInput: cpu-rgb32. It passed 3/3 times on the same machine and same test script:

• stop latency: 79 ms, 140 ms, 138 ms
• every run completed wgc-quiesce -> video-writer-join -> encoder-finalize -> wgc-session-close
• every run produced a readable 5.000 s H.264 fragmented MP4
• repaired helper SHA-256: 1234FA69D894E6AEFB805BFD0CC4E4D9AE042806FF95DE3784DB94AE226893C2

This means the successful result does not depend on my rewritten DXGI path being active.

The remaining difference that matches the timing is the frame-state lock scope. In current rc.2:

• the WGC callback holds mutex across ID3D11DeviceContext::CopyResource (main.cpp around 913-927);
• the writer holds the same mutex across captureVideoSample (main.cpp around 961-1085);
• the CPU capture then calls staging CopyResource and blocking Map(D3D11_MAP_READ) (mf_encoder.cpp around 916-919).

If that CPU readback stalls while the writer owns mutex, an already-entered WGC callback waits on the same mutex. quiesceCapture() then cannot drain that callback, and joining the writer cannot finish. Also, encode_stage=idle appears not to cover the CPU copyFrameToBuffer path, so it does not rule out CopyResource/Map; the stage breadcrumbs are set on the DXGI conversion path.

My repaired helper moves the D3D copy/readback work outside the application frame-state mutex and relies on D3D multithread protection for the shared immediate context. The successful forced-CPU A/B makes that lock-scope change the leading explanation now, rather than the keyed-mutex bridge.