macOS/Arm DXMT crash when creating depth/stencil texture with initial data

[moreaki]

Problem

On macOS/Arm with a Windows 11 Arm guest using the VirtualBox WDDM graphics adapter and 3D acceleration, I can reproduce a VirtualBoxVM host-process crash when starting a WebGL2 workload in Microsoft Edge.

The browser-side test gets as far as creating the WebGL2 context and then the VM process aborts before shader/program setup completes.

Reproduction setup

• Host: macOS/Arm test machine, Mac Studio with Studio Display
• Guest: Windows 11 Arm
• Graphics controller: VMSVGA / VirtualBox Graphics Adapter (WDDM)
• 3D acceleration: enabled
• VRAM: 256 MB
• Workload: local WebGL2 benchmark in Microsoft Edge, using ANGLE/D3D11

Browser event trace from the failing run:

{"event":"script-start","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 Edg/150.0.0.0"}
{"event":"webgl-context","ok":true}

No programs-created event is reached in the failing run.

Crash signature

The macOS crash report shows:

Exception: EXC_BAD_ACCESS / SIGSEGV
Faulting thread: VMSVGA CMD

AGXMetalG14X: agxaAssertBufferIsValid
AGXMetalG14X: AGX::Texture<...>::writeRegion
AGXMetalG14X: -[AGXG14XFamilyTexture replaceRegion:mipmapLevel:slice:withBytes:bytesPerRow:bytesPerImage:]
VBoxDxMt.dylib: WMT::Texture::replaceRegion(...)
VBoxDxMt.dylib: dxmt::InitializeTextureData<D3D11_TEXTURE2D_DESC1>(...)
VBoxDxMt.dylib: dxmt::CreateDeviceTextureInternal<dxmt::tag_texture_2d>(...)
VBoxDxMt.dylib: dxmt::MTLD3D11DeviceImpl::CreateTexture2D(...)
VBoxDD.dylib: dxCreateTexture(...)
VBoxDD.dylib: vmsvga3dBackSurfaceCreateResource(...)
VBoxDD.dylib: dxEnsureResource(...)
VBoxDD.dylib: dxCreateDepthStencilView(...)
VBoxDD.dylib: dxEnsureDepthStencilView(...)
VBoxDD.dylib: vmsvga3dBackDXClearDepthStencilView(...)
VBoxDD.dylib: vmsvga3dCmdDXClearDepthStencilView(...)

The relevant path is depth/stencil resource creation, followed immediately by a depth/stencil clear. The crash happens while DXMT/Metal validates the source bytes passed as initial texture data.

Suspected cause

vmsvga3dBackSurfaceCreateResource() currently passes paInitialData whenever pSurfaceData is non-null, except for multisample and decoder resources.

However, VMSVGA3DMIPMAPLEVEL::pSurfaceData is documented as optional and potentially stale after hardware realization. For depth/stencil resources that are about to be cleared, uploading this host-side backing buffer during texture creation is not required and can hand invalid/stale source memory to DXMT's Metal replaceRegion path.

Verified fix

Skipping initial data for D3D11_BIND_DEPTH_STENCIL resources avoids the crash in the same local reproduction.

After applying the fix and rebuilding/staging VBoxDD.dylib, the same workload completed:

Workload: WebGL2 / ANGLE D3D11, 512 instances, 30 second measurement
Result: VM remained running
Average FPS: 58.9
p50 frame time: 16.70 ms
p95 frame time: 17.30 ms
Renderer: ANGLE (VirtualBox Graphics Adapter (WDDM), Direct3D11 vs_5_0 ps_5_0)

A smaller 8 second smoke run also completed at 59.9 FPS. Both runs previously failed at or shortly after WebGL context creation in the same setup.