[Req]: Improve Windows guest Retina/HiDPI support on macOS ARM

[moreaki]

Problem

Windows 11 ARM guests on macOS/Arm hosts are currently awkward to use with Retina/HiDPI displays and VMSVGA 3D enabled.

The local test system showed these specific functional issues:

• Setting VMSVGA VRAM to 256 MiB did not work with the current ARMv8 PCI MMIO layout. The 256 MiB VRAM BAR left insufficient low 32-bit PCI MMIO space for the VMSVGA3 MMIO BAR, so the Windows guest display stack did not initialize normally.
• A 5760x3240 Retina guest mode needs the additional VRAM. With the smaller practical VRAM configuration, resizing beyond a certain maximum screen size could produce gray or colorful display artifacts, consistent with insufficient framebuffer/surface memory for the requested mode.
• macOS fullscreen/autoresize requested logical host sizes instead of the Retina backing-pixel guest mode, so resizing did not scale correctly and the guest did not reliably enter true Retina resolutions.
• The Windows WDDM Guest Additions mode dropdown exposed the native Retina mode and then mostly legacy 4:3/5:4 fallback modes, with no useful 16:9 intermediate Retina modes.

Functional HiDPI Patch Series

I have split the functional HiDPI work into small pull requests so each subsystem can be reviewed independently:

1. Expand ARMv8 MMIO32 for 256 MiB VMSVGA VRAM #743 Expand ARMv8 PCI 32-bit MMIO to support 256 MiB VMSVGA VRAM.
2. GUI: request Retina backing size in macOS fullscreen #746 Request Retina backing-size guest modes in macOS fullscreen.
3. Additions/WDDM: add Retina-scaled video modes #749 Add WDDM aspect-preserving Retina intermediate video modes.

These are the patches that directly address the user-visible Retina/HiDPI behavior tracked by this issue.

Supporting Fixes Found During the Investigation

The following fixes were found while making the macOS/Arm Windows guest test path buildable and stable enough to validate the HiDPI work. They are intentionally tracked separately from the functional HiDPI series:

Non-DXMT VirtualBox support fixes

• Darwin: fix ARM64 build and runtime compatibility issues #747 - Darwin/Qt ARM64 build and runtime support.
• configure: quote generated shell environment values #752 - Generated POSIX shell environment quoting.
• doc: add macOS Arm runnable build notes #753 - macOS/Arm runnable-build notes.

DXMT and VMSVGA/DXMT support fixes

• DXMT: add LLVM build compatibility fixes #748 - LLVM/AIR compatibility tracking: LLVM 16+ / 17+ / 22 source compatibility issues in AIR conversion code 3Shain/dxmt#185. Upstream Meson link fix: build: derive LLVM link libraries from llvm-config 3Shain/dxmt#193. The VirtualBox kBuild glue remains downstream-specific.
• DXMT: forward D3D10 start instance location #761 - D3D10 start-instance forwarding. Upstream issue/PR: D3D10 DrawIndexedInstanced forwards StartIndexLocation as StartInstanceLocation 3Shain/dxmt#188 / fix(d3d10): forward indexed instanced start instance 3Shain/dxmt#189.
• DXMT: keep indexed draw base vertex signed #758 - Already fixed in DXMT v0.80; this is a VirtualBox vendored-tree backport.
• DXMT: backport dynamic CPU encoding heap #766 - Already fixed in DXMT v0.80; this is a VirtualBox vendored-tree backport.
• DXMT: avoid initial data for depth/stencil textures #764 - VirtualBox VMSVGA caller-side fix; no DXMT upstream issue opened.
• DXMT: validate render draw state before encoding #759 - VirtualBox integration hardening; upstream DXMT PR withdrawn pending a standalone DXMT reproduction.
• DXMT: initialize async compile ready flags #757 - Upstream DXMT PR withdrawn after re-audit; DXMT's C++20 build mode already initializes default-constructed std::atomic_bool.

Performance Improvements Under Separate Validation

The macOS/Arm VMSVGA DX11 output path also showed high CPU cost from repeated readback and frontend repaint work. Those changes are performance improvements, not functional prerequisites for this HiDPI issue, and should be evaluated separately on top of the functional/stability baseline. They are currently draft pull requests until the throughput and latency impact is verified with reproducible 3D/UI benchmarks:

• VMSVGA: coalesce DX11 readbacks and skip unchanged updates #744 VMSVGA DX11 readback coalescing and unchanged-output skipping.
• GUI: reduce macOS repaint and Dock preview overhead #745 macOS Qt repaint and Dock preview overhead reduction.
• GUI: batch framebuffer update notifications #767 GUI framebuffer update-notification batching.

The reproducible benchmark source for this performance validation is https://github.com/moreaki/dxmtbench. I will update the performance pull requests with isolated results from that suite before treating them as final review candidates.

Validation

Local validation was done on a Mac Studio with an Apple Studio Display, running macOS ARM64 with a Windows 11 ARM guest:

• vram=256
• 3D acceleration enabled
• VMSVGA VRAM BAR at 0xe0000000..0xefffffff
• VMSVGA3 MMIO BAR at 0xf0400000..0xf07fffff
• Guest Additions run level 3
• guest mode 5760x3240x32

For the WDDM mode-list change, I also validated the same intermediate modes through the existing driver's custom-mode registry path. Windows Display Settings exposed:

• 5120x2880
• 3840x2160
• 3200x1800
• 2880x1620
• 2560x1440
• 1920x1080

Additional screenshot proof is attached in the issue comments for fullscreen Retina mode and the HiDPI mode dropdown.

Notes

The Windows Guest Additions WDDM driver patch still needs final compile/install validation in a Windows WDK environment. The local macOS ARM64 build tree cannot build VBoxWddm because PATH_SDK_WINSDK10 is not configured.

[moreaki]

Fullscreen Retina proof for the macOS fullscreen/autoresize part of the series (#746):

This screenshot was captured from the running Windows 11 ARM guest with VBoxManage controlvm Win11 screenshotpng after entering macOS fullscreen Retina mode.

Validation details from the same run:

vram=256
accelerate3d="on"
VideoMode="5760,3240,32"@0,0 1
GuestAdditionsRunLevel=3

The PNG itself is exactly 5760 x 3240 pixels, matching the guest mode shown in Windows Display Settings.

[moreaki]

Current fullscreen HiDPI dropdown proof for the WDDM mode-list change (#749):

This screenshot was captured from the currently running Windows 11 ARM guest with VBoxManage controlvm Win11 screenshotpng while the VM was in fullscreen and Windows Display Settings had the resolution dropdown open.

Validation details from the same current run:

vram=256
accelerate3d="on"
VideoMode="2880,1620,32"@0,0 1
GuestAdditionsRunLevel=3

The dropdown shows the useful HiDPI/aspect-correct modes, including 5760x3240, 5120x2880, 3840x2160, 3200x1800, 2880x1620, 2560x1440, and 1920x1080.