Vendored DXMT 0.60 CPU encoding heap can overflow under large command chunks

[moreaki]

VirtualBox currently vendors DXMT 0.60 under src/libs/dxmt-0.60.

In that DXMT version, ArgumentEncodingContext::allocate_cpu_heap() uses one fixed CPU encoding heap and only logs when the offset crosses the heap size:

if (cpu_buffer_offset_ >= kEncodingContextCPUHeapSize) {
  ERR(cpu_buffer_offset_, " - cpu argument heap overflow, expect error.");
}
return ptr_add(cpu_buffer_, aligned);

That means a large encoded command chunk can continue with an out-of-bounds CPU-side command allocation after the warning.

I hit this while stress-testing a Windows 11 Arm guest with VMSVGA 3D acceleration on a macOS/Arm host. The VBox log showed repeated cpu argument heap overflow, expect error. messages followed by many Metal command-buffer failures such as MTLCommandBufferErrorDomain Code=8 / kIOGPUCommandBufferCallbackErrorOutOfMemory, after which the guest display stopped updating correctly.

This has already been addressed upstream in DXMT:

• Upstream fix: 3Shain/dxmt@9987143
• Commit title: feat(dxmt): dynamically grow & shrink cpu encoding buffer
• The fix is included in upstream DXMT tags starting with v0.71; current upstream is newer than the vendored 0.60 snapshot.

A full DXMT v0.60 -> v0.80 refresh is much larger than this single crash fix, so the smallest practical VirtualBox-side fix appears to be backporting the upstream DXMT commit with attribution.