DXMT: validate render draw state before encoding

[moreaki]

Summary

Validate winemetal render draw state before encoding draw calls into Metal.

This patch keeps track of whether the render encoder currently has a valid pipeline state and skips draw commands when no valid PSO is active. It also validates indexed draw buffer presence, type, offset/count range, and indirect draw argument buffer bounds before calling into Metal.

The intent is to keep invalid guest or failed pipeline state from becoming an Objective-C Metal validation exception and VM process crash.

Validation

• git diff --check
• Cherry-picked on top of DXMT: add LLVM build compatibility fixes #748 and built with kmk VBoxDxMt on macOS/Arm using LLVM 15.0.7.

Closes #756.

Signed-off-by: Roberto Nibali rnibali@gmail.com

[oracle-contributor-agreement]

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

• PR author: moreaki
• rnibali@gmail.com (@moreaki)

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

[moreaki]

DXMT v0.80 audit update:

Updating the vendored DXMT tree to v0.80 would not address this. DXMT v0.80 and current DXMT main still forward render draw commands directly to Metal without guarding null/missing render pipeline state or native indexed draw buffer/range issues.

I opened the equivalent upstream DXMT issue and PR:

• Validate render draw state before issuing Metal encoder draws 3Shain/dxmt#190
• fix(winemetal): validate render draw state before encoding 3Shain/dxmt#191

[moreaki]

Re-audit update:

I withdrew the upstream DXMT issue/PR I had opened for this (3Shain/dxmt#190 / 3Shain/dxmt#191). The patch is defensive hardening for invalid render state reaching Metal, but without a standalone DXMT reproduction it is too broad to submit upstream as-is.

For VirtualBox, this remains potentially useful integration hardening only if tied to the VirtualBox/macOS-arm crash path we observed. It should not be presented as an accepted or currently pending upstream DXMT fix until we have a minimal DXMT-native repro.