DXMT: add LLVM build compatibility fixes

[moreaki]

Summary

Add minimal DXMT/LLVM compatibility fixes. These are independent from the earlier GUI repaint and HiDPI work.

The VBoxDxMt link now derives LLVM libraries from the configured llvm-config when available. That avoids hard-coding a Homebrew libLLVM-N.dylib name and supports shared LLVM packages, while preserving the original component-library list as a fallback for Oracle/devtools trees without llvm-config.

The AIR argmemonly attribute handling is version-gated: LLVM 15 and older use the original enum attribute path, while LLVM 16 and newer use the MemoryEffects API required by LLVM 17+.

The AIR/DXBC conversion code now passes explicit element types into GEP construction instead of relying on typed-pointer introspection. For LLVM 15 typed-pointer builds, the vector-array helper preserves the aggregate element type where needed so IRBuilder::CreateGEP does not assert on mismatched source element types. The shuffle-mask sentinel is also kept local so the code is not tied to an LLVM-version-specific mask constant name.

Why

The pointer/API issues are real DXMT/LLVM compatibility bugs:

• getNonOpaquePointerElementType() is invalid once LLVM opaque pointers are in use.
• The GEP crash under LLVM 15 is also real: the IR builder was being given the wrong element type for typed-pointer aggregate accesses.
• ArgMemOnly versus MemoryEffects::argMemOnly() is LLVM API drift.
• The shuffle-mask sentinel naming is LLVM-version drift.

Compatibility note

This patch intentionally does not hard-code libLLVM-15.dylib. The source should build against LLVM 15 and newer LLVM APIs where possible, while keeping the link flexible through llvm-config and preserving the static component-library fallback.

On the macOS/Arm test host, the installed VirtualBox 7.2.12 VBoxDxMt.dylib does not have a dynamic libLLVM-17.dylib dependency according to otool -L. It contains LLVM symbols directly, and strings reports llvm-mc (based on LLVM 15.0.7). That suggests the shipped binary may still be produced through an LLVM 15-era static component-library toolchain.

For the macOS/Arm Metal/AIR path, LLVM 15.0.7 is the currently verified producer. Local LLVM 17/22 builds can get further through the source build, but newer upstream LLVM bitcode may not be accepted by Apple's Metal compiler without a downgrade/normalization layer or a different shader handoff boundary; this showed up locally as Failed to upgrade function bitcode.

Validation

• Applied independently to origin/main.
• git diff --check passes.
• Configured a clean PR worktree on macOS/Arm with kBuild initialized.
• Built VBoxDxMt successfully with Homebrew LLVM 15.0.7:
  • VBOX_LLVM_PATH=/opt/homebrew/opt/llvm@15
  • VBOX_LLVM_CONFIG=/opt/homebrew/opt/llvm@15/bin/llvm-config
• otool -L out/darwin.arm64/release/dist/VirtualBox.app/Contents/MacOS/VBoxDxMt.dylib shows /opt/homebrew/opt/llvm@15/lib/libLLVM.dylib with current version 15.0.7.

Refs #742.

[oracle-contributor-agreement]

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

• PR author: moreaki
• rnibali@gmail.com (@moreaki)

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

[moreaki]

I checked DXMT v0.80 before deciding whether this PR should instead be replaced by a vendored DXMT update.

Result: v0.80 does not make this compatibility patch obsolete. It includes the separate CPU encoding heap fix from upstream DXMT, but it still has LLVM API drift in the AIR converter code:

• src/airconv/nt/air_builder.cpp syntax-checks with LLVM 15 headers, but fails with LLVM 16/17 headers on llvm::Attribute::AttrKind::ArgMemOnly.
• With LLVM 22 headers it also fails on removed typed-pointer APIs such as getNonOpaquePointerElementType() and renamed/removed attributes such as NoCapture.
• The legacy converter path syntax-checks with LLVM 15/16 headers, but LLVM 17 fails on the removed llvm/ADT/Optional.h include.
• DXMT v0.80 still carries fixed LLVM component library lists in its Meson build files, so the VirtualBox kBuild llvm-config handling remains a separate build-system compatibility fix.

So a future DXMT v0.80 vendor refresh may still be valuable, but it would need its own follow-up LLVM compatibility work for the newer nt/ AIR converter paths. It is not a drop-in replacement for this PR against the current DXMT 0.60 vendor snapshot.

[moreaki]

DXMT v0.80 audit update:

Updating the vendored DXMT tree to v0.80 would not currently remove the need for the source-compatibility parts of this PR. I checked DXMT v0.80 (589adb7) and current DXMT main (73d42c5); both still contain LLVM API uses such as Attribute::AttrKind::ArgMemOnly, getNonOpaquePointerElementType(), llvm::Optional, and related LLVM-version drift in the AIR conversion code.

I opened the upstream DXMT source-compatibility issue here: 3Shain/dxmt#185

The VirtualBox Makefile.kmk llvm-config portion remains a downstream integration fix for the vendored/kBuild tree. DXMT upstream also has an analogous hard-coded Meson LLVM link-list issue, so I opened that separately upstream:

• Use llvm-config for AIR converter LLVM link libraries when available 3Shain/dxmt#192
• build: derive LLVM link libraries from llvm-config 3Shain/dxmt#193

If VirtualBox later imports a DXMT version that resolves those upstream items, this PR can be reduced or replaced accordingly.