Conversation
The shipped default is 0, which leaves QEMU's own 32 GiB hole in place. That
cannot host a GPU: an H200 SXM's VRAM aperture is 256 GiB and a B300's is 512
GiB, measured on hardware. Anyone starting from the file in this repository
gets a host that cannot pass a GPU through, while every GPU host in
production has been setting the value by hand.
8T rather than the 1P production runs. The hole is global -- one setting for
every guest on the host -- and its size decides where the top of the guest
address space lands:
unset end 0x980000000 36 bits
8T end 0x80180000000 44 bits
1P end 0x4000180000000 51 bits
51 bits is past the 48-bit ceiling of 4-level paging, so a 1 PiB hole means
5-level EPT for every guest, and on TDX forces GPAW=52. GPU-less guests share
this setting and would pay that for nothing. 8T keeps the top at 44 bits,
under both that ceiling and the 46 physical address bits the fleet's hosts
report, while still covering eight B300s twice over -- 8 x 512 GiB is 4 TiB.
No performance delta was measured; the argument is structural, and 1P is
demonstrably workable since the fleet runs it. But it is 250 times more than
the largest supported topology needs, and crossing a paging boundary for that
is a poor trade to make on behalf of every guest. Deployments needing more
can still set it explicitly.
This moves the measurement baseline. pci_hole64_size feeds qemu-acpi's
MachineConfig, so the generated tables change and RTMR0 with them. Confirmed
with the generator: hole=0, 8T and 1P each produce a distinct tables.bin, and
8T differs from 0 by four bytes -- the _CRS length field. The value is
modelled correctly at all three sizes, so the baseline shifts rather than
breaks.
There was a problem hiding this comment.
🟡 Changes recommended
Other repo config templates/examples still set qemu_pci_hole64_size = 0, so users may still get the broken default and see conflicting guidance versus the new vmm.toml default.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the VMM’s default configuration (vmm.toml) to set a practical 64-bit PCI hole size for modern GPU passthrough, and expands the accompanying rationale to explain why 8 TiB is chosen over the fleet’s 1 PiB setting.
Changes:
- Change the default
qemu_pci_hole64_sizefrom0(QEMU default) to"8T". - Replace/extend the inline comment block to explain the GPU BAR sizing needs and the paging/address-space tradeoffs.
File summaries
| File | Description |
|---|---|
| dstack/vmm/vmm.toml | Changes the default PCI hole size to "8T" and updates the inline documentation/rationale for the new default. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # not free for the GPU-less guests that share this setting. 1P, which the H200 | ||
| # fleet runs today, puts the top at 51 bits, past the 48-bit ceiling of 4-level | ||
| # paging -- so every guest, GPU or not, needs 5-level EPT and, on TDX, GPAW=52. | ||
| # Raise it explicitly if a deployment ever needs more than 8 TiB of BARs. | ||
| qemu_pci_hole64_size = "8T" |
|
Closing without merging — the default should stay Raising it changes the measurements of every guest that was on the default, so an already deployed app would come back from a restart with a different identity and be refused its keys by KMS. That cost lands on existing deployments, which is worse than a default that cannot run a GPU — the latter is a first-run papercut, the former breaks something that was working. The useful part of this PR was the reasoning, and that has moved into #1163 as a comment on the setting:
Measurements were verified either way: |
Why the default is broken
qemu_pci_hole64_size = 0leaves QEMU's own 32 GiB hole in place. Measured on real hardware:So the shipped default cannot pass through a single GPU, while every GPU host in production sets the value by hand. Anyone starting from this repository hits a wall that the fleet solved long ago and never wrote down.
Why 8T and not the 1P production runs
The hole is global — one setting for every guest on the host — and its size decides where the top of the guest address space lands. Measured with QEMU 8.2.2,
-cpu host, 4 GiB RAM:0x9800000008T0x801800000001P0x400018000000051 bits is past the 48-bit ceiling of 4-level paging. A 1 PiB hole therefore means 5-level EPT for every guest, and on TDX forces GPAW=52 (GPAW=48 cannot address that high). GPU-less guests share this setting and would pay that for nothing — on the fleet host, 7 of the 15 running VMs have no GPU attached.
8Tkeeps the top at 44 bits, under both that ceiling and the 46 physical address bits the hosts report, while still covering the largest supported topology twice over: 8 × B300 × 512 GiB = 4 TiB.Caveat, stated plainly: no performance delta was measured, and 1P is demonstrably workable since the fleet runs it. The argument is structural, not benchmarked. But 1P is 250× more than the largest topology needs, and crossing a paging boundary for that is a poor trade to make on behalf of every guest. Deployments needing more than 8 TiB of BARs can still set it explicitly.
Measurement impact
This moves the baseline.
pci_hole64_sizefeedsqemu-acpi'sMachineConfig, so the generated tables change and RTMR0 with them.Verified with the generator itself (
cargo run -p qemu-acpi --example dump), same topology, only the hole varying:Three distinct digests, so the value is modelled correctly at all three sizes — the baseline shifts rather than breaks.
cmpputs the 0 → 8T difference at four bytes: the_CRSlength field.Testing
cargo test -p dstack-vmm— 128 passed.vmm.tomlparses and yields"8T"; it is compiled in viainclude_str!, so a bad value fails the build.