fix(os): let the SWIOTLB bounce buffer grow at runtime

[kvinwang]

Problem

Every DMA in a CVM is bounced through SWIOTLB, because the host cannot reach the guest's private memory. Both guest kernels ship with CONFIG_SWIOTLB_DYNAMIC off, so the pool allocated at boot is the only pool the guest ever gets — arch/x86/mm/mem_encrypt.c sizes it at 6% of RAM clamped to 1 GiB:

size = total_mem * 6 / 100;
size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);

Bursty virtio traffic runs it dry:

virtio-pci 0000:00:02.0: swiotlb buffer is full (sz: 262144 bytes),
total 524288 (slots), used 523564 (slots)

total 524288 slots × 2 KiB is exactly that 1 GiB cap. sz: 262144 is IO_TLB_SIZE * IO_TLB_SEGSIZE, the largest mapping swiotlb can serve at all, so the device was asking for the hardest possible allocation against a 99.9% full pool. The mapping fails with -ENOMEM and the caller reports dropped packets or block I/O errors — nothing on the path to the operator names SWIOTLB, which is what makes this expensive to diagnose.

Neither backend had the symbol:

Backend	Kernel	State before
Yocto	6.18.39	# CONFIG_SWIOTLB_DYNAMIC is not set in the built kernel-config
mkosi	6.18.40	absent from the fragment, absent from x86_64_defconfig, default n in kernel/dma/Kconfig → olddefconfig settles on n

Fix

Set CONFIG_SWIOTLB_DYNAMIC=y in both guest kernel fragments, and assert it in parity.json so the mkosi fragment cannot drift from the Yocto one. check-kernel-config.sh, which both backends already run, turns each fragment line into a build-time assertion, so a defconfig change that flips this fails the build instead of shipping.

Why not just raise swiotlb=

Enlarging the fixed pool from the kernel command line was the alternative. It is worse on three counts:

1. The command line is measured. It is baked into the UKI and feeds the RTMRs, so a change has to stay in step across dstack-uki.bb and os/image/kernel-cmdline.sh — for a value that can only guess at the peak.
2. The boot pool cannot exceed 4 GiB anyway. swiotlb_memblock_alloc() picks memblock_alloc_low() unless SWIOTLB_ANY is set, and on x86 that flag is only set on the Xen path (arch/x86/kernel/pci-dma.c), never for KVM/TDX. So the pool must fit under ARCH_LOW_ADDRESS_LIMIT, competing with the kernel image, e820 reservations and the MMIO hole.
3. An oversized request is silently downgraded. swiotlb_init_remap() halves nslabs and retries until it fits, leaving only a pr_info. Asking for 4 GiB and quietly getting 1 GiB looks exactly like not having changed anything.

Runtime pools have none of these constraints — the per-device transient pool uses min_not_zero(*dev->dma_mask, dev->bus_dma_limit), which for virtio is 64-bit.

Costs, stated plainly

• Per-op overhead on unmap/sync. swiotlb_find_pool() stops being an inline range check against the single pool and becomes smp_rmb() + an RCU walk of mem->pools and dev->dma_io_tlb_pools. In a CVM force_bounce is always on, so this is taken for every bounced DMA. With one pool the walk is one iteration.
• CONFIG_DMA_COHERENT_POOL is load-bearing. The synchronous path allocates with GFP_NOWAIT, and force_dma_unencrypted() is true here, so swiotlb_alloc_tlb() takes the atomic-pool branch and returns NULL outright without that symbol. It is already =y — CONFIG_AMD_MEM_ENCRYPT selects it (arch/x86/Kconfig) and the built Yocto config confirms it.
• No tunables added. The area count is already automatic (swiotlb_adjust_nareas(num_possible_cpus())), and dynamic growth runs as a single work_struct on system_wq, not a pool of threads.

Verification

Applied the symbol to the real production Yocto config and ran olddefconfig against the 6.18.39 source tree to confirm the dependency is satisfiable and nothing else moves:

$ cp .../deploy/images/dstack/kernel-config /tmp/swiotlb-kconfig/.config
$ sed -i 's/^# CONFIG_SWIOTLB_DYNAMIC is not set$/CONFIG_SWIOTLB_DYNAMIC=y/' /tmp/swiotlb-kconfig/.config
$ make -C .../kernel-source O=/tmp/swiotlb-kconfig olddefconfig
$ diff <(sort kernel-config) <(sort /tmp/swiotlb-kconfig/.config) | grep '^[<>]' \
    | grep -Ev 'CC_|GCC_|LD_VERSION|AS_VERSION|RUSTC|PAHOLE|PLUGIN|KSTACK_ERASE|RANDSTRUCT|DEBUG_INFO_COMPRESSED'
< # CONFIG_SWIOTLB_DYNAMIC is not set
> CONFIG_SWIOTLB_DYNAMIC=y

The symbol survives olddefconfig and is the only functional difference; the filtered-out entries are all toolchain probes that differ because the run used the host gcc 13.3 rather than the Yocto cross gcc 15.3. CONFIG_DMA_COHERENT_POOL=y is present in the result.

Not yet verified: a boot on real hardware, and behaviour under an actual exhaustion burst. Both backends need a full image build for that, and this changes the guest kernel, so it wants a boot test on QEMU TDX, GCP and AWS before merge.

Note for reviewers

This changes the guest kernel and therefore the OS image hash and RTMRs. Deployments need the new image registered in the KMS whitelist; it is not a drop-in for running CVMs.

The command line is deliberately untouched, so the three places that have to agree on it stay untouched too.

[Copilot]

🔵 Needs a closer look

It changes guest kernel behavior/config (affecting image hashes/RTMRs) and warrants full platform boot/exhaustion testing before merge.

Pull request overview

Enables runtime growth of the SWIOTLB bounce buffer in dstack guest kernels to avoid DMA mapping failures under bursty virtio I/O, and adds a parity assertion so mkosi can’t drift from the Yocto kernel config.

Changes:

• Enable CONFIG_SWIOTLB_DYNAMIC=y in the Yocto kernel config fragment.
• Enable CONFIG_SWIOTLB_DYNAMIC=y in the mkosi kernel config fragment.
• Require CONFIG_SWIOTLB_DYNAMIC=y via os/mkosi/parity.json to enforce backend parity.

File summaries

File	Description
os/yocto/layers/meta-dstack/recipes-kernel/linux/files/dstack.cfg	Enables CONFIG_SWIOTLB_DYNAMIC for the Yocto guest kernel build.
os/mkosi/components/kernel/kernel.config	Enables CONFIG_SWIOTLB_DYNAMIC for the mkosi guest kernel build.
os/mkosi/parity.json	Adds CONFIG_SWIOTLB_DYNAMIC=y to the required kernel config assertions to prevent drift.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.