Skip to content

fix(vmm): start a VM that is still stopping once QEMU has exited - #1414

Merged
kvinwang merged 1 commit into
nextfrom
fix/vmm-stop-waits-for-exit
Sep 26, 2026
Merged

kvinwang merged 1 commit into
nextfrom
fix/vmm-stop-waits-for-exit

Conversation

@kvinwang

@kvinwang kvinwang commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

StopVm returns as soon as the supervisor has been told to kill QEMU; QEMU exits some time later. A StartVm arriving in that window saw the VM as running, skipped the launch and still returned success. The guest then stayed down until the exited-VM check restarted it about 20s later.

Fix

StopVm stays asynchronous. start_vm now recognises a process that has been told to stop (started == false) but is still running, waits under the launch lock for it to exit (bounded at 60s), and then launches it. If QEMU has not exited by then, the start fails with VM is still stopping, try again once it has exited instead of silently doing nothing. The TPM launcher's graceful-stop poll uses the same wait_for_exit helper.

Verification

  • cargo test -p dstack-vmm -p supervisor passes. The new test uses a fake supervisor whose process keeps running for three polls after the stop; it fails without the fix.
  • cargo clippy -p dstack-vmm -p supervisor --all-targets -- -D warnings is clean.
  • Physical TDX results will follow in a comment.

StopVm only has the supervisor signal QEMU and returns; QEMU exits some
time later. A StartVm arriving in that window saw the VM as running,
skipped the launch and still returned success, so the guest stayed down
until the exited-VM check restarted it about 20s later.

StopVm stays asynchronous. start_vm now recognises a process that has
been told to stop (started == false) but is still running, waits under
the launch lock for it to exit (bounded at 60s) and then launches it. If
QEMU has not exited by then, the start fails with an error instead of
silently doing nothing. The TPM launcher's graceful-stop poll uses the
same wait_for_exit helper.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the fix/vmm-stop-waits-for-exit branch from 66ca977 to 066ed86 Compare September 26, 2026 00:47
@kvinwang kvinwang changed the title fix(vmm): return from a VM stop only once QEMU has exited fix(vmm): start a VM that is still stopping once QEMU has exited Sep 26, 2026
@kvinwang

Copy link
Copy Markdown
Collaborator Author

Physical TDX (tdxlab) check of this revision (066ed869e9), with the same setup as the earlier runs: the guest-readonly fixture from the acceptance suite (physical-tdx provider, guest image dstack-mkosi-58687eb5a, one lease-owned VMM). Each round waits for boot_progress=done, then runs vmm-cli stop --force <id> and vmm-cli start <id> right after it. 4 rounds.

next (before) this PR
stop --force latency ~0.3s 0.32–0.51s (still asynchronous)
start latency — 0.45–0.61s, including the wait for QEMU to exit
status right after start exited in 4/4 rounds running/booting in 4/4 rounds
automatic restarts in the VMM log 4 1, unrelated (below)

In round 2 the new QEMU booted, then the guest's GetSealingKey failed because the host's gramine sealing key provider returned a DCAP error for the quote (Failed to read response length: early eof). The guest reported boot.error and powered off, and the exited-VM check restarted it. That key provider has logged the same error intermittently throughout the day's other runs.

@kvinwang

Copy link
Copy Markdown
Collaborator Author

The full physical-TDX sweep of next + this PR + #1415 + the #841 suite (tree ef0f7b94c) passes tc-gos-setup-008 (stop, then start) and the KMS upgrade-matrix cases, which stop and restart KMS, gateway and client CVMs repeatedly (the 4-worker timeouts in four of those cases pass when rerun one at a time; details in #1415).

@kvinwang
kvinwang merged commit 98defeb into next Sep 26, 2026
11 checks passed
@kvinwang
kvinwang deleted the fix/vmm-stop-waits-for-exit branch September 26, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant