fix(vm): keep a stopped VM restorable after a transient restore failure#89
Merged
Conversation
A hibernated (stopped) VM whose restore failed transiently was marked error, and ResolveForRestore refuses error-state VMs — the wake was permanently bricked until the lease reaped it. FailRestore now spares a stopped origin at the two sequence-level failure exits: nothing ran, record and snapshot are intact, and a retry converges. The run-dir- mutating steps (staged merge, direct populate) still quarantine unconditionally — a partial merge or clean-then-clone leaves mixed- vintage files that vm start must never boot. origin is the pre-kill state, since metering may flip the DB record to stopped after the kill.
Contributor
Author
|
Hardware fault-injection verification (CH, OCI ubuntu:24.04, --nics 0): run → tmpfs marker → |
This was referenced Jul 6, 2026
Closed
CMGS
added a commit
that referenced
this pull request
Jul 6, 2026
KillForRestore marked the VM error on any non-ErrNotRunning failure, including WithRunningVM's fail-closed /proc-scan error. For a hibernated (stopped) VM the VMM is provably gone — hibernate marks error if its terminate fails — so a transient scan hiccup during wake bricked a perfectly restorable VM: ResolveForRestore refuses error state, and nothing on disk had been touched. Route the failure through FailRestore (#89's origin-aware helper): stopped origins stay stopped and the wake remains retryable; a failed kill of a genuinely live VMM on a running origin still quarantines. Closes #91
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A hibernated (stopped) VM whose restore failed transiently — host hiccup at launch, snapshot/load, resume — was marked
error, andResolveForRestorerefuses error-state VMs: the wake was permanently bricked until the lease reaped the sandbox.What
Backend.FailRestore(ctx, vmID, origin): post-kill restore failures now spare a stopped origin (record, snapshot, and disks untouched by the failed attempt — retry is safe and converges), and mark error otherwise. Called at the two sequence-level failure exits (RestoreSequence/DirectRestoreSequence), replacing the per-backend MarkError defers, which also closes two previously-unmarked running-origin gaps (Wrap and BeforeMerge failures).MergeDirInto(stream path) orPopulateclean-then-clone (direct path) leaves mixed-vintage files in the run dir thatvm startmust never silently boot — those failures mark error regardless of origin.originis passed as a parameter deliberately: it is the pre-kill state, and the DB record may already read stopped after the kill's metering flip — re-reading the DB would spare exactly the VMs that must be quarantined.KillForRestore's MarkError is kept: a failed kill of a live VMM is a genuinely unknown state.Tests
TestFailRestoreSparesStoppedOrigin(helper contract),TestDirectRestoreFailureKeepsOriginContract(sequence-level, 3 cases: post-populate failure spares stopped / quarantines running; populate failure quarantines even stopped),TestRestorePartialMergeQuarantinesEvenStoppedOrigin(in-memory tar staged, genuinely partial merge, quarantined).make lint testgreen on linux+darwin.