Skip to content

Avoid metadata writes when deleting instances without vGPUs - #486

Merged
yummybomb merged 1 commit into
mainfrom
hypeship/fix-cpu-delete-enospc
Sep 18, 2026
Merged

yummybomb merged 1 commit into
mainfrom
hypeship/fix-cpu-delete-enospc

Conversation

@yummybomb

@yummybomb yummybomb commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Why

#485 made DELETE return an error when vGPU release fails, instead of logging a warning and continuing. That was the right call for instances that actually hold a vGPU, but it exposed a pre-existing problem: delete called releaseStoredVGPUPersisted for every instance, GPU or not, and that helper always rewrites the metadata file via a temp file and rename.

For an instance with no vGPU that write does nothing useful. Before #485 its failure was swallowed, so nobody noticed. After #485 an ENOSPC on that write aborts the delete before any instance data is removed. On a full disk that means the only way to free space is blocked by the lack of space.

What changed

  • Delete only calls releaseStoredVGPUPersisted when the instance has a vGPU assignment. This matches how start and the vGPU reconciler already gate the same helper.
  • Real vGPU cleanup failures behave exactly as after Wait for exiting VMM tasks and retain failed vGPU cleanup #485: the error is returned, metadata and the GPU claim are retained, and network/device teardown does not run.
  • Restart-policy handling is unchanged. The manual-stop marker is still written before teardown, and only when the instance has a restart policy.

Testing

  • New TestDeleteWithoutVGPUDoesNotRewriteMetadata asserts the metadata file keeps the same inode through delete for a non-GPU instance. It fails on main and passes here.
  • Targeted delete, vGPU release, lifecycle, and reconciliation tests pass with -race; the new test and the existing vGPU delete tests were repeated 25 times.
  • Ran the real delete path against stopped Firecracker metadata with no GPU on an isolated 8 MiB tmpfs under both inode and block exhaustion. Unpatched code fails and leaves the instance behind; patched code deletes it.

Not covered: no hypervisor was started and the full instances suite was not run. Instances that carry a restart policy still need one metadata write during delete to block auto-restart, so a full disk can still fail those; none of our callers set a restart policy today.


Note

Medium Risk
Touches instance delete teardown ordering for vGPU vs non-vGPU paths; wrong gating could skip needed GPU cleanup or regress disk-full delete behavior.

Overview
Delete no longer calls releaseStoredVGPUPersisted for instances without a vGPU assignment, aligning delete with how start already gates that helper. Previously every delete attempted a metadata rewrite (temp file + rename) even when there was nothing to release; after #485 a failed write (e.g. ENOSPC) could abort deletion before any instance data was removed.

When a vGPU is assigned, behavior is unchanged: release runs inside the same guard, failures still return an error and retain metadata for retry, and network/device teardown still does not run on failure.

Adds TestDeleteWithoutVGPUDoesNotRewriteMetadata, which checks the metadata file keeps the same inode through delete for a non-GPU instance (via a hook during device detach).

Reviewed by Cursor Bugbot for commit 50b4444. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb
yummybomb marked this pull request as ready for review September 18, 2026 15:41
@yummybomb
yummybomb merged commit df4d1a5 into main Sep 18, 2026
9 checks passed
@yummybomb
yummybomb deleted the hypeship/fix-cpu-delete-enospc branch September 18, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants