diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md index a0efc4a3..5f57ca7f 100644 --- a/KNOWN_ISSUES.md +++ b/KNOWN_ISSUES.md @@ -15,10 +15,6 @@ After `cocoon vm clone`, the cloned VM resumes with the **original VM's IP addre `cocoon vm clone` inherits CPU, memory, and storage from the snapshot — none of these can be grown at clone time on either backend. NIC count inherits by default; Cloud Hypervisor clones may override it with `--nics N` (cocoon hot-swaps the snapshot's NICs for a fresh set after restore). Firecracker clones must keep the snapshot's NIC topology — FC's `network_overrides` only retargets existing interfaces, so `--nics` is rejected on FC. `cocoon vm restore` is more restrictive: CPU, memory, and storage come from the snapshot (the persisted record is realigned to match), and NIC count must already match the target VM (mismatches are rejected, since restore reuses the existing network namespace). Use `cocoon vm run` to create a fresh VM with different sizing. -## Restore requires a running VM - -`cocoon vm restore` only works on running VMs — it relies on the existing network namespace (netns, tap devices, TC redirect) surviving the CH process restart. A stopped VM's network state may not be intact (e.g., after host reboot the netns is gone). For stopped VMs or cross-VM restore, use `cocoon vm clone` which creates fresh network resources. See [Restore Constraints](README.md#restore-constraints) for all requirements. - ## OCI VM multi-NIC kernel IP limitation OCI VMs use the kernel `ip=` boot parameter for network configuration. While multiple `ip=` parameters can be specified, the Linux kernel only reliably configures **one interface** via this mechanism — subsequent `ip=` parameters may be silently ignored or produce inconsistent results depending on kernel version. diff --git a/README.md b/README.md index 4499c9c8..9c1cd9a8 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,8 @@ cocoon │ ├── exec [flags] VM -- CMD Run a command in a running VM via cocoon-agent (vsock) │ ├── logs [-f] [--tail N] VM Print the per-VM hypervisor log file │ ├── rm [flags] VM [VM...] Delete VM(s) (--force to stop first) -│ ├── restore [flags] VM SNAP Restore a running VM to a snapshot +│ ├── restore [flags] VM SNAP Restore a VM (running or stopped) to a snapshot +│ ├── hibernate [flags] VM Atomically snapshot a running VM and stop it │ ├── status [VM...] Watch VM status in real time │ ├── fs │ │ ├── attach [flags] VM Attach a vhost-user-fs share (CH only) @@ -793,18 +794,29 @@ The `--pull` flag uses the image digest recorded at snapshot time to verify that ### Restore -Restore reverts a **running** VM to a previous snapshot's state in-place: +Restore reverts a VM — running or stopped — to a previous snapshot's state in-place: ```bash # Restore a VM to a previous snapshot cocoon vm restore my-vm my-snap ``` -Cocoon stages the snapshot into a scratch directory first, then restarts the hypervisor process only after the full extraction succeeds — a truncated or corrupt snapshot stream errors out with the running VM still intact. Network is fully preserved — same IP, same MAC, same network namespace. No guest-side reconfiguration is needed (unlike clone). +Cocoon stages the snapshot into a scratch directory first, then (re)starts the hypervisor process only after the full extraction succeeds — a truncated or corrupt snapshot stream errors out with the VM in its prior state. Network is fully preserved — same IP, same MAC, same network namespace; restoring a stopped VM first runs the same network self-heal as `vm start`, so a hibernated VM resumes even after a host reboot. No guest-side reconfiguration is needed (unlike clone). + +### Hibernate + +Hibernate atomically snapshots a running VM and stops it, releasing its memory; resume with `vm restore`: + +```bash +cocoon vm hibernate my-vm --name nap +cocoon vm restore my-vm nap +``` + +Pause, capture, persist, and VMM termination share one pause window: the snapshot point and the stop coincide (nothing the guest does can be lost in between), and the VMM dies only after the snapshot is durably saved — if saving fails (disk full, snapshot DB error), the VM simply resumes running and the command fails. Restore of the hibernated VM preserves machine identity (entropy-only reseed): running processes, sessions, and tmpfs contents continue where they stopped. ### Restore Constraints -- **VM must be running.** Restore operates on a live VM by restarting its CH process with snapshot state. For stopped VMs, use `cocoon vm clone` instead. +- **VM must be running or stopped.** Restore cold-spawns a fresh hypervisor process from the snapshot; a stopped target (e.g. hibernated) gets the `vm start` network self-heal first. - **Snapshot must belong to the VM.** Only snapshots created from the same VM (tracked in `snapshot_ids`) are accepted; pass `--force` with `--from-dir` to opt into a foreign lineage. - **CPU, memory, and storage come from the snapshot.** The hypervisor reconstructs the guest from snapshot state, so these are not configurable at restore time; cocoon realigns the persisted record to match. - **NIC count must match the target VM.** Restore reuses the VM's existing network namespace, TAP devices, and IP allocation; a mismatched count is rejected. diff --git a/cmd/core/utils.go b/cmd/core/utils.go index abef4cb9..c621efb6 100644 --- a/cmd/core/utils.go +++ b/cmd/core/utils.go @@ -10,12 +10,14 @@ import ( "github.com/google/go-containerregistry/pkg/name" "github.com/projecteru2/core/log" + "github.com/spf13/cobra" "github.com/cocoonstack/cocoon/cmd/cliutil" "github.com/cocoonstack/cocoon/config" "github.com/cocoonstack/cocoon/hypervisor" imagebackend "github.com/cocoonstack/cocoon/images" "github.com/cocoonstack/cocoon/progress" + "github.com/cocoonstack/cocoon/snapshot" "github.com/cocoonstack/cocoon/types" "github.com/cocoonstack/cocoon/utils" ) @@ -224,3 +226,47 @@ func resolveVMOwner(ctx context.Context, hypers []hypervisor.Hypervisor, ref str ) return owner, resolved, err } + +// EnsureSnapshotNameFree validates name and rejects a duplicate; empty passes. +func EnsureSnapshotNameFree(ctx context.Context, snapBackend snapshot.Snapshot, name string) error { + if err := (&types.SnapshotConfig{Name: name}).Validate(); err != nil { + return err + } + if name == "" { + return nil + } + if _, err := snapBackend.Inspect(ctx, name); err == nil { + return fmt.Errorf("snapshot name %q already exists", name) + } else if !errors.Is(err, snapshot.ErrNotFound) { + return fmt.Errorf("check snapshot name: %w", err) + } + return nil +} + +// CaptureSnapshot checks the --name preflight, runs capture, and persists the stream, returning the stored snapshot id. +func CaptureSnapshot(ctx context.Context, cmd *cobra.Command, snapBackend snapshot.Snapshot, capture func() (*types.SnapshotConfig, io.ReadCloser, error)) (string, error) { + name, _ := cmd.Flags().GetString("name") + description, _ := cmd.Flags().GetString("description") + if err := EnsureSnapshotNameFree(ctx, snapBackend, name); err != nil { + return "", err + } + cfg, stream, err := capture() + if err != nil { + return "", err + } + return PersistSnapshotStream(ctx, snapBackend, cfg, stream, name, description) +} + +// PersistSnapshotStream labels cfg and stores the stream, closing it either way. +func PersistSnapshotStream(ctx context.Context, snapBackend snapshot.Snapshot, cfg *types.SnapshotConfig, stream io.ReadCloser, name, description string) (string, error) { + defer stream.Close() //nolint:errcheck + defer CloseOnCancel(ctx, stream)() + cfg.Name = name + cfg.Description = description + log.WithFunc("cmdcore.PersistSnapshotStream").Info(ctx, "saving snapshot data ...") + snapID, err := snapBackend.Create(ctx, cfg, stream) + if err != nil { + return "", fmt.Errorf("save snapshot: %w", err) + } + return snapID, nil +} diff --git a/cmd/snapshot/handler.go b/cmd/snapshot/handler.go index 06c1e63b..9dc2c9a2 100644 --- a/cmd/snapshot/handler.go +++ b/cmd/snapshot/handler.go @@ -2,7 +2,6 @@ package snapshot import ( "cmp" - "errors" "fmt" "io" "os" @@ -39,40 +38,14 @@ func (h Handler) Save(cmd *cobra.Command, args []string) error { if err != nil { return err } - name, _ := cmd.Flags().GetString("name") - description, _ := cmd.Flags().GetString("description") - - if err = (&types.SnapshotConfig{Name: name}).Validate(); err != nil { - return err - } - if name != "" { - if _, inspectErr := snapBackend.Inspect(ctx, name); inspectErr == nil { - return fmt.Errorf("snapshot name %q already exists", name) - } else if !errors.Is(inspectErr, snapshot.ErrNotFound) { - return fmt.Errorf("check snapshot name: %w", inspectErr) - } - } logger.Infof(ctx, "snapshotting VM %s ...", vmRef) - - cfg, stream, err := hyper.Snapshot(ctx, vmRef) - if err != nil { - return fmt.Errorf("snapshot VM %s: %w", vmRef, err) - } - defer stream.Close() //nolint:errcheck - - defer cmdcore.CloseOnCancel(ctx, stream)() - - cfg.Name = name - cfg.Description = description - - logger.Info(ctx, "saving snapshot data ...") - - snapID, err := snapBackend.Create(ctx, cfg, stream) + snapID, err := cmdcore.CaptureSnapshot(ctx, cmd, snapBackend, func() (*types.SnapshotConfig, io.ReadCloser, error) { + return hyper.Snapshot(ctx, vmRef) + }) if err != nil { - return fmt.Errorf("save snapshot: %w", err) + return err } - logger.Infof(ctx, "snapshot saved: %s", snapID) return nil } diff --git a/cmd/vm/commands.go b/cmd/vm/commands.go index 27a9e457..9fea4fbb 100644 --- a/cmd/vm/commands.go +++ b/cmd/vm/commands.go @@ -21,6 +21,7 @@ type Actions interface { Logs(cmd *cobra.Command, args []string) error RM(cmd *cobra.Command, args []string) error Restore(cmd *cobra.Command, args []string) error + Hibernate(cmd *cobra.Command, args []string) error Debug(cmd *cobra.Command, args []string) error Status(cmd *cobra.Command, args []string) error FsAttach(cmd *cobra.Command, args []string) error @@ -141,7 +142,7 @@ func Command(h Actions) *cobra.Command { restoreCmd := &cobra.Command{ Use: "restore [flags] VM [SNAPSHOT]", - Short: "Restore a running VM to a previous snapshot (or a directory via --from-dir)", + Short: "Restore a running or stopped VM to a previous snapshot (or a directory via --from-dir)", Args: cobra.RangeArgs(1, 2), PreRunE: func(cmd *cobra.Command, _ []string) error { force, _ := cmd.Flags().GetBool("force") @@ -158,6 +159,15 @@ func Command(h Actions) *cobra.Command { restoreCmd.Flags().Bool("force", false, "skip the snapshot-belongs-to-VM check (only meaningful with --from-dir; risk of restoring to an unrelated lineage)") cliutil.AddOutputFlag(restoreCmd) + hibernateCmd := &cobra.Command{ + Use: "hibernate [flags] VM", + Short: "Atomically snapshot a running VM and stop it (resume with vm restore)", + Args: cobra.ExactArgs(1), + RunE: h.Hibernate, + } + hibernateCmd.Flags().String("name", "", "snapshot name") + hibernateCmd.Flags().String("description", "", "snapshot description") + debugCmd := &cobra.Command{ Use: "debug [flags] IMAGE", Short: "Generate hypervisor launch command (dry run)", @@ -194,6 +204,7 @@ func Command(h Actions) *cobra.Command { logsCmd, rmCmd, restoreCmd, + hibernateCmd, debugCmd, statusCmd, buildFsCommand(h), diff --git a/cmd/vm/hibernate.go b/cmd/vm/hibernate.go new file mode 100644 index 00000000..d15530ab --- /dev/null +++ b/cmd/vm/hibernate.go @@ -0,0 +1,58 @@ +package vm + +import ( + "cmp" + "fmt" + "io" + + "github.com/projecteru2/core/log" + "github.com/spf13/cobra" + + cmdcore "github.com/cocoonstack/cocoon/cmd/core" + "github.com/cocoonstack/cocoon/hypervisor" + "github.com/cocoonstack/cocoon/types" +) + +// Hibernate atomically snapshots a running VM and stops it; the snapshot +// point and the stop coincide, so `vm restore` resumes with nothing lost. +func (h Handler) Hibernate(cmd *cobra.Command, args []string) error { + ctx, conf, err := h.Init(cmd) + if err != nil { + return err + } + logger := log.WithFunc("cmd.vm.hibernate") + vmRef := args[0] + + hyper, err := cmdcore.FindHypervisor(ctx, conf, vmRef) + if err != nil { + return fmt.Errorf("find VM %s: %w", vmRef, err) + } + hib, ok := hyper.(hypervisor.Hibernator) + if !ok { + return fmt.Errorf("backend %s does not support hibernate", hyper.Type()) + } + snapBackend, err := cmdcore.InitSnapshot(ctx, conf) + if err != nil { + return err + } + name, _ := cmd.Flags().GetString("name") + description, _ := cmd.Flags().GetString("description") + // Fail on a taken name before the VM is even paused. + if err = cmdcore.EnsureSnapshotNameFree(ctx, snapBackend, name); err != nil { + return err + } + + logger.Infof(ctx, "hibernating VM %s ...", vmRef) + // persist runs inside the pause window; the VMM dies only after it succeeds. + var snapID string + err = hib.Hibernate(ctx, vmRef, func(cfg *types.SnapshotConfig, stream io.ReadCloser) error { + id, pErr := cmdcore.PersistSnapshotStream(ctx, snapBackend, cfg, stream, name, description) + snapID = id + return pErr + }) + if err != nil { + return err + } + logger.Infof(ctx, "VM hibernated; snapshot %s (resume: cocoon vm restore %s %s)", snapID, vmRef, cmp.Or(name, snapID)) + return nil +} diff --git a/cmd/vm/lifecycle.go b/cmd/vm/lifecycle.go index 4ffb1fc1..d5507771 100644 --- a/cmd/vm/lifecycle.go +++ b/cmd/vm/lifecycle.go @@ -228,6 +228,15 @@ func (h Handler) RM(cmd *cobra.Command, args []string) error { return finishRoutedCmd(ctx, cmd, logTag, "rm", "deleted", allDeleted, lastErr) } +// recoverNetworkForStopped runs Start's network self-heal when the restore target is not running (hibernate resume after a host reboot). +func (h Handler) recoverNetworkForStopped(ctx context.Context, conf *config.Config, hyper hypervisor.Hypervisor, ref string) { + vm, err := hyper.Inspect(ctx, ref) + if err != nil || vm.State == types.VMStateRunning { + return + } + h.recoverNetwork(ctx, conf, hyper, []string{vm.ID}) +} + func (h Handler) recoverNetwork(ctx context.Context, conf *config.Config, hyper hypervisor.Hypervisor, refs []string) { logger := log.WithFunc("cmd.vm.recoverNetwork") diff --git a/cmd/vm/run.go b/cmd/vm/run.go index af05bb4e..83630d5c 100644 --- a/cmd/vm/run.go +++ b/cmd/vm/run.go @@ -157,6 +157,7 @@ func (h Handler) Restore(cmd *cobra.Command, args []string) error { if err != nil { return fmt.Errorf("find VM %s: %w", vmRef, err) } + h.recoverNetworkForStopped(ctx, conf, hyper, vmRef) snapBackend, err := cmdcore.InitSnapshot(ctx, conf) if err != nil { return err @@ -224,6 +225,9 @@ func (h Handler) restoreFromDir(ctx context.Context, cmd *cobra.Command, conf *c if err != nil { return fmt.Errorf("inspect VM: %w", err) } + if vm.State != types.VMStateRunning { + h.recoverNetwork(ctx, conf, hyper, []string{vm.ID}) + } if _, owned := vm.SnapshotIDs[cfg.ID]; !owned { force, _ := cmd.Flags().GetBool("force") if !force { diff --git a/hypervisor/backend.go b/hypervisor/backend.go index f8573152..8318e193 100644 --- a/hypervisor/backend.go +++ b/hypervisor/backend.go @@ -154,3 +154,18 @@ type SnapshotSpec struct { AfterCapture func(rec *VMRecord, tmpDir string) error BuildMeta func(rec *VMRecord, tmpDir string) (*SnapshotMeta, error) } + +// HibernateSpec extends SnapshotSpec with the pause-window terminate hook and the backend's runtime files. +type HibernateSpec struct { + SnapshotSpec + Terminate func(rec *VMRecord, hc *http.Client, pid int) error + RuntimeFiles []string +} + +// runWrapped runs fn under a spec's optional Wrap. +func runWrapped(rec *VMRecord, wrap func(*VMRecord, func() error) error, fn func() error) error { + if wrap != nil { + return wrap(rec, fn) + } + return fn() +} diff --git a/hypervisor/cloudhypervisor/restore.go b/hypervisor/cloudhypervisor/restore.go index 79aa5413..1efc9865 100644 --- a/hypervisor/cloudhypervisor/restore.go +++ b/hypervisor/cloudhypervisor/restore.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "io" + "net/http" "path/filepath" "github.com/projecteru2/core/log" @@ -32,12 +33,16 @@ func (ch *CloudHypervisor) preflightRestore(srcDir string, rec *hypervisor.VMRec } func (ch *CloudHypervisor) killForRestore(ctx context.Context, vmID string, rec *hypervisor.VMRecord) error { - sockPath := hypervisor.SocketPath(rec.RunDir) return ch.KillForRestore(ctx, vmID, rec, func(pid int) error { - return ch.forceTerminate(ctx, utils.NewSocketHTTPClient(sockPath), vmID, sockPath, pid) + return ch.terminateVMM(ctx, rec, utils.NewSocketHTTPClient(hypervisor.SocketPath(rec.RunDir)), pid) }, runtimeFiles) } +// terminateVMM force-terminates rec's VMM over hc; shared by restore and hibernate. +func (ch *CloudHypervisor) terminateVMM(ctx context.Context, rec *hypervisor.VMRecord, hc *http.Client, pid int) error { + return ch.forceTerminate(ctx, hc, rec.ID, hypervisor.SocketPath(rec.RunDir), pid) +} + func (ch *CloudHypervisor) restoreAfterExtract(ctx context.Context, vmID string, vmCfg *types.VMConfig, rec *hypervisor.VMRecord, directBoot bool) (_ *types.VM, err error) { logger := log.WithFunc("cloudhypervisor.Restore") diff --git a/hypervisor/cloudhypervisor/snapshot.go b/hypervisor/cloudhypervisor/snapshot.go index 1e2a6bc8..fe006ff9 100644 --- a/hypervisor/cloudhypervisor/snapshot.go +++ b/hypervisor/cloudhypervisor/snapshot.go @@ -15,7 +15,22 @@ import ( // Snapshot pauses, captures CH state+COW, resumes, and streams the result. func (ch *CloudHypervisor) Snapshot(ctx context.Context, ref string) (*types.SnapshotConfig, io.ReadCloser, error) { - return ch.SnapshotSequence(ctx, ref, hypervisor.SnapshotSpec{ + return ch.SnapshotSequence(ctx, ref, ch.snapshotSpec(ctx)) +} + +// Hibernate captures like Snapshot but persists and then terminates the VMM inside the pause window instead of resuming. +func (ch *CloudHypervisor) Hibernate(ctx context.Context, ref string, persist func(cfg *types.SnapshotConfig, stream io.ReadCloser) error) error { + return ch.HibernateSequence(ctx, ref, hypervisor.HibernateSpec{ + SnapshotSpec: ch.snapshotSpec(ctx), + Terminate: func(rec *hypervisor.VMRecord, hc *http.Client, pid int) error { + return ch.terminateVMM(ctx, rec, hc, pid) + }, + RuntimeFiles: runtimeFiles, + }, persist) +} + +func (ch *CloudHypervisor) snapshotSpec(ctx context.Context) hypervisor.SnapshotSpec { + return hypervisor.SnapshotSpec{ Pause: func(_ *hypervisor.VMRecord, hc *http.Client) error { return pauseVM(ctx, hc) }, Resume: func(_ *hypervisor.VMRecord, hc *http.Client) error { return resumeVM(context.WithoutCancel(ctx), hc) }, Capture: func(rec *hypervisor.VMRecord, hc *http.Client, tmpDir string) error { @@ -43,7 +58,7 @@ func (ch *CloudHypervisor) Snapshot(ctx context.Context, ref string) (*types.Sna return nil }, BuildMeta: buildSnapshotMeta, - }) + } } // buildSnapshotMeta mirrors config.json's disk shape (not activeDisks — diverges for cloudimg post-FirstBooted pre-restart where CH still holds cidata). diff --git a/hypervisor/firecracker/restore.go b/hypervisor/firecracker/restore.go index f283c135..10b99b99 100644 --- a/hypervisor/firecracker/restore.go +++ b/hypervisor/firecracker/restore.go @@ -42,11 +42,15 @@ func (fc *Firecracker) restoreAfterExtractCOW(ctx context.Context, vmID string, func (fc *Firecracker) killForRestore(ctx context.Context, vmID string, rec *hypervisor.VMRecord) error { return fc.KillForRestore(ctx, vmID, rec, func(pid int) error { - sockPath := hypervisor.SocketPath(rec.RunDir) - return fc.forceTerminate(ctx, sockPath, pid) + return fc.terminateVMM(ctx, rec, pid) }, runtimeFiles) } +// terminateVMM force-terminates rec's VMM; shared by restore and hibernate. +func (fc *Firecracker) terminateVMM(ctx context.Context, rec *hypervisor.VMRecord, pid int) error { + return fc.forceTerminate(ctx, hypervisor.SocketPath(rec.RunDir), pid) +} + func (fc *Firecracker) restoreAfterExtract(ctx context.Context, vmID string, vmCfg *types.VMConfig, rec *hypervisor.VMRecord, cowPath string) (_ *types.VM, err error) { logger := log.WithFunc("firecracker.Restore") diff --git a/hypervisor/firecracker/snapshot.go b/hypervisor/firecracker/snapshot.go index e49a1e5a..ebdccba0 100644 --- a/hypervisor/firecracker/snapshot.go +++ b/hypervisor/firecracker/snapshot.go @@ -19,7 +19,22 @@ const ( // Snapshot pauses, captures vmstate+mem+COW, resumes, and streams the result. func (fc *Firecracker) Snapshot(ctx context.Context, ref string) (*types.SnapshotConfig, io.ReadCloser, error) { - return fc.SnapshotSequence(ctx, ref, hypervisor.SnapshotSpec{ + return fc.SnapshotSequence(ctx, ref, fc.snapshotSpec(ctx)) +} + +// Hibernate captures like Snapshot but persists and then terminates the VMM inside the pause window instead of resuming. +func (fc *Firecracker) Hibernate(ctx context.Context, ref string, persist func(cfg *types.SnapshotConfig, stream io.ReadCloser) error) error { + return fc.HibernateSequence(ctx, ref, hypervisor.HibernateSpec{ + SnapshotSpec: fc.snapshotSpec(ctx), + Terminate: func(rec *hypervisor.VMRecord, _ *http.Client, pid int) error { + return fc.terminateVMM(ctx, rec, pid) + }, + RuntimeFiles: runtimeFiles, + }, persist) +} + +func (fc *Firecracker) snapshotSpec(ctx context.Context) hypervisor.SnapshotSpec { + return hypervisor.SnapshotSpec{ Pause: func(_ *hypervisor.VMRecord, hc *http.Client) error { return pauseVM(ctx, hc) }, Resume: func(_ *hypervisor.VMRecord, hc *http.Client) error { return resumeVM(context.WithoutCancel(ctx), hc) }, // createSnapshotFC builds its own client with VMMemTransferTimeout @@ -39,7 +54,7 @@ func (fc *Firecracker) Snapshot(ctx context.Context, ref string) (*types.Snapsho return withSourceWritableDisksLocked(rec.StorageConfigs, fn) }, BuildMeta: buildSnapshotMeta, - }) + } } // buildSnapshotMeta rewrites kernel path to vmlinuz so clones get the portable artifact instead of the FC-specific vmlinux cache. diff --git a/hypervisor/hibernate_test.go b/hypervisor/hibernate_test.go new file mode 100644 index 00000000..393593ee --- /dev/null +++ b/hypervisor/hibernate_test.go @@ -0,0 +1,187 @@ +package hypervisor + +import ( + "errors" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + "time" + + "github.com/cocoonstack/cocoon/types" + "github.com/cocoonstack/cocoon/utils" +) + +// hibernateTestConfig extends the metering stub with a real VMRunDir so prepareSnapshot can MkdirTemp. +type hibernateTestConfig struct { + stubBackendConfig + vmRunRoot string +} + +func (c hibernateTestConfig) VMRunDir(string) string { return c.vmRunRoot } + +type hibernateCalls struct { + resumed bool + terminated bool +} + +func TestHibernateSequencePersistFailureResumesAndRollsBack(t *testing.T) { + b, id := newHibernateTestVM(t) + calls := &hibernateCalls{} + + err := b.HibernateSequence(t.Context(), id, hibernateStubSpec(calls), func(_ *types.SnapshotConfig, stream io.ReadCloser) error { + _, _ = io.Copy(io.Discard, stream) + _ = stream.Close() + return errors.New("store full") + }) + if err == nil || !strings.Contains(err.Error(), "persist snapshot") { + t.Fatalf("HibernateSequence: %v, want persist snapshot error", err) + } + if !calls.resumed { + t.Error("VM was not resumed after persist failure") + } + if calls.terminated { + t.Error("VMM was terminated despite persist failure") + } + + rec, err := b.LoadRecord(t.Context(), id) + if err != nil { + t.Fatalf("load record: %v", err) + } + if rec.State != types.VMStateRunning { + t.Errorf("state %s, want running", rec.State) + } + if len(rec.SnapshotIDs) != 0 { + t.Errorf("snapshot ids not rolled back: %v", rec.SnapshotIDs) + } +} + +func TestHibernateSequenceSuccess(t *testing.T) { + b, id := newHibernateTestVM(t) + calls := &hibernateCalls{} + + var gotCfg *types.SnapshotConfig + err := b.HibernateSequence(t.Context(), id, hibernateStubSpec(calls), func(cfg *types.SnapshotConfig, stream io.ReadCloser) error { + gotCfg = cfg + _, _ = io.Copy(io.Discard, stream) + return stream.Close() + }) + if err != nil { + t.Fatalf("HibernateSequence: %v", err) + } + if !calls.terminated { + t.Error("VMM was not terminated") + } + if calls.resumed { + t.Error("VM was resumed on the success path") + } + if gotCfg == nil || gotCfg.ID == "" { + t.Fatalf("persist got cfg %+v, want a snapshot id", gotCfg) + } + + rec, err := b.LoadRecord(t.Context(), id) + if err != nil { + t.Fatalf("load record: %v", err) + } + if rec.State != types.VMStateStopped { + t.Errorf("state %s, want stopped", rec.State) + } + if _, ok := rec.SnapshotIDs[gotCfg.ID]; !ok { + t.Errorf("snapshot id %s not recorded: %v", gotCfg.ID, rec.SnapshotIDs) + } +} + +func TestHibernateSequenceTerminateFailureMarksError(t *testing.T) { + b, id := newHibernateTestVM(t) + calls := &hibernateCalls{} + + spec := hibernateStubSpec(calls) + spec.Terminate = func(*VMRecord, *http.Client, int) error { return errors.New("kill refused") } + err := b.HibernateSequence(t.Context(), id, spec, func(_ *types.SnapshotConfig, stream io.ReadCloser) error { + _, _ = io.Copy(io.Discard, stream) + return stream.Close() + }) + if err == nil || !strings.Contains(err.Error(), "terminate") { + t.Fatalf("HibernateSequence: %v, want terminate error", err) + } + if calls.resumed { + t.Error("VM was resumed after the snapshot was already persisted") + } + + rec, err := b.LoadRecord(t.Context(), id) + if err != nil { + t.Fatalf("load record: %v", err) + } + if rec.State != types.VMStateError { + t.Errorf("state %s, want error", rec.State) + } + if len(rec.SnapshotIDs) != 1 { + t.Errorf("persisted snapshot id missing from record: %v", rec.SnapshotIDs) + } +} + +// newHibernateTestVM seeds a running VM whose RunDir holds a live stub VMM process, so WithRunningVM lets the sequence proceed. +func newHibernateTestVM(t *testing.T) (*Backend, string) { + t.Helper() + b, _ := newMeteringTestBackend(t) + b.Conf = hibernateTestConfig{ + stubBackendConfig: b.Conf.(stubBackendConfig), + vmRunRoot: t.TempDir(), + } + + const id = "vm-hibernate-test" + runDir := t.TempDir() + seedVMRecord(t, b, id, 1, 512, 1024, true) + if err := b.DB.Update(t.Context(), func(idx *VMIndex) error { + idx.VMs[id].State = types.VMStateRunning + idx.VMs[id].RunDir = runDir + return nil + }); err != nil { + t.Fatalf("seed state: %v", err) + } + + // argv[0] and the socket-path argument satisfy WithRunningVM's cmdline check. + sock := SocketPath(runDir) + if err := os.WriteFile(sock, nil, 0o600); err != nil { + t.Fatalf("touch sock: %v", err) + } + cmd := exec.Command("bash", "-c", fmt.Sprintf("exec -a %s tail -f /dev/null %q", b.Conf.BinaryName(), sock)) + if err := cmd.Start(); err != nil { + t.Fatalf("start stub vmm: %v", err) + } + t.Cleanup(func() { + _ = cmd.Process.Kill() + _, _ = cmd.Process.Wait() + }) + // cmd.Start returns before bash execs into tail; wait until the cmdline check would pass. + deadline := time.Now().Add(2 * time.Second) + for !utils.VerifyProcessCmdline(cmd.Process.Pid, b.Conf.BinaryName(), sock) { + if time.Now().After(deadline) { + t.Fatal("stub vmm did not exec in time") + } + time.Sleep(10 * time.Millisecond) + } + if err := os.WriteFile(b.PIDFilePath(runDir), []byte(strconv.Itoa(cmd.Process.Pid)), 0o600); err != nil { + t.Fatalf("write pidfile: %v", err) + } + return b, id +} + +func hibernateStubSpec(calls *hibernateCalls) HibernateSpec { + return HibernateSpec{ + SnapshotSpec: SnapshotSpec{ + Pause: func(*VMRecord, *http.Client) error { return nil }, + Resume: func(*VMRecord, *http.Client) error { calls.resumed = true; return nil }, + Capture: func(_ *VMRecord, _ *http.Client, tmpDir string) error { + return os.WriteFile(filepath.Join(tmpDir, "mem"), []byte("x"), 0o600) + }, + BuildMeta: func(*VMRecord, string) (*SnapshotMeta, error) { return &SnapshotMeta{}, nil }, + }, + Terminate: func(*VMRecord, *http.Client, int) error { calls.terminated = true; return nil }, + } +} diff --git a/hypervisor/hypervisor.go b/hypervisor/hypervisor.go index 6445e82c..6808ac8e 100644 --- a/hypervisor/hypervisor.go +++ b/hypervisor/hypervisor.go @@ -44,3 +44,8 @@ type Direct interface { DirectClone(ctx context.Context, vmID string, vmCfg *types.VMConfig, net types.NetSetup, snapshotConfig *types.SnapshotConfig, srcDir string) (*types.VM, error) DirectRestore(ctx context.Context, vmRef string, vmCfg *types.VMConfig, srcDir, sourceSnapshotID string) (*types.VM, error) } + +// Hibernator snapshots and stops atomically: capture, persist, and termination share one pause window, and the VMM dies only after persist succeeds — a failed persist leaves the VM running. +type Hibernator interface { + Hibernate(ctx context.Context, ref string, persist func(cfg *types.SnapshotConfig, stream io.ReadCloser) error) error +} diff --git a/hypervisor/restore.go b/hypervisor/restore.go index 3645c226..6bb479e3 100644 --- a/hypervisor/restore.go +++ b/hypervisor/restore.go @@ -34,8 +34,9 @@ func (b *Backend) ResolveForRestore(ctx context.Context, vmRef string) (string, if err != nil { return "", nil, err } - if rec.State != types.VMStateRunning { - return "", nil, fmt.Errorf("vm %s is %s, must be running to restore", vmID, rec.State) + // Stopped restores too (hibernate resume): the sequence cold-spawns a fresh VMM either way and the kill step tolerates a dead one. + if rec.State != types.VMStateRunning && rec.State != types.VMStateStopped { + return "", nil, fmt.Errorf("vm %s is %s, must be running or stopped to restore", vmID, rec.State) } return vmID, &rec, nil } @@ -107,11 +108,7 @@ func (b *Backend) RestoreSequence(ctx context.Context, vmRef string, spec Restor result, afterErr = spec.AfterExtract(ctx, vmID, spec.VMCfg, rec) return afterErr } - if spec.Wrap != nil { - if err := spec.Wrap(rec, inner); err != nil { - return nil, err - } - } else if err := inner(); err != nil { + if err := runWrapped(rec, spec.Wrap, inner); err != nil { return nil, err } b.emitRestoreSuccess(ctx, result, oldShape, spec.SourceSnapshotID) @@ -147,12 +144,8 @@ func (b *Backend) DirectRestoreSequence(ctx context.Context, vmRef string, spec result, afterErr = spec.AfterExtract(ctx, vmID, spec.VMCfg, rec) return afterErr } - if spec.Wrap != nil { - if wrapErr := spec.Wrap(rec, inner); wrapErr != nil { - return nil, wrapErr - } - } else if innerErr := inner(); innerErr != nil { - return nil, innerErr + if err := runWrapped(rec, spec.Wrap, inner); err != nil { + return nil, err } b.emitRestoreSuccess(ctx, result, oldShape, spec.SourceSnapshotID) return result, nil diff --git a/hypervisor/restore_test.go b/hypervisor/restore_test.go new file mode 100644 index 00000000..db56a07e --- /dev/null +++ b/hypervisor/restore_test.go @@ -0,0 +1,47 @@ +package hypervisor + +import ( + "strings" + "testing" + + "github.com/cocoonstack/cocoon/types" +) + +func TestResolveForRestoreStates(t *testing.T) { + b, _ := newMeteringTestBackend(t) + tests := []struct { + name string + state types.VMState + wantErr string + }{ + {"running restorable", types.VMStateRunning, ""}, + {"stopped restorable (hibernate resume)", types.VMStateStopped, ""}, + {"error rejected", types.VMStateError, "must be running or stopped"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + id := "vm-" + string(tt.state) + seedVMRecord(t, b, id, 1, 512, 1024, true) + if err := b.DB.Update(t.Context(), func(idx *VMIndex) error { + idx.VMs[id].State = tt.state + return nil + }); err != nil { + t.Fatalf("seed state: %v", err) + } + + _, rec, err := b.ResolveForRestore(t.Context(), id) + if tt.wantErr == "" { + if err != nil { + t.Fatalf("ResolveForRestore: %v", err) + } + if rec.State != tt.state { + t.Errorf("state %s, want %s", rec.State, tt.state) + } + return + } + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Errorf("got %v, want %q", err, tt.wantErr) + } + }) + } +} diff --git a/hypervisor/snapshot.go b/hypervisor/snapshot.go index 2765750b..33191a9f 100644 --- a/hypervisor/snapshot.go +++ b/hypervisor/snapshot.go @@ -9,6 +9,8 @@ import ( "path/filepath" "strings" + "github.com/projecteru2/core/log" + "github.com/cocoonstack/cocoon/types" "github.com/cocoonstack/cocoon/utils" ) @@ -43,6 +45,20 @@ func (b *Backend) RecordSnapshot(ctx context.Context, vmID string) (string, erro return snapID, nil } +// UnrecordSnapshot drops a snapshot id from the VM's record (persist-failure rollback); best-effort, a leftover id is cosmetic. +func (b *Backend) UnrecordSnapshot(ctx context.Context, vmID, snapID string) { + if err := b.DB.Update(ctx, func(idx *VMIndex) error { + r, err := idx.GetRecord(vmID) + if err != nil { + return err + } + delete(r.SnapshotIDs, snapID) + return nil + }); err != nil { + log.WithFunc(b.Typ+".UnrecordSnapshot").Warnf(ctx, "unrecord snapshot %s on %s: %v", snapID, vmID, err) + } +} + func (b *Backend) BuildSnapshotConfig(snapID string, rec *VMRecord) *types.SnapshotConfig { cfg := &types.SnapshotConfig{ ID: snapID, @@ -56,21 +72,39 @@ func (b *Backend) BuildSnapshotConfig(snapID string, rec *VMRecord) *types.Snaps // SnapshotSequence is the shared capture skeleton; only capture runs in the pause window — AfterCapture (e.g. cidata copy) runs outside. func (b *Backend) SnapshotSequence(ctx context.Context, ref string, spec SnapshotSpec) (_ *types.SnapshotConfig, _ io.ReadCloser, err error) { - vmID, err := b.ResolveRef(ctx, ref) + vmID, rec, tmpDir, err := b.prepareSnapshot(ctx, ref) if err != nil { return nil, nil, err } - rec, err := b.LoadRecord(ctx, vmID) + defer func() { + if err != nil { + os.RemoveAll(tmpDir) //nolint:errcheck,gosec + } + }() + + hc := utils.NewSocketHTTPClient(SocketPath(rec.RunDir)) + pause := func() error { return spec.Pause(&rec, hc) } + resume := func() error { return spec.Resume(&rec, hc) } + captureWindow := func() error { + return b.WithPausedVM(ctx, &rec, pause, resume, func() error { + return spec.Capture(&rec, hc, tmpDir) + }) + } + if err = runWrapped(&rec, spec.Wrap, captureWindow); err != nil { + return nil, nil, fmt.Errorf("snapshot VM %s: %w", vmID, err) + } + cfg, err := b.finalizeSnapshot(ctx, vmID, &rec, spec, tmpDir) if err != nil { return nil, nil, err } - if vErr := types.ValidateStorageConfigs(rec.StorageConfigs); vErr != nil { - return nil, nil, fmt.Errorf("storage invariants violated: %w", vErr) - } + return cfg, utils.TarDirStreamWithRemove(tmpDir), nil +} - tmpDir, err := os.MkdirTemp(b.Conf.VMRunDir(vmID), "snapshot-") +// HibernateSequence is SnapshotSequence with persist inside the pause window and terminate instead of resume: the snapshot point and the stop coincide, any failure before terminate resumes the VM, and a failed terminate marks it error. +func (b *Backend) HibernateSequence(ctx context.Context, ref string, spec HibernateSpec, persist func(cfg *types.SnapshotConfig, stream io.ReadCloser) error) (err error) { + vmID, rec, tmpDir, err := b.prepareSnapshot(ctx, ref) if err != nil { - return nil, nil, fmt.Errorf("create temp dir: %w", err) + return err } defer func() { if err != nil { @@ -78,42 +112,92 @@ func (b *Backend) SnapshotSequence(ctx context.Context, ref string, spec Snapsho } }() + logger := log.WithFunc(b.Typ + ".HibernateSequence") hc := utils.NewSocketHTTPClient(SocketPath(rec.RunDir)) - pause := func() error { return spec.Pause(&rec, hc) } - resume := func() error { return spec.Resume(&rec, hc) } - captureWindow := func() error { - return b.WithPausedVM(ctx, &rec, pause, resume, func() error { - return spec.Capture(&rec, hc, tmpDir) + var killFailed bool + window := func() error { + return b.WithRunningVM(ctx, &rec, func(pid int) error { + if pErr := spec.Pause(&rec, hc); pErr != nil { + return fmt.Errorf("pause: %w", pErr) + } + failResume := func(e error) error { + if rErr := spec.Resume(&rec, hc); rErr != nil { + logger.Warnf(ctx, "resume VM %s after failed hibernate: %v", rec.ID, rErr) + } + return e + } + if cErr := spec.Capture(&rec, hc, tmpDir); cErr != nil { + return failResume(cErr) + } + cfg, sErr := b.finalizeSnapshot(ctx, vmID, &rec, spec.SnapshotSpec, tmpDir) + if sErr != nil { + return failResume(sErr) + } + if pErr := persist(cfg, utils.TarDirStreamWithRemove(tmpDir)); pErr != nil { + b.UnrecordSnapshot(ctx, vmID, cfg.ID) + return failResume(fmt.Errorf("persist snapshot: %w", pErr)) + } + if kErr := spec.Terminate(&rec, hc, pid); kErr != nil { + killFailed = true + return fmt.Errorf("terminate: %w", kErr) + } + return nil }) } - if spec.Wrap != nil { - err = spec.Wrap(&rec, captureWindow) - } else { - err = captureWindow() + if err = runWrapped(&rec, spec.Wrap, window); err != nil { + if killFailed { + b.MarkError(ctx, vmID) + } + return fmt.Errorf("hibernate VM %s: %w", vmID, err) + } + + CleanupRuntimeFiles(ctx, rec.RunDir, spec.RuntimeFiles) + // Warn-and-continue like StopAll: the VMM is dead and the flip self-heals; the snapshot is already durable. + if uErr := b.UpdateStates(ctx, []string{vmID}, types.VMStateStopped); uErr != nil { + logger.Warnf(ctx, "mark stopped %s: %v", vmID, uErr) + } + return nil +} + +// prepareSnapshot resolves ref and stages a capture dir inside the VM's run dir. +func (b *Backend) prepareSnapshot(ctx context.Context, ref string) (string, VMRecord, string, error) { + vmID, err := b.ResolveRef(ctx, ref) + if err != nil { + return "", VMRecord{}, "", err + } + rec, err := b.LoadRecord(ctx, vmID) + if err != nil { + return "", VMRecord{}, "", err + } + if vErr := types.ValidateStorageConfigs(rec.StorageConfigs); vErr != nil { + return "", VMRecord{}, "", fmt.Errorf("storage invariants violated: %w", vErr) } + tmpDir, err := os.MkdirTemp(b.Conf.VMRunDir(vmID), "snapshot-") if err != nil { - return nil, nil, fmt.Errorf("snapshot VM %s: %w", vmID, err) + return "", VMRecord{}, "", fmt.Errorf("create temp dir: %w", err) } + return vmID, rec, tmpDir, nil +} +// finalizeSnapshot writes the sidecar metadata and registers the snapshot on the VM record. +func (b *Backend) finalizeSnapshot(ctx context.Context, vmID string, rec *VMRecord, spec SnapshotSpec, tmpDir string) (*types.SnapshotConfig, error) { if spec.AfterCapture != nil { - if err = spec.AfterCapture(&rec, tmpDir); err != nil { - return nil, nil, err + if err := spec.AfterCapture(rec, tmpDir); err != nil { + return nil, err } } - - meta, err := spec.BuildMeta(&rec, tmpDir) + meta, err := spec.BuildMeta(rec, tmpDir) if err != nil { - return nil, nil, fmt.Errorf("build snapshot metadata: %w", err) + return nil, fmt.Errorf("build snapshot metadata: %w", err) } if err = SaveSnapshotMeta(tmpDir, meta); err != nil { - return nil, nil, fmt.Errorf("save snapshot metadata: %w", err) + return nil, fmt.Errorf("save snapshot metadata: %w", err) } - snapID, err := b.RecordSnapshot(ctx, vmID) if err != nil { - return nil, nil, err + return nil, err } - return b.BuildSnapshotConfig(snapID, &rec), utils.TarDirStreamWithRemove(tmpDir), nil + return b.BuildSnapshotConfig(snapID, rec), nil } func SaveSnapshotMeta(dir string, meta *SnapshotMeta) error {