perf(snapshot): write the guest memory blob sparsely - #1788
Conversation
jprendes
added
the
kind/enhancement
There was a problem hiding this comment.
🟢 Approval recommended
The sparse-write implementation preserves byte-for-byte file contents and is covered by tests validating correctness, reuse, and (where supported) actual sparseness.
Pull request overview
This PR improves snapshot save performance and disk usage by writing the large guest memory blob as a sparse file (skipping all-zero regions) while keeping the on-disk byte stream and digest identical for readers.
Changes:
- Add a sparse-writing path that coalesces contiguous non-zero regions into fewer writes and uses
set_lento preserve full logical length. - On Windows, best-effort mark the blob file as sparse via
FSCTL_SET_SPARSEto prevent NTFS from allocating skipped ranges. - Add tests validating byte-for-byte round-trips, digest verification/reuse behavior, and (on Unix where supported) reduced allocation vs dense writes.
File summaries
| File | Description |
|---|---|
| src/hyperlight_host/src/sandbox/snapshot/file/fsutil.rs | Implements sparse writing for the snapshot memory blob, integrates it into the blob write path, and adds targeted unit tests (including Unix sparseness assertion). |
| src/hyperlight_host/Cargo.toml | Expands windows-sys features needed for DeviceIoControl and FSCTL_SET_SPARSE. |
| CHANGELOG.md | Documents the user-visible snapshot save improvement and its transparency to readers/digests. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
`Snapshot::save` wrote the guest memory image with `write_all`, which allocates blocks for zeros like any other data: the kernel cannot know the caller would accept a hole. A guest memory image is mostly pages the guest never touched, so saving a snapshot wrote hundreds of MB to reach a final artifact of a few tens of MB. Skip all-zero blocks instead, seeking over them and extending the file with `set_len`, so the untouched pages become holes. Runs of adjacent non-zero blocks are coalesced into one write, so a blob with no zeros costs a single `write_all` as before. This is invisible to readers. The digest is computed from the in-RAM buffer before the write, and reads of a hole return zeros, so a sparse blob is byte-for-byte identical to a dense one and hashes the same. The format is unchanged and existing snapshots still load. Unix filesystems make a file sparse implicitly. NTFS does not, and zero-fills any range skipped by a seek unless the file is explicitly marked sparse, so request that first with FSCTL_SET_SPARSE. The request is best-effort: a filesystem without sparse support (FAT32, exFAT, HFS+) rejects it, and the write is then simply dense, which is still correct. Only the large snapshot blob takes this path; the manifest and config blobs are small enough that scanning them for zeros would not pay. Measured on btrfs with a 559 MB image whose pages are ~96% zero: 3.55s and 586 MB on disk, down to 1.70s and 23 MB. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
b54f9e0 to
3546acd
Compare
|
This is a nice improvement! However, I think that the fact that we are seeing snapshots that are e.g. "~96% zero" at all is reflective of deeper issues which we consider addressing additionally or instead. For one thing, I think it might be more impactful to change the snapshot code to simple not include all-zero pages in the snapshot blob in the first place. They could either all be CoW aliases of a single zero page (we could make this a more general page deduplication a la KSM), or be a new "zero-allocate on first read/write" page region type (which we need to introduce for #1690 anyway). And, I'm also still curious where these all-zero pages are coming from in the first place. One source is probably .bss in the executable, which we could change to be a zero-allocate-on-first-read/write mapping as discussed above. The other source is probably the legacy heap region, which as pointed out in #1697, is basically useless and should go away. |
This is the main reason. IIUC, we still need to use heap as our allocator is tied to this heap. |
That's probably true, but IIUC it would also have a perf cost on restore, as we can't just mmap a single blob. |
Snapshot::savewrote the guest memory image withwrite_all, which allocates blocks for zeros like any other data: the kernel cannot know the caller would accept a hole. A guest memory image is mostly pages the guest never touched, so saving a snapshot wrote hundreds of MB to reach a final artifact of a few tens of MB.Skip all-zero blocks instead, seeking over them and extending the file with
set_len, so the untouched pages become holes. Runs of adjacent non-zero blocks are coalesced into one write, so a blob with no zeros costs a singlewrite_allas before.This is invisible to readers. The digest is computed from the in-RAM buffer before the write, and reads of a hole return zeros, so a sparse blob is byte-for-byte identical to a dense one and hashes the same. The format is unchanged and existing snapshots still load.
Unix filesystems make a file sparse implicitly. NTFS does not, and zero-fills any range skipped by a seek unless the file is explicitly marked sparse, so request that first with FSCTL_SET_SPARSE. The request is best-effort: a filesystem without sparse support (FAT32, exFAT, HFS+) rejects it, and the write is then simply dense, which is still correct.
Only the large snapshot blob takes this path; the manifest and config blobs are small enough that scanning them for zeros would not pay.
Measured on btrfs with a 559 MB image whose pages are ~96% zero: 3.55s and 586 MB on disk, down to 1.70s and 23 MB.