Skip to content

feat: transfer manifest + NAS upload tool for large acquisitions - #649

Draft
hongquanli wants to merge 4 commits into
feat/large-acquisitions-pausefrom
feat/large-acquisitions-manifest
Draft

feat: transfer manifest + NAS upload tool for large acquisitions#649
hongquanli wants to merge 4 commits into
feat/large-acquisitions-pausefrom
feat/large-acquisitions-manifest

Conversation

@hongquanli

@hongquanli hongquanli commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Second of three stacked PRs for large acquisitions (stacks on #648; design: AI-docs Squid/to-do/2026-09-14-large-acquisitions-offload-and-pause-design.md). Runs in large acquisition mode now write a transfer manifest listing every file that is complete and safe to move off the disk, and the repo gains an accompanying NAS upload tool that follows it during the run (or uploads any finished acquisition afterwards), for every save format.

Strictly opt-in, unchanged otherwise. With the mode off, no manifest is written, results are handled as before, and the acquisition path is untouched. The only always-on change is that save jobs return a small result dataclass instead of True; the worker ignored those return values before, and the existing OME-TIFF test that asserted truthiness still passes.

The contract — transfer_manifest.jsonl (docs/transfer-manifest.md)

Append-only JSON lines in the experiment folder: start, complete (relative POSIX path, kind file|dir, bytes, t, region, fov), timepoint_done, end. Only complete entries are movable; a dir entry is a subtree. Everything unlisted (run-level metadata, logs, mosaic outputs, Zarr zarr.json, the manifest itself) moves only after end plus a quiet period. timepoint_done is written after a bounded, abort-aware drain of that timepoint's save jobs, so it follows every complete entry of the timepoint. Readers tolerate a truncated last line.

How completion is decided (control/core/transfer_manifest.py)

Save jobs return SaveResult (ZarrWriteResult carries the same fields): plane identity plus immediate_paths (final on arrival) and unit_paths (final once the (t, region, fov) unit is complete). A main-side CompletionTracker, fed from the worker's single result drain (and the inline non-multiprocessing path), counts planes per unit:

format unit listed when
individual TIFF the file its result arrives
multi-page TIFF FOV stack file Z×C planes
Zarr 5D (HCS / per-FOV) <array>/c/<t> Z×C planes (chunk and shard T-extent is 1; writes commit before the job returns)
Zarr 6D (per-region store, FOV axis first) one <array>/c/<fov>/<t> per FOV all FOVs × Z × C
OME-TIFF (unsplit) the FOV file the writer's own finalize (last timepoint)

Files the worker writes directly (per-timepoint coordinates.csv, RGB merges, laser-AF images) are listed right after the write. Incomplete units at the end are logged, never listed. Results are also drained on every pause tick, so files finished during a disk-space pause get listed and the pause can resolve; every drain applies the existing abort-on-failed-job policy. Zarr and OME-TIFF layouts are unchanged.

NAS upload tool — tools/upload_acquisition.py (docs/upload-acquisition.md)

upload_acquisition.py <experiment> <nas_dir> [--mode copy|move] [--follow] [--checksum] [--quiesce-s] [--poll-s] [--dry-run] and verify. Sources of truth: the manifest, else the root .done (a finished run). Without a manifest nothing moves mid-run: a timepoint's own .done only means imaging finished while saves may still be writing. The destination must be disjoint from the experiment folder (the tool refuses one that is, contains, or lies inside it, symlinks resolved). Copies to a .partial name, fsyncs, verifies size (or sha256), renames into place, then unlinks the source in move mode; idempotent and resumable; a failed copy never deletes a source; after end and quiescence it moves the remainder, manifest last. verify checks the destination against the manifest and per-format completeness (timepoint .done markers, no stray OME progress files, Zarr acquisition_complete).

Tests

test_transfer_manifest.py (writer/reader/tracker), test_save_results.py (every job class incl. 6D Zarr and simulation mode, picklability), test_MultiPointWorker_manifest.py (wiring, drain barrier), test_transfer_manifest_integration.py (simulated runs: mode off writes nothing; individual TIFF lists every image, per-timepoint coordinates and ordering; Zarr lists chunk directories only), test_offload_integration.py (a run on a 0.25 GB virtual disk pauses on low space, the tool in --follow --mode move drains it, the run resumes and completes, the destination verifies; individual TIFF and Zarr, whose reassembled store reads back with tensorstore), tests/tools/test_upload_acquisition.py (manifest, legacy and finished-run modes, failure and resume behaviour, verify, CLI).

Follow-ups

  • PR 3: OME-TIFF per-timepoint split (+ ndviewer_light PR).
  • rsync/ssh backend for the tool; GUI surface for the tool.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F7SDD9fYF47FUoL8M1dfvF

hongquanli and others added 2 commits September 15, 2026 00:32
@hongquanli @claude
feat: transfer manifest + NAS upload tool for large acquisitions
8616146 
Second of three stacked PRs for large acquisitions (stacks on the opt-in
large acquisition mode). Runs with the mode on now write
transfer_manifest.jsonl listing every file that is complete and safe to
move off the disk, and tools/upload_acquisition.py follows it to move data
to a NAS during the run, or uploads any finished acquisition afterwards.

Strictly opt-in: with the mode off no manifest is written and the
acquisition path is unchanged. The one always-on change is that save jobs
return a SaveResult dataclass (ZarrWriteResult gains the same fields)
instead of True; the worker ignored those return values before.

- control/core/transfer_manifest.py: append-only JSON-lines writer
  (start / complete / timepoint_done / end; relative POSIX paths; flush
  per line, fsync at timepoint and end), tolerant reader, and the
  main-side CompletionTracker that turns per-plane results into unit
  completion: individual TIFF on arrival, multi-page TIFF and Zarr 5D
  chunk directory <array>/c/<t> at Z x C planes, Zarr 6D one
  <array>/c/<fov>/<t> per FOV after all FOVs x Z x C, OME-TIFF on the
  writer's own finalize.
- Worker: manifest and tracker created only in mode-on runs; results fed
  from the single drain point (and the inline non-multiprocessing path);
  coordinates.csv, RGB merges and laser-AF images listed after writing;
  a bounded, abort-aware drain before timepoint_done so the marker follows
  every complete record of the timepoint; end record after the run.
- tools/upload_acquisition.py: copy|move to a mounted NAS path, --follow
  during a run, size or sha256 verification via .partial + rename,
  idempotent/resumable, never deletes a source on a failed copy, moves the
  unlisted remainder after end + quiescence with the manifest last;
  legacy runs use per-timepoint .done markers or the root .done; verify
  subcommand checks the destination against the manifest and per-format
  completeness. Docs: docs/transfer-manifest.md, docs/upload-acquisition.md.

Tests: manifest writer/reader/tracker, every save job's result (incl. 6D
Zarr and simulation mode), worker wiring and drain barrier, simulated
runs writing the manifest (mode off writes nothing; TIFF and Zarr), and
end-to-end offload: a run on a virtual disk pauses on low space, the tool
drains it in follow+move mode, the run resumes and completes, and the
destination verifies (TIFF; Zarr store reassembled and read back).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F7SDD9fYF47FUoL8M1dfvF
@hongquanli @claude
fix(manifest): list a completed file unit with its on-disk size
c80ea1a 
The transfer manifest recorded the summed pixel bytes of a unit's planes
for multi-plane files (multi-page TIFF stacks, OME-TIFF), which is smaller
than the file on disk (headers, finalized OME-XML), so the upload tool's
size verification rejected them. The unit is final when it is listed, so
stat the file then.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F7SDD9fYF47FUoL8M1dfvF
@hongquanli hongquanli mentioned this pull request Sep 15, 2026
Draft
hongquanli and others added 2 commits September 15, 2026 01:31
@hongquanli @claude
fix: review findings on the manifest and upload tool (deadlock, alias…
13186cf 
…ing, legacy markers, abort policy)

- Pause tick drains job results: save jobs still in flight when a
  disk-space pause began finish during it, but their results stayed
  queued until the next FOV drain, so those files were never listed and
  the upload tool could not free their space; the pause could not
  resolve. The tick now drains every poll, feeding the completion tracker.
- Drains apply the abort-on-failed-job policy: the timepoint barrier and
  the pause tick discarded none_failed, so a failed save consumed there
  vanished before the FOV loop's check and the run continued despite
  abort_on_failed_jobs=True. One helper applies the policy everywhere.
- Upload tool refuses a destination that is, contains, or lies inside
  the experiment folder (resolved, so symlinked aliases count):
  `upload_acquisition.py /data/exp /data` computed /data/exp as the
  destination, every file matched itself and move mode deleted the run.
  transfer_file also refuses a same-file pair.
- Legacy per-timepoint .done markers no longer authorize mid-run moves:
  the marker only means imaging finished; asynchronous saves can still be
  writing after any quiet period. Without a manifest nothing moves until
  the root .done. Docs updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F7SDD9fYF47FUoL8M1dfvF
@hongquanli @claude
fix: process every drained job result even after a failure
eae964c 
The drain combined results with `none_failed = none_failed and
summarize(result)`, which short-circuits after the first failure: later
results were already taken off the queue and were silently dropped. With
the transfer manifest that meant the files of successful saves behind a
failed one were never listed (and, with abort-on-failure off, a
disk-space pause could not resolve). Both the runner-queue and the
inline-queue drains now evaluate every result before combining the
status. Regression tests feed a failure followed by successes through
real queues.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F7SDD9fYF47FUoL8M1dfvF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hongquanli