feat: opt-in large acquisition mode — pause on low disk space, operator pause

[hongquanli]

Summary

First of three stacked PRs for large acquisitions (data offloaded to a NAS during the run; design: AI-docs Squid/to-do/2026-09-14-large-acquisitions-offload-and-pause-design.md). This PR adds an opt-in "large acquisition mode" that pauses the acquisition instead of aborting when the save disk is nearly full, and gives the operator a Pause/Resume control through the same mechanism.

Strictly opt-in — nothing changes with default settings. With LARGE_ACQUISITION_MODE = False (the default) and no per-run opt-in, the acquisition loop, result draining, GUI, TCP status payload, watchdog states and on-disk layout are byte-for-byte today's. The mode is enabled either in Settings › Acquisition › Large Acquisitions, or for one run from the existing "Not Enough Disk Space" pre-flight dialog (which keeps its Cancel path as today).

What it does when the mode is on

• PauseGate (control/core/pause_gate.py): the single pause control point for a run. Holders by reason (disk_space, operator); paused while any holder is active, so an operator Resume cannot override a full disk and freed space cannot override an operator pause. Abort-aware wait (polls the abort flag, so app shutdown still joins in time).
• DiskSpaceGuard (control/core/disk_space.py): at every FOV and timepoint checkpoint, requires reserve + bytes still queued for saving + 2 × one FOV of free space; holds the gate below that, releases with one FOV of hysteresis. Settings are read live so they can be changed mid-run.
• Worker checkpoints: the FOV-loop top (after the result drain, before the stage moves) and the timepoint-loop top. Nothing inside a z-stack or channel sequence is interruptible. While parked the worker keeps heart-beating the watchdog with a new paused run status, re-checks the disk every DISK_SPACE_POLL_INTERVAL_S, and refreshes the GUI.
• Schedule re-anchor: timed acquisitions schedule against a new anchor that shifts by the paused time, so a long pause no longer makes the existing skip-catch-up loop silently drop timepoints (verified: with the shift disabled, a 4 s pause on a 4 s interval loses timepoint 2).
• Watchdog: run.json status paused is treated like running (dead pid → crash, stale heartbeat → hang), never a false hang.
• Surfaces: Pause/Resume button + status line in the flexible and wellplate multipoint widgets (shown only for mode-on runs); TCP/MCP pause_acquisition, resume_acquisition, and get_acquisition_status gains pause_reasons / paused_since / paused_total_s / disk{…} only for mode-on runs; Slack messages on pause and resume (SlackNotifications.NOTIFY_ON_PAUSE).
• Pre-flight: the disk check is now a pure decision function + thin dialog. Mode off: "Enable large acquisition mode for this run" / "Cancel". Mode on: "Continue" / "Cancel". Headless YAML runs take acquisition.large_acquisition_mode from the file (absent = unchanged).
• Dev: SIMULATED_DISK_CAPACITY_GB (Dev tab) computes free space as capacity minus the experiment folder size, so deleting files raises free space — this is how the tests and the GUI walkthrough exercise the disk path without filling a real disk.

Settings ([GENERAL])

key	default
large_acquisition_mode	false	master switch
disk_space_reserve_gb	10.0	free space to keep on the save disk
disk_space_poll_interval_s	5.0	re-check cadence while paused
simulated_disk_capacity_gb	0.0	dev only, 0 = real disk

Tests

New: test_pause_gate.py (12), test_disk_space.py (25), test_MultiPointWorker_pause.py (14, incl. mode-off no-op and drain policy), test_MultiPointController_pause.py (5 end-to-end simulated runs: mode off, operator pause/resume with anchor shift, abort while paused, low-disk pause → resume when space frees, global setting), test_preflight_disk_check.py (10), plus additions to the watchdog, breadcrumb, Slack, server, YAML loader, acquisition-settings, preferences and _def tests. Every touched or new test module passes locally; a 322-test regression subset (engine, controller, settings, watchdog, Slack, server, backpressure) passes. CI (Linux) is the authoritative full-suite run: local macOS full-suite runs abort in a pre-existing memory-profiler test (test_widgets.py::TestMultiPointControllerMemoryMonitoring::test_ram_widget_connect_to_multipoint_monitor, the sampler spawns a footprint subprocess from a background thread; passes alone, aborts in suite order, Linux never takes that path). Tracked as an AI-docs action item; not touched by this PR.

Note for reviewers running locally: the end-to-end tests use the simulated disk so they never depend on the host's free space.

Out of scope / follow-ups

• PR 2: transfer manifest + tools/upload_acquisition.py NAS tool. PR 3: OME-TIFF per-timepoint split (+ ndviewer_light PR).
• GUI Pause controls for runs started over TCP/MCP (the widgets only reveal them for runs they started or that emit a pause signal while they are the current widget); the TCP status/pause commands cover those runs.
• SlackNotifications.NOTIFY_ON_PAUSE is code/INI only, not yet in the Slack settings widget.
• Two pre-existing bugs found during design are deliberately not fixed here (see AI-docs action item): the laser-AF failure path writes to an unpadded timepoint folder; the per-FOV result drain processes one result per queue (this PR drains fully only when the mode is on).

🤖 Generated with Claude Code

https://claude.ai/code/session_01F7SDD9fYF47FUoL8M1dfvF