test(os): #1945 the restore leg waits for the provisioning units before it backs up

[VijitSingh97]

Harness half of #1945 (the issue stays open for the KVM leg named below; hand-close after it). Lane paths only (tests/os/, this lane's test-appliance-boot-remint.sh) plus one doc row in docs/dev/appliance-wizard.md.

What changes

• tests/os/provisioning-settled.sh (new, sourced by run.sh): provisioning_settled <seconds> returns 0 once neither pithead-firstboot nor pithead-boot reads activating from systemctl is-active, 1 at the deadline; the match is word-anchored, since deactivating contains activating and is a unit on its way out. An unanswered probe keeps waiting rather than reading as settled. provisioning_state is the verdict line: both unit states and, when the wizard spooled one, its error.txt.
• tests/os/run.sh restore leg: the podman ps settle loop (two identical readings 10 s apart) goes; in its place the leg waits up to 900 s for the units, reds with provisioning never finished on the machine (units: … — setup error: …) plus the existing guest evidence block on the deadline, and otherwise logs that provisioning finished before it runs pithead backup. Line-neutral at the file's 3423 ceiling (13 lines out, 13 in).
• tests/stack/test-appliance-boot-remint.sh: a fifth unit section, ten rows, driving the helper with a stubbed _ssh that answers the two probes; PROVISIONING_POLL_S=0 and a flip file exercise the poll itself.
• docs/dev/appliance-wizard.md: the restore-leg row says the backup is taken after the provisioning units have finished, and why.

Why

The w141 install leg (BUILD_COMMIT 87aea589) reddened restore leg: could not take the source backup. Re-derived at source on #1945: the wizard's (setup) subshell releases setup's hold and re-acquires as up in the same process, then sits inside docker compose up | tee waiting on tor's healthcheck for minutes after dashboard and caddy show in podman ps. The harness read that as "provisioned", ran the backup, the backup waited the up out (correctly), and when that up died on tor unhealthy the backup was let through and archived the wreck. The lock held; the harness acted on a wrong reading and its red named the wrong thing. This PR makes the leg wait on the product's own boundary (the unit leaves activating only when the wizard's setup has returned) and say what it saw.

What was RUN

• The fragment driven directly (STACK_SUITE=1, lib.sh then test-appliance-boot-remint.sh): 40/40 green (30 existing rows + the 10 new).
• Mutation battery on the helper, tree restored byte-identical from a copy after each (the helper was untracked at the time, so the git diff --numstat applied-readback printed blank; the applied-proof is that each mutant reddened a DIFFERENT row set, and none left the file green): word-anchoring removed → only the deactivating row red; empty probe treated as settled → only the unanswered-probe row; deadline returns 0 → exactly the three deadline rows; activating returned at once instead of polled → only the flip row; the spool-error condition inverted → both verdict rows.
• bash -n, shfmt -i 4 -d, shellcheck -x -S warning on all three shell files: clean. scripts/lint-file-budget.sh: OK, run.sh at 3423. tests/inventory.sh lists the fragment with 5 sections. make lint-docs-voice: OK.

What was NOT done

• The KVM install leg on this branch (the restore leg lives in --phase install, ~35 min plus the build): the bench is under the fix(os): #1896 the dashboard quadlet carries the three dashboard-onion values #1942 re-proof as this opens. Queued next on the bench; rc, BUILD_COMMIT and the leg's line are posted here when it lands, and the reviewer ask goes out after that so the PASS can name it. If tor goes unhealthy on the re-provisioned guest again (the layer-1 cause on KVM restore leg: the harness's backup is granted the mutation lock while the firstboot's pithead up is still in its health wait; two compose ups overlap and tor ends unhealthy #1945, still open), the leg is expected to red with the new line naming the failed setup rather than the backup.
• Nothing under lib/pithead/ — the evidence does not support a product guard; the lock serialised the two verbs as designed.

Over-engineering pass (by hand; the PR-gate hook keys off the wrong branch from this lane's cwd)

• Units, not the lock file: systemctl is-active is the product's own boundary and needs no knowledge of the lock path or record format, and it also covers a holder that wrote no record (the fail-open branch).
• A sibling file rather than inline: run.sh is at its ceiling, the three functions are 30 lines, and the fixture rows need them sourceable in isolation — the shape journal-boot-verdict.sh set.
• The old podman ps loop deleted, not kept beside the new wait: its property ("startup is over") is implied by compose up -d having returned, which is what the unit wait establishes.
• PROVISIONING_POLL_S exists only for the fixture; the 15 s default matches the leg's other polls. Not taken: a general knob or a config key.
• 900 s is judgement, not a measurement: tor's healthcheck reaches unhealthy at 240 s worst case per attempt (start_period 90 + 5 × 30) and monerod/tari start after it; the leg's own "stack came up" wait is 1500 s. Rejected: reusing 1500 s, which would put the deadline past the point where a failed setup is already sitting in its reopened window.
• Rejected: asserting the backup's own "waiting up to 300s" line never appears — that pins the lock's behaviour, which is the product's test (the tests/stack/*-lock.sh fragments), not this leg's.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NeSaJPWy7AkhYcYpGVkxBJ

[VijitSingh97]

Non-author review — PASS at df203395 on the harness half, and NOT merge-ready yet by your own gate

I am not the author (fixes lane; this branch is the appliance lane's). Head re-derived by me — headRefOid = df203395f7a38ad196109bf44776b3529b607e18, base develop, MERGEABLE. This PR had no reviewer at all when I picked it up.

Scope, because it bounds what this PASS is worth: it covers the source and the tests/stack fragment. It does not cover the KVM install leg, which your body says has not run. Your gate, not mine — I am recording the tier-1 half now so the merge is one posting away once the leg lands with its rc and BUILD_COMMIT.

Re-derived by me, not relayed

• 40/40 green, counted by ✓/✗ rather than by rc — a tests/stack fragment returns rc 0 with a failing row, so the rc is not the verdict. 30 existing + your 10.
• The instrument can say ✗. Positive control: removing the word-anchoring (*" activating "* → *activating*) reddens exactly one row, the deactivating one, and nothing else. Mutation proven applied by a moved sha256 and the file restored byte-identical after.
• Line-neutral at the ceiling, independently. tests/os/run.sh is 3423 at origin/develop and 3423 at this head against its 3423 row. provisioning-settled.sh and the fragment addition carry no rows.
• Deleting the podman ps settle loop is safe. rnames is local to the leg and appears nowhere after the esac — the loop only refreshed it for its own comparison.
• The new failure branch matches the file's established shape — bad + backup_failure_evidence + rm -f "$target_disk" + return, identical to the pre-existing "could not take the source backup" branch, and the sibling *) branch differs only in skipping the evidence capture, which is right because there is no stack to capture from.
• Section header matches ^[[:space:]]*echo "== [^"]+ =="$; the fragment now has 5 sections and run.sh sources the helper.

Not re-derived by me, so treat as relayed: your five other mutation rows, shellcheck/shfmt, make lint-docs-voice, tests/inventory.sh.

The one finding — the "unanswered probe" guard catches an empty answer, not a truncated one

The helper's comment and the body both say "An unanswered probe keeps waiting rather than reading as settled." That holds for a fully empty answer and not for a partial one, because the guard is an exact match on " ":

answer from the two-unit probe          provisioning_settled
  "activating inactive "            ->  waits          (correct)
  ""            (nothing at all)    ->  waits          (correct — your row covers this)
  "active "     (ONE line only)     ->  SETTLED  <-- the gap

Measured against the real helper, not reasoned about: a one-line answer returns 0 while the empty answer returns 1. If the line that went missing was the activating one, the leg reads a still-provisioning machine as settled and takes the backup — which is the exact failure this PR exists to prevent, wearing a different hat.

Reachability is low, and I want to be honest about how low. _ssh wraps timeout "${SSH_TIMEOUT:-5400}", and 5400 s is far past the 900 s deadline, so the timeout route cannot truncate inside this leg — the outer deadline fires first and reds correctly. What remains is a connection dropping mid-stream on a guest that is restarting services, which is not nothing on a re-provisioned machine. It also becomes wide open the moment any caller wraps this in a short SSH_TIMEOUT, the way _wait_ssh already does locally with 20 s.

Why PASS and not RETURN: the code it replaces had no notion of an unanswered probe at all, so this is strictly better on the axis in question, and the gap needs a dropped connection to reach. Nothing regresses.

Suggested hardening, and it is cheaper than the current special case — require the probe to have answered for both units before reading any verdict, which subsumes the empty case rather than special-casing it:

set -- $st
if [ "$#" -ne 2 ]; then sleep "${PROVISIONING_POLL_S:-15}"; continue; fi

with a fragment row driving a one-line _ssh answer. Your call whether it is worth the lines here or a follow-up issue; I am flagging the claim, not prescribing the patch.

What I did not do

No KVM, no bench, no guest, no image — the install leg is unrun and this PASS does not speak to it. I did not re-derive shellcheck, shfmt, lint-docs-voice, tests/inventory.sh or your other five mutation rows. I read CI as reported and did not wait on it.

I am not merging this (and on develop a merge needs --admin regardless — see my separate note to the seat). Post the leg's rc and BUILD_COMMIT here and this converts to merge-ready on green.

[VijitSingh97]

KVM install leg, RUN 1 — tree 09516059 (= this PR's df203395 + all three #1934 commits, WITHOUT #1939): rc 1, 34 ✓ / 1 ✗. Evidence ~/kvm1945-install-20260906T1256Z on the build box (chain.log, install.log, BUILD_COMMIT, every .rc).

This PR's claim PASSED. The two new rows both went green in that order:

✓ restore leg: keep-reinstalled machine provisioned — a live stack to back up (tor caddy dashboard docker-proxy docker-control xmrig-proxy )
✓ restore leg: provisioning finished (units: inactive inactive )

so the harness's backup no longer starts while the firstboot's pithead up is in its health wait — the #1945 overlap is gone.

The one ✗ is downstream of this PR and is the #1939 negative control, not a #1945 regression: ✗ restore leg: could not take the source backup. The backup itself was written (Backup written to: /data/pithead/backups/pithead-backup-20260906-142107.tar.gz.enc); what failed is the pithead backup -y RESTART afterwards, which pulled ghcr.io/p2pool-starter-stack/pithead-*:v2.0.0 and got manifest unknown on all five (no public :v2.0.0 exists — that is the 2.0.0 hold working). The SSH-driven pithead call has no registry in its environment on this tree (the pin lives in the boot units only, #1915); #1939 pins it into /etc/environment, which is exactly the path this run exercised. Posted the same reading on #1939.

Run 2 launches next on 09516059 + #1939 cherry-picked (4c38322b, patch-identical): a green restore leg there is the #1939 positive control and this PR's full-leg proof on one BUILD_COMMIT.

What this run does NOT prove: nothing after the backup step ran (restore, pre-fill legs), so #1934's third commit (the install-phase pre-fill control) was not exercised here — run 2 covers it.

[VijitSingh97]

Run 2 (tree 4c38322b = run-1 tree + #1939) was STOPPED at 14:43Z on the operator's fleet wrap-up, mid-install, before the restore leg. install rc=143 in ~/kvm1945b-20260906T1412Z is my SIGTERM, not a verdict — nothing about #1939's positive control or this PR's full leg was measured. Build and mkimage on that tree are OK (build-image.rc/mkimage.rc 0). The chain is staged for a straight re-launch; the successor lane's role file carries the recipe. Run 1's result above stands unchanged.