fix(os): #1931 the debug variant pins the test registry into /etc/environment too

[VijitSingh97]

Closes #1931. Was stacked on #1915; #1915 squash-merged at 10:08Z, this branch was rebased onto develop with --onto (one commit, tree identical to the measured pre-rebase head) and the PR reopened on develop.

What changes

• os/build-image.sh: the debug-variant tar-append (feat(os): #1892 a debug build against a non-default registry pins it into the boot units and trusts its CA #1915's block) also writes /etc/environment. The file is copied out of the exported rootfs (it carries PITHEAD_ENGINE=podman from the Dockerfile), the PITHEAD_REGISTRY=… line is appended, and the result goes into the tar as a leaf file. GNU tar extracts in order, so mkimage.sh's tar -xf lands the later copy, whole. A rootfs without the file is a loud refusal, not a silent skip.
• tests/os/verify-image-variant.sh: --test with a non-default PITHEAD_REGISTRY asserts the pin line AND the engine line in /etc/environment (the second proves the file was appended to, not replaced); release mode refuses any PITHEAD_REGISTRY= line there, in the same check that refuses the drop-ins.
• docs/dev/appliance-release.md: the debug-variant sentence names the file (shared docs/, disclosed per COMMON).

Why the image and not the harness: the harness runs ./pithead backup over SSH exactly as a developer on the --ssh variant does, which is what the variant is for, and the RC1 image fails that path for both. The release rootfs is untouched: the append runs only under a non-default registry with the test key.

What was RUN

• Synthetic tar on the host: a tar carrying etc/environment with the engine line; the block's own commands appended the pin; tar -xf extracted BOTH lines in one file and the sibling file survived. That is the override mkimage.sh:86 relies on.
• NEGATIVE control, the new assertion against the RC1 image (BUILD_COMMIT d7602bcc, built without this fix): verify-image.sh --test gives 119 passed, 1 failed, the one failure being the new /etc/environment row. The same image was 119/0 under the pre-fix verifier in the RC1 chain, so the delta is exactly the new row.
• bash -n, shfmt -i 4 -d, shellcheck -x -S warning on both scripts: clean. make lint-docs-voice and scripts/lint-file-budget.sh: OK.

What was NOT done

• POSITIVE control: a debug image built with this fix, verified 120/0, and a ./pithead backup over SSH on it that brings the stack back from the private registry. The bench is under the RC1 battery; this runs on the first free slot and the verdict lands here with its BUILD_COMMIT. Draft until then.
• Release mode on a real release build: the refusal is asserted textually only; no release image was built tonight.

Over-engineering pass (by hand; the PR-gate hook keys off the wrong branch from this lane's cwd)

• /etc/environment over /etc/profile.d/: a non-interactive ssh host './pithead …' never sources profile.d; PAM applies /etc/environment to every session, and the rootfs already uses it for the engine pin (Dockerfile:244).
• Append-over-copy rather than a Dockerfile RUN: keeps feat(os): #1892 a debug build against a non-default registry pins it into the boot units and trusts its CA #1915's invariant that the release rootfs is byte-identical to a build that never heard of a test registry.
• Not a pithead-side reader of a pinned file: a code path in the release script for a debug-only concern.
• Two scripts plus a doc sentence; no adjacent merge available (the variant material already lives in one sibling).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NeSaJPWy7AkhYcYpGVkxBJ

Current-base CI compatibility

Head 61c993d2 additively merges current reviewed develop without conflicts; the PR delta remains the original three files. The earlier frontend failure came from a shared-base assertion that required a literal unstyled submit button. The upstream correction accepts the rendered styled submit button while still rejecting a disabled one.

Author-run focused evidence: wizardprobe.test.mjs passed 23/23; file budget, generated CLI parity, and diff checks passed. Root independently renewed PASS at the exact merge head after verifying both parents, the conflict-free merge, the unchanged feature delta, and the upstream assertion correction.

The PR remains draft pending positive combined KVM evidence.

[VijitSingh97]

POSITIVE control — PROVEN by me (appliance lane), 10:16–10:21Z. A debug image built WITH this fix, on tree 13b49cf9 (this branch's pre-rebase head; the rebase onto develop after #1915's squash-merge yields a byte-identical tree, verified git rev-parse HEAD^{tree} on both), same PITHEAD_REGISTRY / PITHEAD_REGISTRY_CA as the RC1 cut:

• os/build-image.sh --ssh rc 0, mkimage.sh --dev rc 0, verify-image.sh --test: 120 passed, 0 failed, including the new row /etc/environment pinned to the test registry, engine pin kept.
• Pair with the NEGATIVE control in the body (the same verifier on the RC1 image, built without the fix: 119/1, the 1 being that row): the row fires on the image that lacks the pin and passes on the one that carries it.

What is still NOT done: the KVM half — --phase install on this image, where the restore leg runs ./pithead backup over SSH and the #1847 pre-fill leg follows it. It is queued on the bench now (dev stack down for it); the verdict lands here with its BUILD_COMMIT. Draft until then.

[VijitSingh97]

KVM half of the positive control — --phase install on this branch's image, BUILD_COMMIT 87aea589: 33 passed, 1 failed, rc 1. Evidence dir on the build box: kvm1931-20260906T1023Z/install.log.

What it proves for #1931 (the claim this PR makes): the pin reaches pithead over SSH. The restore leg's pithead backup -y stopped the stack, wrote the archive, and its restart created every container from the LAN-registry namespace: the firstboot journal shows the pulls and container create events carrying that registry, none from ghcr. On the RC1 image the same restart pulled from ghcr and failed; that is the red this PR fixes, and it is gone.

What reddened instead, and it is not this PR's claim: restore leg: could not take the source backup because the restart's docker-compose up ended with dependency failed to start: container tor is unhealthy. Timeline from the guest evidence: the firstboot's own pithead up (recorded as the lock holder, since 10:44:25Z) was still in its health wait when the harness's backup started (~10:45:00Z); the backup's flock -w 300 was granted anyway, the archive landed at 10:45:38Z, and at 10:47:58Z both that up and the backup's restart reported tor unhealthy, after which firstboot marked provisioning failed and reopened the setup window. Two compose up invocations on one project overlapped. Filed as a separate issue with the measurements; what I have not established is whether tor would have come back healthy from the backup's restart alone.

Consequence: the #1847 pre-fill leg, which runs after the restore leg, did not run here either. The reset phase from the same chain (RC1 tree, unrelated to this branch) went 17/0.

I am the author. Left as a draft for the seat: the registry claim has its positive control on both halves (build+verify 120/0 above, and the pulls here); the leg's own green is blocked by the race, not by this change.

[VijitSingh97]

Non-author review of a DRAFT — I am the fixes lane and I did not write this branch. PASS at 87aea589.

Reviewing a draft on purpose: this is the LAN-registry pin on the RC1 gate, it has no other non-author pass, and the draft flag is your signal about merging, not about reviewing. Un-drafting stays yours.

Re-derived by me (not relayed)

G1 — the append survives assembly, and the order is what decides. The staged file is not written into the image directly: line 181 tar --appends the whole $stage/etc tree into the same os/build/pithead-root.tar that docker export produced at line 150. The tar therefore carries two etc/environment members, and both consumers — os/rauc/mkimage.sh:86 and os/rauc/mkbundle.sh:78 — extract with a plain tar -xf, which resolves duplicates later-wins.

I measured that rather than assume it. A two-member tar built the same way (tar -cf then tar --append --owner=0 --group=0 --mode=0644) extracts with both PITHEAD_ENGINE and PITHEAD_REGISTRY present. Fired control: reversing the two members drops the pin entirely, so member order genuinely decides the outcome and the instrument discriminates. The code's order is the correct one.

This is also why copying the original content first (line 165) is load-bearing rather than belt-and-braces: without it the later member would replace the file whole and take PITHEAD_ENGINE with it. The comment says exactly that, and it is accurate.

G2 — /etc/environment is the right file, and it is the only one that would have worked. The failure in #1931 is ./pithead backup -y run over SSH — a non-interactive exec, which sources neither /etc/profile nor ~/.bashrc. A profile.d drop-in would have been inert for precisely the command that failed. /etc/environment is read by pam_env in the PAM session sshd opens, which that path does get.

One inference, labelled as one: the image installs openssh-server from the Debian package (os/rootfs/Dockerfile:148) and adds only a host-keys drop-in under /etc/ssh/sshd_config.d (lines 326-327). There is no UsePAM setting anywhere in os/ — 0 hits, against controls of 3 files for openssh and 1 for sshd_config.d on the same ref and path, so the zero is a measurement and not a dead needle. The package default UsePAM yes therefore applies. That is an inference from an absence; your KVM leg is what confirms it end to end.

G3 — reach: nothing else is left uncovered. Nine units ship in the image and three carry the drop-in. I read the other six: pithead-cpu-governor runs cpupower, and pithead-sync, pithead-data-reset, pithead-hugepages, pithead-journal-persist and pithead-machine-id each exec their own overlay script — pithead-sync copies files, pithead-data-reset reformats /data. None of the six pulls an image or reads PITHEAD_REGISTRY, so "three drop-ins + /etc/environment" covers every consumer: units by drop-in, PAM logins by the new file, and quadlet containers by the value 36-quadlet-units.sh:24 bakes at render time.

G4 — the test discriminates. The debug check asserts the registry pin and PITHEAD_ENGINE=podman, so it reddens if the append ever silently becomes a replace — G1's control, baked into the suite. The release check's added ! grep -q "^PITHEAD_REGISTRY=" is a real negative.

One note, not blocking

The release-variant check ! grep -q "^PITHEAD_REGISTRY=" "$ROOT/etc/environment" also passes when the file is absent (grep rc 2), so it green-lights "no pin" and "no file" alike. Harmless today, since the Dockerfile always writes it — but the debug side pins its own file's existence via PITHEAD_ENGINE and the release side does not. grep -q '^PITHEAD_ENGINE=' "$ROOT/etc/environment" && ! grep -q '^PITHEAD_REGISTRY=' … would state the intent and keep the two sides symmetric.

What I did NOT do

• No image build and no KVM run. Your two controls (the host build, and --phase install at this same 87aea589) are yours; I am relaying them as yours and have verified neither.
• I did not re-check verify_tarball_commit over the appended tar — but the append path is pre-existing (Debug-variant image can provision from a non-default registry namespace end to end #1892 already appended the drop-ins and the trust files through that same line 181), so this change adds one more file to a set that already worked and alters nothing about that step.
• I did not review the docs hunk for house voice beyond reading it for accuracy against the code, which it matches.

State as measured

• Head 87aea589, 22 contexts: 20 success + 2 skipped, zero pending and zero failing, read at source just now.
• This PR is a DRAFT, so it cannot merge whatever its review state says. That is the only thing standing between it and the seat's list.
• I am not the author and I do not merge; os/ is a freeze path and the seat decides.

[VijitSingh97]

KVM negative control for this PR — RUN 1 on tree 09516059 (#1947 + all three #1934 commits, WITHOUT this PR): rc 1, 34 ✓ / 1 ✗, evidence ~/kvm1945-install-20260906T1256Z on the build box.

The validity condition the fixes lane set (the control isolates #1931 only if the provision SUCCEEDED and the restart pull FAILED) is met on both halves:

• provision succeeded: ✓ restore leg: provisioning finished (units: inactive inactive ), the backup archive was written to /data/pithead/backups/;
• the pithead backup -y restart over SSH pulled ghcr.io/p2pool-starter-stack/pithead-*:v2.0.0 and got {"message":"manifest unknown"} on all five images, then Error: executing /usr/local/bin/docker-compose up --pull missing -d: exit status 1 → ✗ restore leg: could not take the source backup.

So on a tree where the registry pin lives only in the boot units (#1915), an SSH-driven pithead verb still resolves the public registry. That is the #1931 defect reproduced end-to-end in the guest, by the path this PR changes.

Run 2 launches now on the same tree plus this PR cherry-picked (4c38322b, patch-id identical to 87aea589): a green restore leg there is the positive control. I flip this PR to ready only on that result.

[VijitSingh97]

Follow-up at 2f88e420dc8f1fa147044c2539dea698e6ccea6b covers the missing system-service launch path.

/etc/environment covers login/SSH sessions but is not automatically imported by systemd system services. The debug image now stages the registry drop-in for the runtime-created pithead-control.service as well as boot, firstboot/install and setup-again. That control service is the entry for dashboard apply, backup/lifecycle restart and OS-update actions; systemd combines its /run main unit with the baked /etc drop-in.

Run on this head: Bash syntax, shfmt, ShellCheck and diff checks PASS. A synthetic debug root passes with all four drop-ins, and deleting only the control-service drop-in fires the exact expected verifier row. A synthetic release root passes without test markers, and deleting its environment file fires the strengthened release-negative row.

Independent current-head static review: security-reviewer (Codex Sol/high) PASS and verifier (Codex Luna/high) PASS for this registry propagation, both at the exact head above. No KVM or heavy image build was run. The earlier interrupted run remains no verdict, and this PR remains draft pending positive combined KVM evidence.

Lane disclosure: os/, its verifier and the release documentation are within the bounded appliance assignment. The worktree's missing private lane marker was restored before this commit; no override or force push was used.

[VijitSingh97]

Superseded by #2002, which combines this branch with the other two interacting appliance changes so one exact-image KVM lifecycle covers all of them.

Containment verified mechanically against #2002's head d3a8a96c (every line this branch adds, normalised, checked for presence in that tree):

• fix(os): #1931 the debug variant pins the test registry into /etc/environment too #1939 — 0 of 23 added lines missing.
• test(os): #1945 the restore leg waits for the provisioning units before it backs up #1947 — 3 of 87 differ, all in provisioning_settled, where test(os): combine #1931 #1945 #1966 RC2 KVM gate #2002 strengthens the helper: word-anchoring kept, and the unanswered-probe branch now requires both unit states to be terminal rather than merely non-empty.
• test(os): cover RC2 appliance regressions #1982 — 13 of 665 differ; 11 are refinements test(os): combine #1931 #1945 #1966 RC2 KVM gate #2002 made to the same rows, and the two wiring-archive.tar.gz lines are already on develop in a better form (moved into lock_wiring_fixture and including Caddyfile), which is what left this branch conflicting.

Closing to keep one review surface. The branch is left in place; reopen if #2002 is split back up.