Release v1.19.2 - #1113
Merged
Merged
Conversation
…1024) A release bundle carries the CLI, the compose file, the config files and cosign.pub — deliberately no docs/. Nine operator messages named a repo-relative doc anyway, so on a release install (most of the fleet) they read like a local path and resolve to nothing. The cosign-missing refusals were the worst case: the one operator who sees them is mid-refusal on a box they cannot upgrade. Add DOCS_URL and route all nine through it — main is the released-only branch, so it is the documentation for the version an operator is actually running. Bundles stay lean; no docs/ is added to the artifact. Guard it with a second rule in lint-operator-strings, which already exists to police this text. The new rule runs at full width — the doctor's dr_* reporters and the runner's _upg_* result strings included — because three of the nine hits live in shapes the #NNN rule never scanned. The #NNN rule stays narrow: those shapes carry twelve references predating the linter, filed as #1026 rather than buried here. The rule found a ninth site by itself, a describe_change preview a hand grep had missed. The script's --self-test harness was never wired to anything, so its scanners had no check of their own; make lint-operator-strings now runs it before the live scan, and it covers both new rules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ers-in-operator-text
…ers-in-operator-text
chore: back-merge main into develop after v1.19.1
A failed upgrade can leave the box-global control units naming a directory that is no longer the install. Nothing reports it. The dashboard keeps writing requests into its own spool, the path unit keeps watching a tree nobody writes to, and the config editor and one-click upgrade simply never complete — no error on either side. Found live on the production box: a v1.18.1 upgrade repointed the units, then aborted at the cosign gate and rolled the stack back to v1.17.0, leaving the units on the half-installed tree. Two operator upgrade requests (v1.19.0, then v1.19.1) sat unread in the live spool for a day. `doctor` reported a healthy box throughout, because it had no check for this at all. #1070 fixed the ordering that causes it, but it cannot reach a box already stranded: the channel the fix would arrive through is the broken one. So the diagnostic has to be able to see it. - `control_units_owner_dir` lifts the ExecStart parse and physical-path resolution out of `provision_control_runner`'s removal branch, which now calls it instead of repeating it. Net fewer lines there. - `check_control_units` reports, under a new "Dashboard control channel" heading: units missing while control is on, units owning a different directory (naming both paths), and a path unit watching a different spool than this install writes to. Disabled control is INFO, not a fault. - The two spellings of one checkout — the versioned dir and the `current` symlink — resolve to one install, so a production box is not told it is broken. Tier 1, per the testing strategy: this is decision logic over a parsed unit file, provable with stubs. Mutation proof, run before this was committed — every assertion goes red under the defect it guards: drop the owner-mismatch arm -> "units point elsewhere" goes green flip that `!=` to `=` -> mismatch AND aligned both flip drop the spool comparison arm -> "different spool" goes green drop the `-z "$owner"` arm -> "no units installed" loses its message That last one is why the no-units assertion checks the message rather than bare FAIL: with no units the owner is empty, so the mismatch arm fires anyway and an assertion for "FAIL" would have stayed green with the arm it names deleted.
The new check landed between check_stratum_exposure's doc comment and the function it describes, so that comment read as documentation for check_control_units. Moved below check_stratum_exposure's closing brace. No behaviour change: mutation proof re-run (all four mutations still turn their assertion red) and tests/stack/run.sh still 1735 passed, 0 failed.
… real
Two blocking defects found by an independent review of this branch before
merge. Both were reproduced in a container harness, and both are fixed here.
1. FALSE FAIL ON THE DOCUMENTED LAYOUT — the worse of the two.
The canonical layout keeps the previous version dir for rollback, and the
one-click upgrade runs in the browser, so an operator's shell is routinely
still sitting in that older dir. Comparing the units against $PWD called
that healthy box broken:
FAIL The control runner units point at .../pithead-v1.19.1, but this
install is .../pithead-v1.19.0 — ... silently never run.
Worse than noise: following the printed fix from there repointed the units
AT the rollback dir, which breaks the live install for real. A check meant
to catch a silent failure could cause it.
A versioned dir that `current` no longer names is a superseded copy, not a
stranded install. It now reports INFO naming the live dir, and gives no
verdict — verdicts about the control channel belong to the live install.
Recognised with the same pattern update_current_symlink already uses.
2. THE PRESCRIBED FIX WAS A NO-OP.
All three messages say to run './pithead apply'. A box whose units point at
a dead install has an unchanged config by definition — the fault is in the
unit files, not config.json — so apply returned at "No configuration
changes detected" nine lines before provision_control_runner. The fix
printed on the only box the check fires for did nothing.
apply now converges the control units before that return. It is idempotent
and sudo-free when they already match, and apply's own comment already
claimed it converged them.
Mutation proof for both new assertions, run before committing:
delete the `current`-symlink guard -> superseded dir goes back to FAIL
remove the call from apply's no-change branch -> "provision" disappears
tests/stack/run.sh: 1739 passed, 0 failed. make lint: clean.
…-unit-target fix(cli): doctor names control units that point at another install
Both scripts opened an absolute path inside a specific user's home directory, which is neither reproducible for anyone else nor something a public repo should carry. They now take the log path as an argument and default to a relative one — one line each, and it makes them runnable. Companion to the develop-v2 fixture cleanup; this is the only occurrence on develop.
…ath-in-research-scripts chore: stop hardcoding a home directory in the research analysis scripts
…ers-in-operator-text # Conflicts: # pithead
…lp's heredoc The rule this PR added reported the tree clean while `show_help` printed a bare `docs/operations.md` to every operator who runs `pithead --help`. msg_lines only ever saw call shapes — log/warn/error/info/echo and describe_change's msg= — so a heredoc body was invisible, and the review that introduced the rule inspected the same blind extractor. The extractor now reads printed heredoc bodies: `cat <<EOF` with nothing after the tag. An opener carrying a redirect writes a file — a unit, a config, a template — which is not operator text, so the `$` anchor keeps those out. Both rules widen together, since a #NNN in a usage block is the same leak as a #NNN in an error. That widening then caught a false positive in the fix itself. `$DOCS_URL/docs/getting-started.md#1-prerequisites` is a GitHub heading anchor, which numbered headings render as `#1-…` — indistinguishable from an issue reference to a pattern looking for `#NNN`. A $DOCS_URL pointer is now stripped before the issue-number check, the same way it already is before the docs-path check. Three self-test cases cover the extractor (printed heredoc flagged, redirected heredoc exempt, #NNN in a heredoc flagged) and one covers the anchor exemption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t by a list of keys Whether `apply` restarts caddy was decided by a hand-maintained list of .env keys. A list like that only works while someone remembers to extend it, and it had already drifted: on the appliance, `dashboard.expose_public_ip` is not in it, so turning OFF the opt-in that serves the dashboard on a globally-routable address re-rendered the Caddyfile without its bind lines and left caddy holding the wildcard listener. The operator sees the setting saved and the file change, and the box stays exposed (#1052). The guard is now the content: snapshot the Caddyfile, render, restart if it differs. That covers every input at once, including the ones nobody has added yet, and it cannot drift — the cause no longer has to be enumerated, only the effect observed. An absent previous file is a fresh install: compose starts caddy on the new one moments later and there is no old configuration to displace, so that case stays quiet. Note the key this was found through does not exist on this branch — it is appliance-only. What is shared is the fragile mechanism, so the fix lands here and reaches the appliance through the routine sync, where the expose_public_ip assertion belongs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…file Emptiness and absence were conflated: a zero-byte Caddyfile left by a crashed render read as "no previous file", so the restart was skipped on exactly the box whose caddy is serving nothing. File existence is now tracked separately from content. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tart-on-any-change fix(cli): restart caddy when the rendered Caddyfile changes, not when a listed key does
…ers-in-operator-text fix(cli): point operator messages at published docs, not repo paths (#1024)
#1108) A release could ship unsigned; an install could not accept one. `release.sh` warned and published when the signing environment was absent, while the one-click upgrade runner refuses any release with no `pithead.tar.gz.sig`. A cut on an unconfigured box therefore produced a bundle every install in the fleet rejects — and release assets are immutable, so the signature could never be attached afterwards. That is how v1.18.0 had to be withdrawn. Three releases shipped unsigned this way while `cosign.pub` was uncommitted and installs merely warned; it is committed now, so the same mistake is a hard refusal. - Signing is mandatory to publish (#960). Preflight aborts when `cosign.pub` is committed and any of {cosign, COSIGN_KEY, COSIGN_PASSWORD} is missing, naming the gap and why it cannot be fixed after the publish. `--unsigned` is the explicit, loud escape hatch. COSIGN_PASSWORD was never checked before: cosign would have prompted for it at stage 6b, after the images were promoted. - The dry run rehearses the decision. It sat inside `if [ "$DRY_RUN" -eq 0 ]`, so the one check that exists to protect a cut could only ever print "signing OFF" — the failure state, unconditionally. - The cosign flags are probed, not assumed. cosign v3 removed `--tlog-upload`, which both signing calls pass; a drifted box passed every other check and then died at stage 6b with the images already promoted. - The pinned verifier is validated before publish (#1084). Preflight pulls COSIGN_IMAGE, signs a probe blob with this box's key, verifies it through the container, and requires a tampered blob to be refused. Since #1072 that image is the trust root for every install, and nothing checked it: the stack tests run against a fake docker, the e2e skips verification on a source checkout, and release-smoke runs after the assets are immutable. The round trip also catches a `cosign.pub` that is no longer the public half of `COSIGN_KEY`. - The release gate stops claiming to gate `main` (#1048). `release-gate.yml` carried a `push: [main]` trigger behind a repo variable nobody set, so every merge recorded a skipped — i.e. green — run of a job that has never executed. It is dispatch-only now, and `releasing.md` names which gates are automated and which are human-run, in one table. - `doctor` answers "can this box take an upgrade" before an upgrade does. It now reports whether the pinned verifier image is present and names the pre-fetch command; if that one fetch fails mid-upgrade the operator sees a *signature* failure, which reads as tampering rather than as an image the host could not pull. The source-checkout line says one-click does not apply there. - The legacy cliff is documented. v1.18.1 and v1.19.0 verify with a host binary and an upgrade runs the installed code, so those boxes refuse until cosign is installed once — with the pinned v2.6.3, because v3 satisfies `command -v` and then fails the verify. Written up in `releasing.md` with the guards read from each released tag, and linked from `operations.md` where a stuck operator looks. Flagged in the doc as not yet driven on a live install. - The prose is reconciled. Signing is no longer "opt-in" anywhere, because the consumer became fail-closed and nothing had updated the docs. Coverage: 51 new tier-1 assertions, each mutation-proven (10 mutations run; every one goes red on exactly the named assertions and nothing else). Re-derived and NOT reproduced: #1108 states the upgrade runner's refusal of an unsigned bundle has no test that would go red if the refusal were deleted. It does — deleting it turns tests/stack/run.sh's three "missing signature" assertions red (1787/3 against a 1790/0 baseline). No test added there. Verified: tier-1 1790/0, dashboard 1721 passed at 97.01%, frontend, compose, integration selftest, fakes, and every lint except lint-proto (needs a Docker daemon this host does not have). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cosign v3 removed the flag from `sign` and `sign-blob` together, so one probe answers the question the loop was asking five lines to ask. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Make the cosign chain one contract, producer to consumer (#1108)
VERSION + dashboard pyproject in lockstep (a shell test fails if they diverge), uv.lock regenerated, and the CHANGELOG section written from the seven issues that shipped since v1.19.1. The headline is the cosign chain (#1108 / #960 / #1084 / #1048): a release could ship unsigned that no install would accept, the dry run could not report the real signing decision, the pinned verifier image was never validated before publish, and the release gate recorded green skipped runs of a job that had never run. Carries an upgrade note for installs still on v1.18.1 or v1.19.0, which verify with a host cosign binary and refuse until it is installed once — with v2.6.3, because v3 satisfies the check and then fails the verification. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
chore(release): prepare v1.19.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release merge for v1.19.2 — published and signature-verified,
maincatching up to the released state.mainis released-only and must stay an ancestor ofdevelop, so this is a real merge, never a squash. The tagv1.19.2isdevelop's tip (20111d6), so nothing landed after the cut and this is the clean straight-merge shape.Verified before this merge
cosign verify-blobon the published bundle →Verified OK; the dashboard image signature verifies and its digest matches what was promoted.pithead.tar.gz.sigis attached (96 bytes) — the asset whose absence this release makes impossible to ship by accident.Release signing ONandRelease verifier validated end to end— and the--dry-runrehearsal beforehand printed the same, which was not possible before this release.1.19.1 → 1.19.2over the download → verify → rollback-guard → extract path.doctorreportsControl runner units target this installand the pinned verifier present.🤖 Generated with Claude Code