ci: detect unreleased work and silently skipped publishes

[abrichr]

Why

Four native releases here never published, and nothing said so.

tag	cause	run
desktop-v0.12.0	failed macOS x86_64 build	30169968004
desktop-v0.8.0	ungranted native-release environment approval	29799291231
desktop-v0.7.0	wholly cancelled run	29756479649
desktop-v0.1.0	ModuleNotFoundError: No module named 'tomllib' on a pre-3.11 runner	29514474536

Four different causes, one shape: nothing alerts when a human-gated publish job is skipped, cancelled, or fails. Each left an immutable public tag promising installers that were never published. native-release.yml has no failure notifier at all, and an if: failure() step could not have seen cancelled or skipped anyway.

The sibling failures elsewhere are the same shape: openadapt-capture PR #51 removed a path that uploaded raw audio waveforms to a third party, merged to main, and sat unreleased while PyPI's only installable version still contained it; and the launcher's Release and PyPI Publish runs 30275153205 / 30226357239 were cancelled seconds in.

What this is not

Nothing here publishes anything. The engine lane stays workflow_dispatch-only and the native lane keeps its native-release reviewer gate on publish-draft, the single point where installer bytes are written. That design exists because Flow 1.13.0 and 1.14.0 were yanked for shipping AGPL benchmark files. This adds visibility and prints the command a human would run.

What it is

scripts/check_release_health.py (stdlib only — no dependency install, no lockfile, no cache) evaluates three state-based detectors per release lane declared in .github/release-health.json (engine: v* + PyPI; native: desktop-v* + installers):

detector	fires when
unreleased-work	main carries commits past the newest release tag that would bump the version under this repo's own [tool.semantic_release.commit_parser_options], main is green, no release run is in flight, and the oldest such commit is past the grace window
tag-without-release	a release-shaped tag has no release object, no run for it is queued/waiting/running, and it is past the grace window — reporting failure/cancelled/skipped, or that the workflow never started at all
unpublished-release	the newest release tag's version is absent from PyPI past the CDN grace window

State-based rather than event-based on purpose. "A release run was cancelled" is not the alarm; "something is still unpublished" is. The launcher's release workflow cancels superseded runs constantly under its concurrency group, and a detector that fired on those would go the way of the launcher's platform-manifest drift check — correct, permanently red after every release, and therefore indistinguishable from noise.

Proof it fires: run live against this repository right now

ALERT [unreleased-work/engine] 3 releasable commit(s) unpublished for 19h 25m
ALERT [tag-without-release/native] 4 tag(s) published no release

The issue body it would file, verbatim:

unreleased-work — 3 releasable commit(s) unpublished for 19h 25m

main has been carrying 3 releasable commit(s) for 19h 25m past v0.14.0. main CI is green and no release run is in flight, so nothing is going to publish these on its own. Next version would be a minor bump.

commit	bump	age	subject
e4244961	minor	19h 25m	feat: capability-aware surface availability with precise remediation (Section 6) (#64)
7ab0d6bc	patch	1h 12m	fix: keep browser provisioning substrate-aware (#67)
3cb7adbc	patch	25m	fix(release): point "Latest" at the native installers (#69)

tag-without-release — 4 tag(s) published no release

tag	age	release run	outcome
desktop-v0.1.0	11d 10h	29514474536	failure
desktop-v0.7.0	6d 23h	29756479649	cancelled
desktop-v0.8.0	6d 11h	29799291231	cancelled
desktop-v0.12.0	1d 21h	30169968004	failure

cancelled and skipped are the outcomes no if: failure() step can see. A run cancelled while waiting on an environment approval gate lands here.

It independently reproduced all four evidence items with their exact run IDs and conclusions, from live state — no fixture involved.

False positives are the failure mode

--self-test enumerates every transient these repositories actually produce and proves the detector stays quiet on each. It runs offline in CI whenever the detector itself changes.

A. It FIRES on the states it exists to catch
  [PASS] unreleased fix: commit, green main, past grace -> alert
  [PASS] `type!:` breaking change -> alert
  [PASS] `BREAKING CHANGE:` footer -> alert
  [PASS] tag with a cancelled/failed/never-started publish -> alert
  [PASS] conclusion `skipped` (invisible to `if: failure()`) -> alert
  [PASS] tag exists but PyPI never got the version -> alert
B. It stays QUIET on every transient this repository actually produces
  [PASS] semantic-release + reconcile commits are `chore` -> quiet
  [PASS] launcher's bare `1.10.0` release commit is not a bump -> quiet
  [PASS] docs/ci/chore(deps) merges -> quiet
  [PASS] a fix merged 1h ago is inside the 4h grace -> quiet
  [PASS] a release run is in flight -> quiet
  [PASS] main is red -> quiet (a red main is its own signal)
  [PASS] main CI still running -> quiet
  [PASS] PyPI CDN has not propagated yet (10m) -> quiet
  [PASS] PyPI unreachable -> warning, never an alert
  [PASS] tag awaiting a human `native-release` approval -> quiet
  [PASS] tag pushed 90m ago, installers still building -> quiet
  [PASS] tag whose artifact IS on PyPI -> warning, never an alert
  [PASS] explicitly acknowledged historical tag -> quiet
  [PASS] launcher concurrency cancellations that left no gap -> quiet (the exact runs 30275153205/30276970325)
C. Commit parsing honours THIS repository's configured tags
  [PASS] `perf:` bumps only when patch_tags says so
  [PASS] `feat(scope): ...` -> minor
  [PASS] Merge commits and free text -> no bump

Two windows matter most here:

• The native-release approval gate. A run still waiting on it suppresses tag-without-release regardless of the tag's age. An unanswered human gate is a pending decision, not a failure. Only once the run reaches a terminal non-success conclusion — which is what happened to desktop-v0.8.0 — does it alert.
• Build duration. The native lane's grace is 6h because a full macOS/Windows/Linux Tauri build, smoke test, SBOM and attest legitimately takes far longer than a wheel.

Triggers and cost

• schedule every 3h — the actual detector; with the 4h grace it bounds "how long can a releasable commit sit unnoticed" to ~7h.
• workflow_run on Release and PyPI Publish and Native Installer Release completed with a non-success conclusion — this is the trigger that sees cancelled and skipped.
• pull_request on the detector's own files — runs --self-test.

One stdlib step, no dependency install, no cache. Python is pinned to 3.12 for the obvious reason, and the script imports tomllib defensively so an old interpreter degrades to documented defaults with a warning instead of repeating run 29514474536.

Issue behaviour

Exactly one open issue per repository, rewritten in place on each run (editing a body does not notify, so a persistent gap is not a daily ping) and closed automatically with a comment once every gap is closed. Never duplicated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM