Skip to content

ci(coverage): give the instrumented lane its own per-test budget so the coverage gate can run - #9474

Merged
os-try-charles merged 1 commit into
mainfrom
claude/issue-9271-coverage-gate-blind
Sep 14, 2026
Merged

ci(coverage): give the instrumented lane its own per-test budget so the coverage gate can run#9474
os-try-charles merged 1 commit into
mainfrom
claude/issue-9271-coverage-gate-blind

Conversation

@claude

@claude claude Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Part of #9271. ⛔ Deliberately NOT Fixes — this closes the proximate half only; the structural half named at the bottom stays open on the card.

What was measured

The blind window is 116 consecutive ci.yml push runs on main — runs 17230 → 17738, 2026-09-11T23:17:11Z2026-09-14T07:09:07Z, 2 days 9 h 42 m. Test (coverage) is failure in 116 of 116. Last run whose coverage lane was green: run 17227 (a94e4d073, 2026-09-11T21:27:02Z), which also published the only coverage-report artifact in the window.

objectui#8479's 66-run streak is NOT this streak. Population: the 500 most recent branch=main, event=push runs of workflow 223145786 (runs 15611 → 17738, from 2026-09-07T06:45Z). It holds 221 successes, the newest being run 17227 — so the 2026-09-07 streak healed and this is a second, later one. ⇒ the card's written re-grade trigger ("longer than roughly a week") does not fire; priority:p2 stands on duration.

The break in the chain, read rather than inferred

⭐ The blobs are not missing. On red run 17738 all four coverage-blob-N artifacts exist and are live (7.8–8.7 MB each); the green run 17227 carries the same four plus coverage-report. blobs=skipped in the sentinel is the counting step being skipped, not an absent blob.

The break is that coverage-report gates every step on needs.test-coverage.result == 'success'. One timed-out row reds a shard, so checkout, download, blob-count and merge are all skipped — and the merge is the only place coverage.thresholds is enforced, because the shard legs override all four to 0. Verbatim from run 17738:

shards=failure blobs=skipped merge=skipped report=skipped — no complete merged report was
produced for db6aa19a94b4893f4e78d0120db7a571a3c033c6, so the coverage thresholds were
never evaluated on this commit

Instrumentation isolated as the single variable

The same-commit CI comparison could never settle this: test-coverage is if: github.event_name == 'push' and test is if: github.event_name != 'push', so instrumentation and trigger event are confounded by construction. Isolated locally instead — same tree, same container, same command, --coverage the only difference:

population: the 146 rows of the two heaviest sweep files, tree db6aa19a9 no coverage coverage
rows exceeding the production 15000 ms budget 0 5
sum of all row durations 30.3 s 115.5 s (3.81×)
heaviest row 6.6 s 25.3 s

The five rows that blow the budget under instrumentation are exactly the five CI sites in those files, and the ordering corroborates: the marginal row (15.0 s locally) is the one that fires 11× in CI, while the 24–25 s rows fire 64–94×.

⚠️ The rows are NEW, not degraded — the corpus-growth hypothesis is false

Same instrumented command, same container, two trees, rows paired by name:

population: the 106 check-i18n-dead-keys rows present on BOTH trees a94e4d073 (last green) db6aa19a9 (today)
total 1.4 s 1.3 s (0.9×)
rows over 15 s 0 0

⇒ the shared rows did not slow down at all. All three of that file's timeout sites are rows that did not exist at the last green run, added 2026-09-12; and the single most frequent site overall, check-object-metadata-write-doors.test.ts, is an entire file that landed 2026-09-12T04:00Z. The tracked-file corpus grew only 3.2% (7492 → 7733) across the window, nowhere near enough to matter.

The repair, and what it does not weaken

A lane-scoped --testTimeout=60000 on the instrumented shard command — 4× the uninstrumented budget, 2.4× the heaviest row observed under instrumentation. ⛔ Not a per-row bump: the wandering site is exactly why it is applied to the lane.

  • vitest.config.mts keeps testTimeout: 15000, so the uninstrumented lane — the one that actually runs before a merge — is untouched. gates_weakened: NONE.
  • No job-budget risk: test-coverage has timeout-minutes: 40 against observed shard wall clocks of 698–1053 s.

Evidence the gate is restored

⭐ A green shard is not evidence a report was produced, so here is the report and the thresholds. CI cannot host this demonstration — the coverage lane runs only on push to [main, develop], never on a PR — so it is reproduced locally over the five CI timeout-site files:

leg shard exit merge thresholds
--testTimeout=15000 1 (10 timeouts, 0 assertion failures) never runs — gated on shard success never evaluated
--testTimeout=60000 0 (379/379 pass) runs evaluated

The merged report from the passing leg, produced by ci.yml's own merge command:

Statements   : 75% ( 1924/2565 )
Branches     : 75.69% ( 1501/1983 )
Functions    : 74.83% ( 232/310 )
Lines        : 77% ( 1634/2122 )

⭐ Firing control that the thresholds are genuinely enforced and not merely printed — same blobs, same command, floor raised above the measured value:

$ pnpm test:coverage --merge-reports --coverage.reporter=text --coverage.thresholds.lines=99
ERROR: Coverage for lines (77%) does not meet global threshold (99%)
exit 1

⛔ The half this PR does NOT fix — it needs a decision above this card

The coverage lane runs only on push to main/develop. An author adding a 6-second repo-sweep row cannot see this lane before merging; the first signal is after the row is already on main, on a red that blocks nothing. That is why the class recurs — this file's own comment records objectui#8545 pricing two earlier episodes at 84 and 87 consecutive unevaluated pushes, and this one ran to 116.

⇒ A budget alone postpones the next episode, it does not prevent it. Making the lane visible pre-merge is the structural fix, and the card rules out making it blocking — so the fork is the maintainer's, not this PR's.

Verification

  • pnpm exec vitest run --project unit scripts/__tests__/157 files / 4625 tests passed, exit 0 (this is the surface holding every gate that reads ci.yml).
  • node scripts/check-changeset-presence.mjs — exit 0, "no changeset is owed".
  • node scripts/check-governed-queue-guard.mjs --test .github/workflows/ci.yml — NOT GOVERNED (control: AGENTS.md returns GOVERNED, exit 3).

🤖 Generated with Claude Code

https://claude.ai/code/session_013VGeMu3p6qEFWR6K6GGLaW


Generated by Claude Code

`vitest.config.mts` sets `testTimeout: 15000`, calibrated for the
uninstrumented lane. The coverage lane runs the same files under v8
instrumentation, where the repo-sweeping rows under `scripts/__tests__/`
take several times longer. Measured on this tree with the coverage flag
set as the only variable: zero of the 146 rows in the two heaviest sweep
files exceed 15s uninstrumented, five exceed it instrumented.

A timed-out row is not a failed gate, it is no gate. `coverage-report`
gates every merge step on `needs.test-coverage.result == 'success'`, so
one dead row skips the merge and the thresholds are never evaluated.

The uninstrumented lane keeps its 15s budget unchanged, so nothing that
guards day-to-day work is relaxed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VGeMu3p6qEFWR6K6GGLaW
@os-try-charles
os-try-charles marked this pull request as ready for review September 14, 2026 08:24
@os-try-charles
os-try-charles added this pull request to the merge queue Sep 14, 2026
Merged via the queue into main with commit 511e402 Sep 14, 2026
35 checks passed
@os-try-charles
os-try-charles deleted the claude/issue-9271-coverage-gate-blind branch September 14, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants