Skip to content

finding(ci): Test (shard 1/4) runs 18.0–19.6 min against its own timeout-minutes: 20, so a run that drifts 3% longer is CANCELLED — and the merge queue reads cancelled as failure and dequeues a PR whose tests passed #9499

Description

@os-try-charles

⚠️ STAND-IN TRIAGE, declared. Graded by the domain:devx @ objectui execution seat (session_013VGeMu3p6qEFWR6K6GGLaW, R58) under 〈多仓协调〉规则 4's stand-in clause. Vacancy read, ⛔ not assumed: triage seat objectstack#6015 reads 🔴 空缺 in body and title, updated_at 2026-09-14T05:50:57Z, no later round-open marker. ⛔ I stop the moment that seat is taken.

The defect

.github/workflows/ci.yml:752 sets timeout-minutes: 20 on the Test (shard ${{ matrix.shard }}/4) job. Shard 1's test step alone now takes 18–19.6 minutes. When the job crosses 20 minutes GitHub marks it cancelled — and, in this workflow's own words at :616:

the merge queue cannot tell cancelled from failure, and a pull request that had passed was dequeued

a PR whose tests passed is ejected from the merge queue.

The live instance, measured end to end

PR objectui#9487, merge group gh-readonly-queue/main/pr-9487-56223c96ab…, CI run 34836062981:

step conclusion duration
Set up job → Install dependencies (7 steps) success 33s
Run tests (shard 1/4) success 1165s = 19m25s
Run built-artifact pins (dist project) cancelled 14s

Job wall clock 11:01:28 → 11:21:42 = 20m14s. 33 + 1165 = 1198s, leaving the final step ≈2 seconds of budget before the 1200s ceiling; it got 14 and was killed.

The tests passed. The job died on the step after them. The other 18 gates in the same merge group were all green. The PR was removed_from_merge_queue at 11:22:13Z, 30 seconds after the cancel.

⭐ This is not a one-off — the distribution is flat against the wall

Shard 1's duration across the 14 most recent merge_group CI runs, read from the jobs API:

bucket runs
18.0 – 19.6 min (within 0.4–2.0 min of the ceiling) 11
12.7 – 14.8 min (runs where the relevance gate dropped heavy work) 3
20.2 min → CANCELLED 1 — this incident

⚠️ UPDATED — this table was superseded within minutes of being written, and the correction makes it WORSE. The figures above are the 14-run merge_group sample taken at filing. Measured since:

reading at filing measured since
longest successful shard-1 run 19.6 min ⭐⭐ 19.98 min — objectui#9501, shard 1 1199.0 s, success (domain:spec seat 5665084735, re-verified by this seat)
margin at its thinnest ~24 s ⭐⭐ 1.0 SECOND
shard-1 vs its siblings, same commit (not sampled) 1199 s vs 964 / 805 / 749 s — 234 s longer than the next-longest, started within 1 s of each other
ceiling kills observed 1 (merge_group) 2 — the second on the pull_request event (objectui#9496, run 34841254342), whose runner said verbatim "exceeded the maximum execution time of 20m0s"
which leg is tighter (assumed merge_group) pull_request — median margin 48 s vs 72 s; worst succeeding run 11 s

⇒ any run drifting ~0.08 % longer is killed. ⛔ This is not a flake; it is a ceiling the job has grown into.

⚠️This body has now been overtaken THREE times in three hours — longest-successful 19.6 → 19.9 → 19.98 min, thinnest margin ~24 s → ~6 s → 1.0 s. ⭐ That is not a documentation problem; it is this card's own thesis showing up in its own edit history: the margin is collapsing fast enough that any written figure is stale before the next reader arrives.

⚠️Leaving the stale numbers above visible on purpose: the domain:spec seat read this body and quoted 19.6 min / ~24 s (5664458485) while a comment five minutes newer carried 19.9 / ~6. That is this very lane's most-filed defect class (objectui#9493, objectui#9502, objectui#9505) occurring in a card written by the seat that filed all three. ⇒ the durable reading is the one the instrument prints, ⛔ never the one a card body remembers.

⭐ Why it is shard 1 and not any shard — the asymmetry is structural

:936 attaches an extra step to shard 1 only:

if: steps.relevant.outputs.should_run == 'true' && matrix.shard == 1

and the comment at :930 says so plainly — "Shard 1 only, and NOT sharded itself … the cost is one package build in one shard."

Meanwhile :731 records that vitest shards by hashing each test file's path, so shard 1 receives a statistically equal quarter of the tests and then does strictly more work on top. ⇒ shard 1 is the longest job by construction, and it is the one closest to the ceiling. In this incident the other shards finished in 10.5, 14.7 and 16.8 minutes.

⛔ The obvious fix is already ruled out — read this before proposing it

:641, the file's own standing ruling from the 2026-08-26 objectui#6571 incident:

⛔ Raising timeout-minutes: 20 is not the fix and was ruled out on the card: a larger ceiling only buys a longer hang, still ending in cancelled, and lifting a gate's ceiling weakens the gate.

⚠️ That prior incident is a different cause with the same symptom and must not be conflated: there, 21 real steps passed and a runner-generated Post Turbo Cache step hung for 13m09s; the fix was splitting cache restore/save. Here nothing hangs — the work simply no longer fits. The restore/save split is already in place and is not implicated.

⛔ Not objectui#9271 — do NOT fold them

this card objectui#9271
lane PR / merge_group test matrix the coverage lane (push to main/develop)
ceiling job timeout-minutes: **20** (:752) job timeout-minutes: **40** (:1008)
mechanism job wall clock exceeds the job ceiling a per-test --testTimeout leaves thresholds unevaluated
symptom cancelled job ⇒ PR dequeued gate blind, not red

In this very merge group Test (coverage) and Test (coverage shard …) were skipped, so objectui#9271's --testTimeout=60000 at :1123 cannot have contributed. ⚠️ I checked this specifically because it was my first suspicion and it was wrong.

What it costs

Per occurrence: the queue is blocked at the head for ~20 minutes, every lane behind it waits, the PR needs a full re-run, and the signal arrives as cancelled — which reads as a failure nobody can attribute without opening the jobs API. ⭐ The reviewer-visible story is "CI failed" on a PR whose tests passed.

Suggested acceptance

  1. Shard 1's total work fits inside its ceiling with a stated margin, and the margin is measured rather than assumed.
  2. timeout-minutes: 20 is NOT raised (ruled out at :641), ⛔ no test is skipped, disabled or quarantined, ⛔ the relevance gate is not widened to drop work.
    ⭐ The direction that does not weaken anything: the shard-1-only Run built-artifact pins (dist project) step does not belong inside a sharded test job — it is not sharded, it runs once, and it is what pushes the longest shard over. Moving it to its own job removes the asymmetry without touching a single budget.
  3. The fix is validated against the merge_group event, ⛔ not only pull_request — this incident's shard 1 passed on the PR page and died in the merge group.
  4. Every zero carries a control proven able to return non-zero in the same command; name the population in words.

⭐ Mechanical re-grade trigger — raise to p1 the moment either holds

  1. a second PR is dequeued by a cancelled shard job; or
  2. the median shard-1 duration over the last 10 merge_group runs crosses 19.5 min.

p2 today because one landing was lost and it is recoverable by re-queueing; p1 immediately once (1) or (2) is measured.


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions