Per-test known issues: green jobs for known failures, honest badges

[yarikoptic]

Most of the matrix is red for reasons we already understand. That hides new failures inside an already-red cell, and nothing notices when a known failure goes away (e.g. a git-annex daily build that fixes something). This PR records known failures per test and judges each cell against that list.

How a cell is judged

Cell outcome	CI job	Badge / status page
all tests pass	green	passing
every failure covered by a known issue	green	failing (known), still red
any failure no issue covers	red	N new failing
suite timed out or died, or its totals disagree with the parse	red	incomplete
a known issue's tests all pass	green, with a notice on the run	+N fixed? appended

What's in it

• evals/ replaces .github/ as the home for these files. evals/matrix.yaml (moved from .github/) says what runs. The new evals/known-issues.yaml records what is expected to fail. Each issue has an id, backend globs × targets, test ids (with prove-style ranges like chown/00.t#107-108,118-120, or "*" for a whole cell not yet narrowed down), expect: fail|flaky, tags from a validated list (fs-limitation, fs-divergence, test-assumption, build-config, harness, fixed-upstream, needs-triage, …), an optional fixed-in, and links. The initial list comes from run 36050917994.

• bin/ci/collect-results.py turns each suite's own output into per-test results:

  • TAP for git: the exact stream prove parses, kept per script as test-results/<script>.tap by a small prove --exec hook (bin/ci/git-prove-exec.sh) that wraps git's own t/run-test.sh;
  • TAP for pjdfstest (prove -v, now the default) and stress-ng (the driver now prints TAP lines);
  • tasty's console output for git annex test. Upstream TODO for a TAP log: provide TAP protocol logging for 'annex test'.

  Each parse is cross-checked against the suite's own totals and exit code. A mismatch makes the cell incomplete (red), so a parser that drifts after a format change cannot silently drop failures.

• bin/ci/known_issues.py has four subcommands:

  • validate checks the YAML against the matrix;
  • check judges one cell and writes verdict.json, a step summary, and annotations;
  • gotchas regenerates GOTCHAS.md's "Known issues" section, which replaces the hand-kept KNOWN_RED list;
  • draft prints issue stubs for new failures.

• Workflow (test.yaml):

  • The suite step is continue-on-error, and bin/ci/check-cell.sh decides the job result.
  • It is skipped on cancelled runs.
  • Validation runs as its own job.
  • results.tsv and verdict.json go into the per-cell artifacts.
  • update-status.py and render-report.py show states, which issues matched, new failures, and a Known-issues section.

• Shellcheck: bin/ci/shellcheck.sh finds scripts by shebang, the same discovery as improveit's shellcheckit, instead of a fixed glob. Master's checks workflow runs it via run-checks.sh shellcheck. All 25 scripts are clean.

Findings from building the initial list

• BeeGFS git-annex: down to 8 tests (export/import and git-remote-annex exporttree) that fail with a busy-device error on rename, identical on 7.4.6 and 8.1.0. Tagged build-config, fixed-upstream and needs-triage: the OsPath build is the working hypothesis.
• Loop git-annex cells: the loop image is 100 MB, the same as git-annex's default annex.diskreserve, so many tests log "not enough free space". This is recorded as a whole-cell harness + needs-triage issue and not fixed here. Until the image is enlarged or the reserve lowered, the ext4 control row can't catch a git-annex regression.

Review

Three independent reviews covered the Python parsers and classifier, the shell and workflow wiring, and the data and docs. All findings were addressed in 539c110, including:

• prove's padded (Wstat: column, which meant no #exit rows were ever produced;
• a repeated TAP number that could overwrite a not ok;
• cancelled runs being recorded as incomplete;
• root-cause hypotheses that had been stated as facts.

What real CI runs caught

The cross-checks did their job: each problem marked the affected cells incomplete (red) rather than hiding a failure.

• git, attempt 1 (--verbose-log .out files): t0202 runs a Perl Test::More script, which adds a second TAP numbering to the .out file. Command output lacking a trailing newline also glues onto the next result line (Hello Worldok 12 - …).
• git, attempt 2 (--write-junit-xml): git v2.55.0, and still master, calls an undefined write_junit_xml_testcase for scripts that skip everything. That garbles their TAP and fails make prove.
• git, now: the per-script copy of prove's own TAP stream. On a local v2.55.0 build over the full t0*/t1* selection CI runs, all 10,359 test points reconcile with prove, t0202 included. It also handles synthetic scripts that fail, fix a known breakage, keep a known breakage, skip everything, or die midway.
• git-annex: merging stderr into the logged pipe spliced git-annex's messages into tasty's result lines. Only stdout is logged now.

pjdfstest -v (including vfat's bail-out) and the stress-ng TAP worked on the first run. Any failures the initial list missed will show up as named "new failures" in the step summary; known_issues.py draft prints stubs for them.

Testing

• The git-annex parser reproduces tasty's own counts on the real CI logs of the vfat and BeeGFS cells.
• The TAP adapters and every classification path (known, flaky, new, not reproduced, stale, fixed-in, timeout, never ran, exit-code/parse disagreement) were exercised on synthetic input.
• The status page render was dry-run locally, with a screenshot check.
• bin/ci/run-checks.sh (shellcheck + bats), codespell, and known_issues.py validate --gotchas pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P8dGtiFfZ6gRt2bWj39Qv3