discriminates: refuse a flag alongside --self-test, with exit 64 — NOT a verdict - #608
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe self-test path now detects unrecognized arguments. It reports a usage error to stderr and returns exit code 64. The module documentation defines this code separately from verdict codes. ChangesSelf-test argument validation
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to Self-test invocations with extra flags now fail clearly with usage exit code 64 rather than reporting a comparison verdict. The intended behavior is covered and no current merge-blocking risk remains. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…rdict is not a diagnosis (#611) The branch reported the exit code and DISCARDED the subject's output: ⛔ discriminates.py CONTROL FAILED (exit 1) — the checker's own control does not pass. ⇒ That is a verdict with no diagnosis. A reader has to reproduce the failure locally to learn anything. ⛔ AND FOR A PLATFORM-DEPENDENT FAILURE THAT IS IMPOSSIBLE FROM THE WRONG PLATFORM. Measured today: discriminates.py's control PASSES on macOS and exits 1 on CI/Linux. The gate's report contained nothing that could distinguish "which assertion failed" from "which machine ran it", so the finding was unreachable from the only machine I have. ⇒ `fout` already holds the output. Printing its FAIL/Traceback/Error lines costs nothing. ★ BOTH POLES, because "echo the output" is only useful if it ALSO says something when there is no output to echo: a control that NAMES its failing assertion -> the line is echoed a control that fails by EXIT CODE ALONE -> "failed by EXIT CODE alone and said nothing about why" The second matters: an empty section reads as "no detail available" when the truth is "the control declined to say". Those are different and the reader should not have to guess which. ⚠ HOW THIS WAS FOUND, and it is worth the line: I fixed discriminates.py's argument surface (#608), the gate then REACHED a control it had never reached — because UNVERIFIABLE preempts the control verdict — and that control failed on Linux. ⇒ UNVERIFIABLE was masking a genuinely failing control, and the gate could not tell me which one. Repairing the flag surface turned an invisible failure into a visible one with no diagnosis attached. gate --self-test 22 checks (was 20), exit 0 · exit-code-gate 0 · check-tools-index 0 · bash -n clean. Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ Co-authored-by: Jonathan Borduas <jonathan.borduas@toolkit3d.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…T a verdict Rebuilt on current main (which now carries #611's CONTROL FAILED diagnosis), so CI can say WHICH assertion fails on Linux. Content identical to 7379984. ⚠ THIS PR IS EXPECTED TO GO RED, and that is the finding. On main this tool is UNVERIFIABLE, which PREEMPTS the control verdict — so the gate has never run its control. Repairing the flag surface makes the gate reach it, and it FAILS on CI/Linux while passing on macOS. ⇒ UNVERIFIABLE was masking a genuinely failing control. #611 was landed first so this run reports which assertion, rather than only that one did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
7379984 to
7eb5169
Compare
… does not have CI/Linux said, via #611's new diagnosis: ⛔ discriminates.py CONTROL FAILED (exit 1) ── its own output, the failing lines only ── FAIL known-negative a self-inconsistent state -> 4: got 0 want 4 ⇒ `run()` uses shell=True, which is `/bin/sh` — bash on macOS, usually DASH on Linux. DASH HAS NO `$RANDOM`. It expands to EMPTY, so `echo $RANDOM$RANDOM$RANDOM` prints the same blank line twice, the two reads AGREE, and this tool CORRECTLY reports a stable state — exit 0 against an expected 4. ★ THE TOOL WAS RIGHT AND THE FIXTURE WAS NOT PORTABLE. Measured, two-poled: /bin/sh (macOS, bash) echo $RANDOM -> 29547 dash echo $RANDOM -> '' (empty) OLD fixture under dash: two calls DIFFER = NO ⇐ why CI failed NEW fixture under dash: two calls DIFFER = YES ⇐ the control can fire ⇒ `od -An -N8 -tx1 /dev/urandom` exists on both and differs per invocation. Control on the other side: `echo SAME` does not differ. ⛔ AND WHY IT SURVIVED TWO WEEKS. This tool's flag surface was UNVERIFIABLE — `--self-test --zzz-not-a-flag` exited 0 — and UNVERIFIABLE PREEMPTS the control verdict. So the gate had never run this control on Linux at all. Repairing the argument surface is what made it reachable; the failure was not introduced, it was UNCOVERED. ⇒ Three changes had to land in order for this to be findable: #608 the flag surface discriminates -> the gate REACHES the control #611 CONTROL FAILED prints its output -> the gate SAYS WHICH assertion this the fixture is portable -> the control passes for a reason ⚠ I could not reproduce it locally: the control passes on macOS before AND after. Every step of the diagnosis came from CI's own output, which is exactly what #611 was landed to make possible. test_discriminates 0 · --self-test 0, all four documented exits reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
The gate measured
discriminates.pyUNVERIFIABLE:--self-test --zzz-not-a-flagexited 0, so"the flag is matched and the rest DISCARDED — a control result here describes an invocation that
was only half read."
Fixing that uncovered a control that had been failing on Linux for two weeks.
1. The argument surface — and why exit 64, not 2
This file's exit space was already full, and every code in it is a verdict about the
comparison:
A mistyped flag is a statement about the invocation. Reusing
2would make "you typed itwrong" indistinguishable from "the check did not discriminate" — the exact exit-2 collision this
file's header records (#58), and the reason it already intercepts
--self-testbeforeparse_args. ⇒ 64 =EX_USAGE, documented beside the four verdicts and marked not a verdict.⚠ The identical one-line fix used exit 2 for
pane-censusandpipe-exit-scan(#607) and wascorrect there — both already document "2 established nothing". A uniform patch would have
collided here and nothing would have said so.
2. ⛔ What that uncovered
With the flag surface repaired, the gate reached a control it had never reached — UNVERIFIABLE
preempts the control verdict — and the control failed on CI/Linux while passing on macOS:
Cause:
run()usesshell=True, i.e./bin/sh— bash on macOS, usually dash on Linux, anddash has no
$RANDOM. It expands to empty, soecho $RANDOM$RANDOM$RANDOMprints the sameblank line twice, the two reads agree, and the tool correctly reports a stable state.
★ The tool was right; the fixture was not portable. Measured, two-poled:
⇒
od -An -N8 -tx1 /dev/urandomexists on both and differs per invocation. Control on the otherside:
echo SAMEdoes not differ.⇒ Three changes had to land in order for this to be findable
⚠ I could not reproduce it locally at any point — the control passes on macOS before and
after. Every step of the diagnosis came from CI's own output, which is precisely what #611 was
landed to make possible.
discriminates.py --self-testtest_discriminates.pyhermetic suites (gating)on CI/Linux🤖 Generated with Claude Code
https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ