gate: on CONTROL FAILED, print the control's own failing lines — a verdict is not a diagnosis - #611
Merged
Merged
Conversation
…rdict is not a diagnosis
The branch reported the exit code and DISCARDED the subject's output:
⛔ discriminates.py CONTROL FAILED (exit 1) — the checker's own control does not pass.
⇒ That is a verdict with no diagnosis. A reader has to reproduce the failure
locally to learn anything.
⛔ AND FOR A PLATFORM-DEPENDENT FAILURE THAT IS IMPOSSIBLE FROM THE WRONG
PLATFORM. Measured today: discriminates.py's control PASSES on macOS and exits 1
on CI/Linux. The gate's report contained nothing that could distinguish "which
assertion failed" from "which machine ran it", so the finding was unreachable
from the only machine I have.
⇒ `fout` already holds the output. Printing its FAIL/Traceback/Error lines costs
nothing.
★ BOTH POLES, because "echo the output" is only useful if it ALSO says something
when there is no output to echo:
a control that NAMES its failing assertion -> the line is echoed
a control that fails by EXIT CODE ALONE -> "failed by EXIT CODE alone and
said nothing about why"
The second matters: an empty section reads as "no detail available" when the
truth is "the control declined to say". Those are different and the reader
should not have to guess which.
⚠ HOW THIS WAS FOUND, and it is worth the line: I fixed discriminates.py's
argument surface (#608), the gate then REACHED a control it had never reached —
because UNVERIFIABLE preempts the control verdict — and that control failed on
Linux. ⇒ UNVERIFIABLE was masking a genuinely failing control, and the gate could
not tell me which one. Repairing the flag surface turned an invisible failure
into a visible one with no diagnosis attached.
gate --self-test 22 checks (was 20), exit 0 · exit-code-gate 0 ·
check-tools-index 0 · bash -n clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
|
Warning Review limit reachedNext included review available in 48 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jobordu
pushed a commit
that referenced
this pull request
Sep 6, 2026
…T a verdict Rebuilt on current main (which now carries #611's CONTROL FAILED diagnosis), so CI can say WHICH assertion fails on Linux. Content identical to 7379984. ⚠ THIS PR IS EXPECTED TO GO RED, and that is the finding. On main this tool is UNVERIFIABLE, which PREEMPTS the control verdict — so the gate has never run its control. Repairing the flag surface makes the gate reach it, and it FAILS on CI/Linux while passing on macOS. ⇒ UNVERIFIABLE was masking a genuinely failing control. #611 was landed first so this run reports which assertion, rather than only that one did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
jobordu
pushed a commit
that referenced
this pull request
Sep 6, 2026
… does not have CI/Linux said, via #611's new diagnosis: ⛔ discriminates.py CONTROL FAILED (exit 1) ── its own output, the failing lines only ── FAIL known-negative a self-inconsistent state -> 4: got 0 want 4 ⇒ `run()` uses shell=True, which is `/bin/sh` — bash on macOS, usually DASH on Linux. DASH HAS NO `$RANDOM`. It expands to EMPTY, so `echo $RANDOM$RANDOM$RANDOM` prints the same blank line twice, the two reads AGREE, and this tool CORRECTLY reports a stable state — exit 0 against an expected 4. ★ THE TOOL WAS RIGHT AND THE FIXTURE WAS NOT PORTABLE. Measured, two-poled: /bin/sh (macOS, bash) echo $RANDOM -> 29547 dash echo $RANDOM -> '' (empty) OLD fixture under dash: two calls DIFFER = NO ⇐ why CI failed NEW fixture under dash: two calls DIFFER = YES ⇐ the control can fire ⇒ `od -An -N8 -tx1 /dev/urandom` exists on both and differs per invocation. Control on the other side: `echo SAME` does not differ. ⛔ AND WHY IT SURVIVED TWO WEEKS. This tool's flag surface was UNVERIFIABLE — `--self-test --zzz-not-a-flag` exited 0 — and UNVERIFIABLE PREEMPTS the control verdict. So the gate had never run this control on Linux at all. Repairing the argument surface is what made it reachable; the failure was not introduced, it was UNCOVERED. ⇒ Three changes had to land in order for this to be findable: #608 the flag surface discriminates -> the gate REACHES the control #611 CONTROL FAILED prints its output -> the gate SAYS WHICH assertion this the fixture is portable -> the control passes for a reason ⚠ I could not reproduce it locally: the control passes on macOS before AND after. Every step of the diagnosis came from CI's own output, which is exactly what #611 was landed to make possible. test_discriminates 0 · --self-test 0, all four documented exits reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
jobordu
added a commit
that referenced
this pull request
Sep 6, 2026
…T a verdict (#608) * discriminates: refuse a flag alongside --self-test, with exit 64 — NOT a verdict Rebuilt on current main (which now carries #611's CONTROL FAILED diagnosis), so CI can say WHICH assertion fails on Linux. Content identical to 7379984. ⚠ THIS PR IS EXPECTED TO GO RED, and that is the finding. On main this tool is UNVERIFIABLE, which PREEMPTS the control verdict — so the gate has never run its control. Repairing the flag surface makes the gate reach it, and it FAILS on CI/Linux while passing on macOS. ⇒ UNVERIFIABLE was masking a genuinely failing control. #611 was landed first so this run reports which assertion, rather than only that one did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ * discriminates: the self-inconsistent control used $RANDOM, which dash does not have CI/Linux said, via #611's new diagnosis: ⛔ discriminates.py CONTROL FAILED (exit 1) ── its own output, the failing lines only ── FAIL known-negative a self-inconsistent state -> 4: got 0 want 4 ⇒ `run()` uses shell=True, which is `/bin/sh` — bash on macOS, usually DASH on Linux. DASH HAS NO `$RANDOM`. It expands to EMPTY, so `echo $RANDOM$RANDOM$RANDOM` prints the same blank line twice, the two reads AGREE, and this tool CORRECTLY reports a stable state — exit 0 against an expected 4. ★ THE TOOL WAS RIGHT AND THE FIXTURE WAS NOT PORTABLE. Measured, two-poled: /bin/sh (macOS, bash) echo $RANDOM -> 29547 dash echo $RANDOM -> '' (empty) OLD fixture under dash: two calls DIFFER = NO ⇐ why CI failed NEW fixture under dash: two calls DIFFER = YES ⇐ the control can fire ⇒ `od -An -N8 -tx1 /dev/urandom` exists on both and differs per invocation. Control on the other side: `echo SAME` does not differ. ⛔ AND WHY IT SURVIVED TWO WEEKS. This tool's flag surface was UNVERIFIABLE — `--self-test --zzz-not-a-flag` exited 0 — and UNVERIFIABLE PREEMPTS the control verdict. So the gate had never run this control on Linux at all. Repairing the argument surface is what made it reachable; the failure was not introduced, it was UNCOVERED. ⇒ Three changes had to land in order for this to be findable: #608 the flag surface discriminates -> the gate REACHES the control #611 CONTROL FAILED prints its output -> the gate SAYS WHICH assertion this the fixture is portable -> the control passes for a reason ⚠ I could not reproduce it locally: the control passes on macOS before AND after. Every step of the diagnosis came from CI's own output, which is exactly what #611 was landed to make possible. test_discriminates 0 · --self-test 0, all four documented exits reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ --------- Co-authored-by: Jonathan Borduas <jonathan.borduas@toolkit3d.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
CONTROL FAILEDbranch reported the exit code and discarded the subject's output:⇒ That is a verdict with no diagnosis. A reader has to reproduce the failure locally to learn
anything.
⛔ And for a platform-dependent failure, that is impossible from the wrong platform
Measured today:
discriminates.py's control passes on macOS and exits 1 on CI/Linux. The gate'sreport contained nothing that could distinguish which assertion failed from which machine ran
it — so the finding was unreachable from the only machine I have.
foutalready holds the output. Printing itsFAIL/Traceback/Errorlines costs nothing.★ Both poles
The second matters: an empty section reads as "no detail available" when the truth is "the
control declined to say." Those are different and the reader should not have to guess which.
⚠ How this was found
I fixed
discriminates.py's argument surface (#608). The gate then reached a control it had neverreached — because UNVERIFIABLE preempts the control verdict — and that control failed on Linux.
⇒ UNVERIFIABLE was masking a genuinely failing control, and the gate could not say which
assertion. Repairing the flag surface turned an invisible failure into a visible one with no
diagnosis attached. This PR attaches the diagnosis; #608 stays open until its Linux failure is
understood.
gate-selftests.sh --self-testscripts/exit-code-gate.sh tools 'test_*.py'scripts/check-tools-index.pybash -n🤖 Generated with Claude Code
https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ