From 8462b78ff2f4efdcc6b21bd79e5b498b6b71de67 Mon Sep 17 00:00:00 2001 From: Jonathan Borduas Date: Sun, 6 Sep 2026 08:56:54 +0100 Subject: [PATCH] =?UTF-8?q?gate:=20on=20CONTROL=20FAILED,=20print=20the=20?= =?UTF-8?q?control's=20own=20failing=20lines=20=E2=80=94=20a=20verdict=20i?= =?UTF-8?q?s=20not=20a=20diagnosis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The branch reported the exit code and DISCARDED the subject's output: ⛔ discriminates.py CONTROL FAILED (exit 1) — the checker's own control does not pass. ⇒ That is a verdict with no diagnosis. A reader has to reproduce the failure locally to learn anything. ⛔ AND FOR A PLATFORM-DEPENDENT FAILURE THAT IS IMPOSSIBLE FROM THE WRONG PLATFORM. Measured today: discriminates.py's control PASSES on macOS and exits 1 on CI/Linux. The gate's report contained nothing that could distinguish "which assertion failed" from "which machine ran it", so the finding was unreachable from the only machine I have. ⇒ `fout` already holds the output. Printing its FAIL/Traceback/Error lines costs nothing. ★ BOTH POLES, because "echo the output" is only useful if it ALSO says something when there is no output to echo: a control that NAMES its failing assertion -> the line is echoed a control that fails by EXIT CODE ALONE -> "failed by EXIT CODE alone and said nothing about why" The second matters: an empty section reads as "no detail available" when the truth is "the control declined to say". Those are different and the reader should not have to guess which. ⚠ HOW THIS WAS FOUND, and it is worth the line: I fixed discriminates.py's argument surface (#608), the gate then REACHED a control it had never reached — because UNVERIFIABLE preempts the control verdict — and that control failed on Linux. ⇒ UNVERIFIABLE was masking a genuinely failing control, and the gate could not tell me which one. Repairing the flag surface turned an invisible failure into a visible one with no diagnosis attached. gate --self-test 22 checks (was 20), exit 0 · exit-code-gate 0 · check-tools-index 0 · bash -n clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ --- scripts/gate-selftests.sh | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/scripts/gate-selftests.sh b/scripts/gate-selftests.sh index 82e0b1e..ea6f504 100755 --- a/scripts/gate-selftests.sh +++ b/scripts/gate-selftests.sh @@ -403,7 +403,23 @@ classify_one() { echo " control that ran and concluded nothing." fi ;; *) echo "@@VERDICT broke $b" - echo " ⛔ $b CONTROL FAILED (exit $rc) — the checker's own control does not pass." ;; + echo " ⛔ $b CONTROL FAILED (exit $rc) — the checker's own control does not pass." + # ⛔ AND SAY WHICH CHECK. Before this, the gate reported the exit code and DISCARDED + # the subject's own output — so a reader had to reproduce the failure locally to + # learn anything. ⚠ For a PLATFORM-DEPENDENT failure that is impossible from the + # wrong platform: discriminates.py's control passes on macOS and exits 1 on + # CI/Linux, and the gate's report contained nothing that could distinguish "which + # assertion" from "which machine". Measured 2026-09-06. + # ⇒ `fout` already holds it. Printing the FAIL lines costs nothing and is the + # difference between a verdict and a diagnosis. + _fails=$(printf '%s\n' "$fout" | grep -E "FAIL|Traceback|Error" | head -6) + if [ -n "$_fails" ]; then + echo " ── its own output, the failing lines only ──" + printf '%s\n' "$_fails" | sed 's/^/ /' + else + echo " ⚠ its output carried no FAIL/Traceback/Error line — the control" + echo " failed by EXIT CODE alone and said nothing about why." + fi ;; esac } @@ -622,6 +638,22 @@ selftest() { out=$(gate "$d" 'c_broken.py' 2>&1); rc=$? check "a FAILING control exits 1 and is named — distinct from UNESTABLISHED" 1 \ "CONTROL FAILED" "c_broken" + + # ⛔ A VERDICT IS NOT A DIAGNOSIS. Both poles, because "echo the output" is only useful + # if it ALSO says something when there is no output to echo. + # ★ POLE A — the control names its failing assertion; the gate must SHOW it. + plant_py "$d/c_talks.py" 'if a == ["--self-test"]:' \ + ' print(" FAIL known-negative pair -> 3: got 1 want 3")' \ + ' sys.exit(3)' 'if a: void()' 'sys.exit(0)' + out=$(gate "$d" 'c_talks.py' 2>&1); rc=$? + check "a failing control's OWN failing line is echoed, not just its exit code" 1 \ + "CONTROL FAILED" "the failing lines only" "known-negative pair -> 3" + # ★ POLE B — c_broken.py fails by exit code alone and prints nothing. The gate must say + # SO, rather than printing an empty section that reads as "no detail available". + out=$(gate "$d" 'c_broken.py' 2>&1); rc=$? + check "a control that fails SILENTLY is reported as saying nothing about why" 1 \ + "CONTROL FAILED" "failed by EXIT CODE alone" + rm -f "$d/c_talks.py" case "$out" in *"c_broken UNESTABLISHED"*) echo " FAIL a failing control was labelled UNESTABLISHED"; ok=1 ;; *) echo " ok known-negative: a failing control never reads UNESTABLISHED" ;;