Skip to content

discriminates: refuse a flag alongside --self-test, with exit 64 — NOT a verdict - #608

Merged
jobordu merged 2 commits into
mainfrom
teamlead/discriminates-argv
Sep 6, 2026
Merged

discriminates: refuse a flag alongside --self-test, with exit 64 — NOT a verdict#608
jobordu merged 2 commits into
mainfrom
teamlead/discriminates-argv

Conversation

@jobordu

@jobordu jobordu commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

The gate measured discriminates.py UNVERIFIABLE: --self-test --zzz-not-a-flag exited 0, so
"the flag is matched and the rest DISCARDED — a control result here describes an invocation that
was only half read."

Fixing that uncovered a control that had been failing on Linux for two weeks.

1. The argument surface — and why exit 64, not 2

This file's exit space was already full, and every code in it is a verdict about the
comparison
:

0 discriminated · 2 non-discriminating · 3 control failed · 4 uninterpretable

A mistyped flag is a statement about the invocation. Reusing 2 would make "you typed it
wrong"
indistinguishable from "the check did not discriminate" — the exact exit-2 collision this
file's header records (#58), and the reason it already intercepts --self-test before
parse_args. ⇒ 64 = EX_USAGE, documented beside the four verdicts and marked not a verdict.

⚠ The identical one-line fix used exit 2 for pane-census and pipe-exit-scan (#607) and was
correct there — both already document "2 established nothing". A uniform patch would have
collided here and nothing would have said so.

2. ⛔ What that uncovered

With the flag surface repaired, the gate reached a control it had never reached — UNVERIFIABLE
preempts the control verdict — and the control failed on CI/Linux while passing on macOS:

FAIL  known-negative  a self-inconsistent state -> 4 (uninterpretable): got 0 want 4

Cause: run() uses shell=True, i.e. /bin/sh — bash on macOS, usually dash on Linux, and
dash has no $RANDOM. It expands to empty, so echo $RANDOM$RANDOM$RANDOM prints the same
blank line twice, the two reads agree, and the tool correctly reports a stable state.

The tool was right; the fixture was not portable. Measured, two-poled:

/bin/sh (macOS, bash)   echo $RANDOM  ->  29547
dash                    echo $RANDOM  ->  ''        (empty)

OLD fixture under dash: two calls DIFFER = NO    ⇐ why CI failed
NEW fixture under dash: two calls DIFFER = YES   ⇐ the control can fire

od -An -N8 -tx1 /dev/urandom exists on both and differs per invocation. Control on the other
side: echo SAME does not differ.

⇒ Three changes had to land in order for this to be findable

#608  the flag surface discriminates      ->  the gate REACHES the control
#611  CONTROL FAILED prints its output    ->  the gate SAYS WHICH assertion
this  the fixture is portable             ->  the control passes for a reason

I could not reproduce it locally at any point — the control passes on macOS before and
after. Every step of the diagnosis came from CI's own output, which is precisely what #611 was
landed to make possible.

BEFORE   43 control(s) passed · 3 UNVERIFIABLE
AFTER    44 control(s) passed · 2 UNVERIFIABLE   (fleet-identity landed separately in #610)
discriminates.py --self-test exit 0, all four documented exits reachable
test_discriminates.py exit 0
hermetic suites (gating) on CI/Linux success

🤖 Generated with Claude Code

https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ

@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 24842fc4-d36d-4cc8-b527-6ed608b0611f

📥 Commits

Reviewing files that changed from the base of the PR and between 5f917b5 and 7379984.

📒 Files selected for processing (1)
  • tools/discriminates.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The self-test path now detects unrecognized arguments. It reports a usage error to stderr and returns exit code 64. The module documentation defines this code separately from verdict codes.

Changes

Self-test argument validation

Layer / File(s) Summary
Document and enforce self-test argument rules
tools/discriminates.py
The module documents exit code 64. The --self-test path rejects extra arguments, reports a usage error, and returns 64 instead of running the self-test.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 7eb51

Self-test invocations with extra flags now fail clearly with usage exit code 64 rather than reporting a comparison verdict. The intended behavior is covered and no current merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: rejecting additional flags with --self-test and returning exit code 64 instead of a verdict.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch teamlead/discriminates-argv

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

jobordu added a commit that referenced this pull request Sep 6, 2026
…rdict is not a diagnosis (#611)

The branch reported the exit code and DISCARDED the subject's output:

    ⛔ discriminates.py CONTROL FAILED (exit 1) — the checker's own control does not pass.

⇒ That is a verdict with no diagnosis. A reader has to reproduce the failure
locally to learn anything.

⛔ AND FOR A PLATFORM-DEPENDENT FAILURE THAT IS IMPOSSIBLE FROM THE WRONG
PLATFORM. Measured today: discriminates.py's control PASSES on macOS and exits 1
on CI/Linux. The gate's report contained nothing that could distinguish "which
assertion failed" from "which machine ran it", so the finding was unreachable
from the only machine I have.

⇒ `fout` already holds the output. Printing its FAIL/Traceback/Error lines costs
nothing.

★ BOTH POLES, because "echo the output" is only useful if it ALSO says something
when there is no output to echo:

    a control that NAMES its failing assertion  -> the line is echoed
    a control that fails by EXIT CODE ALONE     -> "failed by EXIT CODE alone and
                                                   said nothing about why"

The second matters: an empty section reads as "no detail available" when the
truth is "the control declined to say". Those are different and the reader
should not have to guess which.

⚠ HOW THIS WAS FOUND, and it is worth the line: I fixed discriminates.py's
argument surface (#608), the gate then REACHED a control it had never reached —
because UNVERIFIABLE preempts the control verdict — and that control failed on
Linux. ⇒ UNVERIFIABLE was masking a genuinely failing control, and the gate could
not tell me which one. Repairing the flag surface turned an invisible failure
into a visible one with no diagnosis attached.

gate --self-test 22 checks (was 20), exit 0 · exit-code-gate 0 ·
check-tools-index 0 · bash -n clean.


Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ

Co-authored-by: Jonathan Borduas <jonathan.borduas@toolkit3d.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…T a verdict

Rebuilt on current main (which now carries #611's CONTROL FAILED diagnosis), so
CI can say WHICH assertion fails on Linux. Content identical to 7379984.

⚠ THIS PR IS EXPECTED TO GO RED, and that is the finding. On main this tool is
UNVERIFIABLE, which PREEMPTS the control verdict — so the gate has never run its
control. Repairing the flag surface makes the gate reach it, and it FAILS on
CI/Linux while passing on macOS. ⇒ UNVERIFIABLE was masking a genuinely failing
control. #611 was landed first so this run reports which assertion, rather than
only that one did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
@jobordu
jobordu force-pushed the teamlead/discriminates-argv branch from 7379984 to 7eb5169 Compare September 6, 2026 08:00
… does not have

CI/Linux said, via #611's new diagnosis:

    ⛔ discriminates.py CONTROL FAILED (exit 1)
       ── its own output, the failing lines only ──
         FAIL  known-negative  a self-inconsistent state -> 4: got 0 want 4

⇒ `run()` uses shell=True, which is `/bin/sh` — bash on macOS, usually DASH on
Linux. DASH HAS NO `$RANDOM`. It expands to EMPTY, so
`echo $RANDOM$RANDOM$RANDOM` prints the same blank line twice, the two reads
AGREE, and this tool CORRECTLY reports a stable state — exit 0 against an
expected 4.

★ THE TOOL WAS RIGHT AND THE FIXTURE WAS NOT PORTABLE. Measured, two-poled:

    /bin/sh (macOS, bash)   echo $RANDOM  ->  29547
    dash                    echo $RANDOM  ->  ''        (empty)

    OLD fixture under dash: two calls DIFFER = NO    ⇐ why CI failed
    NEW fixture under dash: two calls DIFFER = YES   ⇐ the control can fire

⇒ `od -An -N8 -tx1 /dev/urandom` exists on both and differs per invocation.
Control on the other side: `echo SAME` does not differ.

⛔ AND WHY IT SURVIVED TWO WEEKS. This tool's flag surface was UNVERIFIABLE —
`--self-test --zzz-not-a-flag` exited 0 — and UNVERIFIABLE PREEMPTS the control
verdict. So the gate had never run this control on Linux at all. Repairing the
argument surface is what made it reachable; the failure was not introduced, it
was UNCOVERED.

⇒ Three changes had to land in order for this to be findable:
    #608  the flag surface discriminates      -> the gate REACHES the control
    #611  CONTROL FAILED prints its output    -> the gate SAYS WHICH assertion
    this  the fixture is portable             -> the control passes for a reason

⚠ I could not reproduce it locally: the control passes on macOS before AND after.
Every step of the diagnosis came from CI's own output, which is exactly what #611
was landed to make possible.

test_discriminates 0 · --self-test 0, all four documented exits reachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTMf4EvaTEnDKXY47efRZZ
@jobordu
jobordu merged commit f17e600 into main Sep 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant