Judge test base runs only a local stub, never a real model

[EdbertChan]

Summary

Checker test cases that ask the AI judge use one shared setup.

That setup cleared the runner list, so a test that forgot to pick one fell back to the real model tools.

Now the setup picks a local stand-in that always answers no, and a new test proves it.

Review Claim

Test cases on the shared setup can no longer reach a real model or the real judge queue.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Test-only. The judge's runtime defaults are unchanged; only the shared test setup sets a stand-in. The new stub test fails on the parent slice and passes here.

Slice Rationale

The default flip lands alone so its one new failure mode is easy to see. Directly affected tests stay with it.

Stack position 2 of 7, split from one bundled change. Merge bottom-up.

Non-goals

• No change to the judge's real runner order.
• No other hook's tests change.

Test Plan

Test Plan

$ per-slice checks on split/judge-tests-2
ruff: All checks passed!
[llm-judge: Ran 61 tests in 6.604s OK ] [wrong-check-reflect: Ran 16 tests in 3.266s OK ] [diu-stop: Ran 104 tests in 0.049s OK ]
gate-tests: Ran 8 tests in 0.201s OK
gate: check_hook_test_coverage: OK (37 hook(s) checked)
preflight: ok      preflight passed

• python3 -m unittest tests.test_judge.TestAsk.test_test_base_runs_only_the_local_stub with the parent slice's base — FAILED (failures=1); with this slice — OK
• python3 -m unittest discover -s engine/hooks/llm-judge/tests — Ran 61 tests / OK
• python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests — Ran 16 tests / OK
• python3 -m unittest discover -s engine/hooks/diu-stop/tests — Ran 104 tests / OK
• uvx ruff check . --select E9,F — All checks passed!
• python3 engine/skills/make-pr/scripts/preflight.py --base <stack parent> — ok preflight passed

Revert Plan

Revert Plan

• Safe to revert? Yes, after reverting any later slice in this stack first.
• Revert command: git revert <merge-commit-sha>
• Post-revert steps: None.
• Data migration? No.

🤖 Generated with Claude Code

https://claude.ai/code/session_013smxPCr4XeE7R77zVtN1u4

---

Note

Low Risk
Test-only changes to JudgeTestCase setup and llm-judge unit tests; production judge runner defaults in judge.py are untouched.

Overview
Shared judge test setup now pins a local stub runner (Python subprocess that prints {"match": false}) via RUNNERS_ENV, instead of clearing that env and risking fallback to real codex/claude/cursor tools when a test forgets use_runners.

A new test_test_base_runs_only_the_local_stub locks that behavior. Tests that assert production default runner order or investigate argv explicitly pop(RUNNERS_ENV) so they still exercise real defaults. test_enqueue_writes_only_to_temporary_state_directory was added to ensure enqueue uses the patched temp state dir, not the default under HOME, when STATE_ENV is unset in a nested patch.

^{Reviewed by Cursor Bugbot for commit 1aaedf3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c713de50-91bb-4995-870a-b79f5726ed2b)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_b654833e-1e03-451d-a927-f8dbf6789c84)

[mergify]

Queued — the merge queue status continues in this comment ↓.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_1d6689e0-e592-4490-83eb-2ac966944dd6)

[EdbertChan]

@Mergifyio queue

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-09-14 04:17 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
• ✅ Checks passed · on draft merge queue: checking #525 on main (344657b), stacked on #519 #586
• ✅ Merged — 2026-09-14 04:24 UTC · at 935ee96bf9853c0f33509be825ce2c14e34f9a85 · squash

This pull request spent 7 minutes 28 seconds in the queue, including 6 minutes 2 seconds running CI.

Required conditions to merge

• check-success = lint
• check-success = test
• check-success = validate