Judge tests never reach a model or the shared queue

[EdbertChan]

Summary

Judge-calling hook tests now use a shared base that redirects state to a temporary directory and replaces model runners with a local stub.

The full test runner applies the same isolation, while the coverage gate detects judge-importing hooks whose tests omit the shared base.

This prevents unit tests from making model calls or writing jobs into the user's shared judge queue.

Review Claim

Judge tests are isolated by construction and cannot reach a model or the shared queue.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Test-only and gate-only. No hook decision code changes, and the judge library's runtime behavior outside tests remains untouched.

Slice Rationale

The shared test base, suite-wide isolation, and coverage requirement land together because each closes a different path to the same unsafe test behavior.

Non-goals

• No hook decision logic changes.
• No judge runtime default changes.
• No phrase-list changes.
• No CI workflow changes.

Architecture

Test isolation moves from a per-file convention to shared enforcement at two boundaries.

Before

graph TD
    A["Judge-calling tests"] --> B["Per-file isolation convention"]
    B --> C["Judge state and runners"]
    D["Suite runner"] --> C

Loading

After

graph TD
    A["Judge-calling tests"] --> B["Shared test base"]
    B --> C["Temporary state and local stub runner"]
    D["Suite runner"] --> C
    E["Coverage gate"] --> F["Requires shared base reference"]
    A --> F

Loading

Test Plan

Test Plan

• python3 -m unittest discover -s engine/hooks/llm-judge/tests && python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests && python3 -m unittest discover -s engine/hooks/diu-stop/tests — 60, 16, and 104 tests passed; each runner reported OK.
• python3 scripts/check_hook_test_coverage.py — check_hook_test_coverage: OK (36 hook(s) checked).
• bash scripts/scrub-handoff-artifacts.sh — scrub-handoff-artifacts-ok.
• python3 scripts/check_no_new_comments.py --base origin/main — ok no new comments.
• python3 engine/skills/make-pr/scripts/preflight.py --base origin/main — ok preflight passed.

Revert Plan

Revert Plan

• Safe to revert? Yes.
• Revert command: git revert <merge-commit-sha>.
• Post-revert steps: Re-run the judge-hook test suites and hook-coverage gate.
• Data migration? No.

---

Note

Low Risk
Test infrastructure and coverage gates only; production hook and judge paths are unchanged.

Overview
Introduces a shared JudgeTestCase base in engine/hooks/llm-judge/testing.py so judge-related tests always use a temporary state directory and a local stub runner instead of real model backends or the user’s default judge queue.

Hook and llm-judge test suites are refactored to inherit that base (replacing ad hoc patch.dict / duplicate temp dirs). test_judge.py adds a regression that enqueue only writes under the test state dir, and default-runner-order coverage clears RUNNERS_ENV explicitly.

scripts/run_all_tests.sh exports the same isolated state + stub runners for the full unittest discover run. scripts/check_hook_test_coverage.py fails hooks that import judge but whose tests never reference JudgeTestCase.

No changes to hook decision logic or judge runtime behavior outside tests.

^{Reviewed by Cursor Bugbot for commit 1e6d9be. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_3409cad3-0950-49d9-a987-a560a2d3754f)