[Unverified tag check](1) llm-judge runs a read-only investigate job - #539
Merged
mergify[bot] merged 8 commits intoSep 13, 2026
Merged
Conversation
…n llm-judge job opt into a read-only investigate mode with its own time limit and working folder.
Review claim: A job carrying "mode": "investigate" runs on a second runner set whose Claude argv grants only
the Read, Grep and Glob tools and forbids Write, Edit, NotebookEdit and Bash, with the job's own
timeout_seconds and cwd; a job lacking that key runs exactly as today.
Review lane: behavior
Safety invariant: An investigate job never receives a tool that can write, edit, or execute shell text. The
Claude runner argv always carries --allowedTools Read Grep Glob and --disallowedTools Write Edit NotebookEdit
Bash, the Codex runner always carries --sandbox read-only, and cursor-agent is never used for investigate
jobs because it has no read-only switch.
Effectiveness measurement: New unit tests pin the exact argv of each investigate runner and prove a default
job still gets 60 seconds in an empty temp folder. The PR body pastes the live probe in which claude -p with
this argv was asked to Write a file and to touch one through Bash, answered
{"wrote":false,"read":true,"bash":false}, and neither target file existed afterwards.
Slice rationale: One behavior slice inside the judge, dormant until a hook asks for an investigate job, so a
reviewer can judge the permission argv on its own.
Architectural effect: llm-judge gains a second, read-only runner set chosen per job; the default runner set
and every current caller stay unchanged.
Goal: Give later hooks a safe way to ask a model to check a claim by reading files on this machine.
Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose stated blocker was false;
most could have been checked by reading a log, a transcript, or a source file already on disk. Today each
judge runner gets 60 seconds in an empty temp folder with no tool grants, so it cannot read anything.
Alternative considerations: claude --permission-mode plan was rejected after a live probe wrote a plan file
under ~/.claude/plans. A Bash prefix allowlist such as Bash(git log:*) was rejected after a live probe
blocked even plain git log, so shell access stays out of this slice.
Implementation details: In engine/hooks/llm-judge/judge.py add INVESTIGATE_RUNNERS beside DEFAULT_RUNNERS
holding the codex and claude argv given under Runner argv. Make runners() take the job mode and pick
INVESTIGATE_RUNNERS for "investigate"; CATSTACK_LLM_JUDGE_RUNNERS still replaces whichever set is picked.
Make run_runner accept timeout_seconds (default TIMEOUT_SECONDS, capped at INVESTIGATE_TIMEOUT_CAP = 600)
and cwd (used only when it is an absolute path to a folder that exists, else the temp folder plus a
judge.log line). Thread mode, timeout_seconds and cwd from the job through ask() and run_job().
Runner argv: codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT, and claude -p --model
haiku --settings '{"disableAllHooks": true}' --allowedTools Read Grep Glob --disallowedTools Write Edit
NotebookEdit Bash -- PROMPT.
Non-goals: No hook asks for an investigate job in this slice. No inbox edit, no change to DEFAULT_RUNNERS,
wrong-check-reflect or diu-stop. No shell or network access for the investigator.
Layer: domain
Feature state: dormant
Files:
- engine/hooks/llm-judge/judge.py
- engine/hooks/llm-judge/tests/test_judge.py
Change types:
- engine/hooks/llm-judge/judge.py: modify
- engine/hooks/llm-judge/tests/test_judge.py: modify
Acceptance criteria:
- Tests whose names contain "investigate" pass and pin both investigate runner argv exactly.
- A test proves a job lacking a mode key still runs DEFAULT_RUNNERS with a 60 second limit.
- A test proves timeout_seconds above 600 is capped at 600.
- A test whose name contains "missing" proves a cwd that does not exist falls back to a temp folder and logs it.
- python3 scripts/check_no_new_comments.py --base origin/main exits 0.
Solution:
Let an llm-judge job opt into a read-only investigate mode with its own time limit and working folder.
Review claim: A job carrying "mode": "investigate" runs on a second runner set whose Claude argv grants only
the Read, Grep and Glob tools and forbids Write, Edit, NotebookEdit and Bash, with the job's own
timeout_seconds and cwd; a job lacking that key runs exactly as today.
Review lane: behavior
Safety invariant: An investigate job never receives a tool that can write, edit, or execute shell text. The
Claude runner argv always carries --allowedTools Read Grep Glob and --disallowedTools Write Edit NotebookEdit
Bash, the Codex runner always carries --sandbox read-only, and cursor-agent is never used for investigate
jobs because it has no read-only switch.
Effectiveness measurement: New unit tests pin the exact argv of each investigate runner and prove a default
job still gets 60 seconds in an empty temp folder. The PR body pastes the live probe in which claude -p with
this argv was asked to Write a file and to touch one through Bash, answered
{"wrote":false,"read":true,"bash":false}, and neither target file existed afterwards.
Slice rationale: One behavior slice inside the judge, dormant until a hook asks for an investigate job, so a
reviewer can judge the permission argv on its own.
Architectural effect: llm-judge gains a second, read-only runner set chosen per job; the default runner set
and every current caller stay unchanged.
Goal: Give later hooks a safe way to ask a model to check a claim by reading files on this machine.
Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose stated blocker was false;
most could have been checked by reading a log, a transcript, or a source file already on disk. Today each
judge runner gets 60 seconds in an empty temp folder with no tool grants, so it cannot read anything.
Alternative considerations: claude --permission-mode plan was rejected after a live probe wrote a plan file
under ~/.claude/plans. A Bash prefix allowlist such as Bash(git log:*) was rejected after a live probe
blocked even plain git log, so shell access stays out of this slice.
Implementation details: In engine/hooks/llm-judge/judge.py add INVESTIGATE_RUNNERS beside DEFAULT_RUNNERS
holding the codex and claude argv given under Runner argv. Make runners() take the job mode and pick
INVESTIGATE_RUNNERS for "investigate"; CATSTACK_LLM_JUDGE_RUNNERS still replaces whichever set is picked.
Make run_runner accept timeout_seconds (default TIMEOUT_SECONDS, capped at INVESTIGATE_TIMEOUT_CAP = 600)
and cwd (used only when it is an absolute path to a folder that exists, else the temp folder plus a
judge.log line). Thread mode, timeout_seconds and cwd from the job through ask() and run_job().
Runner argv: codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT, and claude -p --model
haiku --settings '{"disableAllHooks": true}' --allowedTools Read Grep Glob --disallowedTools Write Edit
NotebookEdit Bash -- PROMPT.
Non-goals: No hook asks for an investigate job in this slice. No inbox edit, no change to DEFAULT_RUNNERS,
wrong-check-reflect or diu-stop. No shell or network access for the investigator.
Layer: domain
Feature state: dormant
Files:
- engine/hooks/llm-judge/judge.py
- engine/hooks/llm-judge/tests/test_judge.py
Change types:
- engine/hooks/llm-judge/judge.py: modify
- engine/hooks/llm-judge/tests/test_judge.py: modify
Acceptance criteria:
- Tests whose names contain "investigate" pass and pin both investigate runner argv exactly.
- A test proves a job lacking a mode key still runs DEFAULT_RUNNERS with a 60 second limit.
- A test proves timeout_seconds above 600 is capped at 600.
- A test whose name contains "missing" proves a cwd that does not exist falls back to a temp folder and logs it.
- python3 scripts/check_no_new_comments.py --base origin/main exits 0.
Invoker-Finalize-Id: 890bf8a3-d647-4273-855b-8ca21b411f55
…tep 1 for the llm-judge investigate mode. Review claim: The investigate-mode unit tests run and pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The run exits 0 and reports at least one test; Python exits 5 when -k matches nothing. Slice rationale: One proof step for this workflow's single change. Architectural effect: None; verification only. Goal: Prove the investigate runner argv and time limit deterministically. Motivation: Each slice carries its own executable proof. Alternative considerations: Manual inspection was rejected as non-deterministic. Implementation details: Run the filtered unittest discovery once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 964f7535-7395-4bb3-b657-37a77b357f44
…be the llm-judge investigate mode in the llm-judge README. Review claim: engine/hooks/llm-judge/README.md gains an Investigate mode section that matches judge.py. Review lane: docs Safety invariant: Only engine/hooks/llm-judge/README.md changes; no code or test file is edited. Effectiveness measurement: grep finds the section heading and each named flag in the README. Slice rationale: The README section for the behavior in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README only. Goal: Let the next hook author find the investigate mode and its limits. Motivation: catstack keeps every hook's contract in its README. Alternative considerations: A later README-only pull request was rejected; catstack's playbook counts a README that lands later as a known repair pattern. Implementation details: Add one "## Investigate mode" section to engine/hooks/llm-judge/README.md after "## Runner order". Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `grep -n "## Investigate mode" engine/hooks/llm-judge/README.md` exits 0. - The section names --sandbox read-only, --allowedTools Read Grep Glob, --disallowedTools Write Edit NotebookEdit Bash, and 600. Context: wf-1789279240280-10/implement-judge-investigate-mode (f2435ab): Let an llm-judge job opt into a read-only investigate mode with its own time limit and working folder. Review claim: A job carrying "mode": "investigate" runs on a second runner set whose Claude argv grants only the Read, Grep and Glob tools and forbids Write, Edit, NotebookEdit and Bash, with the job's own timeout_seconds and cwd; a job lacking that key runs exactly as today. Review lane: behavior Safety invariant: An investigate job never receives a tool that can write, edit, or execute shell text. The Claude runner argv always carries --allowedTools Read Grep Glob and --disallowedTools Write Edit NotebookEdit Bash, the Codex runner always carries --sandbox read-only, and cursor-agent is never used for investigate jobs because it has no read-only switch. Effectiveness measurement: New unit tests pin the exact argv of each investigate runner and prove a default job still gets 60 seconds in an empty temp folder. The PR body pastes the live probe in which claude -p with this argv was asked to Write a file and to touch one through Bash, answered {"wrote":false,"read":true,"bash":false}, and neither target file existed afterwards. Slice rationale: One behavior slice inside the judge, dormant until a hook asks for an investigate job, so a reviewer can judge the permission argv on its own. Architectural effect: llm-judge gains a second, read-only runner set chosen per job; the default runner set and every current caller stay unchanged. Goal: Give later hooks a safe way to ask a model to check a claim by reading files on this machine. Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose stated blocker was false; most could have been checked by reading a log, a transcript, or a source file already on disk. Today each judge runner gets 60 seconds in an empty temp folder with no tool grants, so it cannot read anything. Alternative considerations: claude --permission-mode plan was rejected after a live probe wrote a plan file under ~/.claude/plans. A Bash prefix allowlist such as Bash(git log:*) was rejected after a live probe blocked even plain git log, so shell access stays out of this slice. Implementation details: In engine/hooks/llm-judge/judge.py add INVESTIGATE_RUNNERS beside DEFAULT_RUNNERS holding the codex and claude argv given under Runner argv. Make runners() take the job mode and pick INVESTIGATE_RUNNERS for "investigate"; CATSTACK_LLM_JUDGE_RUNNERS still replaces whichever set is picked. Make run_runner accept timeout_seconds (default TIMEOUT_SECONDS, capped at INVESTIGATE_TIMEOUT_CAP = 600) and cwd (used only when it is an absolute path to a folder that exists, else the temp folder plus a judge.log line). Thread mode, timeout_seconds and cwd from the job through ask() and run_job(). Runner argv: codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT, and claude -p --model haiku --settings '{"disableAllHooks": true}' --allowedTools Read Grep Glob --disallowedTools Write Edit NotebookEdit Bash -- PROMPT. Non-goals: No hook asks for an investigate job in this slice. No inbox edit, no change to DEFAULT_RUNNERS, wrong-check-reflect or diu-stop. No shell or network access for the investigator. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/judge.py - engine/hooks/llm-judge/tests/test_judge.py Change types: - engine/hooks/llm-judge/judge.py: modify - engine/hooks/llm-judge/tests/test_judge.py: modify Acceptance criteria: - Tests whose names contain "investigate" pass and pin both investigate runner argv exactly. - A test proves a job lacking a mode key still runs DEFAULT_RUNNERS with a 60 second limit. - A test proves timeout_seconds above 600 is capped at 600. - A test whose name contains "missing" proves a cwd that does not exist falls back to a temp folder and logs it. - python3 scripts/check_no_new_comments.py --base origin/main exits 0. Solution: Describe the llm-judge investigate mode in the llm-judge README. Review claim: engine/hooks/llm-judge/README.md gains an Investigate mode section that matches judge.py. Review lane: docs Safety invariant: Only engine/hooks/llm-judge/README.md changes; no code or test file is edited. Effectiveness measurement: grep finds the section heading and each named flag in the README. Slice rationale: The README section for the behavior in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README only. Goal: Let the next hook author find the investigate mode and its limits. Motivation: catstack keeps every hook's contract in its README. Alternative considerations: A later README-only pull request was rejected; catstack's playbook counts a README that lands later as a known repair pattern. Implementation details: Add one "## Investigate mode" section to engine/hooks/llm-judge/README.md after "## Runner order". Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `grep -n "## Investigate mode" engine/hooks/llm-judge/README.md` exits 0. - The section names --sandbox read-only, --allowedTools Read Grep Glob, --disallowedTools Write Edit NotebookEdit Bash, and 600. Invoker-Finalize-Id: c7674a0c-7e5f-4540-b149-295bbac2133e
…or the llm-judge investigate mode. Review claim: Every llm-judge test still passes after the change. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The whole llm-judge test folder exits 0. Slice rationale: One proof step guarding current judge callers. Architectural effect: None; verification only. Goal: Prove default jobs behave as before. Motivation: wrong-check-reflect and diu-stop already depend on the judge. Alternative considerations: Running only the new tests was rejected; it would miss a regression in the default runner set. Implementation details: Run the whole llm-judge test folder. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: c94db032-2d49-4594-b6e7-b3ac43e7b56b
…p 3 for the llm-judge investigate mode. Review claim: The change adds no explanatory code comments. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The comment gate exits 0 against origin/main. Slice rationale: One proof step for catstack's required comment gate. Architectural effect: None; verification only. Goal: Keep the CI comment gate green. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run the comment gate against origin/main. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 492bf3e7-0cff-46b3-b0d1-89687e47b6b7
…ate-mode/g0.t0.a-af1ca1213-1325fc3c
…ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active Exit code: 0 Invoker-Finalize-Id: 2aafd01f-794a-46f1-8763-0a04ed358458
…a00166eb5-1f479726 — Check that no ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_0cc57b4a-69d3-468b-bb7b-a7c91dd1c569) |
Owner
Author
|
Live probe for the Safety invariant, run by hand on the author's Mac before this plan was submitted. Claude with the investigate argv's tool grants was asked to Write a file, Read a file, and Two alternatives were probed and dropped:
Not probed live: the codex |
Contributor
|
Queued — the merge queue status continues in this comment ↓. |
Owner
Author
Closed
7 tasks
Owner
Author
|
@Mergifyio queue |
Contributor
Merge Queue Status
This pull request spent 5 minutes 33 seconds in the queue, including 4 minutes 31 seconds running CI. Required conditions to merge
|
This was referenced Sep 13, 2026
mergify
Bot
deleted the
plan/unverified-tag-check-1-judge-investigate-mode
branch
September 13, 2026 18:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This part is a helper that asks an AI model a yes-or-no question for other checkers. Right now the model cannot look at anything on this machine.
The problem: many "could not check this" notes were wrong. Of 44 such notes from two weeks, 33 could have been checked by reading a file on disk.
The cause: each model gets one minute in an empty folder with no tools, so it has nothing to read.
The fix: a job can now ask for a look-only mode. The model may read and search files, with more time and a chosen folder.
It still cannot write, edit, or run commands.
No checker asks for this mode yet. Jobs that do not ask run exactly as before.
Review Claim
A job that asks for the look-only mode gets models that can read and search files but cannot write, edit, or run commands, and every other job runs as it did before.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
An investigate job never receives a tool that can write, edit, or run shell text. The Claude runner always carries
--allowedTools Read Grep Globand--disallowedTools Write Edit NotebookEdit Bash, the Codex runner always carries--sandbox read-only, andcursor-agentis never used for investigate jobs because it has no read-only switch. A job without"mode": "investigate"still getsDEFAULT_RUNNERS, 60 seconds, and a fresh empty temp folder.Slice Rationale
This is the first slice of the unverified-tag check. It lands the read-only runner set inside
engine/hooks/llm-judgeon its own, dormant, so the permission argv can be reviewed without a hook that depends on it. The README section ships here because catstack's ship-a-detector playbook requires a hook's README change to land with its code. The hook that sends investigate jobs lands two slices later.Non-goals
DEFAULT_RUNNERS,wrong-check-reflect, ordiu-stop.claude --permission-mode planwas rejected after a live probe wrote a plan file under~/.claude/plans; a Bash prefix allowlist such asBash(git log:*)was rejected after a live probe blocked even plaingit log.Architecture
Before
graph TD J["job file"] --> A["ask(prompt)"] A --> R["DEFAULT_RUNNERS: codex, claude, cursor"] R --> T["60s timeout, empty temp folder, no tool grants"]After
graph TD J["job file"] --> A["ask(prompt, mode, timeout_seconds, cwd)"] A --> M{"mode == investigate?"} M -- "no" --> R["DEFAULT_RUNNERS: codex, claude, cursor"] M -- "yes" --> I["INVESTIGATE_RUNNERS: codex read-only sandbox, claude Read/Grep/Glob only"] R --> T["timeout capped at 600s; cwd used only if absolute existing folder, else temp folder plus judge.log line"] I --> TCATSTACK_LLM_JUDGE_RUNNERSstill replaces whichever runner set is selected.Test Plan
Test Plan
python3 -m unittest discover -s engine/hooks/llm-judge/tests -p test_judge.py -k investigate -vpython3 -m unittest discover -s engine/hooks/llm-judge/testspython3 scripts/check_no_new_comments.py --base origin/main→ok no new comments, exit 0bash scripts/scrub-handoff-artifacts.sh→scrub-handoff-artifacts-ok, exit 0python3 engine/skills/make-pr/scripts/preflight.py --dry-run --base origin/mainclaudeprogram, temp folder holding onlyreadme.txt). The prompt asked it to createwrote.txtwith the Write tool, createbash.txtwith a shell command, and readreadme.txt:Neither
wrote.txtnorbash.txtexists afterwards.cwdexists yet. The refused-cwd fallback (temp folder plusjudge.logline) is injudge.pybut has no unit test in this branch.--sandbox read-only) is checked by unit test only; it was not tried in a real run.Revert Plan
Revert Plan
git revert <merge-sha>🤖 Generated with Claude Code
Note
Medium Risk
Expands model subprocess capabilities (file read/search and real repo cwd) when jobs opt in; default jobs are unchanged but misconfigured investigate jobs could expose paths on disk.
Overview
Adds an opt-in
"mode": "investigate"path on llm-judge jobs so models can answer questions by reading and searching the filesystem under tighter guardrails, without changing default job behavior.When investigate is set,
runners()usesINVESTIGATE_RUNNERS(Codex read-only sandbox + Claude with only Read/Grep/Glob; nocursor-agent) instead of the usual three runners.ask()andrun_job()now accepttimeout_seconds(capped at 600 viabounded_timeout, else 60) and optionalcwd(only an absolute existing directory; otherwise a temp dir plus ajudge.logrefusal line).CATSTACK_LLM_JUDGE_RUNNERSstill overrides whichever set is selected.The README documents investigate mode; unit tests cover runner argv, env override, timeout/cwd threading, the 600s cap, and that non-investigate jobs still get 60s and an empty temp cwd. No hook enqueues investigate jobs in this slice—the feature is dormant until a later change wires it up.
Reviewed by Cursor Bugbot for commit 82f6094. Bugbot is set up for automated code reviews on this repo. Configure here.