merge queue: checking #539 on main (a95db0e), stacked on #493 and #498 - #559
Closed
mergify[bot] wants to merge 19 commits into
Closed
mergify[bot] wants to merge 19 commits into
mergify[bot] wants to merge 19 commits into
Conversation
…the Step 5a passage to call the shipped probe. Review claim: Step 5a names the probe and all three of its outcomes. Review lane: behavior Safety invariant: The blocked-pull-request policy in Step 5 is unchanged; only the passage describing how to test is rewritten. Effectiveness measurement: A fixture asserts the Step 5a passage names the probe and all three outcomes, and that the words `rm -rf` no longer appear in it. Slice rationale: One passage of one document, stacked on the step that ships the executable it names. Architectural effect: The written procedure points at a shipped executable instead of an inline recipe. Goal: Rewrite the Step 5a passage of product/skills/admin-bypass-sweep/SKILL.md to call scripts/probe_branch_rebase.sh and describe its three outcomes. Motivation: The passage already told the reader to tear down with the safe teardown command, and that wording did not prevent an improvised removal that corrupted the next reading. Naming one executable removes the chance to improvise. Alternative considerations: Adding another sentence of guidance was rejected, since equivalent guidance is already present and did not hold. Implementation details: Replace the fenced passage in Step 5a with a call to scripts/probe_branch_rebase.sh and a short list of its three outcomes. Add a fixture under the skill's tests folder asserting the passage names the probe, names all three outcomes, and no longer contains an improvised removal. Non-goals: Does not change Step 5's policy for a blocked pull request, does not touch the merge passage, and does not edit any other written procedure. Layer: domain Feature state: active Files: product/skills/admin-bypass-sweep/SKILL.md, product/skills/admin-bypass-sweep/tests/fires_step5a_names_probe.md Change types: - product/skills/admin-bypass-sweep/SKILL.md: docs-only - product/skills/admin-bypass-sweep/tests/fires_step5a_names_probe.md: create Acceptance criteria: - `python3 -m unittest discover -s product/skills/admin-bypass-sweep/tests -v` exits 0. - The Step 5a passage names scripts/probe_branch_rebase.sh. - The Step 5a passage lists all three outcomes. Solution: Rewrite the Step 5a passage to call the shipped probe. Review claim: Step 5a names the probe and all three of its outcomes. Review lane: behavior Safety invariant: The blocked-pull-request policy in Step 5 is unchanged; only the passage describing how to test is rewritten. Effectiveness measurement: A fixture asserts the Step 5a passage names the probe and all three outcomes, and that the words `rm -rf` no longer appear in it. Slice rationale: One passage of one document, stacked on the step that ships the executable it names. Architectural effect: The written procedure points at a shipped executable instead of an inline recipe. Goal: Rewrite the Step 5a passage of product/skills/admin-bypass-sweep/SKILL.md to call scripts/probe_branch_rebase.sh and describe its three outcomes. Motivation: The passage already told the reader to tear down with the safe teardown command, and that wording did not prevent an improvised removal that corrupted the next reading. Naming one executable removes the chance to improvise. Alternative considerations: Adding another sentence of guidance was rejected, since equivalent guidance is already present and did not hold. Implementation details: Replace the fenced passage in Step 5a with a call to scripts/probe_branch_rebase.sh and a short list of its three outcomes. Add a fixture under the skill's tests folder asserting the passage names the probe, names all three outcomes, and no longer contains an improvised removal. Non-goals: Does not change Step 5's policy for a blocked pull request, does not touch the merge passage, and does not edit any other written procedure. Layer: domain Feature state: active Files: product/skills/admin-bypass-sweep/SKILL.md, product/skills/admin-bypass-sweep/tests/fires_step5a_names_probe.md Change types: - product/skills/admin-bypass-sweep/SKILL.md: docs-only - product/skills/admin-bypass-sweep/tests/fires_step5a_names_probe.md: create Acceptance criteria: - `python3 -m unittest discover -s product/skills/admin-bypass-sweep/tests -v` exits 0. - The Step 5a passage names scripts/probe_branch_rebase.sh. - The Step 5a passage lists all three outcomes. Invoker-Finalize-Id: e859b330-f6c2-4da7-9aaf-2356607f423a
…erministic proof for the Step 5a wiring. Review claim: A fixture asserts the passage names the probe. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The suite fails if the passage stops naming the probe or drops an outcome. Slice rationale: One proof slice for this step. Architectural effect: None; verification only. Goal: Prove the Step 5a wiring deterministically. Motivation: The change is to one written passage, so the proof reads that passage. Alternative considerations: Manual verification was rejected as non-deterministic. Implementation details: Execute the command below as the terminal proof. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 3aa932b8-5cc2-4fba-bc5c-115d7a475b4f
… that no ephemeral inter-task handoff files remain. Review claim: The workflow leaves no scratch handoff artifacts behind. Review lane: cleanup Safety invariant: Read-only; never deletes files, alters the index, or commits caller work. Effectiveness measurement: The gate fails when a plans/invoker-handoff.* or lens-*.json file is still present after the leaf tasks complete. Slice rationale: One terminal hygiene gate for the workflow. Architectural effect: None; check only. Goal: Confirm no ephemeral handoff files survive the run. Motivation: Inter-task scratch files leak into diffs and read as part of the work. Alternative considerations: Deleting them automatically was rejected; the gate reports, it does not mutate. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 2c57922e-973b-4c94-a350-bde1591b77ee
…acd04e326-8748be37 — Terminal check that no ephemeral inter-task handoff files remain. Review claim: The workflow leaves no scratch handoff artifacts behind. Review lane: cleanup Safety invariant: Read-only; never deletes files, alters the index, or commits caller work. Effectiveness measurement: The gate fails when a plans/invoker-handoff.* or lens-*.json file is still present after the leaf tasks complete. Slice rationale: One terminal hygiene gate for the workflow. Architectural effect: None; check only. Goal: Confirm no ephemeral handoff files survive the run. Motivation: Inter-task scratch files leak into diffs and read as part of the work. Alternative considerations: Deleting them automatically was rejected; the gate reports, it does not mutate. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: app_regression Feature state: active
… a full diff read per PR before an admin merge, and mark a narrowed read unchecked. Review claim: A PR whose diff was read through a truncating filter is unchecked, never reviewed. Review lane: behavior Safety invariant: A truncated or filtered diff read marks that PR unchecked, never reviewed; no PR that is reviewable today becomes unmergeable, because the gate reports rather than blocks the merge command. Effectiveness measurement: A fixture review command piped through `head` marks its PR unchecked, and the same command without the pipe marks it reviewed. Both asserted in the skill's test directory. Slice rationale: One step of the sweep skill's review procedure; stacked behind the tally step because both edit the same SKILL.md. Architectural effect: Step 4 gains an explicit read-completeness precondition expressed as the same three-outcome contract the rest of the stack uses. Goal: Change Step 4 of product/skills/admin-bypass-sweep/SKILL.md so each PR's diff is written to a file, fully read, and asserted line-complete before `gh pr merge --admin`, with a narrowed read reported as unchecked. Motivation: A sweep reported 'Reviewed all 19' after two commands that between them truncated one PR entirely and filtered every test file out of five others. The largest diff in the sweep, touching six hook detectors, was cleared for merge on about 21% of its non-test changes. Alternative considerations: Telling the operator not to pipe through head was rejected as unenforceable prose. Blocking the merge outright on a narrowed read was rejected for this slice: the skill's own consent model puts the merge decision with the human, so the gate reports unchecked and the operator decides. Implementation details: In Step 4, replace the ad hoc `gh pr diff` skim with: write each PR's diff to a file, record `wc -l`, read the whole file, and assert lines_read equals the recorded total before merging that PR. Any read through head, tail, grep, awk, or sed marks that PR unchecked. Use the same three outcomes as the rest of the stack: reviewed, flagged, unchecked. Add fixtures under the skill's tests directory for a full read and a truncated read. Non-goals: Does not change what counts as a review finding, does not add an automated code review, does not alter the consent requirements in the STOP section, and does not touch any other skill. Layer: domain Feature state: active Files: product/skills/admin-bypass-sweep/SKILL.md, product/skills/admin-bypass-sweep/tests/ Change types: - product/skills/admin-bypass-sweep/SKILL.md: docs-only - product/skills/admin-bypass-sweep/tests/fires_truncated_diff_read.md: create - product/skills/admin-bypass-sweep/tests/stays_silent_full_diff_read.md: create Acceptance criteria: - `python3 -m unittest discover -s product/skills/admin-bypass-sweep/tests -v` exits 0. - Step 4 requires a recorded line count and a full read before `gh pr merge --admin`. - A read through head, tail, grep, awk, or sed is documented as unchecked. Solution: Require a full diff read per PR before an admin merge, and mark a narrowed read unchecked. Review claim: A PR whose diff was read through a truncating filter is unchecked, never reviewed. Review lane: behavior Safety invariant: A truncated or filtered diff read marks that PR unchecked, never reviewed; no PR that is reviewable today becomes unmergeable, because the gate reports rather than blocks the merge command. Effectiveness measurement: A fixture review command piped through `head` marks its PR unchecked, and the same command without the pipe marks it reviewed. Both asserted in the skill's test directory. Slice rationale: One step of the sweep skill's review procedure; stacked behind the tally step because both edit the same SKILL.md. Architectural effect: Step 4 gains an explicit read-completeness precondition expressed as the same three-outcome contract the rest of the stack uses. Goal: Change Step 4 of product/skills/admin-bypass-sweep/SKILL.md so each PR's diff is written to a file, fully read, and asserted line-complete before `gh pr merge --admin`, with a narrowed read reported as unchecked. Motivation: A sweep reported 'Reviewed all 19' after two commands that between them truncated one PR entirely and filtered every test file out of five others. The largest diff in the sweep, touching six hook detectors, was cleared for merge on about 21% of its non-test changes. Alternative considerations: Telling the operator not to pipe through head was rejected as unenforceable prose. Blocking the merge outright on a narrowed read was rejected for this slice: the skill's own consent model puts the merge decision with the human, so the gate reports unchecked and the operator decides. Implementation details: In Step 4, replace the ad hoc `gh pr diff` skim with: write each PR's diff to a file, record `wc -l`, read the whole file, and assert lines_read equals the recorded total before merging that PR. Any read through head, tail, grep, awk, or sed marks that PR unchecked. Use the same three outcomes as the rest of the stack: reviewed, flagged, unchecked. Add fixtures under the skill's tests directory for a full read and a truncated read. Non-goals: Does not change what counts as a review finding, does not add an automated code review, does not alter the consent requirements in the STOP section, and does not touch any other skill. Layer: domain Feature state: active Files: product/skills/admin-bypass-sweep/SKILL.md, product/skills/admin-bypass-sweep/tests/ Change types: - product/skills/admin-bypass-sweep/SKILL.md: docs-only - product/skills/admin-bypass-sweep/tests/fires_truncated_diff_read.md: create - product/skills/admin-bypass-sweep/tests/stays_silent_full_diff_read.md: create Acceptance criteria: - `python3 -m unittest discover -s product/skills/admin-bypass-sweep/tests -v` exits 0. - Step 4 requires a recorded line count and a full read before `gh pr merge --admin`. - A read through head, tail, grep, awk, or sed is documented as unchecked. Invoker-Finalize-Id: 86232251-f80f-4a4b-a379-36417182163d
…terministic proof for the full-diff-read gate. Review claim: The skill's fixtures assert a truncated diff read is reported unchecked. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The suite fails if the truncated-read fixture is reported as reviewed. Slice rationale: One proof slice for this step. Architectural effect: None; verification only. Goal: Prove the full-diff-read gate deterministically. Motivation: The defect is an over-claimed review, so the proof asserts the narrowed case is named unchecked. Alternative considerations: Manual verification was rejected as non-deterministic. Implementation details: Execute the command below as the terminal proof. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 4cde49f2-d3fe-41d7-ad5f-4c24fb36f491
… that no ephemeral inter-task handoff files remain. Review claim: The workflow leaves no scratch handoff artifacts behind. Review lane: cleanup Safety invariant: Read-only; never deletes files, alters the index, or commits caller work. Effectiveness measurement: The gate fails when a plans/invoker-handoff.* or lens-*.json file is still present after the leaf tasks complete. Slice rationale: One terminal hygiene gate for the workflow. Architectural effect: None; check only. Goal: Confirm no ephemeral handoff files survive the run. Motivation: Inter-task scratch files leak into diffs and read as part of the work. Alternative considerations: Deleting them automatically was rejected; the gate reports, it does not mutate. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 401825e5-c8a7-488b-9472-5b8212a9ca68
…aff8b535d-05d30b31 — Terminal check that no ephemeral inter-task handoff files remain. Review claim: The workflow leaves no scratch handoff artifacts behind. Review lane: cleanup Safety invariant: Read-only; never deletes files, alters the index, or commits caller work. Effectiveness measurement: The gate fails when a plans/invoker-handoff.* or lens-*.json file is still present after the leaf tasks complete. Slice rationale: One terminal hygiene gate for the workflow. Architectural effect: None; check only. Goal: Confirm no ephemeral handoff files survive the run. Motivation: Inter-task scratch files leak into diffs and read as part of the work. Alternative considerations: Deleting them automatically was rejected; the gate reports, it does not mutate. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: app_regression Feature state: active
…n llm-judge job opt into a read-only investigate mode with its own time limit and working folder.
Review claim: A job carrying "mode": "investigate" runs on a second runner set whose Claude argv grants only
the Read, Grep and Glob tools and forbids Write, Edit, NotebookEdit and Bash, with the job's own
timeout_seconds and cwd; a job lacking that key runs exactly as today.
Review lane: behavior
Safety invariant: An investigate job never receives a tool that can write, edit, or execute shell text. The
Claude runner argv always carries --allowedTools Read Grep Glob and --disallowedTools Write Edit NotebookEdit
Bash, the Codex runner always carries --sandbox read-only, and cursor-agent is never used for investigate
jobs because it has no read-only switch.
Effectiveness measurement: New unit tests pin the exact argv of each investigate runner and prove a default
job still gets 60 seconds in an empty temp folder. The PR body pastes the live probe in which claude -p with
this argv was asked to Write a file and to touch one through Bash, answered
{"wrote":false,"read":true,"bash":false}, and neither target file existed afterwards.
Slice rationale: One behavior slice inside the judge, dormant until a hook asks for an investigate job, so a
reviewer can judge the permission argv on its own.
Architectural effect: llm-judge gains a second, read-only runner set chosen per job; the default runner set
and every current caller stay unchanged.
Goal: Give later hooks a safe way to ask a model to check a claim by reading files on this machine.
Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose stated blocker was false;
most could have been checked by reading a log, a transcript, or a source file already on disk. Today each
judge runner gets 60 seconds in an empty temp folder with no tool grants, so it cannot read anything.
Alternative considerations: claude --permission-mode plan was rejected after a live probe wrote a plan file
under ~/.claude/plans. A Bash prefix allowlist such as Bash(git log:*) was rejected after a live probe
blocked even plain git log, so shell access stays out of this slice.
Implementation details: In engine/hooks/llm-judge/judge.py add INVESTIGATE_RUNNERS beside DEFAULT_RUNNERS
holding the codex and claude argv given under Runner argv. Make runners() take the job mode and pick
INVESTIGATE_RUNNERS for "investigate"; CATSTACK_LLM_JUDGE_RUNNERS still replaces whichever set is picked.
Make run_runner accept timeout_seconds (default TIMEOUT_SECONDS, capped at INVESTIGATE_TIMEOUT_CAP = 600)
and cwd (used only when it is an absolute path to a folder that exists, else the temp folder plus a
judge.log line). Thread mode, timeout_seconds and cwd from the job through ask() and run_job().
Runner argv: codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT, and claude -p --model
haiku --settings '{"disableAllHooks": true}' --allowedTools Read Grep Glob --disallowedTools Write Edit
NotebookEdit Bash -- PROMPT.
Non-goals: No hook asks for an investigate job in this slice. No inbox edit, no change to DEFAULT_RUNNERS,
wrong-check-reflect or diu-stop. No shell or network access for the investigator.
Layer: domain
Feature state: dormant
Files:
- engine/hooks/llm-judge/judge.py
- engine/hooks/llm-judge/tests/test_judge.py
Change types:
- engine/hooks/llm-judge/judge.py: modify
- engine/hooks/llm-judge/tests/test_judge.py: modify
Acceptance criteria:
- Tests whose names contain "investigate" pass and pin both investigate runner argv exactly.
- A test proves a job lacking a mode key still runs DEFAULT_RUNNERS with a 60 second limit.
- A test proves timeout_seconds above 600 is capped at 600.
- A test whose name contains "missing" proves a cwd that does not exist falls back to a temp folder and logs it.
- python3 scripts/check_no_new_comments.py --base origin/main exits 0.
Solution:
Let an llm-judge job opt into a read-only investigate mode with its own time limit and working folder.
Review claim: A job carrying "mode": "investigate" runs on a second runner set whose Claude argv grants only
the Read, Grep and Glob tools and forbids Write, Edit, NotebookEdit and Bash, with the job's own
timeout_seconds and cwd; a job lacking that key runs exactly as today.
Review lane: behavior
Safety invariant: An investigate job never receives a tool that can write, edit, or execute shell text. The
Claude runner argv always carries --allowedTools Read Grep Glob and --disallowedTools Write Edit NotebookEdit
Bash, the Codex runner always carries --sandbox read-only, and cursor-agent is never used for investigate
jobs because it has no read-only switch.
Effectiveness measurement: New unit tests pin the exact argv of each investigate runner and prove a default
job still gets 60 seconds in an empty temp folder. The PR body pastes the live probe in which claude -p with
this argv was asked to Write a file and to touch one through Bash, answered
{"wrote":false,"read":true,"bash":false}, and neither target file existed afterwards.
Slice rationale: One behavior slice inside the judge, dormant until a hook asks for an investigate job, so a
reviewer can judge the permission argv on its own.
Architectural effect: llm-judge gains a second, read-only runner set chosen per job; the default runner set
and every current caller stay unchanged.
Goal: Give later hooks a safe way to ask a model to check a claim by reading files on this machine.
Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose stated blocker was false;
most could have been checked by reading a log, a transcript, or a source file already on disk. Today each
judge runner gets 60 seconds in an empty temp folder with no tool grants, so it cannot read anything.
Alternative considerations: claude --permission-mode plan was rejected after a live probe wrote a plan file
under ~/.claude/plans. A Bash prefix allowlist such as Bash(git log:*) was rejected after a live probe
blocked even plain git log, so shell access stays out of this slice.
Implementation details: In engine/hooks/llm-judge/judge.py add INVESTIGATE_RUNNERS beside DEFAULT_RUNNERS
holding the codex and claude argv given under Runner argv. Make runners() take the job mode and pick
INVESTIGATE_RUNNERS for "investigate"; CATSTACK_LLM_JUDGE_RUNNERS still replaces whichever set is picked.
Make run_runner accept timeout_seconds (default TIMEOUT_SECONDS, capped at INVESTIGATE_TIMEOUT_CAP = 600)
and cwd (used only when it is an absolute path to a folder that exists, else the temp folder plus a
judge.log line). Thread mode, timeout_seconds and cwd from the job through ask() and run_job().
Runner argv: codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT, and claude -p --model
haiku --settings '{"disableAllHooks": true}' --allowedTools Read Grep Glob --disallowedTools Write Edit
NotebookEdit Bash -- PROMPT.
Non-goals: No hook asks for an investigate job in this slice. No inbox edit, no change to DEFAULT_RUNNERS,
wrong-check-reflect or diu-stop. No shell or network access for the investigator.
Layer: domain
Feature state: dormant
Files:
- engine/hooks/llm-judge/judge.py
- engine/hooks/llm-judge/tests/test_judge.py
Change types:
- engine/hooks/llm-judge/judge.py: modify
- engine/hooks/llm-judge/tests/test_judge.py: modify
Acceptance criteria:
- Tests whose names contain "investigate" pass and pin both investigate runner argv exactly.
- A test proves a job lacking a mode key still runs DEFAULT_RUNNERS with a 60 second limit.
- A test proves timeout_seconds above 600 is capped at 600.
- A test whose name contains "missing" proves a cwd that does not exist falls back to a temp folder and logs it.
- python3 scripts/check_no_new_comments.py --base origin/main exits 0.
Invoker-Finalize-Id: 890bf8a3-d647-4273-855b-8ca21b411f55
…tep 1 for the llm-judge investigate mode. Review claim: The investigate-mode unit tests run and pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The run exits 0 and reports at least one test; Python exits 5 when -k matches nothing. Slice rationale: One proof step for this workflow's single change. Architectural effect: None; verification only. Goal: Prove the investigate runner argv and time limit deterministically. Motivation: Each slice carries its own executable proof. Alternative considerations: Manual inspection was rejected as non-deterministic. Implementation details: Run the filtered unittest discovery once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 964f7535-7395-4bb3-b657-37a77b357f44
…be the llm-judge investigate mode in the llm-judge README. Review claim: engine/hooks/llm-judge/README.md gains an Investigate mode section that matches judge.py. Review lane: docs Safety invariant: Only engine/hooks/llm-judge/README.md changes; no code or test file is edited. Effectiveness measurement: grep finds the section heading and each named flag in the README. Slice rationale: The README section for the behavior in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README only. Goal: Let the next hook author find the investigate mode and its limits. Motivation: catstack keeps every hook's contract in its README. Alternative considerations: A later README-only pull request was rejected; catstack's playbook counts a README that lands later as a known repair pattern. Implementation details: Add one "## Investigate mode" section to engine/hooks/llm-judge/README.md after "## Runner order". Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `grep -n "## Investigate mode" engine/hooks/llm-judge/README.md` exits 0. - The section names --sandbox read-only, --allowedTools Read Grep Glob, --disallowedTools Write Edit NotebookEdit Bash, and 600. Context: wf-1789279240280-10/implement-judge-investigate-mode (f2435ab): Let an llm-judge job opt into a read-only investigate mode with its own time limit and working folder. Review claim: A job carrying "mode": "investigate" runs on a second runner set whose Claude argv grants only the Read, Grep and Glob tools and forbids Write, Edit, NotebookEdit and Bash, with the job's own timeout_seconds and cwd; a job lacking that key runs exactly as today. Review lane: behavior Safety invariant: An investigate job never receives a tool that can write, edit, or execute shell text. The Claude runner argv always carries --allowedTools Read Grep Glob and --disallowedTools Write Edit NotebookEdit Bash, the Codex runner always carries --sandbox read-only, and cursor-agent is never used for investigate jobs because it has no read-only switch. Effectiveness measurement: New unit tests pin the exact argv of each investigate runner and prove a default job still gets 60 seconds in an empty temp folder. The PR body pastes the live probe in which claude -p with this argv was asked to Write a file and to touch one through Bash, answered {"wrote":false,"read":true,"bash":false}, and neither target file existed afterwards. Slice rationale: One behavior slice inside the judge, dormant until a hook asks for an investigate job, so a reviewer can judge the permission argv on its own. Architectural effect: llm-judge gains a second, read-only runner set chosen per job; the default runner set and every current caller stay unchanged. Goal: Give later hooks a safe way to ask a model to check a claim by reading files on this machine. Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose stated blocker was false; most could have been checked by reading a log, a transcript, or a source file already on disk. Today each judge runner gets 60 seconds in an empty temp folder with no tool grants, so it cannot read anything. Alternative considerations: claude --permission-mode plan was rejected after a live probe wrote a plan file under ~/.claude/plans. A Bash prefix allowlist such as Bash(git log:*) was rejected after a live probe blocked even plain git log, so shell access stays out of this slice. Implementation details: In engine/hooks/llm-judge/judge.py add INVESTIGATE_RUNNERS beside DEFAULT_RUNNERS holding the codex and claude argv given under Runner argv. Make runners() take the job mode and pick INVESTIGATE_RUNNERS for "investigate"; CATSTACK_LLM_JUDGE_RUNNERS still replaces whichever set is picked. Make run_runner accept timeout_seconds (default TIMEOUT_SECONDS, capped at INVESTIGATE_TIMEOUT_CAP = 600) and cwd (used only when it is an absolute path to a folder that exists, else the temp folder plus a judge.log line). Thread mode, timeout_seconds and cwd from the job through ask() and run_job(). Runner argv: codex exec --skip-git-repo-check --sandbox read-only -c notify=[] PROMPT, and claude -p --model haiku --settings '{"disableAllHooks": true}' --allowedTools Read Grep Glob --disallowedTools Write Edit NotebookEdit Bash -- PROMPT. Non-goals: No hook asks for an investigate job in this slice. No inbox edit, no change to DEFAULT_RUNNERS, wrong-check-reflect or diu-stop. No shell or network access for the investigator. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/judge.py - engine/hooks/llm-judge/tests/test_judge.py Change types: - engine/hooks/llm-judge/judge.py: modify - engine/hooks/llm-judge/tests/test_judge.py: modify Acceptance criteria: - Tests whose names contain "investigate" pass and pin both investigate runner argv exactly. - A test proves a job lacking a mode key still runs DEFAULT_RUNNERS with a 60 second limit. - A test proves timeout_seconds above 600 is capped at 600. - A test whose name contains "missing" proves a cwd that does not exist falls back to a temp folder and logs it. - python3 scripts/check_no_new_comments.py --base origin/main exits 0. Solution: Describe the llm-judge investigate mode in the llm-judge README. Review claim: engine/hooks/llm-judge/README.md gains an Investigate mode section that matches judge.py. Review lane: docs Safety invariant: Only engine/hooks/llm-judge/README.md changes; no code or test file is edited. Effectiveness measurement: grep finds the section heading and each named flag in the README. Slice rationale: The README section for the behavior in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README only. Goal: Let the next hook author find the investigate mode and its limits. Motivation: catstack keeps every hook's contract in its README. Alternative considerations: A later README-only pull request was rejected; catstack's playbook counts a README that lands later as a known repair pattern. Implementation details: Add one "## Investigate mode" section to engine/hooks/llm-judge/README.md after "## Runner order". Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `grep -n "## Investigate mode" engine/hooks/llm-judge/README.md` exits 0. - The section names --sandbox read-only, --allowedTools Read Grep Glob, --disallowedTools Write Edit NotebookEdit Bash, and 600. Invoker-Finalize-Id: c7674a0c-7e5f-4540-b149-295bbac2133e
…or the llm-judge investigate mode. Review claim: Every llm-judge test still passes after the change. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The whole llm-judge test folder exits 0. Slice rationale: One proof step guarding current judge callers. Architectural effect: None; verification only. Goal: Prove default jobs behave as before. Motivation: wrong-check-reflect and diu-stop already depend on the judge. Alternative considerations: Running only the new tests was rejected; it would miss a regression in the default runner set. Implementation details: Run the whole llm-judge test folder. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: c94db032-2d49-4594-b6e7-b3ac43e7b56b
…p 3 for the llm-judge investigate mode. Review claim: The change adds no explanatory code comments. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The comment gate exits 0 against origin/main. Slice rationale: One proof step for catstack's required comment gate. Architectural effect: None; verification only. Goal: Keep the CI comment gate green. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run the comment gate against origin/main. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 492bf3e7-0cff-46b3-b0d1-89687e47b6b7
…ate-mode/g0.t0.a-af1ca1213-1325fc3c
…ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active Exit code: 0 Invoker-Finalize-Id: 2aafd01f-794a-46f1-8763-0a04ed358458
…a00166eb5-1f479726 — Check that no ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎉 This pull request has been checked successfully and will be merged soon. 🎉
#539 is queued for merge on branch main (a95db0e).
Stacked behind 2 pull requests queued ahead of this batch, not part of it. These checks run on a tip that also carries their commits, so a failure here can come from them as much as from #539.
Queued ahead of this batch:
This pull request has been created by Mergify to speculatively check the mergeability of #539.
You don't need to do anything. Mergify will close this pull request automatically when it is complete.
Required conditions of queue rule
admin-bypassfor merge:check-success = lintcheck-success = testcheck-success = validateRequired conditions to stay in the queue:
-draftbase=mainlabel=admin-bypass