merge queue: checking #518 on main (0d0daba), stacked on #470 and #483 - #573
Closed
mergify[bot] wants to merge 49 commits into
Closed
mergify[bot] wants to merge 49 commits into
mergify[bot] wants to merge 49 commits into
Conversation
Regex hooks miss new phrasings of the same meaning. This adds a standard-library judge that any hook can hand a question to instead. enqueue() writes the job and starts a detached `judge.py run`, so the hook returns at once and the reply is never delayed. The background run tries codex (gpt-5.3-codex-spark), then claude haiku, then cursor-agent, and keeps the first runner whose stdout has a JSON-object line. verdict() turns the answer into hit, clean, or unchecked; unchecked is never clean. drain(transcript) hands back finished verdicts oldest first and deletes them. Runners get CATSTACK_LLM_JUDGE_CHILD=1, and enqueue() does nothing when it is set, so a judge never starts another judge. Nothing calls this yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hooks/llm-judge/judge.py: a runner fallback chain, a detached background job runner, and a per-transcript verdict store with hit, clean and unchecked outcomes. Review claim: judge.py runs a question through codex (gpt-5.3-codex-spark), then claude (haiku), then cursor-agent, stops at the first runner that returns parseable JSON, records every failed attempt with its reason, and never starts a job from inside a judge child process. Review lane: behavior Safety invariant: Nothing calls the library yet; it adds files only under engine/hooks/llm-judge/. Effectiveness measurement: Tests with fake runners cover fallback order, all-fail as unchecked with each reason, the recursion guard, and a detached job that writes its verdict after the enqueuing call has already returned. Slice rationale: The library is dormant foundation; delivery (step 2) and the first caller (step 3) are separate review claims. Architectural effect: Adds a dormant shared module for model-judged hook decisions; hooks can enqueue a question and a later hook reads the verdict. Goal: Give every catstack hook one reusable, backgrounded, harness-agnostic model judge. Motivation: Regex detectors keep missing new phrasings; the user asked for a reusable model judge across Codex, Claude and Cursor that never interrupts the main flow. Alternative considerations: Claude-only "type: prompt" Stop hooks were rejected (Claude-only and synchronous, adding latency to every reply). Calling a model inline in each hook was rejected (blocks the reply). A per-hook copy of the runner chain was rejected (drift). Implementation details: New directory engine/hooks/llm-judge/ with judge.py and tests/test_judge.py. Three runners in order (codex with model gpt-5.3-codex-spark, claude with model haiku and all hooks disabled, cursor-agent), each with a 60 second timeout, stdin from /dev/null, cwd set to a fresh temp directory, and CATSTACK_LLM_JUDGE_CHILD=1 in its environment; the exact argv for each is in the prompt. A runner whose binary is not on PATH is recorded as not installed. The answer is the last stdout line that parses as a JSON object. CATSTACK_LLM_JUDGE_RUNNERS (a JSON array of name and argv pairs) overrides the chain for tests; CATSTACK_LLM_JUDGE_STATE_DIR overrides the state root (default ~/.cache/catstack-llm-judge). Non-goals: No hook calls the library in this step. No install.sh or settings wiring. No change to any existing hook. Layer: domain Feature state: dormant Files: engine/hooks/llm-judge/judge.py, engine/hooks/llm-judge/tests/test_judge.py, engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/judge.py: create - engine/hooks/llm-judge/tests/test_judge.py: create - engine/hooks/llm-judge/README.md: create Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests` exits 0. - `python3 scripts/check_no_new_comments.py --base origin/main` exits 0. Exit code: 0
…library tests and the repo's no-comments gate. Review claim: The library tests and the no-comments gate pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The command exits 0 on the implemented branch. Slice rationale: One proof task for the library. Architectural effect: None; verification only. Goal: Prove the library deterministically with fake runners. Motivation: Fallback order, recursion guard and backgrounding are behaviors, so they need a test run. Alternative considerations: Calling real model CLIs in CI was rejected (network, auth and usage limits make it non-deterministic). Implementation details: Run the library tests and the comments gate. Non-goals: No product edits. Layer: app_regression Feature state: active Exit code: 0
…nly check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active Exit code: 0
…0a4ff72b-46acc04e — Terminal read-only check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active
A background verdict now reaches the agent at the start of its next turn in Claude (UserPromptSubmit additionalContext), Cursor (stop followup_message) and Codex (notify stderr, chained). A hit shows its on_hit text once; an unchecked verdict names each runner and why it failed; a clean one is silent. Missing transcripts and bad payloads are reported on stderr and exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…udge inbox hook for Claude, Cursor and Codex, and wire it into install.sh. Review claim: On the next turn the inbox hook drains this transcript's llm-judge verdicts and adds each hit's on_hit text, and each unchecked verdict's per-runner reasons, to the agent's context; with none waiting it prints nothing. Review lane: behavior Safety invariant: The inbox only adds text to the next prompt and never blocks; with no verdicts waiting it prints nothing. Effectiveness measurement: Tests seed a verdict store and assert the exact delivered text per harness, silence when empty, and that a delivered verdict is not delivered twice; tests/test_install.py asserts the link and the settings entry. Slice rationale: Delivery is one review claim; the library it reads is step 1 and the first caller is step 3. Architectural effect: Adds a UserPromptSubmit-time reader over llm-judge's verdict store; hooks that enqueue questions get their answers one turn later. Goal: Make background verdicts reach the agent in every harness. Motivation: A verdict nobody reads changes nothing, and an unchecked verdict that is dropped hides a broken judge. Alternative considerations: Delivering from the Stop hook of the same turn was rejected (the verdict is not ready yet, and waiting would block the reply). A Claude-only inbox was rejected (catstack hooks ship for Claude, Cursor and Codex). Implementation details: In engine/hooks/llm-judge/ add claude_prompt_submit.py (Claude UserPromptSubmit, prints hookSpecificOutput.additionalContext), cursor_session.py (Cursor stop, prints followup_message), codex_notify.py (Codex notify, advisory text on stderr), inbox.py (shared formatting over judge.drain), claude.hook.json, install_claude_hook.py, install_cursor_hook.py, install_codex_notify.py, and tests/test_inbox.py. Mirror the file shapes of engine/hooks/wrong-check-reflect/. Wire it like every other hook: link_item in install.sh, the settings merge, an assertion in tests/test_install.py, a README section, and a docs/ecosystem.md row. Follow product/skills/ship-a-detector/playbooks/detector-lifecycle.md steps 14-19 for the wiring. Non-goals: No hook enqueues questions in this step. No change to judge.py behavior. No change to wrong-check-reflect. Layer: app_bridge Feature state: active Files: engine/hooks/llm-judge/inbox.py, engine/hooks/llm-judge/claude_prompt_submit.py, engine/hooks/llm-judge/cursor_session.py, engine/hooks/llm-judge/codex_notify.py, engine/hooks/llm-judge/claude.hook.json, engine/hooks/llm-judge/install_claude_hook.py, engine/hooks/llm-judge/install_cursor_hook.py, engine/hooks/llm-judge/install_codex_notify.py, engine/hooks/llm-judge/tests/test_inbox.py, engine/hooks/llm-judge/README.md, install.sh, tests/test_install.py, docs/ecosystem.md Change types: - engine/hooks/llm-judge/inbox.py: create - engine/hooks/llm-judge/claude_prompt_submit.py: create - engine/hooks/llm-judge/cursor_session.py: create - engine/hooks/llm-judge/codex_notify.py: create - engine/hooks/llm-judge/claude.hook.json: create - engine/hooks/llm-judge/install_claude_hook.py: create - engine/hooks/llm-judge/install_cursor_hook.py: create - engine/hooks/llm-judge/install_codex_notify.py: create - engine/hooks/llm-judge/tests/test_inbox.py: create - engine/hooks/llm-judge/README.md: modify - install.sh: modify - tests/test_install.py: modify - docs/ecosystem.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/llm-judge` exits 0. - `python3 -m unittest tests/test_install.py` exits 0. Exit code: 0
…sts, the hook coverage gate, and the install tests. Review claim: The inbox tests, hook coverage gate and install tests pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The command exits 0 on the implemented branch. Slice rationale: One proof task for the inbox slice. Architectural effect: None; verification only. Goal: Prove delivery and wiring deterministically. Motivation: An unwired hook looks identical to a working one, so the install assertion is part of the proof. Alternative considerations: The full suite was rejected as slower with no extra signal here. Implementation details: Run the three commands in order. Non-goals: No product edits. Layer: app_regression Feature state: active Exit code: 0
…nly check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active Exit code: 0
…dbabd401-e965207c — Terminal read-only check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active
…-reflect sends a background llm-judge question for each reply that answers a user message, alongside its existing regex path. Review claim: After each reply that follows a user message, wrong-check-reflect sends one llm-judge job asking for pushback and self_correction, with hit_if_all_true both keys and on_hit its existing reflect follow-up; the regex path and its once-per-transcript rule are unchanged. Review lane: behavior Safety invariant: The judge runs in the background and never blocks a reply; the worst case is one extra reflect reminder. Effectiveness measurement: A test with a fake runner answering yes to both keys shows the reflect follow-up delivered by the inbox on the next prompt for the exact "You're right. ... I misread which diff you meant." exchange; the Stop hook returns in under one second while the fake runner sleeps two. Slice rationale: The first caller of the judge; the library and inbox are the two slices below it. Architectural effect: wrong-check-reflect gains a second, model-judged path that reports through the llm-judge inbox one turn later; its synchronous regex path stays as the instant backstop. Goal: Catch self-corrections after pushback in any wording. Motivation: Regexes keep missing the next phrasing; the user asked for model judgement after pushback that never interrupts the reply. Alternative considerations: Replacing the regex path was rejected (it is instant and works when every runner is down). Judging only when a regex fires was rejected (that is the case regexes already catch). Implementation details: In engine/hooks/wrong-check-reflect/detect.py add enqueue_judge(payload) that reads the transcript, takes the last user message and the assistant message before it, and when both exist and the current reply is non-empty, imports judge from the sibling llm-judge directory and sends a job whose prompt is the classifier prompt given in the task prompt. Call it from claude_stop_check.py, cursor_session.py and codex_notify.py after the regex decision, except when stop_hook_active is set, when the regex fired this turn, or when the transcript was already prompted. Any exception is written to stderr with context and never changes the hook's exit status. Non-goals: No change to the regex patterns, the follow-up text, or the once-per-transcript rule. No change to llm-judge. Layer: app_bridge Feature state: active Files: engine/hooks/wrong-check-reflect/detect.py, engine/hooks/wrong-check-reflect/claude_stop_check.py, engine/hooks/wrong-check-reflect/cursor_session.py, engine/hooks/wrong-check-reflect/codex_notify.py, engine/hooks/wrong-check-reflect/tests/test_hooks.py, engine/hooks/wrong-check-reflect/README.md Change types: - engine/hooks/wrong-check-reflect/detect.py: modify - engine/hooks/wrong-check-reflect/claude_stop_check.py: modify - engine/hooks/wrong-check-reflect/cursor_session.py: modify - engine/hooks/wrong-check-reflect/codex_notify.py: modify - engine/hooks/wrong-check-reflect/tests/test_hooks.py: modify - engine/hooks/wrong-check-reflect/README.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/wrong-check-reflect` exits 0. Solution: wrong-check-reflect sends a background llm-judge question for each reply that answers a user message, alongside its existing regex path. Review claim: After each reply that follows a user message, wrong-check-reflect sends one llm-judge job asking for pushback and self_correction, with hit_if_all_true both keys and on_hit its existing reflect follow-up; the regex path and its once-per-transcript rule are unchanged. Review lane: behavior Safety invariant: The judge runs in the background and never blocks a reply; the worst case is one extra reflect reminder. Effectiveness measurement: A test with a fake runner answering yes to both keys shows the reflect follow-up delivered by the inbox on the next prompt for the exact "You're right. ... I misread which diff you meant." exchange; the Stop hook returns in under one second while the fake runner sleeps two. Slice rationale: The first caller of the judge; the library and inbox are the two slices below it. Architectural effect: wrong-check-reflect gains a second, model-judged path that reports through the llm-judge inbox one turn later; its synchronous regex path stays as the instant backstop. Goal: Catch self-corrections after pushback in any wording. Motivation: Regexes keep missing the next phrasing; the user asked for model judgement after pushback that never interrupts the reply. Alternative considerations: Replacing the regex path was rejected (it is instant and works when every runner is down). Judging only when a regex fires was rejected (that is the case regexes already catch). Implementation details: In engine/hooks/wrong-check-reflect/detect.py add enqueue_judge(payload) that reads the transcript, takes the last user message and the assistant message before it, and when both exist and the current reply is non-empty, imports judge from the sibling llm-judge directory and sends a job whose prompt is the classifier prompt given in the task prompt. Call it from claude_stop_check.py, cursor_session.py and codex_notify.py after the regex decision, except when stop_hook_active is set, when the regex fired this turn, or when the transcript was already prompted. Any exception is written to stderr with context and never changes the hook's exit status. Non-goals: No change to the regex patterns, the follow-up text, or the once-per-transcript rule. No change to llm-judge. Layer: app_bridge Feature state: active Files: engine/hooks/wrong-check-reflect/detect.py, engine/hooks/wrong-check-reflect/claude_stop_check.py, engine/hooks/wrong-check-reflect/cursor_session.py, engine/hooks/wrong-check-reflect/codex_notify.py, engine/hooks/wrong-check-reflect/tests/test_hooks.py, engine/hooks/wrong-check-reflect/README.md Change types: - engine/hooks/wrong-check-reflect/detect.py: modify - engine/hooks/wrong-check-reflect/claude_stop_check.py: modify - engine/hooks/wrong-check-reflect/cursor_session.py: modify - engine/hooks/wrong-check-reflect/codex_notify.py: modify - engine/hooks/wrong-check-reflect/tests/test_hooks.py: modify - engine/hooks/wrong-check-reflect/README.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/wrong-check-reflect` exits 0.
…k-reflect's tests, llm-judge's tests, and the hook coverage gate. Review claim: Both hooks' tests and the coverage gate pass together. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The command exits 0 on the implemented branch. Slice rationale: One proof task for the adoption slice. Architectural effect: None; verification only. Goal: Prove the model-judged path end to end with fake runners. Motivation: The path crosses two hooks, so both suites run. Alternative considerations: Real model CLIs in CI were rejected as non-deterministic. Implementation details: Run both suites and the coverage gate. Non-goals: No product edits. Layer: app_regression Feature state: active Exit code: 0
…nly check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active Exit code: 0
…be7af30b-bd3c4097 — Terminal read-only check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active
CI failed only on check_no_new_comments: one line in detect.py and six in tests/test_hooks.py. No code changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: I63f3cca645715ef5bd7ab89478b6fdb98e55d844
…er can declare its meaning as a JSON phrase dictionary, and engine/hooks/llm-judge/phrases.py turns that dictionary plus a text into an llm-judge job. Review lane: behavior Safety invariant: No hook or check behavior changes. Only new files under engine/hooks/llm-judge/; no existing hook imports the new module. Pending user confirmation in this session. Effectiveness measurement: Unit tests render the example dictionary into a prompt and assert it contains the meaning, every match phrase, every not_match phrase, and the text; a malformed dictionary raises ValueError naming the file and the bad key. Slice rationale: The dictionary format and loader are one reviewable format. Wiring it into a hook is a later slice so this diff carries no behavior change. Architectural effect: Adds a data format (phrases/<checker>.json) and one pure module that builds llm-judge jobs; checkers stop owning prompt text. Goal: Create phrases.py, one example dictionary, and its tests. Motivation: Checkers that match regexes against prose miss rewordings and fire on quoted or negated text. A model judged against a phrase list catches rewordings: a probe with claude haiku matched "Fair point, I got that backwards earlier." and did not match "The migration finished and the table is live.". Alternative considerations: Keeping prompt strings inside each hook (as PR 425 does with JUDGE_PROMPT) was rejected because every hook would invent its own prompt shape. A synchronous Claude "type: prompt" hook was rejected because diu-stop used one before and it wrote its raw reasoning into the chat (see the docstring at the top of engine/hooks/diu-stop/claude_stop_check.py). Implementation details: Dictionary format, one JSON object per file at engine/hooks/llm-judge/phrases/<checker>.json with keys checker (string equal to the file stem), meaning (one sentence), reads (one of "reply", "user", "exchange"), match (non-empty array of strings), not_match (array of strings), on_hit (string shown to the agent on a hit). phrases.py provides load(checker, directory=None), prompt(dictionary, text), and job(dictionary, transcript, text) as specified in the prompt. Non-goals: Change only the three files listed; no existing hook changes, no inbox or delivery changes, no model calls in tests. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/phrases.py - engine/hooks/llm-judge/phrases/example.json - engine/hooks/llm-judge/tests/test_phrases.py Change types: - engine/hooks/llm-judge/phrases.py: create - engine/hooks/llm-judge/phrases/example.json: create - engine/hooks/llm-judge/tests/test_phrases.py: create Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests -v` exits 0. - `python3 scripts/check_no_new_comments.py --base origin/plan/llm-judge-3-wrong-check-reflect-asks-the-judge-about-pushback-and-take-backs` exits 0. Exit code: 0 Invoker-Finalize-Id: 9d865cdb-b414-49e4-8e28-719759fc61a7
…m: A phrase-judge skill tells authors to declare a checker's meaning as a phrase dictionary and to grow it with real misses instead of writing a regex; the llm-judge README documents the format. Review lane: docs Safety invariant: Documentation and a skill only; no hook, check, or install behavior changes beyond making the new skill visible to Claude, Cursor, and Codex. Pending user confirmation in this session. Effectiveness measurement: The skill test asserts the skill has name and description frontmatter and names phrases/<checker>.json; scripts/check_skills_three_harnesses.py passes with the new skill present. Slice rationale: The prose that teaches the format is reviewed apart from the loader code. Architectural effect: Adds one engine skill; no runtime effect. Goal: Create the phrase-judge skill, its test, and a README section. Motivation: A format with no guidance invites authors to go back to regex the first time a phrasing slips through. Alternative considerations: Putting the guidance only in the README was rejected because agents load skills, not READMEs, when writing a checker. Implementation details: Follow engine/skills/create-skill/SKILL.md. The skill states the dictionary keys, says to seed match and not_match from real pasted texts, to add each new miss or false alarm as a phrase, to use regex only for fixed machine formats, and that dictionary checkers never block because the judge answers in the background. Non-goals: No hook code, no loader changes. Layer: docs Feature state: dormant Files: - engine/skills/phrase-judge/SKILL.md - engine/skills/phrase-judge/tests/test_phrase_judge_skill.py - engine/hooks/llm-judge/README.md Change types: - engine/skills/phrase-judge/SKILL.md: create - engine/skills/phrase-judge/tests/test_phrase_judge_skill.py: create - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/skills/phrase-judge/tests -v` exits 0. - `python3 scripts/check_skills_three_harnesses.py` exits 0. Exit code: 0 Invoker-Finalize-Id: 78a590ff-6980-48f3-abab-1ef3a74ba949
…: The llm-judge unit tests, including the new phrase tests, pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The unittest run is the direct measurement. Slice rationale: One focused proof per claim before review. Architectural effect: None; verification only. Goal: Prove the loader works. Layer exception: allowed. Proof runs after the docs task so it checks the finished branch; it reads files only and changes nothing. Motivation: Tests existing is not proof; running them is. Alternative considerations: The full suite was rejected; this module is the smallest honest proof. Implementation details: Run the llm-judge unittest discover. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when every llm-judge test passes. Exit code: 0 Invoker-Finalize-Id: 76845347-e176-42de-a12c-85f6251b3463
…im: No hook outside engine/hooks/llm-judge/ refers to the phrase module, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing calls the new module yet. Layer exception: allowed. Proof runs after the docs task so it checks the finished branch; it reads files only and changes nothing. Motivation: The safety invariant says no behavior changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the module under engine/hooks, excluding llm-judge itself. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when no hook outside llm-judge mentions the phrase module. Exit code: 0 Invoker-Finalize-Id: c14b40cf-ece4-4c41-9e75-5140ff367c8f
…: The phrase-judge skill test and the three-harness skill check pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The two commands are the direct measurement. Slice rationale: One focused proof per claim before review. Architectural effect: None; verification only. Goal: Prove the skill is well formed and visible to every harness. Layer exception: allowed. The skill check needs the skill file the docs task creates; it reads files only and changes nothing. Motivation: A skill that only one harness can see is unfinished. Alternative considerations: Checking one harness only was rejected. Implementation details: Run the skill unittest discover and the three-harness check. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when both pass. Exit code: 0 Invoker-Finalize-Id: 8386d7ba-ffb5-4005-b1bc-17ec6d5f2c2c
…ks/g0.t0.a-a2da10d77-9902a3ad
…mant/g0.t0.a-a3c533ce6-fa1d2622
…o ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: 5bf9f247-fef6-4e8b-9c38-826d558c5b6d
…f1afd028-112f90c9 — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0.
The prompt listed example phrases but never said to match their meaning.
A real retraction ("Correction: the file I pointed you to earlier is not
the one in use; the real one is src/b.py.") scored match=false in 6 of 6
live claude runs. With one added sentence it scored true in 3 of 3, and the
two not_match examples stayed false in 4 of 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF
Change-Id: Id5112866db12ed375f7677ff411c86fc7c4a7cf1
The engine-only install now links the new engine skill phrase-judge, so TestEngineOnly's hardcoded ENGINE_SKILLS set was one short and CI failed: "Items in the first set but not the second: 'phrase-judge'". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: I9a0c5514443f4558581b562ce0950bdac1bf4808
main now holds #425 as squash commit 5b54fd0. The only conflict was the llm-judge README: this branch adds the "Phrase dictionaries" section where main's side of the block was empty, so the section is kept and nothing from main is dropped. Checked on the merged tree: llm-judge 46 OK, phrase-judge skill 3 OK, check_skills_three_harnesses ok, check_no_new_comments --base origin/main ok, tests.test_install.TestEngineOnly 5 OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: I47726eb7e4e9484fe31419f275963f87cbcf561d
…claim: After any tool call, a finished llm-judge verdict for the current transcript is printed in the harness's after-tool format by one small script per harness (Claude, Cursor, Codex).
Review lane: behavior
Safety invariant: Nothing installs or calls the new scripts in this step; existing delivery at the next prompt is unchanged; every new script exits 0 on every input. Pending user confirmation in this session.
Effectiveness measurement: Tests plant a finished hit verdict, run each script, and assert the on_hit text appears in that harness's JSON exactly once across two runs.
Slice rationale: Delivery scripts and their tests are one reviewable behavior. Installing them into each harness is the next slice, so this diff changes no running agent.
Architectural effect: llm-judge gains a second delivery moment (after a tool call) beside the existing next-prompt delivery; both drain the same verdict store, so each verdict is still delivered once.
Goal: Create the three scripts and one test file.
Motivation: A judge hit found mid-task should reach the agent at its next step, not after the human speaks again.
Alternative considerations: Blocking the tool call until the judge answers was rejected by the user ("keep going and interrupt when the hook fires"). A single shared script with a harness flag was rejected because each harness passes a different payload and expects a different output shape; three thin files match how build-the-lever and the existing llm-judge delivery scripts are laid out.
Implementation details: Claude and Codex print {"hookSpecificOutput": {"hookEventName": "PostToolUse", "additionalContext": <texts joined by a blank line>}} (the shape engine/hooks/answer-overrides-menu/claude_posttooluse.py and engine/hooks/build-the-lever/codex_posttooluse.py already print). Cursor prints {"additional_context": <texts>} (the shape engine/hooks/build-the-lever/cursor_post_tool_use.py prints). The transcript comes from inbox.resolve_transcript(payload). With no transcript, a bad payload, or a drain error, the script writes one stderr line naming the harness and the error and prints nothing on stdout.
Non-goals: No installer, settings, or install.sh change; no change to inbox.py, judge.py, phrases.py, or the existing delivery scripts; no model calls in tests.
Layer: transport
Feature state: dormant
Files:
- engine/hooks/llm-judge/claude_post_tool_use.py
- engine/hooks/llm-judge/cursor_post_tool_use.py
- engine/hooks/llm-judge/codex_post_tool_use.py
- engine/hooks/llm-judge/tests/test_post_tool_use.py
Change types:
- engine/hooks/llm-judge/claude_post_tool_use.py: create
- engine/hooks/llm-judge/cursor_post_tool_use.py: create
- engine/hooks/llm-judge/codex_post_tool_use.py: create
- engine/hooks/llm-judge/tests/test_post_tool_use.py: create
Acceptance criteria:
- `python3 -m unittest discover -s engine/hooks/llm-judge/tests -v` exits 0.
- `python3 scripts/check_no_new_comments.py --base origin/plan/phrase-judge-1-checkers-declare-meaning-as-phrase-dictionaries` exits 0.
Exit code: 0
Invoker-Finalize-Id: da714ff5-5c7d-444f-b6ea-0d4666d0125a
… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Exit code: 1
… claim: The llm-judge tests, including the new after-tool delivery tests, pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The unittest run is the direct measurement. Slice rationale: One focused proof for the delivery behavior before review. Architectural effect: None; verification only. Goal: Prove each script delivers a verdict once and stays quiet otherwise. Motivation: Tests existing is not proof; running them is. Alternative considerations: The full suite was rejected; this module is the smallest honest proof. Implementation details: Run the llm-judge unittest discover. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when every llm-judge test passes. Exit code: 0
… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Exit code: 1
… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Solution: Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them.
… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Exit code: 0
…s-not-installed/g1.t6.a-ac332d93f-93fa90a1
…o ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: cf921902-89e1-4eb2-b9eb-a0ab7656ec3d
…ext — Add a detector for a capability enumeration restated out of error-shaped tool output. Review claim: An outgoing message that restates two or more allowed values from error-shaped tool output beside a capability verb is flagged, unless a non-error source for those values exists or the message attributes them. Review lane: behavior Safety invariant: Every message the shipped hook flags today is still flagged with the same text, and every message it passes today still passes unless it matches this new rule. The 24 existing cases keep passing untouched. Effectiveness measurement: `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0, with new positive cases that exit non-zero before this change and the negative cases below staying silent both before and after. Slice rationale: One conceptual unit: one new rule inside an existing detector, with its fixtures. Architectural effect: None; the hook already parses transcript lines at Stop. This reads tool output lines it already has. Goal: Stop a value lifted out of an error string from going out as the authority it was never sourced from. Motivation: Three outgoing messages asserted a third party's capabilities using a list copied from a second party's error text. The operator believed them and acted. Alternative considerations: A PostToolUse rule firing when the string enters was set aside on measurement -- the value arrived 334 transcript rows before the last assertion, so an entry-time nudge fires far too early to bind the decision. Widening the shipped claim gate was set aside -- that gate asks whether a sentence is hedged; this asks where a value came from, a different question, and three open pull requests are already churning that file. Implementation details: Capture enumerations appearing inside error-shaped tool output in the transcript lines the hook already parses. At Stop, flag an outgoing message restating two or more captured values within range of a capability verb, when no non-error occurrence of those values exists in the transcript yet. Clear on an attribution phrase such as fallback, hardcoded, or in the error. Non-goals: No new hook directory. No change to the shipped hedge or diagnosis rules. No change to the claim gate in engine/hooks/diu-stop. No attempt to judge whether a check was cheap or available -- that is not computable and is out of scope. Layer: domain Feature state: active Files: engine/hooks/hedge-runs-prove-it/detect.py, engine/hooks/hedge-runs-prove-it/tests/test_hooks.py, engine/hooks/hedge-runs-prove-it/README.md Change types: - engine/hooks/hedge-runs-prove-it/detect.py: modify - engine/hooks/hedge-runs-prove-it/tests/test_hooks.py: modify - engine/hooks/hedge-runs-prove-it/README.md: modify Acceptance criteria: - `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0 after the change, with all 24 existing cases still passing. - A message restating two or more values that appear only inside error-shaped tool output, beside a capability verb, is flagged. - The same message is not flagged once a non-error source for those values exists in the transcript. - A message attributing the list as a fallback is not flagged. Solution: Add a detector for a capability enumeration restated out of error-shaped tool output. Review claim: An outgoing message that restates two or more allowed values from error-shaped tool output beside a capability verb is flagged, unless a non-error source for those values exists or the message attributes them. Review lane: behavior Safety invariant: Every message the shipped hook flags today is still flagged with the same text, and every message it passes today still passes unless it matches this new rule. The 24 existing cases keep passing untouched. Effectiveness measurement: `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0, with new positive cases that exit non-zero before this change and the negative cases below staying silent both before and after. Slice rationale: One conceptual unit: one new rule inside an existing detector, with its fixtures. Architectural effect: None; the hook already parses transcript lines at Stop. This reads tool output lines it already has. Goal: Stop a value lifted out of an error string from going out as the authority it was never sourced from. Motivation: Three outgoing messages asserted a third party's capabilities using a list copied from a second party's error text. The operator believed them and acted. Alternative considerations: A PostToolUse rule firing when the string enters was set aside on measurement -- the value arrived 334 transcript rows before the last assertion, so an entry-time nudge fires far too early to bind the decision. Widening the shipped claim gate was set aside -- that gate asks whether a sentence is hedged; this asks where a value came from, a different question, and three open pull requests are already churning that file. Implementation details: Capture enumerations appearing inside error-shaped tool output in the transcript lines the hook already parses. At Stop, flag an outgoing message restating two or more captured values within range of a capability verb, when no non-error occurrence of those values exists in the transcript yet. Clear on an attribution phrase such as fallback, hardcoded, or in the error. Non-goals: No new hook directory. No change to the shipped hedge or diagnosis rules. No change to the claim gate in engine/hooks/diu-stop. No attempt to judge whether a check was cheap or available -- that is not computable and is out of scope. Layer: domain Feature state: active Files: engine/hooks/hedge-runs-prove-it/detect.py, engine/hooks/hedge-runs-prove-it/tests/test_hooks.py, engine/hooks/hedge-runs-prove-it/README.md Change types: - engine/hooks/hedge-runs-prove-it/detect.py: modify - engine/hooks/hedge-runs-prove-it/tests/test_hooks.py: modify - engine/hooks/hedge-runs-prove-it/README.md: modify Acceptance criteria: - `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0 after the change, with all 24 existing cases still passing. - A message restating two or more values that appear only inside error-shaped tool output, beside a capability verb, is flagged. - The same message is not flagged once a non-error source for those values exists in the transcript. - A message attributing the list as a fallback is not flagged. Invoker-Finalize-Id: 8156de0f-c490-468a-a9e5-0fb0dcdcc394
…the hook's own suite as the shared repro for this slice. Review claim: The suite exits 0 only when the new rule flags the copied-from-error shape and stays silent on all four negative shapes. Review lane: proof Safety invariant: The repro is identical before and after; the new positive cases exit non-zero before and 0 after. Effectiveness measurement: The exit status of the suite is the effectiveness signal for this slice. Slice rationale: One conceptual unit: the deterministic repro only. Architectural effect: None. Goal: Deterministically prove the new rule. Motivation: Nothing today asserts where a restated value came from, which is why three messages carried one out of an error string unchallenged. Alternative considerations: Covering only the positive shape was set aside -- without the non-error-source and attributed cases, a rule that fires on every restatement would still exit 0. Implementation details: Execute the suite as the terminal gate. Non-goals: No product edits here. Layer: e2e_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 5910aca8-b6ae-453c-b815-4076d8d90fca
…only gate confirming no ephemeral handoff files were left behind. Review claim: The workflow leaves no ephemeral handoff files in the tree. Review lane: proof Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work. Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal. Slice rationale: One conceptual unit: the hygiene gate. Architectural effect: None. Goal: Confirm no ephemeral handoff files remain after every other task finishes. Motivation: Ephemeral inter-task files leak into the diff and read as part of the change. Alternative considerations: Manual inspection was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: e2e_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 7b20f59a-7962-415f-ad6c-6f9e3a8811b9
…a437a0460-90926507 — Terminal read-only gate confirming no ephemeral handoff files were left behind. Review claim: The workflow leaves no ephemeral handoff files in the tree. Review lane: proof Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work. Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal. Slice rationale: One conceptual unit: the hygiene gate. Architectural effect: None. Goal: Confirm no ephemeral handoff files remain after every other task finishes. Motivation: Ephemeral inter-task files leak into the diff and read as part of the change. Alternative considerations: Manual inspection was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: e2e_regression Feature state: active
Resolve the add/add conflicts by keeping main's files. The task's work already landed through #444, and main's versions are newer: example.json carries the unwired_reason that #480's checker requires. Drop the cursor_post_tool_use.py -> cursor_posttooluse.py renames in bug-complaint-leak and repeat-error-stop from d654dce. A read-only verify task made them, and main's repeat-error-stop/tests/test_hooks.py imports cursor_post_tool_use, so the renames would break that test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
narrow-the-scope fired four times in one session and was wrong all four. Its VERIFY_RE knew pytest, npm test, tsc and friends, but nothing about `bash scripts/test-foo.sh` or `docker build` -- so a shell-and-container project looks exactly like an edit streak with no verification in it, and the reminder becomes noise. A detector that cries wolf trains the reader to skip it, so this adds those patterns plus one more reset rule: a Bash command naming the basename of a file in the current streak clears that file, because running the script you just edited is how this kind of work gets checked and no regex can enumerate every project's entry point. tests/fixtures/shell_verified_streak_2026-09-11.json is the verbatim sequence from that session, silent with the verification row and firing without it. hook-freshness watched the checkout behind ~/.claude/hooks for staleness but never checked that the registered hook scripts still resolve. In that same session ~/.claude/hooks/split-scope pointed into a deleted worktree: the hook was registered, could not run, and said nothing for the whole session, which is "a check that could not run is not a pass" breaking on the gates themselves. The sweep reads settings.json, expands the paths, and names every one that does not resolve. It returns three outcomes, not two -- an unreadable or unparseable settings.json reports unchecked rather than clean, which the first draft of this change got wrong until explicit-failures caught it. install.sh warns when it is linking out of a git worktree, so the next dead link is announced as it is created rather than found a session later. It deliberately does not redirect to the main checkout: tests/test_install.py:290 specifies that install.sh links against its own checkout, because installing from a worktree is how a branch gets tested. An earlier draft did redirect, and that test caught it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZqCxSQVpmM9Rx37C7irgf Change-Id: I09fda6fc2dd315cd16eca766d881cb6f2b7306e6
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎉 This pull request has been checked successfully and will be merged soon. 🎉
#518 is queued for merge on branch main (0d0daba).
Stacked behind 2 pull requests queued ahead of this batch, not part of it. These checks run on a tip that also carries their commits, so a failure here can come from them as much as from #518.
Queued ahead of this batch:
This pull request has been created by Mergify to speculatively check the mergeability of #518.
You don't need to do anything. Mergify will close this pull request automatically when it is complete.
Required conditions of queue rule
admin-bypassfor merge:check-success = lintcheck-success = testcheck-success = validateRequired conditions to stay in the queue:
-draftbase=mainlabel=admin-bypass