Phrase judge (2) finished verdicts reach the agent after its next tool call by EdbertChan · Pull Request #470 · EdbertChan/catstack

EdbertChan · 2026-09-12T05:24:43Z

Summary

This branch changes no files. Its tree is identical to main.

The after-tool delivery scripts it was built to add already landed on main through #444.

The merge commit kept main's copies of every conflicting file, so nothing from the task branch survived.

There is nothing left to review. This PR should be closed instead of merged.

Review Claim

The branch adds no change beyond main: git diff --name-only origin/main HEAD lists zero files.

Review Lane

behavior

Review Unit

routing

Safety Invariant

Merging this branch cannot change any file. Its tree matches origin/main exactly, as the Test Plan output shows.

Slice Rationale

The planned slice (three after-tool delivery scripts plus one test file) already shipped as #444. That leaves this workflow's output as a duplicate with an empty diff.

Non-goals

No change to engine/hooks/llm-judge/claude_post_tool_use.py, cursor_post_tool_use.py, codex_post_tool_use.py, or tests/test_post_tool_use.py.
No change to installers, hook JSON fragments, or install.sh.
The workflow's "nothing installs the scripts yet" invariant does not describe main. On main, claude.tool.hook.json, codex.hook.json, and install_cursor_hook.py already point at these scripts (also from Phrase judge (2) finished verdicts reach the agent after its next tool call #444).

Test Plan

The branch tree matches origin/main:

$ git diff --quiet origin/main HEAD; echo "git diff --quiet exit $?"; git diff --name-only origin/main HEAD | wc -l
git diff --quiet exit 0
       0

llm-judge tests pass:

$ python3 -m unittest discover -s engine/hooks/llm-judge/tests
Ran 59 tests in 5.330s

OK

No handoff artifacts remain:

$ bash scripts/scrub-handoff-artifacts.sh; echo "exit $?"
scrub-handoff-artifacts-ok
exit 0

On main, the hook files already name the scripts:

$ git grep -n -e claude_post_tool_use -e cursor_post_tool_use -e codex_post_tool_use HEAD -- engine/hooks/llm-judge/claude.tool.hook.json engine/hooks/llm-judge/install_cursor_hook.py engine/hooks/llm-judge/codex.hook.json
HEAD:engine/hooks/llm-judge/claude.tool.hook.json:9:            "command": "python3 $HOME/.claude/hooks/llm-judge/claude_post_tool_use.py",
HEAD:engine/hooks/llm-judge/codex.hook.json:9:            "command": "python3 $HOME/.codex/hooks/llm-judge/codex_post_tool_use.py",
HEAD:engine/hooks/llm-judge/install_cursor_hook.py:16:        "command": "python3 $HOME/.cursor/hooks/llm-judge/cursor_post_tool_use.py",
HEAD:engine/hooks/llm-judge/install_cursor_hook.py:23:    "postToolUse": "llm-judge/cursor_post_tool_use.py",

The PR body validator did not check this body. Its preflight step refuses a branch with no changed files:

$ node scripts/validate-pr-body-local.mjs --body-file /tmp/pr-body-phrase-judge-2.md --base main
no changed files vs origin/main
validate-pr-body-local: engine/skills/make-pr/scripts/preflight.py exited 2 against origin/main

Revert Plan

Safe to revert? Yes
Revert command: none needed; closing the PR leaves main unchanged
Post-revert steps: None
Data migration? No

🤖 Generated with Claude Code

Regex hooks miss new phrasings of the same meaning. This adds a standard-library judge that any hook can hand a question to instead. enqueue() writes the job and starts a detached `judge.py run`, so the hook returns at once and the reply is never delayed. The background run tries codex (gpt-5.3-codex-spark), then claude haiku, then cursor-agent, and keeps the first runner whose stdout has a JSON-object line. verdict() turns the answer into hit, clean, or unchecked; unchecked is never clean. drain(transcript) hands back finished verdicts oldest first and deletes them. Runners get CATSTACK_LLM_JUDGE_CHILD=1, and enqueue() does nothing when it is set, so a judge never starts another judge. Nothing calls this yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…hooks/llm-judge/judge.py: a runner fallback chain, a detached background job runner, and a per-transcript verdict store with hit, clean and unchecked outcomes. Review claim: judge.py runs a question through codex (gpt-5.3-codex-spark), then claude (haiku), then cursor-agent, stops at the first runner that returns parseable JSON, records every failed attempt with its reason, and never starts a job from inside a judge child process. Review lane: behavior Safety invariant: Nothing calls the library yet; it adds files only under engine/hooks/llm-judge/. Effectiveness measurement: Tests with fake runners cover fallback order, all-fail as unchecked with each reason, the recursion guard, and a detached job that writes its verdict after the enqueuing call has already returned. Slice rationale: The library is dormant foundation; delivery (step 2) and the first caller (step 3) are separate review claims. Architectural effect: Adds a dormant shared module for model-judged hook decisions; hooks can enqueue a question and a later hook reads the verdict. Goal: Give every catstack hook one reusable, backgrounded, harness-agnostic model judge. Motivation: Regex detectors keep missing new phrasings; the user asked for a reusable model judge across Codex, Claude and Cursor that never interrupts the main flow. Alternative considerations: Claude-only "type: prompt" Stop hooks were rejected (Claude-only and synchronous, adding latency to every reply). Calling a model inline in each hook was rejected (blocks the reply). A per-hook copy of the runner chain was rejected (drift). Implementation details: New directory engine/hooks/llm-judge/ with judge.py and tests/test_judge.py. Three runners in order (codex with model gpt-5.3-codex-spark, claude with model haiku and all hooks disabled, cursor-agent), each with a 60 second timeout, stdin from /dev/null, cwd set to a fresh temp directory, and CATSTACK_LLM_JUDGE_CHILD=1 in its environment; the exact argv for each is in the prompt. A runner whose binary is not on PATH is recorded as not installed. The answer is the last stdout line that parses as a JSON object. CATSTACK_LLM_JUDGE_RUNNERS (a JSON array of name and argv pairs) overrides the chain for tests; CATSTACK_LLM_JUDGE_STATE_DIR overrides the state root (default ~/.cache/catstack-llm-judge). Non-goals: No hook calls the library in this step. No install.sh or settings wiring. No change to any existing hook. Layer: domain Feature state: dormant Files: engine/hooks/llm-judge/judge.py, engine/hooks/llm-judge/tests/test_judge.py, engine/hooks/llm-judge/README.md Change types: - engine/hooks/llm-judge/judge.py: create - engine/hooks/llm-judge/tests/test_judge.py: create - engine/hooks/llm-judge/README.md: create Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests` exits 0. - `python3 scripts/check_no_new_comments.py --base origin/main` exits 0. Exit code: 0

…library tests and the repo's no-comments gate. Review claim: The library tests and the no-comments gate pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The command exits 0 on the implemented branch. Slice rationale: One proof task for the library. Architectural effect: None; verification only. Goal: Prove the library deterministically with fake runners. Motivation: Fallback order, recursion guard and backgrounding are behaviors, so they need a test run. Alternative considerations: Calling real model CLIs in CI was rejected (network, auth and usage limits make it non-deterministic). Implementation details: Run the library tests and the comments gate. Non-goals: No product edits. Layer: app_regression Feature state: active Exit code: 0

…nly check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active Exit code: 0

…0a4ff72b-46acc04e — Terminal read-only check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active

A background verdict now reaches the agent at the start of its next turn in Claude (UserPromptSubmit additionalContext), Cursor (stop followup_message) and Codex (notify stderr, chained). A hit shows its on_hit text once; an unchecked verdict names each runner and why it failed; a clean one is silent. Missing transcripts and bad payloads are reported on stderr and exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…udge inbox hook for Claude, Cursor and Codex, and wire it into install.sh. Review claim: On the next turn the inbox hook drains this transcript's llm-judge verdicts and adds each hit's on_hit text, and each unchecked verdict's per-runner reasons, to the agent's context; with none waiting it prints nothing. Review lane: behavior Safety invariant: The inbox only adds text to the next prompt and never blocks; with no verdicts waiting it prints nothing. Effectiveness measurement: Tests seed a verdict store and assert the exact delivered text per harness, silence when empty, and that a delivered verdict is not delivered twice; tests/test_install.py asserts the link and the settings entry. Slice rationale: Delivery is one review claim; the library it reads is step 1 and the first caller is step 3. Architectural effect: Adds a UserPromptSubmit-time reader over llm-judge's verdict store; hooks that enqueue questions get their answers one turn later. Goal: Make background verdicts reach the agent in every harness. Motivation: A verdict nobody reads changes nothing, and an unchecked verdict that is dropped hides a broken judge. Alternative considerations: Delivering from the Stop hook of the same turn was rejected (the verdict is not ready yet, and waiting would block the reply). A Claude-only inbox was rejected (catstack hooks ship for Claude, Cursor and Codex). Implementation details: In engine/hooks/llm-judge/ add claude_prompt_submit.py (Claude UserPromptSubmit, prints hookSpecificOutput.additionalContext), cursor_session.py (Cursor stop, prints followup_message), codex_notify.py (Codex notify, advisory text on stderr), inbox.py (shared formatting over judge.drain), claude.hook.json, install_claude_hook.py, install_cursor_hook.py, install_codex_notify.py, and tests/test_inbox.py. Mirror the file shapes of engine/hooks/wrong-check-reflect/. Wire it like every other hook: link_item in install.sh, the settings merge, an assertion in tests/test_install.py, a README section, and a docs/ecosystem.md row. Follow product/skills/ship-a-detector/playbooks/detector-lifecycle.md steps 14-19 for the wiring. Non-goals: No hook enqueues questions in this step. No change to judge.py behavior. No change to wrong-check-reflect. Layer: app_bridge Feature state: active Files: engine/hooks/llm-judge/inbox.py, engine/hooks/llm-judge/claude_prompt_submit.py, engine/hooks/llm-judge/cursor_session.py, engine/hooks/llm-judge/codex_notify.py, engine/hooks/llm-judge/claude.hook.json, engine/hooks/llm-judge/install_claude_hook.py, engine/hooks/llm-judge/install_cursor_hook.py, engine/hooks/llm-judge/install_codex_notify.py, engine/hooks/llm-judge/tests/test_inbox.py, engine/hooks/llm-judge/README.md, install.sh, tests/test_install.py, docs/ecosystem.md Change types: - engine/hooks/llm-judge/inbox.py: create - engine/hooks/llm-judge/claude_prompt_submit.py: create - engine/hooks/llm-judge/cursor_session.py: create - engine/hooks/llm-judge/codex_notify.py: create - engine/hooks/llm-judge/claude.hook.json: create - engine/hooks/llm-judge/install_claude_hook.py: create - engine/hooks/llm-judge/install_cursor_hook.py: create - engine/hooks/llm-judge/install_codex_notify.py: create - engine/hooks/llm-judge/tests/test_inbox.py: create - engine/hooks/llm-judge/README.md: modify - install.sh: modify - tests/test_install.py: modify - docs/ecosystem.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/llm-judge` exits 0. - `python3 -m unittest tests/test_install.py` exits 0. Exit code: 0

…sts, the hook coverage gate, and the install tests. Review claim: The inbox tests, hook coverage gate and install tests pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The command exits 0 on the implemented branch. Slice rationale: One proof task for the inbox slice. Architectural effect: None; verification only. Goal: Prove delivery and wiring deterministically. Motivation: An unwired hook looks identical to a working one, so the install assertion is part of the proof. Alternative considerations: The full suite was rejected as slower with no extra signal here. Implementation details: Run the three commands in order. Non-goals: No product edits. Layer: app_regression Feature state: active Exit code: 0

…nly check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active Exit code: 0

…dbabd401-e965207c — Terminal read-only check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active

…-reflect sends a background llm-judge question for each reply that answers a user message, alongside its existing regex path. Review claim: After each reply that follows a user message, wrong-check-reflect sends one llm-judge job asking for pushback and self_correction, with hit_if_all_true both keys and on_hit its existing reflect follow-up; the regex path and its once-per-transcript rule are unchanged. Review lane: behavior Safety invariant: The judge runs in the background and never blocks a reply; the worst case is one extra reflect reminder. Effectiveness measurement: A test with a fake runner answering yes to both keys shows the reflect follow-up delivered by the inbox on the next prompt for the exact "You're right. ... I misread which diff you meant." exchange; the Stop hook returns in under one second while the fake runner sleeps two. Slice rationale: The first caller of the judge; the library and inbox are the two slices below it. Architectural effect: wrong-check-reflect gains a second, model-judged path that reports through the llm-judge inbox one turn later; its synchronous regex path stays as the instant backstop. Goal: Catch self-corrections after pushback in any wording. Motivation: Regexes keep missing the next phrasing; the user asked for model judgement after pushback that never interrupts the reply. Alternative considerations: Replacing the regex path was rejected (it is instant and works when every runner is down). Judging only when a regex fires was rejected (that is the case regexes already catch). Implementation details: In engine/hooks/wrong-check-reflect/detect.py add enqueue_judge(payload) that reads the transcript, takes the last user message and the assistant message before it, and when both exist and the current reply is non-empty, imports judge from the sibling llm-judge directory and sends a job whose prompt is the classifier prompt given in the task prompt. Call it from claude_stop_check.py, cursor_session.py and codex_notify.py after the regex decision, except when stop_hook_active is set, when the regex fired this turn, or when the transcript was already prompted. Any exception is written to stderr with context and never changes the hook's exit status. Non-goals: No change to the regex patterns, the follow-up text, or the once-per-transcript rule. No change to llm-judge. Layer: app_bridge Feature state: active Files: engine/hooks/wrong-check-reflect/detect.py, engine/hooks/wrong-check-reflect/claude_stop_check.py, engine/hooks/wrong-check-reflect/cursor_session.py, engine/hooks/wrong-check-reflect/codex_notify.py, engine/hooks/wrong-check-reflect/tests/test_hooks.py, engine/hooks/wrong-check-reflect/README.md Change types: - engine/hooks/wrong-check-reflect/detect.py: modify - engine/hooks/wrong-check-reflect/claude_stop_check.py: modify - engine/hooks/wrong-check-reflect/cursor_session.py: modify - engine/hooks/wrong-check-reflect/codex_notify.py: modify - engine/hooks/wrong-check-reflect/tests/test_hooks.py: modify - engine/hooks/wrong-check-reflect/README.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/wrong-check-reflect` exits 0. Solution: wrong-check-reflect sends a background llm-judge question for each reply that answers a user message, alongside its existing regex path. Review claim: After each reply that follows a user message, wrong-check-reflect sends one llm-judge job asking for pushback and self_correction, with hit_if_all_true both keys and on_hit its existing reflect follow-up; the regex path and its once-per-transcript rule are unchanged. Review lane: behavior Safety invariant: The judge runs in the background and never blocks a reply; the worst case is one extra reflect reminder. Effectiveness measurement: A test with a fake runner answering yes to both keys shows the reflect follow-up delivered by the inbox on the next prompt for the exact "You're right. ... I misread which diff you meant." exchange; the Stop hook returns in under one second while the fake runner sleeps two. Slice rationale: The first caller of the judge; the library and inbox are the two slices below it. Architectural effect: wrong-check-reflect gains a second, model-judged path that reports through the llm-judge inbox one turn later; its synchronous regex path stays as the instant backstop. Goal: Catch self-corrections after pushback in any wording. Motivation: Regexes keep missing the next phrasing; the user asked for model judgement after pushback that never interrupts the reply. Alternative considerations: Replacing the regex path was rejected (it is instant and works when every runner is down). Judging only when a regex fires was rejected (that is the case regexes already catch). Implementation details: In engine/hooks/wrong-check-reflect/detect.py add enqueue_judge(payload) that reads the transcript, takes the last user message and the assistant message before it, and when both exist and the current reply is non-empty, imports judge from the sibling llm-judge directory and sends a job whose prompt is the classifier prompt given in the task prompt. Call it from claude_stop_check.py, cursor_session.py and codex_notify.py after the regex decision, except when stop_hook_active is set, when the regex fired this turn, or when the transcript was already prompted. Any exception is written to stderr with context and never changes the hook's exit status. Non-goals: No change to the regex patterns, the follow-up text, or the once-per-transcript rule. No change to llm-judge. Layer: app_bridge Feature state: active Files: engine/hooks/wrong-check-reflect/detect.py, engine/hooks/wrong-check-reflect/claude_stop_check.py, engine/hooks/wrong-check-reflect/cursor_session.py, engine/hooks/wrong-check-reflect/codex_notify.py, engine/hooks/wrong-check-reflect/tests/test_hooks.py, engine/hooks/wrong-check-reflect/README.md Change types: - engine/hooks/wrong-check-reflect/detect.py: modify - engine/hooks/wrong-check-reflect/claude_stop_check.py: modify - engine/hooks/wrong-check-reflect/cursor_session.py: modify - engine/hooks/wrong-check-reflect/codex_notify.py: modify - engine/hooks/wrong-check-reflect/tests/test_hooks.py: modify - engine/hooks/wrong-check-reflect/README.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/wrong-check-reflect` exits 0.

…k-reflect's tests, llm-judge's tests, and the hook coverage gate. Review claim: Both hooks' tests and the coverage gate pass together. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The command exits 0 on the implemented branch. Slice rationale: One proof task for the adoption slice. Architectural effect: None; verification only. Goal: Prove the model-judged path end to end with fake runners. Motivation: The path crosses two hooks, so both suites run. Alternative considerations: Real model CLIs in CI were rejected as non-deterministic. Implementation details: Run both suites and the coverage gate. Non-goals: No product edits. Layer: app_regression Feature state: active Exit code: 0

…nly check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active Exit code: 0

…be7af30b-bd3c4097 — Terminal read-only check that no inter-task handoff files remain. Review claim: No handoff artifacts are left in the tree. Review lane: proof Safety invariant: Read-only; never deletes files or commits. Effectiveness measurement: Exits 0 when no handoff files remain. Slice rationale: Required terminal gate for implementation plans. Architectural effect: None. Goal: Keep ephemeral handoff files out of the PR. Motivation: Required by the plan linter. Alternative considerations: None. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No edits. Layer: app_regression Feature state: active

CI failed only on check_no_new_comments: one line in detect.py and six in tests/test_hooks.py. No code changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: I63f3cca645715ef5bd7ab89478b6fdb98e55d844

…er can declare its meaning as a JSON phrase dictionary, and engine/hooks/llm-judge/phrases.py turns that dictionary plus a text into an llm-judge job. Review lane: behavior Safety invariant: No hook or check behavior changes. Only new files under engine/hooks/llm-judge/; no existing hook imports the new module. Pending user confirmation in this session. Effectiveness measurement: Unit tests render the example dictionary into a prompt and assert it contains the meaning, every match phrase, every not_match phrase, and the text; a malformed dictionary raises ValueError naming the file and the bad key. Slice rationale: The dictionary format and loader are one reviewable format. Wiring it into a hook is a later slice so this diff carries no behavior change. Architectural effect: Adds a data format (phrases/<checker>.json) and one pure module that builds llm-judge jobs; checkers stop owning prompt text. Goal: Create phrases.py, one example dictionary, and its tests. Motivation: Checkers that match regexes against prose miss rewordings and fire on quoted or negated text. A model judged against a phrase list catches rewordings: a probe with claude haiku matched "Fair point, I got that backwards earlier." and did not match "The migration finished and the table is live.". Alternative considerations: Keeping prompt strings inside each hook (as PR 425 does with JUDGE_PROMPT) was rejected because every hook would invent its own prompt shape. A synchronous Claude "type: prompt" hook was rejected because diu-stop used one before and it wrote its raw reasoning into the chat (see the docstring at the top of engine/hooks/diu-stop/claude_stop_check.py). Implementation details: Dictionary format, one JSON object per file at engine/hooks/llm-judge/phrases/<checker>.json with keys checker (string equal to the file stem), meaning (one sentence), reads (one of "reply", "user", "exchange"), match (non-empty array of strings), not_match (array of strings), on_hit (string shown to the agent on a hit). phrases.py provides load(checker, directory=None), prompt(dictionary, text), and job(dictionary, transcript, text) as specified in the prompt. Non-goals: Change only the three files listed; no existing hook changes, no inbox or delivery changes, no model calls in tests. Layer: domain Feature state: dormant Files: - engine/hooks/llm-judge/phrases.py - engine/hooks/llm-judge/phrases/example.json - engine/hooks/llm-judge/tests/test_phrases.py Change types: - engine/hooks/llm-judge/phrases.py: create - engine/hooks/llm-judge/phrases/example.json: create - engine/hooks/llm-judge/tests/test_phrases.py: create Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests -v` exits 0. - `python3 scripts/check_no_new_comments.py --base origin/plan/llm-judge-3-wrong-check-reflect-asks-the-judge-about-pushback-and-take-backs` exits 0. Exit code: 0 Invoker-Finalize-Id: 9d865cdb-b414-49e4-8e28-719759fc61a7

…m: A phrase-judge skill tells authors to declare a checker's meaning as a phrase dictionary and to grow it with real misses instead of writing a regex; the llm-judge README documents the format. Review lane: docs Safety invariant: Documentation and a skill only; no hook, check, or install behavior changes beyond making the new skill visible to Claude, Cursor, and Codex. Pending user confirmation in this session. Effectiveness measurement: The skill test asserts the skill has name and description frontmatter and names phrases/<checker>.json; scripts/check_skills_three_harnesses.py passes with the new skill present. Slice rationale: The prose that teaches the format is reviewed apart from the loader code. Architectural effect: Adds one engine skill; no runtime effect. Goal: Create the phrase-judge skill, its test, and a README section. Motivation: A format with no guidance invites authors to go back to regex the first time a phrasing slips through. Alternative considerations: Putting the guidance only in the README was rejected because agents load skills, not READMEs, when writing a checker. Implementation details: Follow engine/skills/create-skill/SKILL.md. The skill states the dictionary keys, says to seed match and not_match from real pasted texts, to add each new miss or false alarm as a phrase, to use regex only for fixed machine formats, and that dictionary checkers never block because the judge answers in the background. Non-goals: No hook code, no loader changes. Layer: docs Feature state: dormant Files: - engine/skills/phrase-judge/SKILL.md - engine/skills/phrase-judge/tests/test_phrase_judge_skill.py - engine/hooks/llm-judge/README.md Change types: - engine/skills/phrase-judge/SKILL.md: create - engine/skills/phrase-judge/tests/test_phrase_judge_skill.py: create - engine/hooks/llm-judge/README.md: modify Acceptance criteria: - `python3 -m unittest discover -s engine/skills/phrase-judge/tests -v` exits 0. - `python3 scripts/check_skills_three_harnesses.py` exits 0. Exit code: 0 Invoker-Finalize-Id: 78a590ff-6980-48f3-abab-1ef3a74ba949

…: The llm-judge unit tests, including the new phrase tests, pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The unittest run is the direct measurement. Slice rationale: One focused proof per claim before review. Architectural effect: None; verification only. Goal: Prove the loader works. Layer exception: allowed. Proof runs after the docs task so it checks the finished branch; it reads files only and changes nothing. Motivation: Tests existing is not proof; running them is. Alternative considerations: The full suite was rejected; this module is the smallest honest proof. Implementation details: Run the llm-judge unittest discover. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when every llm-judge test passes. Exit code: 0 Invoker-Finalize-Id: 76845347-e176-42de-a12c-85f6251b3463

…im: No hook outside engine/hooks/llm-judge/ refers to the phrase module, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing calls the new module yet. Layer exception: allowed. Proof runs after the docs task so it checks the finished branch; it reads files only and changes nothing. Motivation: The safety invariant says no behavior changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the module under engine/hooks, excluding llm-judge itself. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when no hook outside llm-judge mentions the phrase module. Exit code: 0 Invoker-Finalize-Id: c14b40cf-ece4-4c41-9e75-5140ff367c8f

…: The phrase-judge skill test and the three-harness skill check pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The two commands are the direct measurement. Slice rationale: One focused proof per claim before review. Architectural effect: None; verification only. Goal: Prove the skill is well formed and visible to every harness. Layer exception: allowed. The skill check needs the skill file the docs task creates; it reads files only and changes nothing. Motivation: A skill that only one harness can see is unfinished. Alternative considerations: Checking one harness only was rejected. Implementation details: Run the skill unittest discover and the three-harness check. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when both pass. Exit code: 0 Invoker-Finalize-Id: 8386d7ba-ffb5-4005-b1bc-17ec6d5f2c2c

…ks/g0.t0.a-a2da10d77-9902a3ad

…mant/g0.t0.a-a3c533ce6-fa1d2622

…o ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: 5bf9f247-fef6-4e8b-9c38-826d558c5b6d

…f1afd028-112f90c9 — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0.

The prompt listed example phrases but never said to match their meaning. A real retraction ("Correction: the file I pointed you to earlier is not the one in use; the real one is src/b.py.") scored match=false in 6 of 6 live claude runs. With one added sentence it scored true in 3 of 3, and the two not_match examples stayed false in 4 of 4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: Id5112866db12ed375f7677ff411c86fc7c4a7cf1

The engine-only install now links the new engine skill phrase-judge, so TestEngineOnly's hardcoded ENGINE_SKILLS set was one short and CI failed: "Items in the first set but not the second: 'phrase-judge'". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: I9a0c5514443f4558581b562ce0950bdac1bf4808

main now holds #425 as squash commit 5b54fd0. The only conflict was the llm-judge README: this branch adds the "Phrase dictionaries" section where main's side of the block was empty, so the section is kept and nothing from main is dropped. Checked on the merged tree: llm-judge 46 OK, phrase-judge skill 3 OK, check_skills_three_harnesses ok, check_no_new_comments --base origin/main ok, tests.test_install.TestEngineOnly 5 OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tzk7khYCUbjYwGJf6dTWUF Change-Id: I47726eb7e4e9484fe31419f275963f87cbcf561d

…claim: After any tool call, a finished llm-judge verdict for the current transcript is printed in the harness's after-tool format by one small script per harness (Claude, Cursor, Codex). Review lane: behavior Safety invariant: Nothing installs or calls the new scripts in this step; existing delivery at the next prompt is unchanged; every new script exits 0 on every input. Pending user confirmation in this session. Effectiveness measurement: Tests plant a finished hit verdict, run each script, and assert the on_hit text appears in that harness's JSON exactly once across two runs. Slice rationale: Delivery scripts and their tests are one reviewable behavior. Installing them into each harness is the next slice, so this diff changes no running agent. Architectural effect: llm-judge gains a second delivery moment (after a tool call) beside the existing next-prompt delivery; both drain the same verdict store, so each verdict is still delivered once. Goal: Create the three scripts and one test file. Motivation: A judge hit found mid-task should reach the agent at its next step, not after the human speaks again. Alternative considerations: Blocking the tool call until the judge answers was rejected by the user ("keep going and interrupt when the hook fires"). A single shared script with a harness flag was rejected because each harness passes a different payload and expects a different output shape; three thin files match how build-the-lever and the existing llm-judge delivery scripts are laid out. Implementation details: Claude and Codex print {"hookSpecificOutput": {"hookEventName": "PostToolUse", "additionalContext": <texts joined by a blank line>}} (the shape engine/hooks/answer-overrides-menu/claude_posttooluse.py and engine/hooks/build-the-lever/codex_posttooluse.py already print). Cursor prints {"additional_context": <texts>} (the shape engine/hooks/build-the-lever/cursor_post_tool_use.py prints). The transcript comes from inbox.resolve_transcript(payload). With no transcript, a bad payload, or a drain error, the script writes one stderr line naming the harness and the error and prints nothing on stdout. Non-goals: No installer, settings, or install.sh change; no change to inbox.py, judge.py, phrases.py, or the existing delivery scripts; no model calls in tests. Layer: transport Feature state: dormant Files: - engine/hooks/llm-judge/claude_post_tool_use.py - engine/hooks/llm-judge/cursor_post_tool_use.py - engine/hooks/llm-judge/codex_post_tool_use.py - engine/hooks/llm-judge/tests/test_post_tool_use.py Change types: - engine/hooks/llm-judge/claude_post_tool_use.py: create - engine/hooks/llm-judge/cursor_post_tool_use.py: create - engine/hooks/llm-judge/codex_post_tool_use.py: create - engine/hooks/llm-judge/tests/test_post_tool_use.py: create Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/llm-judge/tests -v` exits 0. - `python3 scripts/check_no_new_comments.py --base origin/plan/phrase-judge-1-checkers-declare-meaning-as-phrase-dictionaries` exits 0. Exit code: 0 Invoker-Finalize-Id: da714ff5-5c7d-444f-b6ea-0d4666d0125a

… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Exit code: 1

… claim: The llm-judge tests, including the new after-tool delivery tests, pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The unittest run is the direct measurement. Slice rationale: One focused proof for the delivery behavior before review. Architectural effect: None; verification only. Goal: Prove each script delivers a verdict once and stays quiet otherwise. Motivation: Tests existing is not proof; running them is. Alternative considerations: The full suite was rejected; this module is the smallest honest proof. Implementation details: Run the llm-judge unittest discover. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when every llm-judge test passes. Exit code: 0

… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Exit code: 1

… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Solution: Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them.

… Review claim: No installer, settings fragment, or install.sh line mentions the new after-tool scripts, so the change is dormant. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep printing nothing is the direct measurement of the safety invariant. Slice rationale: The safety invariant gets its own proof. Architectural effect: None; verification only. Goal: Prove nothing installs the new scripts yet. Motivation: The safety invariant says no running agent changes; this checks it. Alternative considerations: Reading the diff by eye was rejected as non-deterministic. Implementation details: git grep for the three script names outside their own files and tests. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when nothing outside the scripts and their test file names them. Exit code: 0

…s-not-installed/g1.t6.a-ac332d93f-93fa90a1

…o ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: cf921902-89e1-4eb2-b9eb-a0ab7656ec3d

cursor · 2026-09-12T05:24:49Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_13df1f13-2f3b-42e2-83db-7065d30cf29e)

mergify · 2026-09-12T06:41:15Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

cursor · 2026-09-12T07:07:28Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c0c129ae-eef6-434b-9c56-b18de26258c3)

Resolve the add/add conflicts by keeping main's files. The task's work already landed through #444, and main's versions are newer: example.json carries the unwired_reason that #480's checker requires. Drop the cursor_post_tool_use.py -> cursor_posttooluse.py renames in bug-complaint-leak and repeat-error-stop from d654dce. A read-only verify task made them, and main's repeat-error-stop/tests/test_hooks.py imports cursor_post_tool_use, so the renames would break that test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

cursor · 2026-09-12T20:26:09Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_8f2f0e02-11f4-444b-a3ab-36b9a907b7ec)

edbert-bot and others added 30 commits September 11, 2026 02:19

Add llm judge phrase loader

2e0bb00

docs: add phrase judge skill

a3013d7

Invoker: merge experiment/wf-1789159008754-1/verify-phrase-skill-chec…

4e78d32

…ks/g0.t0.a-a2da10d77-9902a3ad

Invoker: merge experiment/wf-1789159008754-1/verify-phrase-module-dor…

2dbd843

…mant/g0.t0.a-a3c533ce6-fa1d2622

llm-judge: add after-tool delivery scripts

6ad6afd

Invoker Bot and others added 8 commits September 12, 2026 02:20

Invoker: merge experiment/wf-1789160500325-2/verify-after-tool-script…

afb5f6a

…s-not-installed/g1.t6.a-ac332d93f-93fa90a1

EdbertChan added the admin-bypass label Sep 12, 2026

EdbertChan force-pushed the plan/phrase-judge-2-finished-verdicts-reach-the-agent-after-its-next-tool-call branch from 4ccb37d to 0b5f0f4 Compare September 12, 2026 20:26

Navigation Menu

Phrase judge (2) finished verdicts reach the agent after its next tool call - #470

Phrase judge (2) finished verdicts reach the agent after its next tool call#470
EdbertChan wants to merge 39 commits into
mainfrom
plan/phrase-judge-2-finished-verdicts-reach-the-agent-after-its-next-tool-call

EdbertChan commented Sep 12, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Sep 12, 2026

Uh oh!

mergify Bot commented Sep 12, 2026

Uh oh!

cursor Bot commented Sep 12, 2026

Uh oh!

cursor Bot commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

EdbertChan commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review Claim

Review Lane

Review Unit

Safety Invariant

Slice Rationale

Non-goals

Test Plan

Revert Plan

Uh oh!

cursor Bot commented Sep 12, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

mergify Bot commented Sep 12, 2026

Uh oh!

cursor Bot commented Sep 12, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot commented Sep 12, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

EdbertChan commented Sep 12, 2026 •

edited

Loading