[Unverified tag check](3) a finished reply's unverified tags get a background read-only check - #550
Closed
EdbertChan wants to merge 13 commits into
Closed
EdbertChan wants to merge 13 commits into
EdbertChan wants to merge 13 commits into
Conversation
added 3 commits
September 13, 2026 06:28
…— Build the unverified-tag-check hook folder: the detector, the three harness entry scripts, and their installers.
Review claim: For each well-formed unverified tag in a finished reply, outside code fences and inline code,
at most 3 per reply and once per claim per transcript, the hook builds one llm-judge job with
mode investigate, timeout_seconds 300, the payload cwd, hit_if_all_true [] and a prompt asking for
blocker_false, claim_status and report; the entry scripts always exit 0.
Review lane: behavior
Safety invariant: The hook never blocks or delays a reply: every entry script exits 0, catches every
exception and prints it to stderr as "catstack-hook-error unverified-tag-check: <type>: <message>", and never
waits on a verdict. Every job it builds sets "mode": "investigate", so the model only gets read-only tools.
A reply the hook cannot read produces no job and a stderr line, never a verdict of true.
Effectiveness measurement: Unit tests feed the real tag texts from the backtest (for example
"{{CAT-UNVERIFIED: DO1's repair worker is off or stuck -- cannot verify: can't log in to DO1 to look}}") and
prove one job per tag with the exact job keys, and prove silence for a malformed tag, a tag inside a code
fence, a tag inside inline code, a repeated claim, and a reply with no tag. After install, one live reply
carrying a tag produces a verdict file under ~/.cache/catstack-llm-judge/verdicts/, pasted in the PR body.
Slice rationale: One behavior slice holding the detector and its per-harness entry scripts, dormant until the
next task wires them into install.sh.
Architectural effect: A new Stop-time hook that feeds llm-judge's investigate mode; it depends on the
investigate mode and the inbox report line from the two earlier slices.
Goal: Check every "cannot verify" excuse in the background and tell the agent whether it held.
Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose blocker was false and 39
whose claim was never checked afterward. Today every evidence hook treats a well-formed tag as a pass.
Alternative considerations: A blocking Stop hook was rejected by the user in favor of a background check that
reports and never blocks. A text-pattern check of the blocker was rejected; it cannot tell a real blocker from
a false one. Shell and network access for the checker were rejected by the user; it reads files only.
Implementation details: Create engine/hooks/unverified-tag-check/detect.py modeled on
engine/hooks/wrong-check-reflect/detect.py. Read the reply from payload last_assistant_message or else the
last assistant text in transcript_path. Strip fenced code blocks (only closed fences) and inline code spans
before matching, then take markers.well_formed_tags() from engine/hooks/_markers/markers.py (import it the
way that module's docstring says). Divide each tag body at "cannot verify:" into claim and blocker. Keep a
state file per transcript at ~/.cache/catstack-unverified-tag-check/<first 16 hex of sha1(transcript)>.json
mapping a normalized claim to a timestamp, with a 2 hour TTL; a missing, unreadable or malformed state file
reads as empty. Build the job described under Job shape and hand it to llm-judge. Create
claude_stop_check.py, codex_notify.py and cursor_session.py plus install_claude_hook.py,
install_codex_notify.py, install_cursor_hook.py and claude.hook.json by copying the wrong-check-reflect
versions and changing the hook name and marker.
Job shape: {"id": uuid4 hex, "hook": "unverified-tag-check", "transcript": path, "mode": "investigate",
"timeout_seconds": 300, "cwd": payload cwd, "hit_if_all_true": [], "on_hit": "unverified-tag-check: checked
\"<claim clipped to 120 chars>\". Tell the user this result in plain words:", "prompt": see the prompt field}.
Non-goals: No install.sh edit, no README, no docs/ecosystem.md row in this task. No blocking, no waiting on
verdicts, no shell or network tools for the checker, no phone notification.
Layer: contact_surface
Feature state: dormant
Files:
- engine/hooks/unverified-tag-check/detect.py
- engine/hooks/unverified-tag-check/claude_stop_check.py
- engine/hooks/unverified-tag-check/codex_notify.py
- engine/hooks/unverified-tag-check/cursor_session.py
- engine/hooks/unverified-tag-check/claude.hook.json
- engine/hooks/unverified-tag-check/install_claude_hook.py
- engine/hooks/unverified-tag-check/install_codex_notify.py
- engine/hooks/unverified-tag-check/install_cursor_hook.py
- engine/hooks/unverified-tag-check/tests/test_hooks.py
Change types:
- engine/hooks/unverified-tag-check/detect.py: create
- engine/hooks/unverified-tag-check/claude_stop_check.py: create
- engine/hooks/unverified-tag-check/codex_notify.py: create
- engine/hooks/unverified-tag-check/cursor_session.py: create
- engine/hooks/unverified-tag-check/claude.hook.json: create
- engine/hooks/unverified-tag-check/install_claude_hook.py: create
- engine/hooks/unverified-tag-check/install_codex_notify.py: create
- engine/hooks/unverified-tag-check/install_cursor_hook.py: create
- engine/hooks/unverified-tag-check/tests/test_hooks.py: create
Acceptance criteria:
- `python3 -m unittest discover -s engine/hooks/unverified-tag-check/tests -v` exits 0.
- `python3 scripts/check_hook_test_coverage.py engine/hooks/unverified-tag-check` exits 0.
- `python3 scripts/check_no_silent_hook_except.py` exits 0.
- `python3 scripts/check_no_new_comments.py --base origin/main` exits 0.
Solution:
Build the unverified-tag-check hook folder: the detector, the three harness entry scripts, and their installers.
Review claim: For each well-formed unverified tag in a finished reply, outside code fences and inline code,
at most 3 per reply and once per claim per transcript, the hook builds one llm-judge job with
mode investigate, timeout_seconds 300, the payload cwd, hit_if_all_true [] and a prompt asking for
blocker_false, claim_status and report; the entry scripts always exit 0.
Review lane: behavior
Safety invariant: The hook never blocks or delays a reply: every entry script exits 0, catches every
exception and prints it to stderr as "catstack-hook-error unverified-tag-check: <type>: <message>", and never
waits on a verdict. Every job it builds sets "mode": "investigate", so the model only gets read-only tools.
A reply the hook cannot read produces no job and a stderr line, never a verdict of true.
Effectiveness measurement: Unit tests feed the real tag texts from the backtest (for example
"{{CAT-UNVERIFIED: DO1's repair worker is off or stuck -- cannot verify: can't log in to DO1 to look}}") and
prove one job per tag with the exact job keys, and prove silence for a malformed tag, a tag inside a code
fence, a tag inside inline code, a repeated claim, and a reply with no tag. After install, one live reply
carrying a tag produces a verdict file under ~/.cache/catstack-llm-judge/verdicts/, pasted in the PR body.
Slice rationale: One behavior slice holding the detector and its per-harness entry scripts, dormant until the
next task wires them into install.sh.
Architectural effect: A new Stop-time hook that feeds llm-judge's investigate mode; it depends on the
investigate mode and the inbox report line from the two earlier slices.
Goal: Check every "cannot verify" excuse in the background and tell the agent whether it held.
Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose blocker was false and 39
whose claim was never checked afterward. Today every evidence hook treats a well-formed tag as a pass.
Alternative considerations: A blocking Stop hook was rejected by the user in favor of a background check that
reports and never blocks. A text-pattern check of the blocker was rejected; it cannot tell a real blocker from
a false one. Shell and network access for the checker were rejected by the user; it reads files only.
Implementation details: Create engine/hooks/unverified-tag-check/detect.py modeled on
engine/hooks/wrong-check-reflect/detect.py. Read the reply from payload last_assistant_message or else the
last assistant text in transcript_path. Strip fenced code blocks (only closed fences) and inline code spans
before matching, then take markers.well_formed_tags() from engine/hooks/_markers/markers.py (import it the
way that module's docstring says). Divide each tag body at "cannot verify:" into claim and blocker. Keep a
state file per transcript at ~/.cache/catstack-unverified-tag-check/<first 16 hex of sha1(transcript)>.json
mapping a normalized claim to a timestamp, with a 2 hour TTL; a missing, unreadable or malformed state file
reads as empty. Build the job described under Job shape and hand it to llm-judge. Create
claude_stop_check.py, codex_notify.py and cursor_session.py plus install_claude_hook.py,
install_codex_notify.py, install_cursor_hook.py and claude.hook.json by copying the wrong-check-reflect
versions and changing the hook name and marker.
Job shape: {"id": uuid4 hex, "hook": "unverified-tag-check", "transcript": path, "mode": "investigate",
"timeout_seconds": 300, "cwd": payload cwd, "hit_if_all_true": [], "on_hit": "unverified-tag-check: checked
\"<claim clipped to 120 chars>\". Tell the user this result in plain words:", "prompt": see the prompt field}.
Non-goals: No install.sh edit, no README, no docs/ecosystem.md row in this task. No blocking, no waiting on
verdicts, no shell or network tools for the checker, no phone notification.
Layer: contact_surface
Feature state: dormant
Files:
- engine/hooks/unverified-tag-check/detect.py
- engine/hooks/unverified-tag-check/claude_stop_check.py
- engine/hooks/unverified-tag-check/codex_notify.py
- engine/hooks/unverified-tag-check/cursor_session.py
- engine/hooks/unverified-tag-check/claude.hook.json
- engine/hooks/unverified-tag-check/install_claude_hook.py
- engine/hooks/unverified-tag-check/install_codex_notify.py
- engine/hooks/unverified-tag-check/install_cursor_hook.py
- engine/hooks/unverified-tag-check/tests/test_hooks.py
Change types:
- engine/hooks/unverified-tag-check/detect.py: create
- engine/hooks/unverified-tag-check/claude_stop_check.py: create
- engine/hooks/unverified-tag-check/codex_notify.py: create
- engine/hooks/unverified-tag-check/cursor_session.py: create
- engine/hooks/unverified-tag-check/claude.hook.json: create
- engine/hooks/unverified-tag-check/install_claude_hook.py: create
- engine/hooks/unverified-tag-check/install_codex_notify.py: create
- engine/hooks/unverified-tag-check/install_cursor_hook.py: create
- engine/hooks/unverified-tag-check/tests/test_hooks.py: create
Acceptance criteria:
- `python3 -m unittest discover -s engine/hooks/unverified-tag-check/tests -v` exits 0.
- `python3 scripts/check_hook_test_coverage.py engine/hooks/unverified-tag-check` exits 0.
- `python3 scripts/check_no_silent_hook_except.py` exits 0.
- `python3 scripts/check_no_new_comments.py --base origin/main` exits 0.
Invoker-Finalize-Id: bfd7f5b8-14a9-48c6-b813-f8089b5d7ade
…f step 1 for the unverified-tag-check hook. Review claim: The hook's unit tests run and pass. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The hook test folder exits 0. Slice rationale: One proof step for the detector. Architectural effect: None; verification only. Goal: Prove the detector's positive and negative cases deterministically. Motivation: Each slice carries its own executable proof. Alternative considerations: Manual inspection was rejected as non-deterministic. Implementation details: Run the hook's unittest discovery once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 06936678-f3aa-41e8-b7c3-4391b470e54b
…roof step 2 for the unverified-tag-check hook. Review claim: The hook meets catstack's hook coverage gate. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The coverage gate exits 0 for the hook folder. Slice rationale: One proof step for the positive, negative and unreadable-input coverage rule. Architectural effect: None; verification only. Goal: Prove the hook has positive, negative and unreadable-input cases. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run the coverage gate on the hook folder. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 896a9ad9-e822-400e-823b-71b44bff744b
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_38e8bfc2-720a-4f0a-adc8-2a21633cc58e) |
… the unverified-tag-check hook into install.sh for Claude, Cursor and Codex, and assert it in the install suite. Review claim: install.sh links engine/hooks/unverified-tag-check into the Claude, Cursor and Codex hook folders and runs its three installers, and tests/test_install.py asserts the Claude Stop entry, the Cursor stop entry and the Codex notify entry each appear once after install. Review lane: behavior Safety invariant: Only lines for unverified-tag-check are added to install.sh; every other link_item and installer line stays byte-for-byte the same, and shellcheck install.sh still exits 0. Effectiveness measurement: A new install-suite case named for unverified_tag_check passes, and the class-level case that requires every claude_*.py entry point to be wired passes. Slice rationale: One wiring slice that turns the hook on, kept apart from the detector so a reviewer sees the install change alone. Architectural effect: Turns the hook on for every harness at the next ./install.sh. Goal: Make the hook actually run after install. Motivation: catstack's ship-a-detector playbook records hooks that shipped but never ran because install.sh linked them without merging their settings. Alternative considerations: Claude-only wiring was rejected; catstack hooks work the same across all three harnesses. Implementation details: In install.sh add link_item lines for unverified-tag-check next to the wrong-check-reflect lines in the .claude, .cursor and .codex sections, and add its install_claude_hook.py, install_cursor_hook.py and install_codex_notify.py calls next to wrong-check-reflect's. In tests/test_install.py add a case modeled on the llm-judge wiring case. Non-goals: No change to the hook code. No README or docs/ecosystem.md edit in this task. Layer: ui_activation Feature state: active Files: - install.sh - tests/test_install.py Change types: - install.sh: modify - tests/test_install.py: modify Acceptance criteria: - `python3 -m unittest discover -s tests -p test_install.py -k unverified_tag_check -v` exits 0 and runs at least one test. - `shellcheck install.sh` exits 0. Context: wf-1789279289772-12/implement-unverified-tag-check-detector (4cee07c): Build the unverified-tag-check hook folder: the detector, the three harness entry scripts, and their installers. Review claim: For each well-formed unverified tag in a finished reply, outside code fences and inline code, at most 3 per reply and once per claim per transcript, the hook builds one llm-judge job with mode investigate, timeout_seconds 300, the payload cwd, hit_if_all_true [] and a prompt asking for blocker_false, claim_status and report; the entry scripts always exit 0. Review lane: behavior Safety invariant: The hook never blocks or delays a reply: every entry script exits 0, catches every exception and prints it to stderr as "catstack-hook-error unverified-tag-check: <type>: <message>", and never waits on a verdict. Every job it builds sets "mode": "investigate", so the model only gets read-only tools. A reply the hook cannot read produces no job and a stderr line, never a verdict of true. Effectiveness measurement: Unit tests feed the real tag texts from the backtest (for example "{{CAT-UNVERIFIED: DO1's repair worker is off or stuck -- cannot verify: can't log in to DO1 to look}}") and prove one job per tag with the exact job keys, and prove silence for a malformed tag, a tag inside a code fence, a tag inside inline code, a repeated claim, and a reply with no tag. After install, one live reply carrying a tag produces a verdict file under ~/.cache/catstack-llm-judge/verdicts/, pasted in the PR body. Slice rationale: One behavior slice holding the detector and its per-harness entry scripts, dormant until the next task wires them into install.sh. Architectural effect: A new Stop-time hook that feeds llm-judge's investigate mode; it depends on the investigate mode and the inbox report line from the two earlier slices. Goal: Check every "cannot verify" excuse in the background and tell the agent whether it held. Motivation: A backtest of 44 unverified tags from the last 14 days found 33 whose blocker was false and 39 whose claim was never checked afterward. Today every evidence hook treats a well-formed tag as a pass. Alternative considerations: A blocking Stop hook was rejected by the user in favor of a background check that reports and never blocks. A text-pattern check of the blocker was rejected; it cannot tell a real blocker from a false one. Shell and network access for the checker were rejected by the user; it reads files only. Implementation details: Create engine/hooks/unverified-tag-check/detect.py modeled on engine/hooks/wrong-check-reflect/detect.py. Read the reply from payload last_assistant_message or else the last assistant text in transcript_path. Strip fenced code blocks (only closed fences) and inline code spans before matching, then take markers.well_formed_tags() from engine/hooks/_markers/markers.py (import it the way that module's docstring says). Divide each tag body at "cannot verify:" into claim and blocker. Keep a state file per transcript at ~/.cache/catstack-unverified-tag-check/<first 16 hex of sha1(transcript)>.json mapping a normalized claim to a timestamp, with a 2 hour TTL; a missing, unreadable or malformed state file reads as empty. Build the job described under Job shape and hand it to llm-judge. Create claude_stop_check.py, codex_notify.py and cursor_session.py plus install_claude_hook.py, install_codex_notify.py, install_cursor_hook.py and claude.hook.json by copying the wrong-check-reflect versions and changing the hook name and marker. Job shape: {"id": uuid4 hex, "hook": "unverified-tag-check", "transcript": path, "mode": "investigate", "timeout_seconds": 300, "cwd": payload cwd, "hit_if_all_true": [], "on_hit": "unverified-tag-check: checked \"<claim clipped to 120 chars>\". Tell the user this result in plain words:", "prompt": see the prompt field}. Non-goals: No install.sh edit, no README, no docs/ecosystem.md row in this task. No blocking, no waiting on verdicts, no shell or network tools for the checker, no phone notification. Layer: contact_surface Feature state: dormant Files: - engine/hooks/unverified-tag-check/detect.py - engine/hooks/unverified-tag-check/claude_stop_check.py - engine/hooks/unverified-tag-check/codex_notify.py - engine/hooks/unverified-tag-check/cursor_session.py - engine/hooks/unverified-tag-check/claude.hook.json - engine/hooks/unverified-tag-check/install_claude_hook.py - engine/hooks/unverified-tag-check/install_codex_notify.py - engine/hooks/unverified-tag-check/install_cursor_hook.py - engine/hooks/unverified-tag-check/tests/test_hooks.py Change types: - engine/hooks/unverified-tag-check/detect.py: create - engine/hooks/unverified-tag-check/claude_stop_check.py: create - engine/hooks/unverified-tag-check/codex_notify.py: create - engine/hooks/unverified-tag-check/cursor_session.py: create - engine/hooks/unverified-tag-check/claude.hook.json: create - engine/hooks/unverified-tag-check/install_claude_hook.py: create - engine/hooks/unverified-tag-check/install_codex_notify.py: create - engine/hooks/unverified-tag-check/install_cursor_hook.py: create - engine/hooks/unverified-tag-check/tests/test_hooks.py: create Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/unverified-tag-check/tests -v` exits 0. - `python3 scripts/check_hook_test_coverage.py engine/hooks/unverified-tag-check` exits 0. - `python3 scripts/check_no_silent_hook_except.py` exits 0. - `python3 scripts/check_no_new_comments.py --base origin/main` exits 0. Solution: Wire the unverified-tag-check hook into install.sh for Claude, Cursor and Codex, and assert it in the install suite. Review claim: install.sh links engine/hooks/unverified-tag-check into the Claude, Cursor and Codex hook folders and runs its three installers, and tests/test_install.py asserts the Claude Stop entry, the Cursor stop entry and the Codex notify entry each appear once after install. Review lane: behavior Safety invariant: Only lines for unverified-tag-check are added to install.sh; every other link_item and installer line stays byte-for-byte the same, and shellcheck install.sh still exits 0. Effectiveness measurement: A new install-suite case named for unverified_tag_check passes, and the class-level case that requires every claude_*.py entry point to be wired passes. Slice rationale: One wiring slice that turns the hook on, kept apart from the detector so a reviewer sees the install change alone. Architectural effect: Turns the hook on for every harness at the next ./install.sh. Goal: Make the hook actually run after install. Motivation: catstack's ship-a-detector playbook records hooks that shipped but never ran because install.sh linked them without merging their settings. Alternative considerations: Claude-only wiring was rejected; catstack hooks work the same across all three harnesses. Implementation details: In install.sh add link_item lines for unverified-tag-check next to the wrong-check-reflect lines in the .claude, .cursor and .codex sections, and add its install_claude_hook.py, install_cursor_hook.py and install_codex_notify.py calls next to wrong-check-reflect's. In tests/test_install.py add a case modeled on the llm-judge wiring case. Non-goals: No change to the hook code. No README or docs/ecosystem.md edit in this task. Layer: ui_activation Feature state: active Files: - install.sh - tests/test_install.py Change types: - install.sh: modify - tests/test_install.py: modify Acceptance criteria: - `python3 -m unittest discover -s tests -p test_install.py -k unverified_tag_check -v` exits 0 and runs at least one test. - `shellcheck install.sh` exits 0. Invoker-Finalize-Id: bbc45c89-e649-4e64-800d-ff9d3b389ac9
…ng — Proof step 3 for the unverified-tag-check hook. Review claim: The install suite passes with the hook wired for all three harnesses. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The whole install suite exits 0. Slice rationale: One proof step for the wiring change. Architectural effect: None; verification only. Goal: Prove install.sh wires the hook and nothing else changed. Motivation: Hooks have shipped that never ran after install. Alternative considerations: Running only the new case was rejected; the class-level wiring cases must pass too. Implementation details: Run the install suite once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0
…ng — Proof step 3 for the unverified-tag-check hook. Review claim: The install suite passes with the hook wired for all three harnesses. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The whole install suite exits 0. Slice rationale: One proof step for the wiring change. Architectural effect: None; verification only. Goal: Prove install.sh wires the hook and nothing else changed. Motivation: Hooks have shipped that never ran after install. Alternative considerations: Running only the new case was rejected; the class-level wiring cases must pass too. Implementation details: Run the install suite once. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0
… Proof step 4 for the unverified-tag-check hook. Review claim: install.sh still passes shellcheck. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: shellcheck exits 0 on install.sh. Slice rationale: One proof step for CI's shellcheck gate. Architectural effect: None; verification only. Goal: Keep the CI shellcheck gate green. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run shellcheck on install.sh. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0
…nts — Proof step 5 for the unverified-tag-check hook. Review claim: The change adds no explanatory code comments. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: The comment gate exits 0 against origin/main. Slice rationale: One proof step for catstack's required comment gate. Architectural effect: None; verification only. Goal: Keep the CI comment gate green. Motivation: CI runs this gate on every pull request. Alternative considerations: Waiting for CI was rejected; it is slower to find. Implementation details: Run the comment gate against origin/main. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 68f12bea-3aa8-4a46-b1d1-446f609962e3
… Proof step 6 for the unverified-tag-check hook. Review claim: catstack's whole suite passes with the hook added. Review lane: proof Safety invariant: Proof-only; adds no product behavior. Effectiveness measurement: scripts/run_all_tests.sh exits 0. Slice rationale: One proof step matching CI's main suite run. Architectural effect: None; verification only. Goal: Prove the hook breaks no sibling hook or class-level gate. Motivation: Several catstack gates assert properties across every hook folder. Alternative considerations: Running only the hook's suite was rejected; class-level gates live elsewhere. Implementation details: Run the repository's full suite script. Non-goals: No product edits here; proof only. Layer: app_regression Feature state: active Exit code: 0
…e unverified-tag-check README, add its docs/ecosystem.md row, and point the prove-it principle at it. Review claim: engine/hooks/unverified-tag-check/README.md states what the hook fires on, what it stays silent on, the message the agent sees, the fail direction and the checker's read-only limits; docs/ecosystem.md gains one inventory row; corpus/skills/principle-prove-it/SKILL.md names the hook next to the tag rule. Review lane: docs Safety invariant: Only the README, the docs/ecosystem.md row and one sentence in corpus/skills/principle-prove-it/SKILL.md change; no code or test file is edited. Effectiveness measurement: grep finds the row in docs/ecosystem.md and the hook name in the skill, and the skill file-reference gate passes. Slice rationale: The README, inventory row and skill pointer for the hook in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README and inventory only. Goal: Make the hook findable and its limits explicit. Motivation: catstack's playbook found hooks shipped without inventory rows and fixed them in later pull requests. Alternative considerations: A later README-only pull request was rejected; the playbook counts that as a known repair pattern. Implementation details: Write engine/hooks/unverified-tag-check/README.md, add one row to the engine inventory table in docs/ecosystem.md, and add one sentence to corpus/skills/principle-prove-it/SKILL.md where it describes the CAT-UNVERIFIED tag. Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/unverified-tag-check/README.md - docs/ecosystem.md - corpus/skills/principle-prove-it/SKILL.md Change types: - engine/hooks/unverified-tag-check/README.md: create - docs/ecosystem.md: modify - corpus/skills/principle-prove-it/SKILL.md: modify Acceptance criteria: - `grep -n "unverified-tag-check" docs/ecosystem.md` exits 0. - `grep -n "unverified-tag-check" corpus/skills/principle-prove-it/SKILL.md` exits 0. - `python3 scripts/check_skill_file_refs.py` exits 0. Context: wf-1789279289772-12/wire-unverified-tag-check-install (9edf2fb): Wire the unverified-tag-check hook into install.sh for Claude, Cursor and Codex, and assert it in the install suite. Review claim: install.sh links engine/hooks/unverified-tag-check into the Claude, Cursor and Codex hook folders and runs its three installers, and tests/test_install.py asserts the Claude Stop entry, the Cursor stop entry and the Codex notify entry each appear once after install. Review lane: behavior Safety invariant: Only lines for unverified-tag-check are added to install.sh; every other link_item and installer line stays byte-for-byte the same, and shellcheck install.sh still exits 0. Effectiveness measurement: A new install-suite case named for unverified_tag_check passes, and the class-level case that requires every claude_*.py entry point to be wired passes. Slice rationale: One wiring slice that turns the hook on, kept apart from the detector so a reviewer sees the install change alone. Architectural effect: Turns the hook on for every harness at the next ./install.sh. Goal: Make the hook actually run after install. Motivation: catstack's ship-a-detector playbook records hooks that shipped but never ran because install.sh linked them without merging their settings. Alternative considerations: Claude-only wiring was rejected; catstack hooks work the same across all three harnesses. Implementation details: In install.sh add link_item lines for unverified-tag-check next to the wrong-check-reflect lines in the .claude, .cursor and .codex sections, and add its install_claude_hook.py, install_cursor_hook.py and install_codex_notify.py calls next to wrong-check-reflect's. In tests/test_install.py add a case modeled on the llm-judge wiring case. Non-goals: No change to the hook code. No README or docs/ecosystem.md edit in this task. Layer: ui_activation Feature state: active Files: - install.sh - tests/test_install.py Change types: - install.sh: modify - tests/test_install.py: modify Acceptance criteria: - `python3 -m unittest discover -s tests -p test_install.py -k unverified_tag_check -v` exits 0 and runs at least one test. - `shellcheck install.sh` exits 0. Solution: Write the unverified-tag-check README, add its docs/ecosystem.md row, and point the prove-it principle at it. Review claim: engine/hooks/unverified-tag-check/README.md states what the hook fires on, what it stays silent on, the message the agent sees, the fail direction and the checker's read-only limits; docs/ecosystem.md gains one inventory row; corpus/skills/principle-prove-it/SKILL.md names the hook next to the tag rule. Review lane: docs Safety invariant: Only the README, the docs/ecosystem.md row and one sentence in corpus/skills/principle-prove-it/SKILL.md change; no code or test file is edited. Effectiveness measurement: grep finds the row in docs/ecosystem.md and the hook name in the skill, and the skill file-reference gate passes. Slice rationale: The README, inventory row and skill pointer for the hook in this same pull request, which catstack's ship-a-detector playbook requires to ship together. Architectural effect: None; README and inventory only. Goal: Make the hook findable and its limits explicit. Motivation: catstack's playbook found hooks shipped without inventory rows and fixed them in later pull requests. Alternative considerations: A later README-only pull request was rejected; the playbook counts that as a known repair pattern. Implementation details: Write engine/hooks/unverified-tag-check/README.md, add one row to the engine inventory table in docs/ecosystem.md, and add one sentence to corpus/skills/principle-prove-it/SKILL.md where it describes the CAT-UNVERIFIED tag. Non-goals: No code or test edits. Layer: docs Feature state: active Files: - engine/hooks/unverified-tag-check/README.md - docs/ecosystem.md - corpus/skills/principle-prove-it/SKILL.md Change types: - engine/hooks/unverified-tag-check/README.md: create - docs/ecosystem.md: modify - corpus/skills/principle-prove-it/SKILL.md: modify Acceptance criteria: - `grep -n "unverified-tag-check" docs/ecosystem.md` exits 0. - `grep -n "unverified-tag-check" corpus/skills/principle-prove-it/SKILL.md` exits 0. - `python3 scripts/check_skill_file_refs.py` exits 0. Invoker-Finalize-Id: e968bbe3-cd2c-4159-8272-412e217e1792
…ephemeral inter-task handoff files remain before the merge gate. Review claim: Only ephemeral handoff files are checked; no product or test file is touched. Review lane: cleanup Safety invariant: The scrub script runs read-only and never touches the home Invoker ledger. Effectiveness measurement: The scrub script exits 0 and reports no handoff path. Slice rationale: One cleanup slice required on every implementation plan that opens a pull request, kept separate from behavior and proof work. Architectural effect: None. Goal: Leave a clean worktree for the pull request. Motivation: Ephemeral inter-task files must not leak into the reviewed diff. Alternative considerations: Leaving this step out was rejected; it is a hard requirement for every plan that opens a pull request. Implementation details: Run the repository's handoff-scrub script without --apply. Non-goals: No product edits in this task. Layer: docs Feature state: active Exit code: 0 Invoker-Finalize-Id: c0a6949c-aaf2-47cf-a39e-9dcfb3c66e75
…validate; Exit code: 0
EdbertChan
force-pushed
the
plan/unverified-tag-check-3-hook
branch
from
September 13, 2026 10:05
3b999ec to
5c76091
Compare
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_580d4efd-3dc4-40a5-ac8e-464c6e238eac) |
8 tasks
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Finished replies carrying an unverified claim now receive a background read-only check.
The check asks whether the stated blocker was real and whether the claim was true, false, or still unknown.
It reports the result on the agent's next step without delaying or blocking the reply.
The hook is installed for Claude, Cursor, and Codex, with documentation covering its limits and delivery path.
Review Claim
Approve a non-blocking check that reviews each well-formed unverified claim once and reports a plain-language verdict later.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
Every entry point exits 0, catches failures, reports them to stderr, and never waits for or blocks on the background verdict.
Slice Rationale
This slice completes the hook from detector through three-harness installation, documentation, and inbox delivery.
The detector, activation wiring, tests, README, and inventory row stay together so the shipped behavior is discoverable and executable.
Non-goals
Architecture
Before
graph TD A["finished reply"] --> B["agent continues"]After
graph TD A["finished reply with unverified tag"] --> B["three-harness hook entry"] B --> C["read-only investigate job"] C --> D["inbox report on next step"] B --> E["reply returns immediately"]Test Plan
Test Plan
python3 -m unittest discover -s engine/hooks/unverified-tag-check/tests -vpython3 scripts/check_hook_test_coverage.py engine/hooks/unverified-tag-checkpython3 -m unittest discover -s tests -p test_install.py -vshellcheck install.shpython3 scripts/check_no_new_comments.py --base origin/mainbash scripts/run_all_tests.shbash scripts/scrub-handoff-artifacts.shRevert Plan
Revert Plan
git revert <merge-commit-sha>Note
Medium Risk
New Stop/notify hook on all three harnesses plus background llm-judge jobs, but advisory and fail-open with no blocking path.
Overview
Adds a new
unverified-tag-checkadvisory hook that runs when an assistant turn finishes (ClaudeStop, Cursorstop, Codexnotifyonagent-turn-complete). It scans the reply for well-formed{{CAT-UNVERIFIED: … -- cannot verify: …}}tags—skipping code fences, malformed tags, repeats within two hours, and anything beyond the first three per reply—and enqueues read-onlyinvestigatejobs tollm-judgewithout blocking or delaying the reply.Verdicts are delivered on the next turn via the shared judge inbox (
on_hitprompts the agent to tell the user the result in plain language). Fail-open behavior is preserved across entrypoints and broken state.install.shsymlinks the hook for Claude, Cursor, and Codex and runs the three merge installers;docs/ecosystem.mddocuments it. Unit tests cover detection/deduping and install wiring end-to-end.Reviewed by Cursor Bugbot for commit 5c76091. Bugbot is set up for automated code reviews on this repo. Configure here.