Skip to content

[CAT-UNVERIFIED tag](1) One shared escape-hatch tag; bare UNVERIFIED: excuses nothing - #415

Merged
mergify[bot] merged 1 commit into
mainfrom
stack/EdbertChan/cat-unverified-tag/1-shared-tag-in-hooks
Sep 11, 2026
Merged

[CAT-UNVERIFIED tag](1) One shared escape-hatch tag; bare UNVERIFIED: excuses nothing#415
mergify[bot] merged 1 commit into
mainfrom
stack/EdbertChan/cat-unverified-tag/1-shared-tag-in-hooks

Conversation

@EdbertChan

@EdbertChan EdbertChan commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Summary

Every evidence hook now accepts one escape hatch: {{CAT-UNVERIFIED: <claim> -- cannot verify: <reason>}}. Bare UNVERIFIED: excuses nothing.

The old marker cost four characters and silenced a whole message. It got used to stop checking, not to report a check that could not run.

Six hooks also kept private copies of the marker regex, and they had drifted apart on whether a blocker must be named.

One shared module now defines the tag. All seven hooks that read a marker use it, and a tag with no named blocker excuses nothing.

Supersedes #383: same paragraph scoping and must-name-a-blocker rule, moved onto the new tag.

Before and After

The same three messages through engine/hooks/diu-stop/claude_stop_check.py, on 509b9cd and on this branch:

legacy marker in front of a real claim
  before exit=2  This message contains an `UNVERIFIED:` claim...
  after  exit=2  This message makes an unverified-shaped claim ("because")...
well-formed tag beside the claim
  before exit=2  This message contains an `UNVERIFIED:` claim...
  after  exit=0  (silent)
tag naming no blocker
  before exit=2  This message contains an `UNVERIFIED:` claim...
  after  exit=2  This message makes an unverified-shaped claim ("because")...

Review Claim

Approve one shared definition of the escape-hatch tag, read by all seven hooks that honour a marker, with bare UNVERIFIED: no longer excusing anything.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Every message blocked on 509b9cd is still blocked, except one carrying a tag that names a blocker -- syntax that did not exist before. Messages that relied on bare UNVERIFIED: to pass are now blocked; that is the intended change.

Slice Rationale

The seven hooks move together because external-claim-gate borrows diu-stop's matcher (detect.py:13). Splitting them left external-claim-gate failing 2 of 40 tests. The retry change is the next PR.

Non-goals

  • No change to what a retry skips. That is the next PR in this stack.
  • No change to the prose that tells authors what to write. That is PRs 3 to 5.
  • scripts/check_history_claims.py, the CI twin of history-claim-check, keeps its own copy. It is a script, not a hook, so it cannot reach _markers by the sibling path.

Test Plan

Test Plan
  • python3 engine/skills/make-pr/scripts/preflight.py --base <this PR's base>
unit    engine-runtime: 27 file(s)
neutral 1 file(s): tests/scenarios/prove-it-claims.json
declare Review Unit: engine-runtime
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/_markers
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/diu-stop
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/external-claim-gate
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/hedge-runs-prove-it
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/history-claim-check
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/incidence-needs-repetition
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/named-verb-guard
        check_hook_test_coverage: OK (1 hook(s) checked)
gate    python3 scripts/check_hook_test_coverage.py engine/hooks/prove-it-ship-gate
        check_hook_test_coverage: OK (1 hook(s) checked)
ok      preflight passed
  • bash scripts/run_all_tests.sh -- exit 0, 45 suites OK, 0 failed, 1668 tests
  • python3 -m unittest discover -s engine/hooks/_markers/tests -- includes test_installed_layout, which builds the symlinked $HOME/.claude/hooks/ layout and imports through it
  • Before/after above: the three messages piped as {"last_assistant_message": ...} into the hook from 509b9cd and from this branch

Revert Plan

Revert Plan
  • Safe to revert? Yes, once the PRs stacked on it are reverted first
  • Revert command: git revert <sha>
  • Post-revert steps: rerun ./install.sh so the _markers link is removed
  • Data migration? No

🤖 Generated with Claude Code

https://claude.ai/code/session_01VKsvxJk65w6q7KnPSRYvNg


Note

Medium Risk
Changes enforcement behavior across multiple Stop/PreToolUse hooks: messages that relied on bare UNVERIFIED: will now block, though coverage is broad via tests and shared marker logic.

Overview
Introduces a shared engine/hooks/_markers module so every evidence hook uses one definition of the escape hatch: {{CAT-UNVERIFIED: <claim> -- cannot verify: <reason>}}. Only tags that name a blocker after cannot verify: excuse the paragraph they sit in; malformed tags and unverified claims still block.

Bare UNVERIFIED: is retired as a free pass—diu-stop and the other hooks now treat it as legacy prose, surface guidance toward the new tag, and no longer silence whole messages. Seven hooks (diu-stop, external-claim-gate, hedge-runs-prove-it, history-claim-check, incidence-needs-repetition, named-verb-guard, prove-it-ship-gate) import the shared module via the symlinked $HOME/.claude/hooks/ layout; install.sh adds the _markers link. Unit and layout tests cover tag parsing, installer wiring, and block messages that preserve double braces in .format() templates.

Scenario fixtures in tests/scenarios/prove-it-claims.json are updated for well-formed tags, tags without blockers, and legacy markers still firing ship/hedge gates.

Reviewed by Cursor Bugbot for commit a25cdb5. Bugbot is set up for automated code reviews on this repo. Configure here.

Bare `UNVERIFIED:` silenced every check in the message it appeared in. It
cost four characters and bought a whole turn, so it got used to stop
checking rather than to report a check that could not run.

Six hooks read that marker, each with a private regex, and they had
drifted: hedge-runs-prove-it required a named blocker, diu-stop did not.
engine/hooks/_markers now holds one definition of
{{CAT-UNVERIFIED: <claim> -- cannot verify: <reason>}}, the names-a-blocker
test, and the retired bare marker. install.sh links it beside the hooks.

Every suppressor reads it: diu-stop, external-claim-gate (which borrows
diu-stop's matcher at detect.py:13), hedge-runs-prove-it,
history-claim-check, incidence-needs-repetition, named-verb-guard and
prove-it-ship-gate. A well-formed tag excuses its paragraph. A tag that
names no blocker excuses nothing. Bare UNVERIFIED: is prose, and the block
text names the tag that replaced it.

Same three messages through diu-stop, on 509b9cd and on this commit:

  legacy marker in front of a real claim
    before exit=2  contains an `UNVERIFIED:` claim...
    after  exit=2  unverified-shaped claim ("because")...
  well-formed tag beside the claim
    before exit=2  contains an `UNVERIFIED:` claim...
    after  exit=0  (silent)
  tag naming no blocker
    before exit=2  contains an `UNVERIFIED:` claim...
    after  exit=2  unverified-shaped claim ("because")...

Two bugs found by tests while building this, both now pinned:
- A hook directory is itself a symlink, so joining ".." onto it resolves
  the link first and lands beside the checkout. Two textual dirnames reach
  the sibling module; test_installed_layout builds the symlinked layout.
- Block texts are .format() templates, so a literal {{CAT-UNVERIFIED}}
  rendered as {CAT-UNVERIFIED}. Every existing test passed through it,
  because they asserted the substring CAT-UNVERIFIED. The tag is now a
  format argument, and _markers tests assert on the braces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VKsvxJk65w6q7KnPSRYvNg
@cursor

cursor Bot commented Sep 11, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_841d75b0-61da-4b3c-80fe-622bcfca960c)

@mergify

mergify Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Queued — the merge queue status continues in this comment ↓.

@EdbertChan

Copy link
Copy Markdown
Owner Author

Mergify repair stopped: PR #415 is ready to land, but upper stack PR(s) #416, #417, #418, #419 are open without admin-bypass; a human must decide whether to include them in the admin-bypass landing stack or land them separately.

@EdbertChan

Copy link
Copy Markdown
Owner Author

@Mergifyio queue

@mergify

mergify Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

This pull request spent 5 minutes 9 seconds in the queue, including 3 minutes 44 seconds running CI.

Required conditions to merge
  • check-success = lint
  • check-success = test

@mergify mergify Bot added the queued label Sep 11, 2026
@mergify
mergify Bot merged commit 8d2039a into main Sep 11, 2026
4 checks passed
@mergify mergify Bot removed the queued label Sep 11, 2026
EdbertChan added a commit that referenced this pull request Sep 11, 2026
Three detectors each had a hand-rolled backtest.py. scripts/backtest_detector.py
now replays any PATH:CALLABLE detector over local Claude Code and OMP
transcripts, with --compare against another detector or a git ref. The hooks
gain replay functions beside their unchanged decision code, and the three
per-detector scripts are deleted.

Carved out of #371 onto current main, with two fixes its tests needed there:
- materialize_revision resolves symlinks before taking the path relative to
  the git toplevel. On macOS /var is a link to /private/var, so the relative
  path climbed out of the archived copy and --compare HEAD loaded the working
  tree instead of HEAD.
- the near-miss test used a bare UNVERIFIED: line, which diu-stop no longer
  honours (#415); it now uses inline code excused by fenced output.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gnua5cuAAF1ey1b7zRqJCC
Change-Id: I3066797e7031407d5a7074b62a57d7e0e2db25b7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants