Flag a capability list restated from an error message as if it were the tool's own answer - #483
Open
EdbertChan wants to merge 4 commits into
Conversation
…ext — Add a detector for a capability enumeration restated out of error-shaped tool output. Review claim: An outgoing message that restates two or more allowed values from error-shaped tool output beside a capability verb is flagged, unless a non-error source for those values exists or the message attributes them. Review lane: behavior Safety invariant: Every message the shipped hook flags today is still flagged with the same text, and every message it passes today still passes unless it matches this new rule. The 24 existing cases keep passing untouched. Effectiveness measurement: `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0, with new positive cases that exit non-zero before this change and the negative cases below staying silent both before and after. Slice rationale: One conceptual unit: one new rule inside an existing detector, with its fixtures. Architectural effect: None; the hook already parses transcript lines at Stop. This reads tool output lines it already has. Goal: Stop a value lifted out of an error string from going out as the authority it was never sourced from. Motivation: Three outgoing messages asserted a third party's capabilities using a list copied from a second party's error text. The operator believed them and acted. Alternative considerations: A PostToolUse rule firing when the string enters was set aside on measurement -- the value arrived 334 transcript rows before the last assertion, so an entry-time nudge fires far too early to bind the decision. Widening the shipped claim gate was set aside -- that gate asks whether a sentence is hedged; this asks where a value came from, a different question, and three open pull requests are already churning that file. Implementation details: Capture enumerations appearing inside error-shaped tool output in the transcript lines the hook already parses. At Stop, flag an outgoing message restating two or more captured values within range of a capability verb, when no non-error occurrence of those values exists in the transcript yet. Clear on an attribution phrase such as fallback, hardcoded, or in the error. Non-goals: No new hook directory. No change to the shipped hedge or diagnosis rules. No change to the claim gate in engine/hooks/diu-stop. No attempt to judge whether a check was cheap or available -- that is not computable and is out of scope. Layer: domain Feature state: active Files: engine/hooks/hedge-runs-prove-it/detect.py, engine/hooks/hedge-runs-prove-it/tests/test_hooks.py, engine/hooks/hedge-runs-prove-it/README.md Change types: - engine/hooks/hedge-runs-prove-it/detect.py: modify - engine/hooks/hedge-runs-prove-it/tests/test_hooks.py: modify - engine/hooks/hedge-runs-prove-it/README.md: modify Acceptance criteria: - `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0 after the change, with all 24 existing cases still passing. - A message restating two or more values that appear only inside error-shaped tool output, beside a capability verb, is flagged. - The same message is not flagged once a non-error source for those values exists in the transcript. - A message attributing the list as a fallback is not flagged. Solution: Add a detector for a capability enumeration restated out of error-shaped tool output. Review claim: An outgoing message that restates two or more allowed values from error-shaped tool output beside a capability verb is flagged, unless a non-error source for those values exists or the message attributes them. Review lane: behavior Safety invariant: Every message the shipped hook flags today is still flagged with the same text, and every message it passes today still passes unless it matches this new rule. The 24 existing cases keep passing untouched. Effectiveness measurement: `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0, with new positive cases that exit non-zero before this change and the negative cases below staying silent both before and after. Slice rationale: One conceptual unit: one new rule inside an existing detector, with its fixtures. Architectural effect: None; the hook already parses transcript lines at Stop. This reads tool output lines it already has. Goal: Stop a value lifted out of an error string from going out as the authority it was never sourced from. Motivation: Three outgoing messages asserted a third party's capabilities using a list copied from a second party's error text. The operator believed them and acted. Alternative considerations: A PostToolUse rule firing when the string enters was set aside on measurement -- the value arrived 334 transcript rows before the last assertion, so an entry-time nudge fires far too early to bind the decision. Widening the shipped claim gate was set aside -- that gate asks whether a sentence is hedged; this asks where a value came from, a different question, and three open pull requests are already churning that file. Implementation details: Capture enumerations appearing inside error-shaped tool output in the transcript lines the hook already parses. At Stop, flag an outgoing message restating two or more captured values within range of a capability verb, when no non-error occurrence of those values exists in the transcript yet. Clear on an attribution phrase such as fallback, hardcoded, or in the error. Non-goals: No new hook directory. No change to the shipped hedge or diagnosis rules. No change to the claim gate in engine/hooks/diu-stop. No attempt to judge whether a check was cheap or available -- that is not computable and is out of scope. Layer: domain Feature state: active Files: engine/hooks/hedge-runs-prove-it/detect.py, engine/hooks/hedge-runs-prove-it/tests/test_hooks.py, engine/hooks/hedge-runs-prove-it/README.md Change types: - engine/hooks/hedge-runs-prove-it/detect.py: modify - engine/hooks/hedge-runs-prove-it/tests/test_hooks.py: modify - engine/hooks/hedge-runs-prove-it/README.md: modify Acceptance criteria: - `python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py` exits 0 after the change, with all 24 existing cases still passing. - A message restating two or more values that appear only inside error-shaped tool output, beside a capability verb, is flagged. - The same message is not flagged once a non-error source for those values exists in the transcript. - A message attributing the list as a fallback is not flagged. Invoker-Finalize-Id: 8156de0f-c490-468a-a9e5-0fb0dcdcc394
…the hook's own suite as the shared repro for this slice. Review claim: The suite exits 0 only when the new rule flags the copied-from-error shape and stays silent on all four negative shapes. Review lane: proof Safety invariant: The repro is identical before and after; the new positive cases exit non-zero before and 0 after. Effectiveness measurement: The exit status of the suite is the effectiveness signal for this slice. Slice rationale: One conceptual unit: the deterministic repro only. Architectural effect: None. Goal: Deterministically prove the new rule. Motivation: Nothing today asserts where a restated value came from, which is why three messages carried one out of an error string unchallenged. Alternative considerations: Covering only the positive shape was set aside -- without the non-error-source and attributed cases, a rule that fires on every restatement would still exit 0. Implementation details: Execute the suite as the terminal gate. Non-goals: No product edits here. Layer: e2e_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 5910aca8-b6ae-453c-b815-4076d8d90fca
…only gate confirming no ephemeral handoff files were left behind. Review claim: The workflow leaves no ephemeral handoff files in the tree. Review lane: proof Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work. Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal. Slice rationale: One conceptual unit: the hygiene gate. Architectural effect: None. Goal: Confirm no ephemeral handoff files remain after every other task finishes. Motivation: Ephemeral inter-task files leak into the diff and read as part of the change. Alternative considerations: Manual inspection was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: e2e_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 7b20f59a-7962-415f-ad6c-6f9e3a8811b9
…a437a0460-90926507 — Terminal read-only gate confirming no ephemeral handoff files were left behind. Review claim: The workflow leaves no ephemeral handoff files in the tree. Review lane: proof Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work. Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal. Slice rationale: One conceptual unit: the hygiene gate. Architectural effect: None. Goal: Confirm no ephemeral handoff files remain after every other task finishes. Motivation: Ephemeral inter-task files leak into the diff and read as part of the change. Alternative considerations: Manual inspection was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: e2e_regression Feature state: active
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_6494c46e-0a87-4e9d-9065-0320edb538a2) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This checker reviews outgoing replies for claims about what a tool accepts or supports.
The problem was a capability list copied from an error message and presented as the tool's own answer.
The change records enumerated values found only in error-shaped tool output and checks later capability claims against that source.
It flags repeated values without another source, while allowing attributed fallback lists and values supplied by a non-error result.
Review Claim
The checker flags a capability list repeated from error output unless the reply attributes it or another tool result supports it.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
Existing hedge and diagnosis decisions remain unchanged; only replies matching the new error-only capability-list shape receive additional feedback.
Slice Rationale
This is one behavioral slice: one detector rule, its focused regression cases, and the matching documentation.
Non-goals
Test Plan
Test Plan
python3 engine/skills/make-pr/scripts/preflight.py --base origin/main—preflight passed; hook coverage:OK (1 hook(s) checked).python3 engine/hooks/hedge-runs-prove-it/tests/test_hooks.py—Ran 28 tests;OK.python3 scripts/check_hook_test_coverage.py engine/hooks/hedge-runs-prove-it—OK (1 hook(s) checked).bash scripts/scrub-handoff-artifacts.sh—scrub-handoff-artifacts-ok.Revert Plan
Revert Plan
git revert <merge-commit-sha>.Note
Low Risk
Additive Stop-hook rule with exemptions and a two-value threshold; mis-fires could block benign replies but do not change auth, data, or existing hedge/diagnosis gates.
Overview
Adds a third check to the
hedge-runs-prove-itStop hook: replies that sound like verified capability facts (accepts,supports,only, etc.) but repeat two or more model/token values that appeared only in error-shaped tool results.The hook now scans turn transcripts for bracketed or comma-joined enumerations in error output, then blocks nearby restatements unless a non-error tool result also lists those values or the reply attributes the list (fallback, hardcoded, built-in, “in the error”, retract).
Hedge and diagnosis behavior is unchanged;
decide()only loads the transcript when capability verbs appear (not just when hedges fire). README and focused unit/hook tests cover the new path.Reviewed by Cursor Bugbot for commit 6707b0b. Bugbot is set up for automated code reviews on this repo. Configure here.