agent-relay-attribution: a two-digit number in tool output no longer clears a relayed claim - #487
Conversation
Review evidence:
- Before regex change: test_two_digit_tool_output_does_not_corroborate_relay ... FAIL
AssertionError: unexpectedly None
- After regex change: Ran 12 tests in 0.009s; OK
Acceptance checks:
- check_hook_test_coverage: OK (1 hook(s) checked)
- ok no new comments
- git diff origin/main -- engine/hooks/agent-relay-attribution/detect.py: only EVIDENCE_NUMBER_RE changed from \b\d{2,}\b to \b\d{3,}\b
…e — Review claim: agent-relay-attribution counts a number as corroborating evidence only when it has three or more digits, so a two-digit value in unrelated tool output no longer clears a relayed claim.
Review lane: behavior
Safety invariant: Every existing fires entry still fires and every existing silent entry stays silent; the hook remains advisory. Pending user confirmation in this session.
Effectiveness measurement: The new fires entry is silent before the regex change and fires after; the new silent entry stays silent both times; both outputs in the commit body.
Slice rationale: One regex floor in one hook with the fixture pair that proves it, so the diff carries one behavior claim.
Architectural effect: None; the evidence-number threshold changes from two digits to three.
Goal: Change the floor and add a fixture pair built from the hook's own decision rule.
Motivation: The fires fixture in the earlier attempt had no status word, no "because", and no digit, so decide_from_lines returned None before reaching the evidence comparison and nothing could fail before the change.
Alternative considerations: Matching only numbers adjacent to a count word was considered and left for later; the digit floor is the smallest change that separates the real pair.
Implementation details: The decision in decide_from_lines fires only when the reply has a fact (a status word, the word because, or a digit), has no attribution phrase, a task-notification arrived within the recent turns, and no evidence token of the reply appears in a verification tool's output this turn; evidence tokens are status words plus EVIDENCE_NUMBER_RE matches. The fires entry therefore uses the reply "The cheaper catch is our existing repeat-error-stop hook, which today resets its counter on every edit, so it stayed silent in both loops over the last 30 days." with a transcript whose last human turn is a task-notification, followed in the same turn by an assistant Bash tool_use and its tool_result whose text is a line containing "timeout=30" and no other run of two or more digits. With the current regex the token 30 is shared and the hook is silent; with the three-digit floor the reply has no evidence token, so claim_shown_this_turn is False and the hook fires. The silent entry uses the same transcript with the reply "UNVERIFIED: the subagent reports its counter resets on every edit, and it stayed silent over the last 30 days." whose phrase "the subagent reports" matches ATTRIBUTION_RE, so it is silent before and after. Add one test per entry in tests/test_hooks.py following the existing case() helper; run the suite with the entries added and detect.py unchanged, paste the FAIL line, then change line 31 to \b\d{3,}\b, run again and paste the OK line, both in the commit body.
Non-goals: No change to the attribution phrases, the status words, the turn window, or the teammate envelope handling.
Layer: domain
Feature state: active
Files:
- engine/hooks/agent-relay-attribution/detect.py
- engine/hooks/agent-relay-attribution/tests/fixtures/relay_fires.json
- engine/hooks/agent-relay-attribution/tests/fixtures/relay_silent.json
- engine/hooks/agent-relay-attribution/tests/test_hooks.py
Change types:
- engine/hooks/agent-relay-attribution/detect.py: modify
- engine/hooks/agent-relay-attribution/tests/fixtures/relay_fires.json: modify
- engine/hooks/agent-relay-attribution/tests/fixtures/relay_silent.json: modify
- engine/hooks/agent-relay-attribution/tests/test_hooks.py: modify
Acceptance criteria:
- `git diff origin/main -- engine/hooks/agent-relay-attribution/detect.py` shows line 31 changed to \b\d{3,}\b and nothing else.
- `python3 -m unittest discover -s engine/hooks/agent-relay-attribution/tests -v` exits 0 and includes the two new tests.
- `python3 scripts/check_hook_test_coverage.py engine/hooks/agent-relay-attribution` exits 0.
- `python3 scripts/check_no_new_comments.py --base origin/main` exits 0.
Exit code: 0
…m: the hook coverage gate passes on the finished branch. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The gate run is the direct measurement. Slice rationale: One focused proof per gate before review. Architectural effect: None; verification only. Goal: Prove the fixture pair satisfies the coverage gate. Layer exception: allowed. Proof runs after the implementation task so it checks the finished branch; it reads files only and changes nothing. Motivation: A fixture pair that the gate cannot see is not coverage. Alternative considerations: Trusting the suite alone was rejected; the gate checks fixture presence separately. Implementation details: Run the coverage gate for this hook. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when the gate passes. Exit code: 0
…: the regex change is present on the finished branch and the suite and coverage gate pass with it. Review lane: proof Safety invariant: Verification is read-only and does not alter any file. Effectiveness measurement: The grep for the new regex and the suite run are the direct measurement. Slice rationale: One focused proof before review, including the check the earlier attempt lacked: that the code change exists. Architectural effect: None; verification only. Goal: Prove the floor changed and every fixture holds. Layer exception: allowed. Proof runs after the implementation task so it checks the finished branch; it reads files only and changes nothing. Motivation: The earlier attempt passed its proof with no code change because the proof never checked for one. Alternative considerations: The full repository suite was rejected; this is the module the fix touches. Implementation details: Grep detect.py for the three-digit regex, then run the suite. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when the regex is present and the suite passes. Exit code: 0
…ate/g0.t0.a-a8e3a9b3f-3193f1d6
…No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: bbedd11c-52da-48a1-aab8-54461e4f304a
…a4c6a95c1-b0e0f67b — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_e5164505-6c29-48a1-a625-f8656c4aa1fe) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The relay-attribution hook no longer treats a two-digit value in unrelated tool output as corroboration.
Focused fixtures cover both outcomes: the unverified relay fires, while the attributed relay remains silent.
Review Claim
Evidence numbers corroborate a relayed claim only when they contain three or more digits.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
Every existing fires fixture still fires and every existing silent fixture stays silent; the hook remains advisory, with no registration, installer, or settings changes.
Slice Rationale
This slice isolates one regex threshold and its focused fixture coverage on top of the four-checker parent change.
Non-goals
Test Plan
Test Plan
python3 engine/skills/make-pr/scripts/preflight.py --base origin/main—ok preflight passed.OK.python3 scripts/check_hook_test_coverage.py engine/hooks/agent-relay-attribution—OK.python3 scripts/check_no_new_comments.py --base origin/main—ok no new comments.bash scripts/scrub-handoff-artifacts.sh—scrub-handoff-artifacts-ok.Revert Plan
Revert Plan
git revert <merge-commit-sha>.Note
Low Risk
Narrow regex change in an advisory Stop hook with fixture-backed tests; may flag slightly more unattributed relays but does not block execution.
Overview
Tightens agent-relay-attribution so unrelated Bash output can’t “verify” a relayed claim just because it shares a two-digit number (e.g.
timeout=30vs “last 30 days”).EVIDENCE_NUMBER_REindetect.pynow requires three or more digits for numeric corroboration viaevidence_tokens/claim_shown_this_turn.Unattributed relays in that situation fire again; properly attributed replies still stay silent. New
relay_fires/relay_silentfixtures and focused unittest cases lock in both sides.Reviewed by Cursor Bugbot for commit 2d56e6d. Bugbot is set up for automated code reviews on this repo. Configure here.