A scope correction needs evidence the agent actually did unrequested work - #496
Open
EdbertChan wants to merge 4 commits into
Open
Conversation
…rrection — Record a scope correction only when unrequested mutating work actually happened. Review claim: A correction phrasing alone no longer records a correction; the agent must also have performed mutating work in the turn opened by the user's previous message. Review lane: behavior Safety invariant: A correction phrasing accompanied by real mutating work records a correction exactly as it does today, at the same stage, with the same message. Explicit expansion stays excluded. The enforcement side, the contract wording, and the hard-stop release conditions are untouched. Effectiveness measurement: `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0, with a new case that exits non-zero before this change: a correction phrasing plus a transcript showing zero mutating work records nothing. Slice rationale: One conceptual unit: what evidence `correction_class` requires before returning a correction. Architectural effect: `correction_class` gains a parameter for the corroborating evidence, so the decision stops being a pure function of the user's text. Callers pass what the module already reads. Goal: Stop a status question from being treated as proof that the agent drifted. Motivation: Two status questions escalated to a hard stop that froze a session for hours, while the measured mutating-work count in both turns was zero. The escalation was unearned and the agent then invented a matching confession. Alternative considerations: Adding the question shapes to the expansion exclusion was set aside -- it suppresses the phrasing rather than asking for evidence, and a genuinely drifting agent asked "what are you doing" would then go uncaught. Comparing work against a stated scope was set aside for this slice -- it needs a recorded contract to compare against, which only exists after the first stage, so it cannot gate the first stage. Implementation details: Give `correction_class` the corroborating evidence as an argument rather than computing it inline, so it stays testable as a pure decision. Derive that evidence from the transcript the module already receives: whether any mutating tool call appears after the user's previous message. Treat unreadable or absent transcript evidence as its own outcome rather than as either answer, per the repo's rule that a check which could not run is not a pass; record which way it resolves and pin it with a case. Non-goals: No change to enforcement, to which tools are blocked at each stage, to the contract wording, or to the hard-stop release conditions. No change to the correction phrasing list. No new hook. Layer: domain Feature state: active Files: engine/hooks/scope-lock/detect.py, engine/hooks/scope-lock/claude_prompt_scope.py, engine/hooks/scope-lock/tests/test_hooks.py, engine/hooks/scope-lock/README.md Change types: - engine/hooks/scope-lock/detect.py: modify - engine/hooks/scope-lock/claude_prompt_scope.py: modify - engine/hooks/scope-lock/tests/test_hooks.py: modify - engine/hooks/scope-lock/README.md: modify Acceptance criteria: - `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0 after the change, with all 54 existing cases still passing. - A correction phrasing with zero mutating work since the user's previous message records nothing. - The same phrasing with real mutating work records a correction, unchanged from today. - Transcript evidence that cannot be read resolves to a named third outcome, pinned by a case. Solution: Record a scope correction only when unrequested mutating work actually happened. Review claim: A correction phrasing alone no longer records a correction; the agent must also have performed mutating work in the turn opened by the user's previous message. Review lane: behavior Safety invariant: A correction phrasing accompanied by real mutating work records a correction exactly as it does today, at the same stage, with the same message. Explicit expansion stays excluded. The enforcement side, the contract wording, and the hard-stop release conditions are untouched. Effectiveness measurement: `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0, with a new case that exits non-zero before this change: a correction phrasing plus a transcript showing zero mutating work records nothing. Slice rationale: One conceptual unit: what evidence `correction_class` requires before returning a correction. Architectural effect: `correction_class` gains a parameter for the corroborating evidence, so the decision stops being a pure function of the user's text. Callers pass what the module already reads. Goal: Stop a status question from being treated as proof that the agent drifted. Motivation: Two status questions escalated to a hard stop that froze a session for hours, while the measured mutating-work count in both turns was zero. The escalation was unearned and the agent then invented a matching confession. Alternative considerations: Adding the question shapes to the expansion exclusion was set aside -- it suppresses the phrasing rather than asking for evidence, and a genuinely drifting agent asked "what are you doing" would then go uncaught. Comparing work against a stated scope was set aside for this slice -- it needs a recorded contract to compare against, which only exists after the first stage, so it cannot gate the first stage. Implementation details: Give `correction_class` the corroborating evidence as an argument rather than computing it inline, so it stays testable as a pure decision. Derive that evidence from the transcript the module already receives: whether any mutating tool call appears after the user's previous message. Treat unreadable or absent transcript evidence as its own outcome rather than as either answer, per the repo's rule that a check which could not run is not a pass; record which way it resolves and pin it with a case. Non-goals: No change to enforcement, to which tools are blocked at each stage, to the contract wording, or to the hard-stop release conditions. No change to the correction phrasing list. No new hook. Layer: domain Feature state: active Files: engine/hooks/scope-lock/detect.py, engine/hooks/scope-lock/claude_prompt_scope.py, engine/hooks/scope-lock/tests/test_hooks.py, engine/hooks/scope-lock/README.md Change types: - engine/hooks/scope-lock/detect.py: modify - engine/hooks/scope-lock/claude_prompt_scope.py: modify - engine/hooks/scope-lock/tests/test_hooks.py: modify - engine/hooks/scope-lock/README.md: modify Acceptance criteria: - `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0 after the change, with all 54 existing cases still passing. - A correction phrasing with zero mutating work since the user's previous message records nothing. - The same phrasing with real mutating work records a correction, unchanged from today. - Transcript evidence that cannot be read resolves to a named third outcome, pinned by a case. Invoker-Finalize-Id: 5a0246f5-8c0f-402b-91e3-ae8e12197dfd
…he hook's own suite as the shared repro for this slice. Review claim: The suite exits 0 only when a correction phrasing without corroborating work records nothing and one with it still records a correction. Review lane: proof Safety invariant: The repro is identical before and after; the new negative case exits non-zero before and 0 after. Effectiveness measurement: The exit status of the suite is the effectiveness signal for this slice. Slice rationale: One conceptual unit: the deterministic repro only. Architectural effect: None. Goal: Deterministically prove the change. Motivation: 54 cases exist and none of them asserts that a correction needs corroborating work, which is why wording alone could freeze a session. Alternative considerations: Asserting only the negative case was set aside -- without the positive one, a change that disables detection entirely would still exit 0. Implementation details: Execute the suite as the terminal gate. Non-goals: No product edits here. Layer: e2e_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 2e537274-f32f-48a9-8fb7-61dffeb18567
…only gate confirming no ephemeral handoff files were left behind. Review claim: The workflow leaves no ephemeral handoff files in the tree. Review lane: proof Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work. Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal. Slice rationale: One conceptual unit: the hygiene gate. Architectural effect: None. Goal: Confirm no ephemeral handoff files remain after every other task finishes. Motivation: Ephemeral inter-task files leak into the diff and read as part of the change. Alternative considerations: Manual inspection was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: e2e_regression Feature state: active Exit code: 0 Invoker-Finalize-Id: 7603b996-8545-498f-a695-0ba70ed69c6c
…ad1813c48-145c3c0e — Terminal read-only gate confirming no ephemeral handoff files were left behind. Review claim: The workflow leaves no ephemeral handoff files in the tree. Review lane: proof Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work. Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal. Slice rationale: One conceptual unit: the hygiene gate. Architectural effect: None. Goal: Confirm no ephemeral handoff files remain after every other task finishes. Motivation: Ephemeral inter-task files leak into the diff and read as part of the change. Alternative considerations: Manual inspection was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply. Non-goals: No deletion, no index changes, no commits. Layer: e2e_regression Feature state: active
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_63794f85-cbf0-41e3-abcb-23ab4cfad486) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The scope lock protects work after a user says the agent went off task.
Previously, correction wording alone could trigger that lock, including ordinary status questions.
The detector now checks the prior agent turn for a mutating tool call before recording a correction.
Read-only, missing, unreadable, or unparseable transcript evidence does not count as mutating work.
Review Claim
Approve requiring evidence of unrequested mutating work before correction wording can trigger the scope lock.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
When correction wording follows real mutating work, the same correction is recorded at the same stage with the same message. Expansion wording remains excluded, and enforcement and release behavior are unchanged.
Slice Rationale
This is one behavior slice: the correction decision now requires transcript evidence already available to the hook. Tests and documentation describe that same decision.
Non-goals
Architecture
Before
graph TD A["Current user wording"] --> B["Correction phrase matcher"] B --> C["Record scope correction"]After
graph TD A["Current user wording"] --> B["Correction phrase matcher"] D["Prior user turn transcript"] --> E["Mutating-work evidence"] B --> F["Correction decision"] E --> F F --> C["Record scope correction"]Test Plan
Test Plan
python3 engine/hooks/scope-lock/tests/test_hooks.py—Ran 58 tests in 0.072s;OK.python3 engine/skills/make-pr/scripts/preflight.py --base origin/main—ok preflight passed; hook coverageOK (1 hook(s) checked).bash scripts/scrub-handoff-artifacts.sh—scrub-handoff-artifacts-ok.python3 scripts/check_no_new_comments.py --base main—ok no new comments.Revert Plan
Revert Plan
git revert <merge-commit-sha>.python3 engine/hooks/scope-lock/tests/test_hooks.py.Note
Medium Risk
Changes when the scope-lock state machine arms, which directly affects tool blocking in live harness sessions; behavior is intentionally more conservative when transcript evidence is missing.
Overview
Scope corrections no longer lock the session from wording alone.
correction_classnow needsmutating_work is True, supplied by scanning the session transcript for a non–read-only tool call in the agent turn after the previous user message.mutating_work_after_previous_userparses the JSONL transcript (including nestedresponse_itemshapes), aggregates tool names per user turn, and treats read-only tools like Read/Grep as non-corroborating. Missing, unreadable, or bad transcript data returnsNone(unknown), which does not record a correction—so the hook fails open when evidence cannot be verified. Pure read-only prior turns returnFalse, so status-style questions after inspection no longer trip the lock.Contract gates, hard-stop rules, and pretool allowlists are unchanged aside from sharing
_is_local_read_only_toolfor normalization. Docs and tests cover the new corroboration rule and the tradeoff that drift stated only in a plan, before any mutating call, is no longer caught.Reviewed by Cursor Bugbot for commit 3a579a1. Bugbot is set up for automated code reviews on this repo. Configure here.