Skip to content

A scope correction needs evidence the agent actually did unrequested work - #496

Open
EdbertChan wants to merge 4 commits into
mainfrom
plan/a-scope-correction-needs-evidence-the-agent-actually-did-unrequested-work
Open

A scope correction needs evidence the agent actually did unrequested work#496
EdbertChan wants to merge 4 commits into
mainfrom
plan/a-scope-correction-needs-evidence-the-agent-actually-did-unrequested-work

Conversation

@EdbertChan

@EdbertChan EdbertChan commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Summary

The scope lock protects work after a user says the agent went off task.

Previously, correction wording alone could trigger that lock, including ordinary status questions.

The detector now checks the prior agent turn for a mutating tool call before recording a correction.

Read-only, missing, unreadable, or unparseable transcript evidence does not count as mutating work.

Review Claim

Approve requiring evidence of unrequested mutating work before correction wording can trigger the scope lock.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

When correction wording follows real mutating work, the same correction is recorded at the same stage with the same message. Expansion wording remains excluded, and enforcement and release behavior are unchanged.

Slice Rationale

This is one behavior slice: the correction decision now requires transcript evidence already available to the hook. Tests and documentation describe that same decision.

Non-goals

  • No change to enforcement, blocked-tool stages, contract wording, or hard-stop release conditions.
  • No change to correction phrases or addition of a new hook.
  • No comparison against a stated scope; this slice only establishes whether mutating work occurred.

Architecture

Before

graph TD
    A["Current user wording"] --> B["Correction phrase matcher"]
    B --> C["Record scope correction"]
Loading

After

graph TD
    A["Current user wording"] --> B["Correction phrase matcher"]
    D["Prior user turn transcript"] --> E["Mutating-work evidence"]
    B --> F["Correction decision"]
    E --> F
    F --> C["Record scope correction"]
Loading

Test Plan

Test Plan
  • python3 engine/hooks/scope-lock/tests/test_hooks.pyRan 58 tests in 0.072s; OK.
  • python3 engine/skills/make-pr/scripts/preflight.py --base origin/mainok preflight passed; hook coverage OK (1 hook(s) checked).
  • bash scripts/scrub-handoff-artifacts.shscrub-handoff-artifacts-ok.
  • python3 scripts/check_no_new_comments.py --base mainok no new comments.

Revert Plan

Revert Plan
  • Safe to revert? Yes.
  • Revert command: git revert <merge-commit-sha>.
  • Post-revert steps: Run python3 engine/hooks/scope-lock/tests/test_hooks.py.
  • Data migration? No.

Note

Medium Risk
Changes when the scope-lock state machine arms, which directly affects tool blocking in live harness sessions; behavior is intentionally more conservative when transcript evidence is missing.

Overview
Scope corrections no longer lock the session from wording alone. correction_class now needs mutating_work is True, supplied by scanning the session transcript for a non–read-only tool call in the agent turn after the previous user message.

mutating_work_after_previous_user parses the JSONL transcript (including nested response_item shapes), aggregates tool names per user turn, and treats read-only tools like Read/Grep as non-corroborating. Missing, unreadable, or bad transcript data returns None (unknown), which does not record a correction—so the hook fails open when evidence cannot be verified. Pure read-only prior turns return False, so status-style questions after inspection no longer trip the lock.

Contract gates, hard-stop rules, and pretool allowlists are unchanged aside from sharing _is_local_read_only_tool for normalization. Docs and tests cover the new corroboration rule and the tradeoff that drift stated only in a plan, before any mutating call, is no longer caught.

Reviewed by Cursor Bugbot for commit 3a579a1. Bugbot is set up for automated code reviews on this repo. Configure here.

Invoker Bot and others added 4 commits September 12, 2026 18:14
…rrection — Record a scope correction only when unrequested mutating work actually happened.

Review claim: A correction phrasing alone no longer records a correction; the agent must also have performed mutating work in the turn opened by the user's previous message.
Review lane: behavior
Safety invariant: A correction phrasing accompanied by real mutating work records a correction exactly as it does today, at the same stage, with the same message. Explicit expansion stays excluded. The enforcement side, the contract wording, and the hard-stop release conditions are untouched.
Effectiveness measurement: `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0, with a new case that exits non-zero before this change: a correction phrasing plus a transcript showing zero mutating work records nothing.
Slice rationale: One conceptual unit: what evidence `correction_class` requires before returning a correction.
Architectural effect: `correction_class` gains a parameter for the corroborating evidence, so the decision stops being a pure function of the user's text. Callers pass what the module already reads.
Goal: Stop a status question from being treated as proof that the agent drifted.
Motivation: Two status questions escalated to a hard stop that froze a session for hours, while the measured mutating-work count in both turns was zero. The escalation was unearned and the agent then invented a matching confession.
Alternative considerations: Adding the question shapes to the expansion exclusion was set aside -- it suppresses the phrasing rather than asking for evidence, and a genuinely drifting agent asked "what are you doing" would then go uncaught. Comparing work against a stated scope was set aside for this slice -- it needs a recorded contract to compare against, which only exists after the first stage, so it cannot gate the first stage.
Implementation details: Give `correction_class` the corroborating evidence as an argument rather than computing it inline, so it stays testable as a pure decision. Derive that evidence from the transcript the module already receives: whether any mutating tool call appears after the user's previous message. Treat unreadable or absent transcript evidence as its own outcome rather than as either answer, per the repo's rule that a check which could not run is not a pass; record which way it resolves and pin it with a case.
Non-goals: No change to enforcement, to which tools are blocked at each stage, to the contract wording, or to the hard-stop release conditions. No change to the correction phrasing list. No new hook.
Layer: domain
Feature state: active
Files: engine/hooks/scope-lock/detect.py, engine/hooks/scope-lock/claude_prompt_scope.py, engine/hooks/scope-lock/tests/test_hooks.py, engine/hooks/scope-lock/README.md
Change types:
- engine/hooks/scope-lock/detect.py: modify
- engine/hooks/scope-lock/claude_prompt_scope.py: modify
- engine/hooks/scope-lock/tests/test_hooks.py: modify
- engine/hooks/scope-lock/README.md: modify
Acceptance criteria:
- `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0 after the change, with all 54 existing cases still passing.
- A correction phrasing with zero mutating work since the user's previous message records nothing.
- The same phrasing with real mutating work records a correction, unchanged from today.
- Transcript evidence that cannot be read resolves to a named third outcome, pinned by a case.

Solution:
  Record a scope correction only when unrequested mutating work actually happened.
Review claim: A correction phrasing alone no longer records a correction; the agent must also have performed mutating work in the turn opened by the user's previous message.
Review lane: behavior
Safety invariant: A correction phrasing accompanied by real mutating work records a correction exactly as it does today, at the same stage, with the same message. Explicit expansion stays excluded. The enforcement side, the contract wording, and the hard-stop release conditions are untouched.
Effectiveness measurement: `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0, with a new case that exits non-zero before this change: a correction phrasing plus a transcript showing zero mutating work records nothing.
Slice rationale: One conceptual unit: what evidence `correction_class` requires before returning a correction.
Architectural effect: `correction_class` gains a parameter for the corroborating evidence, so the decision stops being a pure function of the user's text. Callers pass what the module already reads.
Goal: Stop a status question from being treated as proof that the agent drifted.
Motivation: Two status questions escalated to a hard stop that froze a session for hours, while the measured mutating-work count in both turns was zero. The escalation was unearned and the agent then invented a matching confession.
Alternative considerations: Adding the question shapes to the expansion exclusion was set aside -- it suppresses the phrasing rather than asking for evidence, and a genuinely drifting agent asked "what are you doing" would then go uncaught. Comparing work against a stated scope was set aside for this slice -- it needs a recorded contract to compare against, which only exists after the first stage, so it cannot gate the first stage.
Implementation details: Give `correction_class` the corroborating evidence as an argument rather than computing it inline, so it stays testable as a pure decision. Derive that evidence from the transcript the module already receives: whether any mutating tool call appears after the user's previous message. Treat unreadable or absent transcript evidence as its own outcome rather than as either answer, per the repo's rule that a check which could not run is not a pass; record which way it resolves and pin it with a case.
Non-goals: No change to enforcement, to which tools are blocked at each stage, to the contract wording, or to the hard-stop release conditions. No change to the correction phrasing list. No new hook.
Layer: domain
Feature state: active
Files: engine/hooks/scope-lock/detect.py, engine/hooks/scope-lock/claude_prompt_scope.py, engine/hooks/scope-lock/tests/test_hooks.py, engine/hooks/scope-lock/README.md
Change types:
- engine/hooks/scope-lock/detect.py: modify
- engine/hooks/scope-lock/claude_prompt_scope.py: modify
- engine/hooks/scope-lock/tests/test_hooks.py: modify
- engine/hooks/scope-lock/README.md: modify
Acceptance criteria:
- `python3 engine/hooks/scope-lock/tests/test_hooks.py` exits 0 after the change, with all 54 existing cases still passing.
- A correction phrasing with zero mutating work since the user's previous message records nothing.
- The same phrasing with real mutating work records a correction, unchanged from today.
- Transcript evidence that cannot be read resolves to a named third outcome, pinned by a case.

Invoker-Finalize-Id: 5a0246f5-8c0f-402b-91e3-ae8e12197dfd
…he hook's own suite as the shared repro for this slice.

Review claim: The suite exits 0 only when a correction phrasing without corroborating work records nothing and one with it still records a correction.
Review lane: proof
Safety invariant: The repro is identical before and after; the new negative case exits non-zero before and 0 after.
Effectiveness measurement: The exit status of the suite is the effectiveness signal for this slice.
Slice rationale: One conceptual unit: the deterministic repro only.
Architectural effect: None.
Goal: Deterministically prove the change.
Motivation: 54 cases exist and none of them asserts that a correction needs corroborating work, which is why wording alone could freeze a session.
Alternative considerations: Asserting only the negative case was set aside -- without the positive one, a change that disables detection entirely would still exit 0.
Implementation details: Execute the suite as the terminal gate.
Non-goals: No product edits here.
Layer: e2e_regression
Feature state: active

Exit code: 0
Invoker-Finalize-Id: 2e537274-f32f-48a9-8fb7-61dffeb18567
…only gate confirming no ephemeral handoff files were left behind.

Review claim: The workflow leaves no ephemeral handoff files in the tree.
Review lane: proof
Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work.
Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal.
Slice rationale: One conceptual unit: the hygiene gate.
Architectural effect: None.
Goal: Confirm no ephemeral handoff files remain after every other task finishes.
Motivation: Ephemeral inter-task files leak into the diff and read as part of the change.
Alternative considerations: Manual inspection was set aside as non-deterministic.
Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply.
Non-goals: No deletion, no index changes, no commits.
Layer: e2e_regression
Feature state: active

Exit code: 0
Invoker-Finalize-Id: 7603b996-8545-498f-a695-0ba70ed69c6c
…ad1813c48-145c3c0e — Terminal read-only gate confirming no ephemeral handoff files were left behind.

Review claim: The workflow leaves no ephemeral handoff files in the tree.
Review lane: proof
Safety invariant: Read-only -- never deletes files, alters the index, or commits caller work.
Effectiveness measurement: A non-zero exit when ephemeral handoff files remain is the signal.
Slice rationale: One conceptual unit: the hygiene gate.
Architectural effect: None.
Goal: Confirm no ephemeral handoff files remain after every other task finishes.
Motivation: Ephemeral inter-task files leak into the diff and read as part of the change.
Alternative considerations: Manual inspection was set aside as non-deterministic.
Implementation details: Run scripts/scrub-handoff-artifacts.sh without --apply.
Non-goals: No deletion, no index changes, no commits.
Layer: e2e_regression
Feature state: active
@cursor

cursor Bot commented Sep 12, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_63794f85-cbf0-41e3-abcb-23ab4cfad486)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant