feat(agent): stuck/runaway guard + surface the failure behind a max_turns cap (ABCA-662)#600
Conversation
…loop Live-caught ABCA-483: a one-line README task burned ALL 100 turns (~22 min, $1.53) because the agent re-ran the SAME failing command (mise //cdk:test → JS-heap OOM, exit 134) over and over, yak-shaving the build env instead of finishing. Nothing noticed until the hard max_turns cap killed it. New `stuck_guard.py` (pure, dep-free): PostToolUse records each tool result; a coarse (tool_name, command) signature tracks consecutive failures. A new between-turns hook (registered after cancel, before nudge — same short-circuit rationale) then: - STEER once at 3 repeats: inject 'stop retrying X — work around it or finish with what you have'; - BAIL at 6 repeats: end the turn loop early (continue_=False) so the task fails fast instead of grinding to the cap. Conservative by design: keys on a REPEATING FAILURE (not a raw turn count), so a large task making varied progress never trips it; unknown tool output counts as success. Fail-open everywhere — a guard bug can't wedge the agent. Platform-agnostic, MAIN-BOUND: lives entirely in agent/src (stuck_guard.py + hooks.py); surfaces via the channel-neutral error_message. The honest reason is then rendered per-channel by the classifier / failure-reply (several fixes). 1158 agent tests pass (ruff + ty clean).
Reviewing the guard for false-positive risk (user: 'make sure the turn limit isn't too aggressive') showed the bail path was unsafe: distinguishing a true spin from a legitimately-iterating agent (re-running the same test command as it fixes failures one by one) from raw output is genuinely fragile — a digit- normalizing fingerprint that ignores volatile GC timings ALSO collapses 'test file_0' vs 'file_1' (the progress signal), so it would kill a working agent. A false-positive KILL is far worse than a false-positive nudge. So: removed BAIL entirely. The guard now only ever injects a ONE-TIME advisory steer when the same command fails with IDENTICAL output N times; the max_turns cap (with a fix's honest 'Exceeded max turns' reason) is the real runaway backstop. A false positive costs exactly one extra advisory comment. Platform-agnostic, main-bound.
…ns early (ABCA-662)
ABCA-662 capped at max_turns, but the CAUSE was masked: it had finished the code
and then thrashed ~15 turns retrying a failing 'git push' (remote: invalid
credentials) every which way — a DIFFERENT command each turn, so the existing
per-signature stuck-guard (same command × identical output ≥3) never tripped,
and the terminal reason read only 'Exceeded max turns'. max_turns is a CORRECT
classification, but the user can't tell 'genuinely needed more turns' from 'spun
on an error' — and raising the cap just burns more tokens.
Two changes, both advisory (no hard kill — keeps the K10 no-false-kill stance):
- stuck_guard: add a signature-agnostic TRAILING WINDOW (last 6 tool outcomes).
When ≥5 share the SAME byte-identical failure fingerprint across DIFFERENT
commands, steer once ('you're spinning on one failing op — stop, it won't
resolve by retrying'). EXACT fingerprint (no digit-blur) so a healthy
iterate-and-fix loop (same cmd, a different test failing each run) never trips
it — the K10 case, still covered by its test. Also exposes recent_failure_
summary().
- pipeline/hooks: the between-turns hook latches that summary; the terminal path
appends it to a max_turns reason → 'error_max_turns … — spinning on failing
tool calls (last: git push → invalid credentials)'. Only enriches max_turns;
a productive run adds nothing.
- error-classifier: a new bucket for 'error_max_turns … spinning on failing tool
calls' → 'Ran out of turns while stuck on a failing step' with the honest
remedy (raising --max-turns won't help; fix the failing op / check the env),
ordered before the generic max_turns bucket. Tests: window-spin + K10 no-trip
+ summary + pipeline enrichment + classifier bucket.
…es (ABCA-662 review) The prior 'spinning on failing tool calls' classifier bucket asserted 'raising --max-turns won't help'. That over-claims: the trailing window can't distinguish (a) a HARD blocker no turn count fixes from (b) a LONG task that hit a transient/recoverable snag near the end and just ran out — which is 662 itself (siblings 661/663 pushed fine with the same token → its 'invalid credentials' was a transient race that more turns / a retry WOULD have cleared). Reframe the bucket to SURFACE what it was stuck on and present BOTH paths (environment blocker → fix + retry; transient/recoverable or a shorter sibling got past it → just retry / raise --max-turns), letting the failure KIND in the detail tell the reader which applies. Title 'Ran out of turns retrying a failing step'. The early window-steer (agent told to STOP and report while it still has turns) is the real mechanism that lets a long task escape the thrash; this copy is just the honest post-hoc explanation. Test updated to assert both remedies, not the over-claim.
…ted failure, don't diagnose it
Reviewer concern on the ABCA-662 work: re-titling a max_turns failure as "Ran out
of turns retrying a failing step" over-claims. The stuck-guard's trailing window is
only the last ~6 tool calls, which genuinely CANNOT distinguish a hard blocker (bad
creds, no permission — more turns won't help) from a long task that made real
progress and hit a recoverable snag only at the tail (more turns / a retry WOULD
help — 662 itself: siblings pushed fine with the same token). Framing the whole run
around its last 6 calls misrepresents the latter.
- error-classifier: drop the separate "retrying a failing step" bucket. A max_turns
failure stays the plain "Exceeded max turns"; the description/remedy point the
reader at the observed detail and still surface the environment-blocker path,
but assert NO cause.
- stuck_guard.recent_failure_summary: emit a NEUTRAL observation ("last tool calls
repeated: `<cmd>` → <err>") instead of "spinning on failing tool calls". The
mid-run steer TO the agent (an advisory nudge, not a user surface) still says
"spinning" — that's fine, it's coaching the agent, not classifying the outcome.
- tests updated to pin the neutral wording + assert the title is NOT re-framed.
scottschreckengaust
left a comment
There was a problem hiding this comment.
Review — Principal Architect (Stack B: agent error handling, PR 2/2 · stacked on #599)
Base verified against fresh origin PR head (diffed vs #599's branch pr/error-classifier-hardening). All mandatory agents run (table below).
Verdict: Approve (with nits)
Excellent stuck/runaway-guard work. The StuckGuard catches two distinct spin shapes — a per-signature identical-failure streak (ABCA-483) and the signature-agnostic trailing-window loop-of-variations (ABCA-662: the agent retries a failing git push every which way, each getting "invalid credentials", so no single command streak grows but the window is failure-dominated). It's advisory-only by deliberate design — a false nudge costs one comment; the max_turns cap is the real backstop — and the K10 reasoning (do NOT digit-blur fingerprints, so a healthy iterate-and-fix loop with a different failing test each run reads as progress and never trips) is a subtle, correct call. Cleanly stacked on #599's ErrorClass.
Verified myself: _LAST_STUCK_SUMMARY module-global is correctly reset per-task (reset_stuck_summary() at run_task top, before hooks run — mirrors the existing blocker-latch pattern), the deferred from hooks import … is intentional (avoids a circular import, matches the pre-existing convention), and the max_turns copy stays neutral (appends an observation, makes no causal claim about whether more turns would help) — a mature UX decision.
Non-blocking suggestions / nits
- N3 (crit-8 test seam) — The max_turns reason-enrichment chain is tested only at its two pure endpoints:
stuck_guard.recent_failure_summary()(unit) and the classifier consuming a hand-built string. The wiring between them is unverified — neither thepipeline.py:477append (if err and "error_max_turns" in err: … last_stuck_summary()) nor thehooks.py:1465latch-write is exercised end-to-end. Add aTestResolveOverallTaskStatuscase monkeypatchinghooks.last_stuck_summary(assert append on max_turns, no-append on a non-max_turns error, no double-append), and one driving_stuck_guard_between_turns_hook/stop_hookthen assertinglast_stuck_summary()— including the load-bearing "recovered → clears to None" behavior the comment claims. - N4 (crit-6 boundary) — The window-steer boundary is untested at exactly
WINDOW_FAIL_THRESHOLD(5/6): the tests straddle it at 6 (steers) and 4 (doesn't) but skip the==5>=edge where an off-by-one would hide. Also add "window not full" (5 identical failures in a length-5 window → no steer, since_dominant_window_failurerequires a full window). Two ~3-line tests;WINDOW_FAIL_THRESHOLDisn't even imported into the test file, so nothing currently pins the constant's boundary semantics. - N5 (cosmetic) —
record_tool_resultcalls_looks_failed(tool_response)twice (window bookkeeping + streak); results are identical, could hoist to one call. And the window-steer's double-nudge guard keys on_last_failing_sig, so a per-signature steer + a window steer can both fire for one broad spin — benign (advisory;_window_steeredstill caps window steers at one).
Tests & CI
- CI: all green (title, CodeQL ×3, dead-code advisory, secrets/deps/actions,
build (agentcore));MERGEABLE. - Coverage: stuck-guard core detection and hook integration are genuinely well-covered — the
test_post_tool_use_record_error_never_blocks_screening(a guard raise must not break the fail-open screening path),test_stop_hook_steers_not_bails(advisory-only contract),test_iterating_agent_same_command_DIFFERENT_failures_never_steers(K10 false-positive guard), and the neutral-wordingrecent_failure_summaryassertions are all high-value, behavior-not-implementation tests. Gaps are the integration seams → N3/N4. - Bootstrap synth-coverage: N/A (Python agent code + a small classifier copy tweak; no CDK construct/stack/resource change).
Review agents run (Stage 3 — mandatory)
| Agent | Ran? | Outcome |
|---|---|---|
code-reviewer |
✅ | No blocking defects; stuck-guard streak/window logic, per-task reset, and deferred imports all verified correct. |
silent-failure-hunter |
✅ | The three advisory swallow-paths are all appropriate fail-open: PostToolUse record_tool_result swallow can't mask screening (runs before/independent of the scanner); the between-turns swallow only disables an advisory nudge, never a safety behavior; _looks_failed unknown→success is the correct bias for an additive guard (a miss costs nothing — max_turns is the floor). No silent-failure defects. |
pr-test-analyzer |
✅ | Core detection well-covered; integration-seam gaps → N3, boundary → N4. |
type-design-analyzer |
⏭️ omitted | StuckAction/_SigState are simple dataclasses; no complex invariant to analyze. |
comment-analyzer |
⏭️ omitted | Comment accuracy spot-checked by hand + the other agents; the neutral-observation wording is correct and well-documented. |
/security-review |
⏭️ omitted | No IAM/Cedar/network/secrets change; the guard previews untrusted repo output but truncates (_CMD_PREVIEW_LEN) and never executes it. |
Human heuristics
- Proportionality — ✅ Deliberately minimal (advisory, no kill); matches the ABCA-483/662 incidents.
- Coherence — ✅ Mirrors existing conventions (dedup set, per-task reset, between-turns hook ordering with the cancel<stuck<nudge invariant).
- Clarity — ✅ Exemplary "why" comments, especially the K10 don't-blur-digits rationale and the neutral-observation design.
- Appropriateness — ✅ The "surface what was observed, make no causal claim" max_turns copy is a notably mature UX call.
Governance
Branch pr/agent-stuck-guard carries no issue number (ADR-003) — same item as #599; attach an approved issue + conforming branch before merge. Code is approve-ready.
Bottom line: approve-ready. N3/N4 are worthwhile follow-up tests (the integration seams and the exact window boundary), not blockers. The advisory-only design and K10 false-positive avoidance are high quality. Merge after #599 lands (stacked).
🤖 Generated with Claude Code
…ary (#600 N3/N4) Follow-up coverage from the #600 review (feature approved as-is; tests only): - N3 (crit-8 wiring seam): the max_turns reason-enrichment was tested only at its two pure endpoints (stuck_guard.recent_failure_summary + the classifier on a hand-built string) — the append in _resolve_overall_task_status that joins them was unverified. Add TestMaxTurnsStuckEnrichment (monkeypatching hooks.last_stuck_summary): appends on error_max_turns, does NOT append on a non-max_turns error, leaves the reason unchanged when there's no summary, and does not double-append when the summary is already present. - N4 (crit-6 boundary): the window-steer was tested at 6/6 (steers) and 4/6 (doesn't) but not at the exact WINDOW_FAIL_THRESHOLD `>=` edge. Add test_window_steers_at_exactly_the_threshold (5/6 same-fingerprint fails in a full window → steers) and test_no_steer_when_window_not_yet_full (5 fails in a length-5 history → no steer, since _dominant_window_failure requires a full window). Imports WINDOW/WINDOW_FAIL_THRESHOLD so the constants' boundary semantics are now pinned. Full agent suite green (1218); ruff format + check clean.
|
Follow-up on the N3/N4 nits — since this merged into the Stack B base branch (
All green (50 tests across the two files, ruff clean). |
What
Backports the agent-side stuck-guard subsystem from
linear-vercel, plus a modest max-turns diagnostic.feat(agent): stuck/runaway guard — newStuckGuard(agent/src/stuck_guard.py) that records tool results via the PostToolUse hook and detects a repeating-failing-command loop (the ABCA-483 spin).fix: stuck-guard advisory-only — drop the hard bail; keep a one-time steer (preserves the K10 "no false kill" stance).fix(agent): catch loop-of-variations early (ABCA-662) — a signature-agnostic trailing window (last 6 tool outcomes): when ≥5 share a byte-identical failure fingerprint across different commands, steer the agent once, mid-run.max_turnscap coincides with a failure-dominated trailing window, the pipeline appends a neutral observation to the reason (last tool calls repeated: \git push` → invalid credentials`). The failure stays classified as the plain "Exceeded max turns": it surfaces what was observed and lets the reader decide — it does not re-title the run or claim more turns wouldn't help (the last 6 calls can't distinguish a hard blocker from a long task that hit a recoverable snag at the tail). The mid-run steer to the agent still coaches assertively — that's advice to the agent, not a classification of the outcome.Conflict resolution (keep-both / partial)
hooks.py— 4 add-vs-add interleaves withmain's feat(agent): observable blocker signal + bounded self-remediation for environmental faults #251 egress-denial detection: kept both the#251progress/egress path and the newstuck_guardrecording inpost_tool_use_hook.pipeline.py— the incoming hunk carried context from a not-included commit (8035b71, ENOSPC/OOMbuild_infra_failed, which depends on the unmerged feat(linear): parent/sub-issue orchestration with dependency-aware child tasks #247 build subsystem). Dropped that context; kept only the ABCA-662 stuck-summary enrichment. Verifiedbuild_infra_failedis referenced nowhere.Test
tyclean,ruff check+ruff format --checkclean, 1212 tests pass (coverage 80.83%;stuck_guard.py89%).--fixno mutations.