Skip to content

fix(acp): dead-letter usage-limit refusals instead of retrying - #7787

Open
jramiresbrito wants to merge 1 commit into
block:mainfrom
jramiresbrito:fix/acp-usage-limit-no-retry-storm
Open

jramiresbrito wants to merge 1 commit into
block:mainfrom
jramiresbrito:fix/acp-usage-limit-no-retry-storm

Conversation

@jramiresbrito

Copy link
Copy Markdown

Problem

A provider usage or rate limit is wall-clock bound: it resets on the provider's schedule, never inside the retry ladder's ~21-minute budget. Today every such refusal falls through to queue.requeue(), so the batch spends all 10 attempts on guaranteed failures and is dead-lettered anyway — 21 minutes later, with the same outcome and no reset time surfaced to the user.

Observed in production against claude-agent-acp 0.78.0:

WARN  agent_returned (application error — pipe intact)
      error=Agent reported error (code -32603): Internal error:
      You've hit your session limit · resets 4:50pm (America/Sao_Paulo)
WARN  requeueing failed batch with backoff attempt=10 max=10
ERROR dead-lettering batch after 10 retries — discarding 1 events

This is the self-amplifying retry loop AGENTS.md rule 4 warns about — "a persistent failure must not self-amplify into an unbounded refresh loop" — and the batch ends in the same dead-letter either way.

Change

Classify usage-limit refusals like the existing non-retryable auth and model not found cases:

  • is_usage_limit_error sits beside is_auth_error and matches the harness shapes seen in the wild: session limit, usage limit, 429, rate_limit_error.
  • The dispatch branch in handle_prompt_result dead-letters immediately and echoes the provider's own message, so the failure notice carries the reset time the user needs in order to know when to re-send.

No behaviour change for any other error class — transport errors still respawn, and everything unclassified still uses the full retry ladder.

Tests

Bound to the production seam, not to the classifier alone (AGENTS.md rule 3):

  • usage_limit_error_dead_letters_batch_without_consuming_retries drives handle_prompt_result and asserts queued_event_count == 0.
  • Falsifiability checked: disabling the new branch makes it fail with left: 1, right: 0.
  • Three classifier tests cover the verbatim production message, the usage limit / 429 / rate_limit_error variants, and rejection of unrelated and transport errors.
cargo test -p buzz-acp --lib
test result: ok. 945 passed; 0 failed; 1 ignored

cargo fmt and cargo clippy -p buzz-acp --all-targets are clean.

A provider usage or rate limit is wall-clock bound: it resets on the
provider's schedule, never inside the retry ladder's ~21-minute budget.
Today every such refusal falls through to `queue.requeue()`, so the batch
spends all 10 attempts on guaranteed failures and is dead-lettered
anyway — 21 minutes later, with the same outcome and no reset time shown
to the user. Observed in production against claude-agent-acp 0.78.0:

  WARN agent_returned (application error — pipe intact)
       error=Agent reported error (code -32603): Internal error:
       You've hit your session limit · resets 4:50pm (America/Sao_Paulo)
  WARN requeueing failed batch with backoff attempt=10 max=10
  ERROR dead-lettering batch after 10 retries — discarding 1 events

This is the self-amplifying retry loop AGENTS.md rule 4 warns about: a
persistent failure must have a terminal state rather than consume its
whole budget.

Classify it like the existing non-retryable auth and model-not-found
cases. `is_usage_limit_error` sits beside `is_auth_error` and matches the
harness shapes seen in the wild (session limit, usage limit, 429,
rate_limit_error). The dispatch branch dead-letters immediately and
echoes the provider's own message, so the failure notice carries the
reset time the user needs to know when to re-send.

Tests bind the production seam, not the classifier alone:
`usage_limit_error_dead_letters_batch_without_consuming_retries` drives
`handle_prompt_result` and asserts the queue stays empty. Removing the
new branch makes it fail (queued_event_count 1, expected 0).

Signed-off-by: Joao Victor Ramires Guimaraes Brito <joaoramires.engsoft@gmail.com>
@jramiresbrito
jramiresbrito requested a review from a team as a code owner September 21, 2026 20:25
@github-actions

Copy link
Copy Markdown

🔐 Codex Security Review

Status: review required for the current range.

The current range is 77729abfb692b25a0f4ec4a69add86af2e32c0dd...26537ac72a31c7218b52fc26cfc56c0e96b710c5.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review 26537ac72a31c7218b52fc26cfc56c0e96b710c5 to authorize a new review.
Any previous review applies only to its recorded range.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant