fix(acp): dead-letter usage-limit refusals instead of retrying - #7787
Open
jramiresbrito wants to merge 1 commit into
Open
jramiresbrito wants to merge 1 commit into
jramiresbrito wants to merge 1 commit into
Conversation
A provider usage or rate limit is wall-clock bound: it resets on the
provider's schedule, never inside the retry ladder's ~21-minute budget.
Today every such refusal falls through to `queue.requeue()`, so the batch
spends all 10 attempts on guaranteed failures and is dead-lettered
anyway — 21 minutes later, with the same outcome and no reset time shown
to the user. Observed in production against claude-agent-acp 0.78.0:
WARN agent_returned (application error — pipe intact)
error=Agent reported error (code -32603): Internal error:
You've hit your session limit · resets 4:50pm (America/Sao_Paulo)
WARN requeueing failed batch with backoff attempt=10 max=10
ERROR dead-lettering batch after 10 retries — discarding 1 events
This is the self-amplifying retry loop AGENTS.md rule 4 warns about: a
persistent failure must have a terminal state rather than consume its
whole budget.
Classify it like the existing non-retryable auth and model-not-found
cases. `is_usage_limit_error` sits beside `is_auth_error` and matches the
harness shapes seen in the wild (session limit, usage limit, 429,
rate_limit_error). The dispatch branch dead-letters immediately and
echoes the provider's own message, so the failure notice carries the
reset time the user needs to know when to re-send.
Tests bind the production seam, not the classifier alone:
`usage_limit_error_dead_letters_batch_without_consuming_retries` drives
`handle_prompt_result` and asserts the queue stays empty. Removing the
new branch makes it fail (queued_event_count 1, expected 0).
Signed-off-by: Joao Victor Ramires Guimaraes Brito <joaoramires.engsoft@gmail.com>
🔐 Codex Security Review
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A provider usage or rate limit is wall-clock bound: it resets on the provider's schedule, never inside the retry ladder's ~21-minute budget. Today every such refusal falls through to
queue.requeue(), so the batch spends all 10 attempts on guaranteed failures and is dead-lettered anyway — 21 minutes later, with the same outcome and no reset time surfaced to the user.Observed in production against
claude-agent-acp0.78.0:This is the self-amplifying retry loop AGENTS.md rule 4 warns about — "a persistent failure must not self-amplify into an unbounded refresh loop" — and the batch ends in the same dead-letter either way.
Change
Classify usage-limit refusals like the existing non-retryable auth and
model not foundcases:is_usage_limit_errorsits besideis_auth_errorand matches the harness shapes seen in the wild:session limit,usage limit,429,rate_limit_error.handle_prompt_resultdead-letters immediately and echoes the provider's own message, so the failure notice carries the reset time the user needs in order to know when to re-send.No behaviour change for any other error class — transport errors still respawn, and everything unclassified still uses the full retry ladder.
Tests
Bound to the production seam, not to the classifier alone (AGENTS.md rule 3):
usage_limit_error_dead_letters_batch_without_consuming_retriesdriveshandle_prompt_resultand assertsqueued_event_count == 0.left: 1, right: 0.usage limit/429/rate_limit_errorvariants, and rejection of unrelated and transport errors.cargo fmtandcargo clippy -p buzz-acp --all-targetsare clean.