Skip to content

Keep Account Pooler accounts usable when upstream rejects fresh tokens - #4385

Merged
SawyerHood merged 2 commits into
mainfrom
fix/account-pool-upstream-credential-rejection
Sep 26, 2026
Merged

SawyerHood merged 2 commits into
mainfrom
fix/account-pool-upstream-credential-rejection

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

During the 2026-09-25 Codex outage, chatgpt.com/backend-api/codex answered every request with HTTP 401 invalid_api_key ("Incorrect API key provided: sk-svcac…"), a key the pooler never sends. The hub refreshed the OAuth token, the refresh succeeded, the fresh token got the same 401, and markAuthError stored a permanent account error. Stored errors are never cleared automatically: selection skips errored accounts, freshSecret rethrows the stored error before refreshing, and bb pool account refresh <id> goes through the same path. Both Codex accounts stayed errored until they were removed and logged in again. A 403 marked the account without even trying a refresh.

What changed

  • hub.ts: 401 and 403 both trigger one forced refresh. If the upstream rejects the newly issued token, the credential is proven valid, so the hub returns 503 (with the upstream detail) and records a token-keyed upstream backoff that holds the account out for one minute instead of marking an error. Rejection checks still guard against a late response poisoning a newer token. API-key accounts keep the old behavior.
  • provider-adapter.ts: only HTTP 400/401 from the OAuth token endpoint are permanent refresh failures; other statuses and unreadable 200 bodies (parseOAuthRefreshResponse, used by both adapters) are transient.
  • bb pool account refresh <id> / account.refreshUsage: for an errored OAuth account, forces a refresh past the stored error and clears it on success; a permanent failure updates the error message, and the CLI reports the account still in error.
  • Docs: skill reference and CLI summary (PLUGIN_OVERVIEW.md is at the 4,000-character bundled-plugin cap).
  • Deviation from the investigation's proposal: no automatic background retry of errored accounts. After this change an error means the OAuth server rejected the refresh token, so periodic retries cannot succeed, and replaying a rotated Codex refresh token can trip reuse detection.

How you verified

  • New tests: Claude and Codex × 401/403 replay the outage's invalid_api_key body (503, no account error, no upstream or refresh traffic during the hold, recovery on the same token afterwards); manual refresh clears a stored invalid_grant error; refresh 403/404 and unreadable bodies are transient. All 10 fail against the previous source and pass now.
  • Updated three existing expectations from 401/429 to 503 for fresh-token rejection.
  • pnpm exec turbo run typecheck test --filter=bb-plugin-account-pool: 323 passed.

AGENT GENERATED

🤖 Generated with Claude Code

SawyerHood and others added 2 commits September 26, 2026 02:55
A 401 on a freshly refreshed OAuth token (the ChatGPT outage's
invalid_api_key responses) was stored as a permanent account error that
nothing cleared. The hub now refreshes once on 401 or 403, and when the
new token is still rejected it returns 503 and holds that token out of
routing for one minute instead of marking the account. Only 400/401
refresh responses are permanent credential failures; unreadable refresh
bodies and other statuses are transient. `bb pool account refresh <id>`
now forces a refresh for an errored OAuth account and clears the error
on success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SawyerHood
SawyerHood merged commit 2726ab1 into main Sep 26, 2026
18 checks passed
@SawyerHood
SawyerHood deleted the fix/account-pool-upstream-credential-rejection branch September 26, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant