rpc: retry -32005 rate-limit errors instead of crashing the ingester - #83
Open
SQD-Trevor-Agent wants to merge 1 commit into
Open
rpc: retry -32005 rate-limit errors instead of crashing the ingester#83SQD-Trevor-Agent wants to merge 1 commit into
SQD-Trevor-Agent wants to merge 1 commit into
Conversation
The RPC connection classified JSON-RPC error -32005 ("rate-limit
exceeded") as non-retryable, so a transient rate limit from the
upstream endpoint propagated as a fatal RpcError and crashed the
ingest process into a restart loop. Add -32005 to the retryable
codes so the connection backs off and retries, mirroring the other
transient server codes already handled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
taiko-mainnet(and any network on a rate-limited endpoint) crash-loops its stage-2rpc-ingestprocess. The ingester resumes from the last written chunk, fetches ~2k blocks, then dies with:(top of stack:
_fetch_receipts→RpcClient.batch_call→RpcConnection._unpack_result)Root cause
RpcConnection._is_retryable_error(sqa/util/rpc/connection.py) lists the JSON-RPC error codes it treats as transient/retryable:-32005("rate-limit exceeded" / limit-exceeded, as returned by the public Taiko RPC and by most providers) is not in the set. A rate limit is inherently transient, but because it is unrecognised theRpcErrorpropagates out of_handle_requestinstead of triggering the connection's back-off/retry, and the ingest program crashes. Kubernetes restarts it, it hits the same rate limit again → restart loop. HTTP-level 429s are already retried (isinstance(e, httpx.HTTPStatusError)branch); this only fixes the case where the provider returns the rate limit as a JSON-RPC error body with HTTP 200.Fix
Add
-32005to the retryable code set so a rate-limit response backs off (_backoff, 10ms→20s schedule) and retries instead of crashing — the same handling already applied to-32000/-32002/-32007/429.Test
tests/test_rpc_connection.pyasserts-32005is retryable, the existing transient codes stay retryable, and an unknown code (-32099) stays non-retryable. Fails on the pre-fix code, passes after:Falsification
If, after this change, the ingester still crash-loops on Taiko with a different error code (not -32005), or the crash is a genuine non-transient failure rather than rate limiting, this fix is insufficient. Separately, retry only makes the process survive the rate limit — if the public endpoint's sustained throughput is below what ingest needs, throughput/lag is a provider-capacity matter (a higher-limit/dedicated Taiko endpoint) and is being handed to infra on-call; that is orthogonal to the crash-loop this PR removes.