Document RPC rate-limit handling design (Solana retry + Ethereum normalize)

[fengtality]

Purpose

Track the current RPC rate-limit handling design for Gateway (Solana + Ethereum) as introduced/refined in PR #656, so the behavior is documented somewhere durable without committing a standalone doc file into the repo.

This is a tracking / documentation issue. The longer-term cleanup is captured separately in #657 (viem migration), which would replace most of the Ethereum-side custom code below — see it as the follow-up to this design.

---

RPC Rate‑Limit Handling

How Gateway reacts when an RPC endpoint answers a request with HTTP 429 Too Many Requests.
Covers both Solana (@solana/web3.js Connection) and Ethereum (ethers Provider).

Source: src/rpc/rpc-connection-interceptor.ts, wired up in src/chains/solana/solana.ts and src/chains/ethereum/ethereum.ts.

TL;DR

• Every Connection / Provider is wrapped in a Proxy that intercepts every method call.
• Solana reads (get… methods) are retried up to 3 times, 5 s apart on a 429, because web3.js's own
  retry budget is short. After retries are exhausted (or on a write) it throws a clean statusCode: 429.
• Ethereum does NOT retry in the Proxy — ethers already retries 429s internally (up to throttleLimit,
  default 12, with Retry-After support). The Proxy only normalizes the 429 into a clean
  statusCode: 429 error so routes surface it instead of masking it as a 0 balance or a 500.
• Writes (send…) are never retried on either chain — they fail fast.

The asymmetry is deliberate: Solana's client retries too little, Ethereum's retries plenty. We add a retry only
where it's missing, and otherwise just translate the error.

Why the two chains differ

	Solana — web3.js 1.98.2	Ethereum — ethers 5.8.0
Client's built‑in 429 retry	up to 5 attempts, exp backoff 500ms→1s→2s→4s (~7.5 s), no Retry-After	up to throttleLimit = 12 attempts, exp backoff, honors Retry-After
Toggle	disableRetryOnRateLimit	throttleLimit
Gateway Proxy adds	retry ×3 (5 s each) on reads + normalize	normalize only (no retry)
Rationale	client gives up quickly under sustained load; extra retries help bridge bursts	client already retries heavily; a second layer would only compound blocking into minutes

Verified against the installed sources: web3.js node_modules/@solana/web3.js/lib/index.cjs.js:5054,
ethers @ethersproject/web/lib/index.js:85.

Layer 2 in detail (the Proxy interceptor)

createRateLimitAwareSolanaConnection(connection, rpcUrl) and
createRateLimitAwareEthereumProvider(provider, rpcUrl) each return a Proxy whose get trap wraps every function
property. They share the is429Error detector and the clean‑error factory, but differ in retry behavior.

Solana wrapper — retry on reads

const maxAttempts = isRetryableReadMethod(prop) ? READ_RETRY_DELAYS_MS.length + 1 : 1; // 4 or 1
for (let attempt = 0; attempt < maxAttempts; attempt++) {
  try {
    return await realMethod.apply(target, args);   // bound to the RAW object, not the proxy
  } catch (error) {
    if (!is429Error(error)) throw error;            // non‑429 → propagate untouched
    if (attempt < maxAttempts - 1) {                // retries remain → log + sleep + continue
      await sleep(READ_RETRY_DELAYS_MS[attempt]);   // 5000, 5000, 5000
      continue;
    }
    throw createRateLimitError(rpcUrl, 'solana');   // budget exhausted → clean 429
  }
}

• Read = method name starts with get… (or, on Ethereum, equals call) — see isRetryableReadMethod.
• Reads get maxAttempts = 4 (1 initial + 3 retries); everything else gets 1 (fail fast).

Ethereum wrapper — normalize only, no retry

try {
  return await realMethod.apply(target, args);
} catch (error) {
  if (!is429Error(error)) throw error;              // non‑429 → propagate untouched
  throw createRateLimitError(rpcUrl, 'ethereum');   // 429 → clean error, NO sleep, NO retry
}

Shared behavior

• Non‑429 errors are never retried or wrapped — they propagate exactly as thrown.
• The thrown rate‑limit error carries statusCode: 429, name: 'TooManyRequestsError', and a human message telling
  the operator to configure an API‑keyed RPC provider. Fastify turns statusCode: 429 into an HTTP 429 response.
• realMethod.apply(target, …) binds to the raw object, so a wrapped method's internal this.foo() calls go to
  the unwrapped object — this prevents nested/compounding retries, but also means only the method you call on the
  proxy is intercepted (see Gotcha Feat/certs #1).

is429Error detection

A 429 surfaces in many shapes across providers, so detection is intentionally broad:

error.message includes '429' | 'too many requests' | 'rate limit'
  || error.code === 429 || error.status === 429 || error.response.status === 429
  || JSON.stringify(error) includes '"code":429' / '"status":429'

Provider	Typical 429 shape	Matched by
ethers v5	SERVER_ERROR with error.status === 429	error.status === 429
web3.js	Error: 429 Too Many Requests …	message.includes('429' / 'too many requests')

Scenarios by RPC call type

R = retried in the Proxy (Solana reads only). N = normalized to clean 429, not retried in the Proxy.
W = write, fail fast.

Solana

Call	Class	On sustained 429
getBalance / getAccountInfo / getMultipleAccountsInfo	R	web3.js retries ~5×, then Proxy retries 3× more (5 s each), then throws 429 — the core of the Orca/CLMM fix
getProgramAccounts / getTokenAccountsByOwner / getLatestBlockhash / getSignatureStatuses	R	same as above
simulateTransaction	N	name isn't get… → not retried; normalized to 429
sendRawTransaction / sendTransaction	W	throws 429 immediately (no double‑broadcast risk)
confirmTransaction	W	not retried itself; its internal polling bypasses the proxy too (Gotcha #1)

Ethereum

Call	Class	On sustained 429
getBalance (native)	N	ethers retries ≤12× internally, then Proxy normalizes to a clean 429
contract.balanceOf → provider.call (token balance)	N	ethers retries internally; normalized 429 re‑thrown by getBalances
contract.allowance → provider.call	N	normalized; propagated by the allowances route
getTransactionReceipt (receipt polling)	N	normalized; handleTransactionExecution re‑throws the 429
estimateGas	N	normalized; propagated by the estimate‑gas route
sendTransaction	W	normalized to 429 immediately

Worked examples

A. Transient 429 that clears (Solana read)

t=0.0s  getAccountInfo()  → 429  (after web3.js' own internal retries)   attempt 0   warn "Retrying in 5000ms (1/3)"
t=5.0s  retry             → 429                                          attempt 1   warn "Retrying in 5000ms (2/3)"
t=10.0s retry             → 200 OK                                       attempt 2   ✅ returns the data

Caller never sees an error; added latency ≈ 10 s on top of web3.js' internal backoff.

B. Sustained 429 (Ethereum native balance)

ethers retries internally up to throttleLimit (12) with Retry-After → still 429
→ Proxy does NOT retry → throws TooManyRequestsError(statusCode 429)
→ getBalances() sees statusCode === 429 and re‑throws → HTTP 429 to the client

The balance is not silently reported as 0. (Before this PR, the route caught the error and returned 0/500.)

C. Write under load (Solana send)

t=0.0s  sendRawTransaction() → 429  (maxAttempts = 1) → throw TooManyRequestsError immediately

D. Non‑429 error (any call)

getAccountInfo() → "fetch failed: ECONNRESET" → is429Error == false → re‑thrown unchanged, no retry

Gotchas & caveats

Gotcha #1 — interception is single‑layer

realMethod.apply(target, …) binds to the raw object. Only the method you call on the proxy is intercepted. If a
high‑level method fans out to several RPC calls internally (e.g. Solana confirmTransaction polling
getSignatureStatuses), those inner calls bypass the proxy and are neither retried nor normalized. Coverage relies
on connectors calling connection.getXxx / provider.call directly on the wrapped object — which they do.

Gotcha #2 — classification is by name, not semantics

simulateTransaction (Solana) and estimateGas (Ethereum) are read‑only but aren't classified as retryable reads
because their names don't start with get (and aren't call). On Solana they're normalized but not retried; on
Ethereum everything is normalize‑only anyway. The rule is purely lexical, so a future get…‑named mutating method
would be wrongly classified as a read — unlikely, but worth knowing.

Gotcha #3 — Solana retry is fixed delay, no backoff or jitter

Solana retries are a flat 5s, 5s, 5s. Under a sustained limit, parallel reads wake together and re‑hit the
endpoint at once (mild thundering herd). Fine for momentary bursts; the durable fix for sustained limits is an
API‑keyed provider (Helius), which the error message points the operator to.

Gotcha #4 — added latency (Solana)

A single rate‑limited Solana read can block up to ~15 s in the Proxy (on top of web3.js' own backoff) before failing.
Parallel batches (Promise.all) sleep concurrently; a sequential loop adds ~15 s per rate‑limited item until the
first exhausts and throws.

Where it's wired up

• Solana: src/chains/solana/solana.ts wraps every Connection it builds (retry + normalize).
• Ethereum: src/chains/ethereum/ethereum.ts, plus src/rpc/infura-service.ts and
  src/rpc/chainstack-service.ts wrap every Provider (normalize only).
• Ethereum routes re‑throw statusCode === 429 instead of converting to 0/500:
  routes/balances.ts, routes/allowances.ts, routes/estimate-gas.ts, and Ethereum.getBalances.

Future direction

A single retry policy would be simpler than today's per‑chain split. The cleanest long‑term shape is multiple
endpoints with automatic failover rather than retrying one rate‑limited endpoint — see the viem migration issue
(#657), which would replace most of this custom code with viem's fallback transport (built‑in 429 retry +
Retry-After + latency/stability ranking across providers).