solana/rpc: wait as long as a rate-limited endpoint asks

[bgm-malbeclabs]

Stacked on #4161, which adds the transport wrapper this needs. Review that one first; the base here is solana/observe-ratelimit-headers.

Summary

• A retrying ledger RPC client now waits as long as a rate-limited endpoint asks it to, instead of retrying inside the window that just refused it.
• The endpoint's Retry-After is carried from the transport to the retry loop, which cannot otherwise see it: solana-go discards the http.Response and its error types keep only a code.
• One call holds for at most 15s of endpoint-requested waiting. A call asked for longer stops and returns the rate limit rather than waiting a shorter time.
• New counter doublezero_solana_rpc_retry_after_exceeded_total says when that allowance is sized wrong.
• Endpoints that send no Retry-After keep the existing jittered backoff untouched.

Why

The retry budget is shorter than the window that refuses it, so it cannot rescue a rate-limited call. Measured against the package defaults:

before attempt 2: nominal 500ms  jittered [250ms, 500ms]
before attempt 3: nominal 1s     jittered [500ms, 1s]
before attempt 4: nominal 1.5s   jittered [750ms, 1.5s]

total backoff across 4 attempts: [1.5s, 3s]

The provider fronting mainnet-beta enforces its limits over a rolling 10s window. Even the unluckiest 3s total lands inside the window the first attempt was refused in. Four requests are spent to be told the same thing, aimed at an endpoint that is already shedding load. A lake indexer activity sat in this state for 4.5 hours.

Guessing a longer number is worse in both directions: a fixed 10s wait stalls a call the endpoint would have served again in 1s, and it is still too short for a provider whose window is longer. Only the endpoint knows its window, and it says so in Retry-After — which the provider confirms it sets on every refusal.

How

solana-go discards the http.Response inside CallForInto, and *HTTPError is {Code int, err error} while *RPCError is {Code, Message, Data}. The header is gone before any error reaches the retry loop. So the transport writes it into an unexported per-call slot on the context, and the retry loop takes it before waiting.

The header is recorded on every response, not only on a 429. The sustained refusals that motivated this arrived as HTTP 200 with the rate limit inside the JSON-RPC envelope, a shape no status-code check matches. It costs nothing, because the retry loop reads the value only after an attempt has failed retryably.

Waits are spread upward only. Every client refused in the same window otherwise resumes together and re-spikes the endpoint as the window rolls — the same reason the existing backoff is jittered. Arriving early is arriving refused, so the spread never subtracts.

The bound

The allowance is 15s per call, summed across attempts. Two things fix that number:

• It has to clear the 10s window it is meant to outlast, or honoring the header changes nothing.
• It has to keep the worst case inside what callers were sized for. defaultRequestTimeout documents the constraint as state-ingest's 60s tick, because state-ingest passes its root context straight to GetProgramData with no call-site bound. 4 attempts of 10s plus 15s is 55s, still inside it.

A call asked for longer than the allowance stops rather than waiting a shorter time: a partial wait spends the wait and is refused anyway, since the window has not rolled. It returns the rate-limit error itself, so the caller sees the cause instead of a deadline, and the caller's own next poll arrives after the window regardless.

That comment and the test asserting it both hardcoded the old 3s backoff total against the 60s tick. Both now carry the allowance instead, because it is the term that bounds the wait.

Testing Verification

• Both real refusal shapes, end to end through rpc.New: HTTP 429 status, and HTTP 200 with {"error":{"code":429}} in the envelope. Both wait the header and then succeed.
• The end-to-end tests go through the real constructor rather than the retry helper on purpose: the value has to cross a transport boundary the helper cannot see, so a unit test of the helper passes with the wiring absent. Removing the single wiring line drops the observed wait from 1.1s to 2.8ms and both shapes fail.
• Retry-After: 600 ends the call after one attempt, returns an error naming 429, and does not hold the caller while deciding.
• Same server and same refusal with the allowance off retries in under a second, which is what pins the timing to the header rather than to anything else in the stack.
• An endpoint sending no header still retries and still succeeds, on the unchanged backoff.
• ParseRetryAfter over delta-seconds, an HTTP-date ahead, an HTTP-date already past, zero, negative, garbage, and absent.
• NoteRetryAfter on a context with no slot is a no-op, since the transport cannot know which client it serves.
• take clears the slot, so a header from one attempt cannot pace the next.
• GOOS=linux go build ./tools/... ./controlplane/... clean; go test -race ./tools/solana/... green.