feat(http): surface Retry-After, and retry a rate limit when asked

README.md

@@ -483,11 +483,50 @@ Rail0::Client.new(
   timeout:     30,                                # seconds (default 30)
   max_retries: 0,                                 # network-error retries (default 0)
   retry_delay: 0.2,                               # base delay, doubles each attempt
+  retry_on_429: false,                            # retry a rate limit (default false)
+  retry_after_cap: 60,                            # longest Retry-After to honour, seconds
   logger:      Rail0::DEFAULT_LOGGER               # optional
 )
 ```
 
-Only network errors and timeouts are retried; HTTP error responses are not.
+### Rate limits
+
+The gateway throttles two surfaces independently: the public, unauthenticated one **per
+IP** (100 requests / 60s by default — SIWE nonce + verify, `/payment_methods`, the
+catalog reads, `/health`) and everything authenticated **per session**, keyed on the
+JWT's subject (300 / 60s). Over budget it answers **429** with `code: "rate_limited"` and
+a `Retry-After`.
+
+`Rail0::ApiError#retry_after` carries that header as whole seconds — nil on every other
+error, and nil when the header is absent or unusable. Read it rather than guessing:
+
+```ruby
+begin
+  client.payments.list
+rescue Rail0::ApiError => e
+  raise unless e.error == "rate_limited"
+  sleep(e.retry_after || 5)   # the gateway's own pacing
+end
+```
+
+Note what the number means: the gateway sends **the whole throttle period**, not the time
+left in the current window, so it is an upper bound on the wait rather than a measurement.
+
+`retry_on_429: true` makes the SDK do that waiting for you — Retry-After, clamped to
+`retry_after_cap`, plus a little jitter (see `Rail0::Backoff`; callers sharing one session
+are told the same number and would otherwise wake in lockstep). The jitter never shortens
+a wait below what it is for: additive on the server's own number, and equal jitter — half
+fixed, half random — on a guessed one. It is **off by default**
+on purpose: an automatic sleep hides back-pressure from the process that could react to
+it, and in a request/response app it turns a rate limit into a stalled page. Turn it on in
+a job — and note it sleeps the **calling thread**. It also works on its own: you do not
+need to set `max_retries` as well (that pairing would make the flag a silent no-op).
+
+Only network errors, timeouts and — when opted in — a 429 are retried; no other HTTP
+error is. The 429 is safe to retry on **any** method, `POST` included, because the
+gateway rejects it in middleware before the request reaches the application: nothing ran,
+so nothing can run twice. That is not true of a 502 or a timeout on a capture, where the
+broadcast may already be in flight.
 
 ## Project structure
 

lib/rail0.rb

@@ -11,6 +11,7 @@
 require_relative "rail0/version"
 require_relative "rail0/error_hints"
 require_relative "rail0/api_error"
+require_relative "rail0/backoff"
 require_relative "rail0/default_logger"
 require_relative "rail0/request"
 require_relative "rail0/http_client"

lib/rail0/api_error.rb

@@ -15,18 +15,29 @@ class ApiError < StandardError
     # @!attribute [r] detail
     #   @return [String, nil] One or two sentences fit to show a user verbatim. Also this
     #     exception's message.
-    attr_reader :status, :error, :title, :detail
+    # @!attribute [r] retry_after
+    #   @return [Integer, nil] Seconds the gateway asked the caller to wait, from the
+    #     `Retry-After` header — present on a 429 (`error == "rate_limited"`) and nil
+    #     otherwise. Surfaced because the alternative is a caller guessing: the SDK used
+    #     to drop the header, so "rate limited" arrived with no idea of for how long.
+    #
+    #     Note it is the WHOLE window the gateway throttles over, not the time left in it
+    #     — the limiter sends its period verbatim — so it is an upper bound on the wait,
+    #     not a measurement. Rail0::Backoff clamps it for that reason.
+    attr_reader :status, :error, :title, :detail, :retry_after
 
     # @param status [Integer]
     # @param error [String]
     # @param message [String] The detail; kept positional for compatibility.
     # @param title [String, nil]
-    def initialize(status, error, message, title: nil)
+    # @param retry_after [Integer, nil]
+    def initialize(status, error, message, title: nil, retry_after: nil)
       super(message)
       @status = status
       @error = error
       @title = title
       @detail = message
+      @retry_after = retry_after
       freeze
     end
 

lib/rail0/backoff.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module Rail0
+  # How long to wait before retrying a request the gateway rate-limited.
+  #
+  # Pure, and its own module, because the two interesting decisions here are easy to get
+  # backwards and impossible to notice once they are wrong — a client that waits too
+  # little walks straight back into the limiter, and one that waits too long looks hung.
+  #
+  # 1. JITTER NEVER SHORTENS THE WAIT BELOW WHAT IT IS FOR.
+  #    On a server-instructed wait it is ADDITIVE: scaling a Retry-After DOWN means retrying
+  #    before the window the server named has passed, which is a second 429 by construction.
+  #    So the instruction is honoured in full and a small random tail is added.
+  #
+  #    On a guessed wait it is EQUAL jitter — half the delay fixed, half random — not the
+  #    textbook "full jitter" that multiplies the whole delay by rand(). Full jitter can
+  #    land arbitrarily close to zero, which makes a real pause indistinguishable from the
+  #    bug where a Retry-After of "0" is honoured as a duration and the retry fires
+  #    immediately. A floor spreads the herd just as well and leaves "did we actually wait"
+  #    observable.
+  #
+  #    Why any jitter at all when the server told us the time: because callers align on
+  #    it. rail0-admin proxies every merchant over ONE session, so they share the
+  #    per-session bucket and would all be told the same Retry-After, wake together, and
+  #    recreate the burst the limiter just rejected.
+  #
+  # 2. THE CAP IS NOT PARANOIA. The gateway sends the WHOLE period as Retry-After
+  #    (rack_attack.rb: `headers["retry-after"] = match_data[:period].to_s`), not the time
+  #    remaining in the window — so hitting the limit one second in is told to wait the
+  #    full 60. Capping bounds both that over-wait and a hostile or misconfigured value
+  #    from anything between the client and the gateway.
+  module Backoff
+    module_function
+
+    # @param retry_after [Integer, Float, nil] the server's Retry-After, in seconds.
+    #   Absent, unparseable, zero or negative all mean "no instruction" — and zero is the
+    #   trap: it is a valid duration, so treating it as one produces a burst of
+    #   back-to-back requests against the very limiter that asked for a pause.
+    # @param attempt [Integer] 1 for the first retry, 2 for the second, …
+    # @param base [Float] the exponential backoff's first delay, in seconds.
+    # @param cap [Float] the longest wait to allow, in seconds.
+    # @param jitter [Float, nil] randomness in [0,1); injected only by tests. Nil draws it.
+    # @return [Float] seconds to sleep.
+    def throttle_delay(retry_after:, attempt:, base:, cap:, jitter: nil)
+      random = jitter || Kernel.rand
+      instructed = positive_number(retry_after)
+
+      if instructed
+        # Honour it in full (clamped), plus a fraction of one base delay so aligned
+        # callers do not wake in lockstep.
+        [instructed, cap].min + (random * base)
+      else
+        # No instruction: exponential from `base`, EQUAL jitter (half fixed, half random),
+        # clamped.
+        full = base * (2**(attempt - 1))
+        [(full / 2.0) + ((full / 2.0) * random), cap].min
+      end
+    end
+
+    # @return [Float, nil] the value when it is a positive number, else nil.
+    def positive_number(value)
+      number = Float(value, exception: false)
+      number&.positive? ? number : nil
+    end
+  end
+end

lib/rail0/client.rb

@@ -58,11 +58,17 @@ class Client
     # @param logger [#call, nil] Optional logger. Pass Rail0::DEFAULT_LOGGER for built-in output.
     # @param max_retries [Integer] Extra attempts after a network failure. Default: 0.
     # @param retry_delay [Numeric] Base delay in seconds between retries (exponential backoff). Default: 0.2.
+    # @param retry_on_429 [Boolean] Retry a rate-limited request, waiting the gateway's
+    #   Retry-After. Default: false — an automatic sleep hides back-pressure from the
+    #   process that could react to it, and stalls a request/response app. Turn it on in a
+    #   job. It works on its own: no need to set +max_retries+ as well.
+    # @param retry_after_cap [Numeric] Longest Retry-After to honour, in seconds. Default: 60.
     def initialize(base_url:, headers: {}, token: nil, timeout: 30, logger: nil,
-                   max_retries: 0, retry_delay: 0.2)
+                   max_retries: 0, retry_delay: 0.2, retry_on_429: false, retry_after_cap: 60)
       http = HttpClient.new(
         base_url: base_url, headers: headers, token: token, timeout: timeout,
-        logger: logger, max_retries: max_retries, retry_delay: retry_delay
+        logger: logger, max_retries: max_retries, retry_delay: retry_delay,
+        retry_on_429: retry_on_429, retry_after_cap: retry_after_cap
       )
       @auth            = Resources::Auth.new(http)
       @chains          = Resources::Chains.new(http)

lib/rail0/http_client.rb

@@ -6,17 +6,31 @@
 module Rail0
   # @!visibility private
   class HttpClient
-    attr_reader :base_url, :timeout, :logger, :max_retries, :retry_delay
+    attr_reader :base_url, :timeout, :logger, :max_retries, :retry_delay,
+                :retry_on_429, :retry_after_cap
 
+    # @param retry_on_429 [Boolean] retry a rate-limited request, waiting the gateway's
+    #   Retry-After (see Rail0::Backoff). OFF by default, deliberately: an automatic
+    #   sleep hides back-pressure from the one process that could react to it, and in a
+    #   request/response app it turns a 429 into a stalled page. Turn it on for a job.
+    #
+    #   It does NOT need `max_retries` to be set as well. That pairing is a footgun —
+    #   the flag would silently do nothing — so on its own it allows one retry.
+    # @param retry_after_cap [Numeric] longest wait to honour, in seconds. The gateway
+    #   sends its whole throttle period as Retry-After rather than the time left in it,
+    #   so this bounds both the over-wait and any hostile value from in between.
     def initialize(base_url:, headers: {}, token: nil, timeout: 30, logger: nil,
-                   max_retries: 0, retry_delay: 0.2)
+                   max_retries: 0, retry_delay: 0.2, retry_on_429: false,
+                   retry_after_cap: 60)
       @base_url        = base_url.chomp("/")
       @static_headers  = { "Content-Type" => "application/json" }.merge(headers)
       @token           = token
       @timeout         = timeout
       @logger          = logger || NULL_LOGGER
       @max_retries     = max_retries
       @retry_delay     = retry_delay
+      @retry_on_429    = retry_on_429
+      @retry_after_cap = retry_after_cap
       freeze
     end
 

lib/rail0/request.rb

@@ -5,6 +5,7 @@
 require "uri"
 require "forwardable"
 require_relative "api_error"
+require_relative "backoff"
 require_relative "default_logger"
 
 module Rail0
@@ -25,7 +26,8 @@ class Request
               put: Net::HTTP::Put, patch: Net::HTTP::Patch,
               delete: Net::HTTP::Delete }.freeze
 
-    def_delegators :client, :base_url, :headers, :timeout, :logger, :max_retries, :retry_delay
+    def_delegators :client, :base_url, :headers, :timeout, :logger, :max_retries, :retry_delay,
+                   :retry_on_429, :retry_after_cap
 
     attr_reader :client, :method, :path, :body, :paginated, :extra_headers
 
@@ -46,7 +48,8 @@ def call
       unless response.is_a?(Net::HTTPSuccess)
         error_body = parse_error_body(response)
         api_error  = ApiError.new(response.code.to_i, error_code(error_body),
-                                  error_message(error_body, response), title: error_body[:title])
+                                  error_message(error_body, response), title: error_body[:title],
+                                  retry_after: retry_after_seconds(response))
         logger.call(LogEntry.new(
           method: method.to_s.upcase, url: url, duration_ms: duration_ms, attempt: attempt,
           request_body: body, status: response.code.to_i, response_body: error_body, error: api_error
@@ -65,21 +68,71 @@ def call
 
     private
 
+    # One loop for the two things worth retrying, which fail in different ways: a network
+    # error raises, a rate limit comes back as a perfectly good 429 response.
+    #
+    # A 429 is the one status this SDK retries, and the reason is not that it is common.
+    # The gateway rejects it in middleware (Rack::Attack), BEFORE the request reaches the
+    # application — so nothing was executed, and retrying carries no risk of doing the
+    # work twice. That is not true of a 502 or a timeout on, say, a capture, where the
+    # broadcast may already be in flight. Which is why the method does not matter here and
+    # a POST is retried like a GET.
+    #
+    # The sleep is on the CALLING thread. There is no thread pool in this SDK and no
+    # promise to wait on: a job that turns retry_on_429 on is choosing to block.
     def with_retries(url)
       attempt = 1
-      start  = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      result = yield
-      [result, elapsed_ms(start), attempt]
-    rescue *ERRORS => e
-      will_retry = attempt <= max_retries
-      logger.call(LogEntry.new(
-        method: method.to_s.upcase, url: url, duration_ms: elapsed_ms(start), attempt: attempt,
-        request_body: body, error: e, will_retry: will_retry
-      ))
-      raise unless will_retry
-      attempt += 1
-      sleep(retry_delay * (2**(attempt - 2)))
-      retry
+      loop do
+        start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        begin
+          response = yield
+        rescue *ERRORS => e
+          will_retry = attempt <= max_retries
+          logger.call(LogEntry.new(
+            method: method.to_s.upcase, url: url, duration_ms: elapsed_ms(start),
+            attempt: attempt, request_body: body, error: e, will_retry: will_retry
+          ))
+          raise unless will_retry
+
+          attempt += 1
+          sleep(retry_delay * (2**(attempt - 2)))
+          next
+        end
+
+        return [response, elapsed_ms(start), attempt] unless retry_throttled?(response, attempt)
+
+        delay = Backoff.throttle_delay(
+          retry_after: response["retry-after"], attempt: attempt,
+          base: retry_delay, cap: retry_after_cap
+        )
+        logger.call(LogEntry.new(
+          method: method.to_s.upcase, url: url, duration_ms: elapsed_ms(start), attempt: attempt,
+          request_body: body, status: response.code.to_i,
+          response_body: parse_error_body(response), will_retry: true
+        ))
+        attempt += 1
+        sleep(delay)
+      end
+    end
+
+    # Whether this response is a rate limit the client opted into retrying, and whether
+    # there is budget left.
+    #
+    # `max_retries` is what bounds it, except that its default is 0 — so requiring both
+    # flags would make retry_on_429 a silent no-op. One retry is the floor when the
+    # caller asked for the behaviour at all.
+    def retry_throttled?(response, attempt)
+      return false unless retry_on_429 && response.code.to_i == 429
+
+      attempt <= [max_retries, 1].max
+    end
+
+    # @return [Integer, nil] the Retry-After header as whole seconds, when it is a
+    #   positive number. HTTP-date form is not parsed: the gateway never sends one, and
+    #   guessing at a date would be worse than admitting we have no instruction.
+    def retry_after_seconds(response)
+      seconds = Backoff.positive_number(response["retry-after"])
+      seconds&.round
     end
 
     def parse_body(response)