From 564084afbbdf0c25cb61d2585c0240ef4a769400 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Wed, 8 Jul 2026 22:27:59 -0700 Subject: [PATCH 1/5] Add proposals for client-identity rate limiting (write and query paths) Cortex's ingestion and query rate limits are enforced at the tenant level only. When multiple independent services or teams share a tenant, one noisy or misbehaving client can exhaust the whole tenant's budget for everyone else, and there is no way to distinguish who, within a tenant, is responsible. Adds two companion design proposals: - client-identity-rate-limiting.md: write-path (distributor) enforcement, keyed on a new trusted X-User-ID header, following the same trust model Cortex already uses for X-Scope-OrgID. - client-identity-query-rate-limiting.md: the read-path (query-frontend) follow-on, reusing the same header and identity approach. Both are opt-in, additive to existing limits, and covered as experimental features on the proposed rollout plan. Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 242 ++++++++++++++ .../client-identity-rate-limiting.md | 315 ++++++++++++++++++ 2 files changed, 557 insertions(+) create mode 100644 docs/proposals/client-identity-query-rate-limiting.md create mode 100644 docs/proposals/client-identity-rate-limiting.md diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md new file mode 100644 index 00000000000..249f35375a2 --- /dev/null +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -0,0 +1,242 @@ +--- +title: "Client-Identity Query Rate Limiting" +linkTitle: "Client-Identity Query Rate Limiting" +weight: 1 +slug: client-identity-query-rate-limiting +--- + +- Author: Krunal Jain +- Date: July 2026 +- Status: Proposed + +## Background + +This is a follow-on to +[Client-Identity Rate Limiting](./client-identity-rate-limiting.md) (write path), applying the same +idea to queries. + +Cortex has no per-request-rate limiter on the read path today. What exists is a different kind of +control: a per-tenant limit on how many requests can be *queued at once*, and a family of per-query +cost limits that bound how *expensive* a single query can be. Neither bounds *how many queries per +second* a tenant's traffic issues, and, same as the write path, neither can distinguish one query +source from another within a shared tenant. + +The query-frontend already reads and logs a couple of per-request identifying headers today, purely +for observability (Grafana's dashboard and panel identifiers, included in slow-query log lines). +These headers are not used for enforcement today, only logging, and this proposal does not use them +for identity either; it uses a new, generic header instead (see Proposed Design). + +A single noisy dashboard, a runaway alert rule evaluating too frequently, or one team's ad hoc +Explore usage can all generate a query load that consumes a shared tenant's query-processing +capacity (queriers, store-gateways) at the expense of every other user/dashboard under that same +`X-Scope-OrgID`, mirroring exactly the ingestion-side problem the write-path proposal addresses, +just on reads. + +## Problem + +Provide a query-rate-limiting dimension *below* the tenant, at the query-frontend, so that one noisy +query source cannot degrade query performance for every other user sharing a tenant. + +Requirements (mirroring the write-path proposal): + +- **Opt-in and backward compatible.** Deployments that don't set the identity header, or don't + enable this feature, see no behavior change. +- **Per-tenant configurable**, following the same default-plus-per-tenant-override pattern used + throughout Cortex. +- **Presence-gated, not mandatory.** A request without the identity header is not rejected; it is + simply not subject to the additional per-identity check, and counts only toward whatever + tenant-level behavior already exists today (which, notably, currently has no *rate* limit to fall + back to on the read path, see Interaction with Existing Limits). +- **Same generic, gateway-set identity header as the write-path proposal (`X-User-ID`), not a + per-client-application-specific one.** Consistent with the write-path proposal's principle: + Cortex trusts one opaque identity value, set by whatever sits in front of it, the same way it + already trusts `X-Scope-OrgID`. It does not special-case any particular client application. + +## Out of Scope + +- The write (ingestion) path: covered by the companion proposal, + [Client-Identity Rate Limiting](./client-identity-rate-limiting.md). The two share a design + philosophy (trusted, gateway-forwarded identity header, opt-in, additive to existing limits) and, + as of this revision, the *same* identity header. They are kept as separate proposals because they + have different enforcement points (distributor vs. query-frontend), not because of any difference + in identity source. +- Per-query cost limiting: unaffected and orthogonal; this proposal bounds *frequency*, those bound + *cost per query*. +- Any new authentication mechanism, and any client-application-specific identity header. Exactly + like the write-path proposal, this reuses a header trusted the same way `X-Scope-OrgID` already + is; it does not add token validation, a new identity protocol, or special-cased handling for any + particular upstream client. +- Enforcement in the querier itself (as opposed to the query-frontend/scheduler). The query-frontend + is the natural analog to the distributor here: it is the first internal component every query + passes through, exactly as the distributor is for writes. + +## Proposed Design + +### Identity extraction + +The same generic, trusted header as the write-path proposal, `X-User-ID`, read once when a query +request enters the query-frontend, alongside the existing Grafana dashboard/panel header reads. +Using the exact same header and identity-extraction approach as the write-path proposal means a +deployment that sets `X-User-ID` once, at its gateway, for all Cortex-bound traffic gets consistent +per-client identity across both ingestion and queries with no additional configuration: one +identity mechanism, reused at both enforcement points, rather than two different ones. + +### Configuration + +New limits, following the same pattern as the write-path proposal: + +```yaml +# Default (flags), applied to all tenants without an override +frontend: + client_identity_query_rate_limit: 0 # 0 = disabled (default; no behavior change) + client_identity_query_burst_size: 0 + # Same bounded-tracking cap as the write-path proposal, sized independently since the + # frontend and distributor are separate processes with separate memory budgets. + client_identity_query_tracked_clients_limit: 10000 + +# Per-tenant override via runtime config +overrides: + tenant-123: + client_identity_query_rate_limit: 50 + client_identity_query_burst_size: 100 +``` + +Units are queries per second (matching how Cortex's existing outstanding-requests limit is already +a count, not a byte/sample rate; this is a request-frequency limit, deliberately independent of +query cost). + +### Enforcement + +Same two-gate structure as the write-path proposal: + +1. The tenant has a non-zero `client_identity_query_rate_limit` configured. +2. The request carries a non-empty `X-User-ID` value. + +``` +Query request arrives at query-frontend + │ + v +┌───────────────────────────┐ no (either gate) ┌───────────────┐ +│ X-User-ID present AND │──────────────────────>│ Continue │ +│ tenant limit configured? │ │ (queue/forward) │ +└───────────────────────────┘ └───────────────┘ + │ yes + v +┌────────────────────────────┐ fail ┌──────────────────┐ +│ Per-(tenant, client) rate │────────────────>│ 429 Too Many │ +│ limit check (new) │ │ Requests │ +└────────────────────────────┘ └──────────────────┘ + │ pass + v +Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) +``` + +A new rate limiter instance, reusing the same local rate-limiting mechanism as the write-path +proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses +the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection +under this feature is observable the same way existing query-frontend rejections already are. + +The rate-limiter strategy backing it mirrors the write-path proposal's approach exactly: it splits +the combined tenant/client key and looks up the per-tenant configured limit, the same per-tenant +limits interface pattern already used elsewhere for query-side limits. + +### Where this runs relative to the frontend/scheduler split + +Cortex's query-frontend can run with or without a separate query-scheduler. The query-frontend's +request handler runs in the query-frontend process in both configurations: it is the entrypoint +before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather +than inside the scheduler's own queueing logic, covers both deployment topologies with one check and +avoids needing per-scheduler-replica state coordination for something that is already a +per-frontend-replica local rate limit, the same "local" semantics the write-path proposal's default +strategy already has. + +### Bounding memory: capped tracking, not an unbounded map + +Same concern as the write-path proposal, and the same fix: the number of distinct tenant+client +keys here is bounded only by how many distinct `X-User-ID` values are presented to the +query-frontend, so tracking is capped with least-recently-used eviction rather than kept in a plain +unbounded map, exactly as described in the write-path proposal's equivalent section. The two caps +are configured independently (see Configuration above) since the query-frontend and distributor are +separate processes with independent memory budgets, but the mechanism and rationale are identical. + +### Local enforcement precision, and interaction with the results cache + +The query-frontend, like the distributor, typically runs multiple replicas behind a load balancer, +so the same local-enforcement precision caveat raised in the write-path proposal applies here too: +one client's query traffic is a smaller, lumpier stream than a tenant's aggregate traffic, so +which replica a given request lands on matters more than it does for tenant-wide limits. The same +resolution applies: ship local-only first, treat a global variant as a near-term follow-up based on +operational experience, not indefinitely deferred work. + +A second, read-path-specific question: the query-frontend serves some requests entirely from its +results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check +before knowing whether a request will be a cache hit means a client issuing frequent but cheap, +repeated, cacheable queries (for example, an auto-refreshing dashboard hitting the same query +window) could be throttled even though those requests would have cost negligible querier/store +gateway capacity. Given this feature's stated goal is protecting shared query-processing capacity, +not query volume for its own sake, this is worth resolving before this feature is considered +complete: either check cache-eligibility first and only rate-limit the cache-miss path, or accept +the imprecision in a first version and document it clearly as a known limitation. This proposal +does not resolve that choice; it is called out explicitly as an open question below rather than +silently deferred. + +### Metrics + +- Reuses the query-frontend's existing rejected-query counter with a new discard reason for this + limit; no new metric family needed, consistent with how other query-frontend rejections are + already tracked. +- Per-client rejection visibility is obtained from query-frontend logs rather than a dedicated + metric label, the same way dashboard/panel identifiers already are, and for the same reason as + the write-path proposal: adding a client label to an existing metric that already carries several + labels would multiply its cardinality by however many distinct clients are tracked. + +### Interaction with existing limits + +- **Orthogonal to per-query cost limits**: those bound how expensive one query is; this bounds how + many queries per second one client issues. +- **Orthogonal to the existing per-tenant outstanding-requests limit**: that bounds queue depth (how + many requests can be waiting at once); this bounds arrival rate. A client could stay under the + queue depth limit while still issuing queries fast enough to starve other queue occupants of + querier time; this closes that gap. +- **Consistent with the write-path proposal's philosophy**: purely additive, off by default, + identical two-gate opt-in structure, same underlying rate-limiting mechanism. + +## Rollout Plan + +Same phased approach as the write-path proposal: introduced as an **experimental** feature, local +enforcement only, disabled by default (`client_identity_query_rate_limit: 0`). A global variant and +graduation out of experimental status follow once operational experience from Phase 1 shows how +much the local-only imprecision, and the cache-interaction question raised above, matter in +practice. + +## Alternatives Considered + +- **Enforce in the scheduler instead of the frontend.** Would require either scheduler-side rate + limiting to be per-scheduler-replica (fine, but then it needs its own identical mechanism since + the scheduler is a separate component from the frontend) or shared/global state across scheduler + replicas, a much larger change mirroring the local-vs-global split the write-path proposal already + has to consider. Enforcing once in the frontend, before a request is ever queued or forwarded, is + simpler and sufficient for a first version; a global variant could be considered later exactly as + it was for the write path. +- **A single combined proposal covering both ingestion and query paths.** Considered and explicitly + rejected; see the write-path proposal's cross-reference to this one. Different enforcement + points and independent review/merge risk argue for two focused proposals over one broad one, even + though both now share the same identity mechanism. + +## Open Questions + +- Should this feature ship together with the write-path proposal, or fully independently, given they + touch different components and teams (distributor vs. query-frontend) might review them + separately? Leaning towards shipping independently, sharing only the small identity-extraction + logic, given the review benefits of keeping them as separate changes discussed above. +- Should the per-client query limit be expressible as a percentage of the tenant's overall query + capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports + fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow + refinement rather than blocking the initial absolute-number implementation. +- Should the per-client check run before or after results-cache eligibility is known (see "Local + enforcement precision, and interaction with the results cache" above)? Checking after would avoid + throttling cheap, cache-served requests, at the cost of being a slightly more invasive change to + the request handling order; checking before (as currently proposed) is simpler but risks + throttling based on request *count* rather than actual resource consumption. This should be + resolved before the feature is considered ready to implement, not left as a known limitation by + default. diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md new file mode 100644 index 00000000000..0885dfbf959 --- /dev/null +++ b/docs/proposals/client-identity-rate-limiting.md @@ -0,0 +1,315 @@ +--- +title: "Client-Identity Rate Limiting" +linkTitle: "Client-Identity Rate Limiting" +weight: 1 +slug: client-identity-rate-limiting +--- + +- Author: Krunal Jain +- Date: July 2026 +- Status: Proposed + +## Background + +Cortex enforces ingestion rate limits at the tenant level only. Every request is identified solely +by the `X-Scope-OrgID` header value, and the distributor's rate limiter applies a single shared +budget, local or global, across everything sent under that tenant. + +In practice, a single tenant is very often not a single writer. Multiple independent services, +teams, or clusters frequently share one `X-Scope-OrgID` (splitting tenants further has real +operational cost: more ring shards, more per-tenant limits to tune, more override entries to +maintain). When that happens, today's rate limiter cannot distinguish a well-behaved writer from a +noisy or misbehaving one sharing the same tenant. A single client scaling up unexpectedly (bad +config, retry storm, new deployment) can exhaust the *entire tenant's* ingestion budget and start +throttling every other legitimate writer sharing that org ID, even though none of the throttled +traffic was the cause. + +Cortex's trust model already has precedent for exactly this shape of problem at the tenant level: +`X-Scope-OrgID` itself is a plain, unauthenticated-by-Cortex header. Cortex trusts it entirely +because it assumes something in front of it (a reverse proxy, an auth gateway such as the one +described in the [Authentication Gateway](./auth-gateway.md) proposal) authenticates the caller and +sets the header correctly, and that untrusted clients can never reach Cortex directly. There is no +equivalent mechanism today for identifying *who, within a tenant,* sent a given write. + +## Problem + +Provide a second, optional rate-limiting dimension *below* the tenant: throttle by client identity, +where identity is an opaque string supplied by the same trusted gateway/proxy layer that already +sets `X-Scope-OrgID`, so that one noisy client cannot exhaust a shared tenant's ingestion budget for +everyone else. + +Requirements: + +- **Opt-in and backward compatible.** Deployments that don't set the identity header, or don't + enable this feature, see no behavior change; the tenant-level limiter continues to work exactly + as it does today. +- **Per-tenant configurable**, following the same default-plus-per-tenant-override pattern as every + other Cortex limit. +- **Consistent trust model with the rest of Cortex.** The identity header is trusted the same way + `X-Scope-OrgID` already is: because it arrives from a trusted network path, not because Cortex + independently verifies it. This proposal does not attempt to add a stronger guarantee than + Cortex's existing multi-tenancy model provides; it reuses the same assumption rather than + introducing a new one. +- **Presence-gated, not mandatory.** A request without the identity header is not rejected or + treated as suspicious; it simply isn't subject to the additional per-identity check, and its + usage counts only toward the tenant's existing aggregate budget, unchanged from today. + +## Out of Scope + +- Any form of authentication Cortex does not already support (JWT validation, OAuth, SSO, mTLS + client-cert identity). An earlier draft of this proposal considered deriving identity from a + verified TLS client certificate; that was dropped because it only works when Cortex itself + terminates client TLS, which excludes the common case of a service mesh (Istio, Linkerd) or + gateway terminating mTLS in front of Cortex, a real gap given Cortex's own case studies document + exactly this kind of mesh-fronted deployment. Reusing the existing `X-Scope-OrgID` trust model + avoids that gap entirely. +- Cross-tenant limiting or identity: this is purely a sub-division of the existing per-tenant + ingestion rate limit. +- Read-path (query) rate limiting by client identity is covered by a **separate, companion + proposal**, [Client-Identity Query Rate Limiting](./client-identity-query-rate-limiting.md), kept + independent of this one because it has a different enforcement point (query-frontend, not + distributor); it reuses the same `X-User-ID` header and identity-extraction approach introduced + here. Filed alongside this proposal rather than folded into it, so each can be reviewed and + merged on its own timeline. +- Defining *how* an operator's gateway derives the identity value (API key ID, service account, + mesh workload identity, job name, etc.). Exactly like Cortex never defines what a "tenant" is + organizationally, this proposal treats the identity as an opaque string the gateway is trusted to + set consistently, which is out of scope for Cortex itself to prescribe. + +## Proposed Design + +### Identity extraction + +Add a new trusted header, `X-User-ID`, read once when a write request enters Cortex. This mirrors +the existing `X-Scope-OrgID` header handling: like `X-Scope-OrgID`, `X-User-ID` is trusted as-is, +because Cortex assumes it is set by a trusted gateway/proxy in front of it, not supplied directly by +an untrusted caller. If the header is absent, the request is unaffected by anything in this +proposal. + +Cortex's HTTP library dependency already defines an `X-Scope-UserID` header, distinct from the +`X-User-ID` proposed here, but it is unused anywhere in Cortex today and its original intent isn't +documented. Reusing it was considered and rejected: adopting an existing-but-dormant header with no +clear record of its intended semantics risks silently changing behavior for any deployment that +happens to already send that header for an unrelated reason, whereas a new, clearly-scoped header +name carries no such risk. The similarity in naming is coincidental and worth calling out explicitly +so it doesn't read as an oversight during review. + +The extracted identity is threaded through the request the same way source IPs are today (already +carried from the HTTP entrypoint through to the distributor for logging purposes), so it survives +the handoff into the write path without changing any internal request/response shapes. + +### Configuration + +New limits, following the same default-plus-per-tenant-override pattern as every other Cortex +limit: + +```yaml +# Default (flags), applied to all tenants without an override +distributor: + client_identity_ingestion_rate_limit: 0 # 0 = disabled (default; no behavior change) + client_identity_ingestion_burst_size: 0 + # Global cap (not per-tenant) on how many distinct tracked tenant+client entries the + # per-client rate limiter keeps in memory at once; oldest-used entries are evicted once + # this is reached. See "Bounding memory" below. + client_identity_ingestion_tracked_clients_limit: 10000 + +# Per-tenant override via runtime config +overrides: + tenant-123: + client_identity_ingestion_rate_limit: 5000 + client_identity_ingestion_burst_size: 10000 +``` + +`client_identity_ingestion_rate_limit: 0` is the default and means "disabled", the same convention +Cortex already uses elsewhere for a limit that should only take effect once explicitly configured. +This guarantees zero behavior change for any tenant that doesn't explicitly opt in. + +### Enforcement + +Enforcement is gated on **two independent conditions**, both of which must hold: + +1. The tenant has a non-zero `client_identity_ingestion_rate_limit` configured. +2. The request carries a non-empty `X-User-ID` value. + +If either is false, the request is subject only to the existing tenant-level ingestion rate check, +exactly as today: a missing header is not an error, and a tenant that hasn't opted in never pays +any cost for this feature. + +``` +Push request + │ + v +┌─────────────────────┐ fail ┌──────────────────┐ +│ Tenant IngestionRate │────────────────>│ 429 Too Many │ +│ check (existing) │ │ Requests │ +└─────────────────────┘ └──────────────────┘ + │ pass + v +┌───────────────────────────┐ no (either gate) ┌───────────────┐ +│ X-User-ID present AND │──────────────────────>│ Continue to │ +│ tenant limit configured? │ │ ingestion │ +└───────────────────────────┘ └───────────────┘ + │ yes + v +┌────────────────────────────┐ fail ┌──────────────────┐ +│ Per-(tenant, client) rate │────────────────>│ 429 Too Many │ +│ limit check (new) │ │ Requests │ +└────────────────────────────┘ └──────────────────┘ + │ pass + v +Continue to ingestion +``` + +A second rate limiter instance, reusing Cortex's existing local rate-limiting mechanism unmodified, +is added to the distributor, keyed by tenant and client identity together rather than by tenant +alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing +tenant-level rate limit; this only prevents one client from consuming the entire budget. + +The per-tenant limit and burst are looked up the same way every other tenant limit is: the +enforcement point resolves the tenant from the combined key and reads the configured value from +that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other +Cortex limit; only the token bucket enforcing it is split per client. + +### Local enforcement is less precise here than at the tenant level + +Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces +its share of the limit independently) and a "global" mode (the limit is divided evenly across +healthy distributors, trading a little coordination overhead for much tighter enforcement). +"Local" is imprecise by design: which distributor a given request lands on is effectively random, +so a client's traffic isn't evenly spread across replicas in the short term, and a burst can land +disproportionately on one distributor. + +That imprecision matters less at the tenant level, where traffic volume is usually large enough +relative to distributor count for the law of large numbers to smooth it out. At the per-client +granularity this proposal adds, that assumption is weaker: one client's traffic is a smaller, +lumpier stream, so local-only enforcement is more likely to either erroneously throttle a +well-behaved client (bad luck concentrates its requests on an already-busy distributor) or let a +misbehaving client through for longer than intended (its requests happen to spread across +distributors, each individually staying under its local share). This is a real precision gap in a +local-only first version, not merely a style choice inherited from the tenant-level limiter. + +Given that, this proposal's initial version still ships local-only, matching how Cortex introduces +most new limits, but a global variant (dividing the per-client limit across healthy distributors, +mirroring the existing global tenant-level strategy) should be treated as a near-term follow-up +once operational experience shows how much the imprecision matters in practice, not as indefinitely +deferred future work. + +### Bounding memory: capped tracking, not an unbounded map + +Unlike the existing tenant-level limiter, whose per-tenant map stays small because the number of +tenants is small and operator-controlled, the number of distinct tenant+client keys here is +bounded only by how many distinct `X-User-ID` values a gateway ever sends. Left as a plain +unbounded map, this becomes both a slow memory leak under normal churn (clients renamed, rotated, +or retired over time never get cleaned up) and, combined with the header trust boundary above, a +denial-of-service vector: a caller able to set arbitrary `X-User-ID` values could grow that map +without limit simply by rotating identities. + +Cortex already solves an analogous problem elsewhere: the tenant-federation regex resolver bounds +a similar per-key cache with a fixed-size, least-recently-used eviction cache rather than a plain +map, sized by an operator-configurable limit. This proposal adopts the same approach: the per-client +rate limiter tracking is capped by a new limit (default sized generously enough for typical +deployments, e.g. on the order of 10,000 tracked clients) using least-recently-used eviction once +the cap is reached. This bounds worst-case memory regardless of how many distinct identities are +ever presented, at the cost of possible churn (an idle client's tracked state being evicted to make +room for an active one) under sustained high-cardinality abuse, a fairness tradeoff, not a resource +leak, and one already covered by the existing guidance to keep the identity header behind a trusted +gateway. + +### Header trust boundary + +Because `X-User-ID` is trusted the same way `X-Scope-OrgID` is, deployments that expose Cortex's +HTTP endpoints directly to untrusted clients (rather than through a gateway/proxy that strips and +re-sets both headers) would let a malicious caller set an arbitrary identity, for example to evade +throttling by rotating identity values, or to frame another client. This is not a new risk specific +to this feature: it is the exact risk profile `X-Scope-OrgID` already carries today, and Cortex's +existing guidance (run behind a reverse proxy/gateway that authenticates callers and controls these +headers) applies unchanged. This should be stated explicitly in the docs for this feature, the same +way multi-tenancy setup docs already caution about `X-Scope-OrgID` exposure. The bounded tracking +above ensures that even under this threat model, the failure mode is bounded churn, not unbounded +memory growth. + +### Metrics + +- The existing discarded-samples metric gains a new discard reason for this rate limit, following + the same reason-labeling convention already used for the tenant-level rate limiter, so it's + observable per tenant the same way tenant-level rate limiting already is. +- Per-client rejection visibility (which specific client tripped the limit) is available from + request logs rather than a dedicated per-client metric label. Adding a client label directly to + the discarded-samples metric would multiply its existing cardinality (already keyed by discard + reason and tenant) by the number of distinct tracked clients; logs are the better fit for a + dimension that's expected to have many more distinct values than tenants do. This mirrors the + same choice made in the companion query-path proposal, for consistency between the two. + +A cardinality caveat on tracking itself, separate from the metrics question above: client identity +values are gateway-controlled, but a deployment that sets a distinct identity per end user (rather +than per service/team) could still produce many distinct tracked clients. The bounded-tracking cap +described above already limits the worst case; this is called out here because it's the same +underlying cardinality concern Cortex's existing per-labelset limits feature already warns about, +just showing up in a different place (limiter memory) rather than metrics. + +### Interaction with existing limits + +This is purely additive: + +- **The tenant-level ingestion rate limit is unaffected** and remains the hard ceiling for the + tenant as a whole. +- **Per-labelset limits** partition by data content (label matchers); this proposal partitions by + data origin (who sent it). They are orthogonal and can be used together. +- If a tenant has no client-identity limit configured (the default), or the request has no + `X-User-ID` header, the new check is a no-op and behavior is identical to today. + +## Rollout Plan + +Introduced as an **experimental** feature, consistent with how Cortex introduces most new limits +(disabled by default, documented as experimental, graduated to stable after operational +experience). Rollout in two phases: + +**Phase 1**: Ship local-only enforcement (see "Local enforcement is less precise" above) behind the +`client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` (disabled) so existing +deployments see no change. Validate with a small number of opted-in tenants. + +**Phase 2**: Based on operational feedback on how much the local-only imprecision matters in +practice, add the global variant (dividing the per-client limit across healthy distributors) and +consider graduating the feature out of experimental status. + +## Alternatives Considered + +- **mTLS client certificate identity.** Cryptographically stronger than a trusted header, and + Cortex's server already supports client-cert authentication. Rejected as the primary mechanism + because it only works when Cortex itself terminates client TLS: a common deployment pattern is a + service mesh or gateway terminating mTLS *before* Cortex, in which case the certificate Cortex + sees reflects the mesh sidecar, not the original caller. It would also reopen an unresolved + Common-Name-vs-Subject-Alternative-Name identity question. The header-based approach sidesteps + both problems and matches Cortex's existing `X-Scope-OrgID` trust model rather than introducing a + second, different one. Could still be revisited later as an alternate identity *source* feeding + the same enforcement path, for deployments that do terminate TLS at Cortex. +- **JWT claim-based identity.** Would require Cortex to parse and validate bearer tokens itself, + which is a meaningfully larger surface (token validation, key rotation, clock skew) than reading a + header the same way `X-Scope-OrgID` already is. Left as potential future work if operators need + cryptographic identity guarantees stronger than the trusted-header model provides. +- **Basic Auth username.** Only applicable if a deployment terminates HTTP Basic Auth at Cortex + itself, which is uncommon for the write path in practice (most Cortex deployments put + authentication at a gateway/proxy in front of Cortex, per the Authentication Gateway proposal), + the same gateway that would set `X-User-ID` under this proposal, making a separate Basic Auth + path redundant. + +## Open Questions + +- Should the client-identity limiter default to enabled with a very high limit (encouraging + visibility/metrics even for tenants that don't want enforcement), or fully opt-in (0 = off, as + described above)? This proposal defaults to fully opt-in to guarantee zero behavior change on + upgrade, consistent with how most new Cortex limits are introduced as disabled-by-default + experimental features. +- Should the header name be configurable, the way Cortex already allows configuring the header used + for source-IP logging, rather than a hardcoded `X-User-ID`? Leaning toward configurable, to avoid + collisions with header names deployments may already use for a similar purpose. +- Should this eventually extend to the query path (read-side per-client throttling, reusing this + same `X-User-ID` header)? **Yes: see the companion proposal, + [Client-Identity Query Rate Limiting](./client-identity-query-rate-limiting.md),** filed alongside + this one and deliberately kept separate rather than expanding this proposal's scope. +- What's the right default for the tracked-clients cap, and should evictions under sustained + pressure be observable (e.g. a counter for evictions caused by hitting the cap, distinct from + normal idle-entry turnover)? A visible eviction-rate metric would let operators tell "the cap is + properly sized" apart from "we're actively being hit with identity churn," which matters for + diagnosing the tradeoff described above. From ebf105ec12d6caaa63ee664eb7eea6f1265092c9 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Thu, 9 Jul 2026 20:57:38 -0700 Subject: [PATCH 2/5] proposals: use global counters for per-client rate limiting Both proposals now specify global enforcement (dividing the per-client limit across healthy replicas via the existing ring/distributor-count mechanism) rather than local-only with global as a follow-up. - Write path: rename section 'Local enforcement is less precise' -> 'Global enforcement'; update enforcement description and rollout plan - Query path: rename section 'Local enforcement precision...' -> 'Global enforcement and interaction with the results cache'; update frontend/scheduler-split section, rate-limiter description, rollout plan, and open-questions cross-reference Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 35 ++++++-------- .../client-identity-rate-limiting.md | 46 ++++++++----------- 2 files changed, 35 insertions(+), 46 deletions(-) diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md index 249f35375a2..c1f21e071ba 100644 --- a/docs/proposals/client-identity-query-rate-limiting.md +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -131,7 +131,7 @@ Query request arrives at query-frontend Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) ``` -A new rate limiter instance, reusing the same local rate-limiting mechanism as the write-path +A new rate limiter instance, reusing the same global rate-limiting mechanism as the write-path proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection under this feature is observable the same way existing query-frontend rejections already are. @@ -145,10 +145,10 @@ limits interface pattern already used elsewhere for query-side limits. Cortex's query-frontend can run with or without a separate query-scheduler. The query-frontend's request handler runs in the query-frontend process in both configurations: it is the entrypoint before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather -than inside the scheduler's own queueing logic, covers both deployment topologies with one check and -avoids needing per-scheduler-replica state coordination for something that is already a -per-frontend-replica local rate limit, the same "local" semantics the write-path proposal's default -strategy already has. +than inside the scheduler's own queueing logic, covers both deployment topologies with one check. +Using global counters (shared across all query-frontend replicas via the same mechanism as the +write-path proposal) means a client cannot route around the limit by landing on a less-loaded +frontend replica. ### Bounding memory: capped tracking, not an unbounded map @@ -159,16 +159,13 @@ unbounded map, exactly as described in the write-path proposal's equivalent sect are configured independently (see Configuration above) since the query-frontend and distributor are separate processes with independent memory budgets, but the mechanism and rationale are identical. -### Local enforcement precision, and interaction with the results cache +### Global enforcement and interaction with the results cache -The query-frontend, like the distributor, typically runs multiple replicas behind a load balancer, -so the same local-enforcement precision caveat raised in the write-path proposal applies here too: -one client's query traffic is a smaller, lumpier stream than a tenant's aggregate traffic, so -which replica a given request lands on matters more than it does for tenant-wide limits. The same -resolution applies: ship local-only first, treat a global variant as a near-term follow-up based on -operational experience, not indefinitely deferred work. +This proposal uses global counters, consistent with the write-path proposal. The same enforcement +precision that global counters provide on the write path applies here: a client cannot route around +the limit by spreading requests across frontend replicas. -A second, read-path-specific question: the query-frontend serves some requests entirely from its +A read-path-specific question remains: the query-frontend serves some requests entirely from its results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check before knowing whether a request will be a cache hit means a client issuing frequent but cheap, repeated, cacheable queries (for example, an auto-refreshing dashboard hitting the same query @@ -203,11 +200,10 @@ silently deferred. ## Rollout Plan -Same phased approach as the write-path proposal: introduced as an **experimental** feature, local -enforcement only, disabled by default (`client_identity_query_rate_limit: 0`). A global variant and -graduation out of experimental status follow once operational experience from Phase 1 shows how -much the local-only imprecision, and the cache-interaction question raised above, matter in -practice. +Same phased approach as the write-path proposal: introduced as an **experimental** feature with +global enforcement, disabled by default (`client_identity_query_rate_limit: 0`). Graduation out of +experimental status follows once operational experience from Phase 1, including resolution of the +cache-interaction question above, confirms the design is sound. ## Alternatives Considered @@ -233,8 +229,7 @@ practice. capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow refinement rather than blocking the initial absolute-number implementation. -- Should the per-client check run before or after results-cache eligibility is known (see "Local - enforcement precision, and interaction with the results cache" above)? Checking after would avoid +- Should the per-client check run before or after results-cache eligibility is known (see "Global enforcement and interaction with the results cache" above)? Checking after would avoid throttling cheap, cache-served requests, at the cost of being a slightly more invasive change to the request handling order; checking before (as currently proposed) is simpler but risks throttling based on request *count* rather than actual resource consumption. This should be diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index 0885dfbf959..9fdf71c74a1 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -160,9 +160,8 @@ Push request Continue to ingestion ``` -A second rate limiter instance, reusing Cortex's existing local rate-limiting mechanism unmodified, -is added to the distributor, keyed by tenant and client identity together rather than by tenant -alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +A second rate limiter instance, reusing Cortex's existing global rate-limiting mechanism, is added +to the distributor, keyed by tenant and client identity together rather than by tenant alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing tenant-level rate limit; this only prevents one client from consuming the entire budget. @@ -171,29 +170,25 @@ enforcement point resolves the tenant from the combined key and reads the config that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other Cortex limit; only the token bucket enforcing it is split per client. -### Local enforcement is less precise here than at the tenant level +### Global enforcement Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces its share of the limit independently) and a "global" mode (the limit is divided evenly across healthy distributors, trading a little coordination overhead for much tighter enforcement). -"Local" is imprecise by design: which distributor a given request lands on is effectively random, -so a client's traffic isn't evenly spread across replicas in the short term, and a burst can land -disproportionately on one distributor. - -That imprecision matters less at the tenant level, where traffic volume is usually large enough -relative to distributor count for the law of large numbers to smooth it out. At the per-client -granularity this proposal adds, that assumption is weaker: one client's traffic is a smaller, -lumpier stream, so local-only enforcement is more likely to either erroneously throttle a -well-behaved client (bad luck concentrates its requests on an already-busy distributor) or let a -misbehaving client through for longer than intended (its requests happen to spread across -distributors, each individually staying under its local share). This is a real precision gap in a -local-only first version, not merely a style choice inherited from the tenant-level limiter. - -Given that, this proposal's initial version still ships local-only, matching how Cortex introduces -most new limits, but a global variant (dividing the per-client limit across healthy distributors, -mirroring the existing global tenant-level strategy) should be treated as a near-term follow-up -once operational experience shows how much the imprecision matters in practice, not as indefinitely -deferred future work. + +At the per-client granularity this proposal adds, local enforcement is more likely to either +erroneously throttle a well-behaved client (bad luck concentrates its requests on an already-busy +distributor) or let a misbehaving client through for longer than intended (its requests happen to +spread across distributors, each individually staying under its local share). One client's traffic +is a smaller, lumpier stream than a tenant's aggregate traffic, so the law-of-large-numbers +smoothing that makes local enforcement tolerable at the tenant level is weaker here. + +This proposal therefore ships with **global enforcement by default**, mirroring Cortex's existing +`ingestion_rate_strategy: global` option for tenant-level limits. The per-client limit is divided +evenly across healthy distributor replicas using the same ring membership and distributor count +already tracked for global tenant-level enforcement. This ensures a client cannot route around the +limit by having its requests spread across replicas, and avoids the erroneous-throttle / bypass +risks that a local-only strategy would introduce at this granularity. ### Bounding memory: capped tracking, not an unbounded map @@ -265,13 +260,12 @@ Introduced as an **experimental** feature, consistent with how Cortex introduces (disabled by default, documented as experimental, graduated to stable after operational experience). Rollout in two phases: -**Phase 1**: Ship local-only enforcement (see "Local enforcement is less precise" above) behind the +**Phase 1**: Ship with global enforcement (see "Global enforcement" above) behind the `client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` (disabled) so existing deployments see no change. Validate with a small number of opted-in tenants. -**Phase 2**: Based on operational feedback on how much the local-only imprecision matters in -practice, add the global variant (dividing the per-client limit across healthy distributors) and -consider graduating the feature out of experimental status. +**Phase 2**: Based on operational feedback, consider graduating the feature out of experimental +status. ## Alternatives Considered From 19f20b7b757f79a2a03858914537b92c23703237 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Thu, 9 Jul 2026 21:01:55 -0700 Subject: [PATCH 3/5] proposals: clarify global counters use gossip, not divide-by-N The previous commit said 'global enforcement' but implied Cortex's existing divide-by-replica-count approximation. This updates both proposals to describe the intended gossip-based mechanism: - Each replica tracks a local per-client counter - Counters are gossiped to peers via the existing memberlist transport - Enforcement is against the cluster-wide sum, not a per-replica share This is more accurate than divide-by-N: it handles uneven load distribution and replica count changes without reconfiguration. The gossip convergence lag (order of memberlist gossip interval) is noted as an acceptable property of this best-effort limit. Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 25 +++++++----- .../client-identity-rate-limiting.md | 39 ++++++++++++------- 2 files changed, 41 insertions(+), 23 deletions(-) diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md index c1f21e071ba..a4a4122a246 100644 --- a/docs/proposals/client-identity-query-rate-limiting.md +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -131,8 +131,8 @@ Query request arrives at query-frontend Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) ``` -A new rate limiter instance, reusing the same global rate-limiting mechanism as the write-path -proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses +A new rate limiter instance, using the same gossip-based global counter mechanism as the +write-path proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection under this feature is observable the same way existing query-frontend rejections already are. @@ -146,9 +146,12 @@ Cortex's query-frontend can run with or without a separate query-scheduler. The request handler runs in the query-frontend process in both configurations: it is the entrypoint before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather than inside the scheduler's own queueing logic, covers both deployment topologies with one check. -Using global counters (shared across all query-frontend replicas via the same mechanism as the -write-path proposal) means a client cannot route around the limit by landing on a less-loaded -frontend replica. + +Each query-frontend replica maintains a local per-client counter, gossips it to peers via the +existing memberlist transport (the same mechanism as the write-path proposal), and enforces against +the cluster-wide sum. A client cannot route around the limit by landing on a less-loaded frontend +replica, and enforcement stays accurate as replicas scale up or down without any reconfiguration of +per-replica shares. ### Bounding memory: capped tracking, not an unbounded map @@ -159,11 +162,13 @@ unbounded map, exactly as described in the write-path proposal's equivalent sect are configured independently (see Configuration above) since the query-frontend and distributor are separate processes with independent memory budgets, but the mechanism and rationale are identical. -### Global enforcement and interaction with the results cache +### Global enforcement via gossip and interaction with the results cache -This proposal uses global counters, consistent with the write-path proposal. The same enforcement -precision that global counters provide on the write path applies here: a client cannot route around -the limit by spreading requests across frontend replicas. +This proposal uses the same gossip-based global counters as the write-path proposal: each +query-frontend replica maintains a local per-client counter, gossips it to peers via memberlist, +and enforces against the cluster-wide sum. A client cannot route around the limit by spreading +requests across frontend replicas, and the gossip convergence lag (on the order of the memberlist +gossip interval) is acceptable for a best-effort rate limit, the same as on the write path. A read-path-specific question remains: the query-frontend serves some requests entirely from its results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check @@ -229,7 +234,7 @@ cache-interaction question above, confirms the design is sound. capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow refinement rather than blocking the initial absolute-number implementation. -- Should the per-client check run before or after results-cache eligibility is known (see "Global enforcement and interaction with the results cache" above)? Checking after would avoid +- Should the per-client check run before or after results-cache eligibility is known (see "Global enforcement via gossip and interaction with the results cache" above)? Checking after would avoid throttling cheap, cache-served requests, at the cost of being a slightly more invasive change to the request handling order; checking before (as currently proposed) is simpler but risks throttling based on request *count* rather than actual resource consumption. This should be diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index 9fdf71c74a1..1629a92c1d3 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -160,8 +160,10 @@ Push request Continue to ingestion ``` -A second rate limiter instance, reusing Cortex's existing global rate-limiting mechanism, is added -to the distributor, keyed by tenant and client identity together rather than by tenant alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +A second rate limiter instance is added to the distributor, keyed by tenant and client identity +together rather than by tenant alone. It tracks a local per-client counter, gossips it to peers +via memberlist, and enforces against the cluster-wide sum — see "Global enforcement via gossip" +above. This check runs as an *additional* step alongside, not instead of, the existing tenant-level limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing tenant-level rate limit; this only prevents one client from consuming the entire budget. @@ -170,25 +172,36 @@ enforcement point resolves the tenant from the combined key and reads the config that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other Cortex limit; only the token bucket enforcing it is split per client. -### Global enforcement +### Global enforcement via gossip Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces its share of the limit independently) and a "global" mode (the limit is divided evenly across -healthy distributors, trading a little coordination overhead for much tighter enforcement). +healthy distributors by replica count). Neither is quite right for per-client enforcement. At the per-client granularity this proposal adds, local enforcement is more likely to either erroneously throttle a well-behaved client (bad luck concentrates its requests on an already-busy distributor) or let a misbehaving client through for longer than intended (its requests happen to spread across distributors, each individually staying under its local share). One client's traffic -is a smaller, lumpier stream than a tenant's aggregate traffic, so the law-of-large-numbers -smoothing that makes local enforcement tolerable at the tenant level is weaker here. - -This proposal therefore ships with **global enforcement by default**, mirroring Cortex's existing -`ingestion_rate_strategy: global` option for tenant-level limits. The per-client limit is divided -evenly across healthy distributor replicas using the same ring membership and distributor count -already tracked for global tenant-level enforcement. This ensures a client cannot route around the -limit by having its requests spread across replicas, and avoids the erroneous-throttle / bypass -risks that a local-only strategy would introduce at this granularity. +is a smaller, lumpier stream than a tenant's aggregate, so the law-of-large-numbers smoothing that +makes local enforcement tolerable at the tenant level is weaker here. + +The existing "global" divide-by-N mode improves on this but still assumes even load distribution: +when a client's traffic lands disproportionately on one replica, that replica's share of the budget +is exhausted faster than others and enforcement becomes uneven again. + +This proposal therefore uses **gossip-based global counters**. Each distributor maintains a local +per-client counter, gossips it to peers via the existing memberlist transport (the same mechanism +already used for the distributor ring), and enforces against the **cluster-wide sum** of all +replicas' local counts. This means: + +- A client's rate is enforced against its true cluster-wide arrival rate, not an approximation + derived from assuming even distribution across replicas. +- Enforcement remains accurate as replicas are added, removed, or experience uneven load — no + reconfiguration of per-replica shares is needed. +- The gossip convergence window introduces a small lag (on the order of the memberlist gossip + interval), during which a burst can transiently exceed the limit before all replicas see the + updated sum. This is an inherent property of eventual consistency and is acceptable for a + best-effort rate limit of this kind. ### Bounding memory: capped tracking, not an unbounded map From 5780a0fc06588c7adaf862cfbe911c553a4e4818 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Fri, 10 Jul 2026 13:36:04 -0700 Subject: [PATCH 4/5] proposals: address review feedback on global/per-tenant language and enforcement design - Replace 'global enforcement via gossip' with ring-based distributed enforcement: shard (tenant, client) onto the existing ring so one replica owns the authoritative counter per key; works on any ring backend (Consul/Etcd/memberlist), not memberlist-only. - Rename 'global' counter language throughout both proposals to 'distributed per-tenant' to avoid confusion with cross-tenant scope. - Move tracked_clients_limit from a process-wide cap to a per-tenant override, consistent with every other Cortex limit. - Clarify LRU eviction semantics under rate limiting: eviction resets a client's counter (allowing a brief burst), which is the intentional tradeoff over refusing new clients when the cap is full. Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 63 +++++--- .../client-identity-rate-limiting.md | 140 +++++++++++------- 2 files changed, 133 insertions(+), 70 deletions(-) diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md index a4a4122a246..34e51ec640b 100644 --- a/docs/proposals/client-identity-query-rate-limiting.md +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -90,15 +90,16 @@ New limits, following the same pattern as the write-path proposal: frontend: client_identity_query_rate_limit: 0 # 0 = disabled (default; no behavior change) client_identity_query_burst_size: 0 - # Same bounded-tracking cap as the write-path proposal, sized independently since the - # frontend and distributor are separate processes with separate memory budgets. - client_identity_query_tracked_clients_limit: 10000 # Per-tenant override via runtime config overrides: tenant-123: client_identity_query_rate_limit: 50 client_identity_query_burst_size: 100 + # Per-tenant cap on how many distinct tracked client entries the per-client rate limiter + # keeps in memory for this tenant; oldest-used entries are evicted once this is reached. + # See "Bounding memory" in the write-path proposal for eviction semantics. + client_identity_query_tracked_clients_limit: 1000 ``` Units are queries per second (matching how Cortex's existing outstanding-requests limit is already @@ -131,7 +132,7 @@ Query request arrives at query-frontend Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) ``` -A new rate limiter instance, using the same gossip-based global counter mechanism as the +A new rate limiter instance, using the same ring-based distributed enforcement as the write-path proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection under this feature is observable the same way existing query-frontend rejections already are. @@ -147,28 +148,29 @@ request handler runs in the query-frontend process in both configurations: it is before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather than inside the scheduler's own queueing logic, covers both deployment topologies with one check. -Each query-frontend replica maintains a local per-client counter, gossips it to peers via the -existing memberlist transport (the same mechanism as the write-path proposal), and enforces against -the cluster-wide sum. A client cannot route around the limit by landing on a less-loaded frontend -replica, and enforcement stays accurate as replicas scale up or down without any reconfiguration of -per-replica shares. +Each query-frontend replica uses the same ring-based distributed enforcement as the write-path +proposal: the `(tenant, client)` key is hashed onto the existing ring, and the owning replica +maintains the authoritative per-tenant, per-client counter. A client cannot route around the limit +by landing on a less-loaded frontend replica, and enforcement stays accurate as replicas scale up +or down without any reconfiguration of per-replica shares. ### Bounding memory: capped tracking, not an unbounded map Same concern as the write-path proposal, and the same fix: the number of distinct tenant+client keys here is bounded only by how many distinct `X-User-ID` values are presented to the -query-frontend, so tracking is capped with least-recently-used eviction rather than kept in a plain -unbounded map, exactly as described in the write-path proposal's equivalent section. The two caps -are configured independently (see Configuration above) since the query-frontend and distributor are +query-frontend, so tracking is capped per tenant with least-recently-used eviction rather than kept +in a plain unbounded map, exactly as described in the write-path proposal's equivalent section +(including its discussion of the enforcement consequence of eviction). The two caps are configured +independently per tenant (see Configuration above) since the query-frontend and distributor are separate processes with independent memory budgets, but the mechanism and rationale are identical. -### Global enforcement via gossip and interaction with the results cache +### Distributed enforcement and interaction with the results cache -This proposal uses the same gossip-based global counters as the write-path proposal: each -query-frontend replica maintains a local per-client counter, gossips it to peers via memberlist, -and enforces against the cluster-wide sum. A client cannot route around the limit by spreading -requests across frontend replicas, and the gossip convergence lag (on the order of the memberlist -gossip interval) is acceptable for a best-effort rate limit, the same as on the write path. +This proposal uses the same ring-based distributed enforcement as the write-path proposal: each +`(tenant, client)` key is sharded onto the existing ring, and the owning query-frontend replica +maintains the authoritative per-tenant, per-client counter. A client cannot route around the +per-tenant limit by spreading requests across frontend replicas, and enforcement stays accurate as +replicas scale without any per-replica reconfiguration. A read-path-specific question remains: the query-frontend serves some requests entirely from its results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check @@ -203,6 +205,26 @@ silently deferred. - **Consistent with the write-path proposal's philosophy**: purely additive, off by default, identical two-gate opt-in structure, same underlying rate-limiting mechanism. +## Value of the Identity Header Beyond Rate Limiting + +The same `X-User-ID` header established on the write path (see the companion proposal) carries +equivalent value on the read path once it is available: + +- **Query resource consumption per client.** Query execution time, querier CPU, and store-gateway + reads can all be attributed per client in logs, enabling identification of expensive query + sources — runaway alert rules, auto-refreshing dashboards, ad hoc exploration — without manual + log correlation. +- **Per-client slow query attribution.** The query-frontend already logs slow queries; with + `X-User-ID` present, slow query log lines carry the client identity directly, making it + straightforward to identify which client or dashboard is responsible without cross-referencing + external systems. +- **Future per-client query cost quotas.** Once client identity is a reliable, trusted dimension on + the read path, per-client cost controls — for example, a cap on total querier time or + store-gateway bytes scanned per client per interval — become implementable without any additional + identity plumbing. + +These are downstream opportunities enabled by this proposal's identity header, not in scope here. + ## Rollout Plan Same phased approach as the write-path proposal: introduced as an **experimental** feature with @@ -230,6 +252,11 @@ cache-interaction question above, confirms the design is sound. touch different components and teams (distributor vs. query-frontend) might review them separately? Leaning towards shipping independently, sharing only the small identity-extraction logic, given the review benefits of keeping them as separate changes discussed above. +- What's the right default for the per-tenant tracked-clients cap? The cap is configurable per + tenant via `client_identity_query_tracked_clients_limit` in the overrides; evictions caused by + hitting the cap are observable via a dedicated eviction counter, letting operators distinguish + "cap is properly sized" from "we're actively being hit with identity churn" without needing to + inspect memory profiles. - Should the per-client query limit be expressible as a percentage of the tenant's overall query capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index 1629a92c1d3..bb045f2c6ff 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -108,16 +108,16 @@ limit: distributor: client_identity_ingestion_rate_limit: 0 # 0 = disabled (default; no behavior change) client_identity_ingestion_burst_size: 0 - # Global cap (not per-tenant) on how many distinct tracked tenant+client entries the - # per-client rate limiter keeps in memory at once; oldest-used entries are evicted once - # this is reached. See "Bounding memory" below. - client_identity_ingestion_tracked_clients_limit: 10000 # Per-tenant override via runtime config overrides: tenant-123: client_identity_ingestion_rate_limit: 5000 client_identity_ingestion_burst_size: 10000 + # Per-tenant cap on how many distinct tracked client entries the per-client rate limiter + # keeps in memory for this tenant; oldest-used entries are evicted once this is reached. + # See "Bounding memory" below. + client_identity_ingestion_tracked_clients_limit: 1000 ``` `client_identity_ingestion_rate_limit: 0` is the default and means "disabled", the same convention @@ -161,9 +161,9 @@ Continue to ingestion ``` A second rate limiter instance is added to the distributor, keyed by tenant and client identity -together rather than by tenant alone. It tracks a local per-client counter, gossips it to peers -via memberlist, and enforces against the cluster-wide sum — see "Global enforcement via gossip" -above. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +together rather than by tenant alone. It enforces against the authoritative per-tenant, per-client +counter via ring-based sharding — see "Distributed enforcement via the distributor ring" +below. This check runs as an *additional* step alongside, not instead of, the existing tenant-level limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing tenant-level rate limit; this only prevents one client from consuming the entire budget. @@ -172,36 +172,34 @@ enforcement point resolves the tenant from the combined key and reads the config that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other Cortex limit; only the token bucket enforcing it is split per client. -### Global enforcement via gossip - -Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces -its share of the limit independently) and a "global" mode (the limit is divided evenly across -healthy distributors by replica count). Neither is quite right for per-client enforcement. - -At the per-client granularity this proposal adds, local enforcement is more likely to either -erroneously throttle a well-behaved client (bad luck concentrates its requests on an already-busy -distributor) or let a misbehaving client through for longer than intended (its requests happen to -spread across distributors, each individually staying under its local share). One client's traffic -is a smaller, lumpier stream than a tenant's aggregate, so the law-of-large-numbers smoothing that -makes local enforcement tolerable at the tenant level is weaker here. - -The existing "global" divide-by-N mode improves on this but still assumes even load distribution: -when a client's traffic lands disproportionately on one replica, that replica's share of the budget -is exhausted faster than others and enforcement becomes uneven again. - -This proposal therefore uses **gossip-based global counters**. Each distributor maintains a local -per-client counter, gossips it to peers via the existing memberlist transport (the same mechanism -already used for the distributor ring), and enforces against the **cluster-wide sum** of all -replicas' local counts. This means: - -- A client's rate is enforced against its true cluster-wide arrival rate, not an approximation - derived from assuming even distribution across replicas. -- Enforcement remains accurate as replicas are added, removed, or experience uneven load — no - reconfiguration of per-replica shares is needed. -- The gossip convergence window introduces a small lag (on the order of the memberlist gossip - interval), during which a burst can transiently exceed the limit before all replicas see the - updated sum. This is an inherent property of eventual consistency and is acceptable for a - best-effort rate limit of this kind. +### Distributed enforcement via the distributor ring + +Per-client limits are enforced using the existing distributor ring rather than gossip-based counter +aggregation. Each `(tenant, client)` key is sharded onto the ring: the distributor instance that +owns the shard for that key maintains the authoritative rate-limit counter for it. + +When a distributor receives a write request carrying an `X-User-ID` value, it hashes +`(tenant_id, client_id)` to determine the owning ring member, then: + +- If it owns that shard, it increments and enforces the counter locally. +- If another replica owns the shard, it forwards the rate-limit check to that replica via an + internal RPC before accepting or rejecting the request. + +This approach: + +- **Works on any ring backend** (Consul, Etcd, memberlist) — not memberlist-only — because the + ring is used only for key ownership, not as a gossip transport. +- **Eliminates convergence lag**: the owning replica holds the single authoritative counter; there + is no eventual-consistency window during which a burst can transiently exceed the configured + per-tenant limit before all replicas converge. +- **Stays accurate under uneven load**: a client's requests landing on different distributors always + count against the same shard owner, regardless of which replica physically received each request. + +The tradeoff is a potential per-request remote call to the shard owner when the receiving +distributor does not own the relevant shard. For deployments with random load balancing in front of +distributors, a majority of checks will require this hop. Whether the added latency is acceptable +depends on the intra-cluster RPC latency for a given deployment; this is a known cost and worth +calling out explicitly in the implementation phase. ### Bounding memory: capped tracking, not an unbounded map @@ -216,13 +214,25 @@ without limit simply by rotating identities. Cortex already solves an analogous problem elsewhere: the tenant-federation regex resolver bounds a similar per-key cache with a fixed-size, least-recently-used eviction cache rather than a plain map, sized by an operator-configurable limit. This proposal adopts the same approach: the per-client -rate limiter tracking is capped by a new limit (default sized generously enough for typical -deployments, e.g. on the order of 10,000 tracked clients) using least-recently-used eviction once -the cap is reached. This bounds worst-case memory regardless of how many distinct identities are -ever presented, at the cost of possible churn (an idle client's tracked state being evicted to make -room for an active one) under sustained high-cardinality abuse, a fairness tradeoff, not a resource -leak, and one already covered by the existing guidance to keep the identity header behind a trusted -gateway. +tracking for each tenant is capped by a new per-tenant limit (configurable via +`client_identity_ingestion_tracked_clients_limit` in the tenant's overrides, with a sensible default +on the order of 1,000 tracked clients per tenant) using least-recently-used eviction once the cap +is reached. + +Unlike the federation regex resolver, where eviction only affects latency (a cache miss means a +slower regex rebuild), eviction here has an enforcement consequence: an evicted client's counter is +discarded, so if it reappears it starts from zero and can issue a fresh burst before its token +bucket fills and enforcement kicks in again. This is a deliberate tradeoff — the alternative +(refusing new clients once the cap is full) risks denying service to legitimate new clients when a +tenant has many active writers. The LRU approach bounds memory and preserves fairness under normal +cardinality, at the cost of a brief enforcement gap when a previously-evicted client resumes. Under +sustained high-cardinality abuse (an attacker rotating identities to flood the LRU), the failure +mode is bounded churn (evictions) rather than unbounded memory growth, which is acceptable given +the existing guidance to keep `X-User-ID` behind a trusted gateway. + +Operators can tune the per-tenant cap upward for tenants with many legitimate concurrent writers, +or observe the eviction counter (a dedicated counter incremented in the LRU eviction callback) to +distinguish "cap is properly sized" from "we are actively seeing identity churn." ### Header trust boundary @@ -267,15 +277,41 @@ This is purely additive: - If a tenant has no client-identity limit configured (the default), or the request has no `X-User-ID` header, the new check is a no-op and behavior is identical to today. +## Value of the Identity Header Beyond Rate Limiting + +Establishing `X-User-ID` as a first-class header in the write path opens up a broader set of +per-client observability and control capabilities that are not in scope for this proposal but +become straightforward extensions once the identity is available: + +- **Resource consumption attribution.** Samples ingested, bytes written, and active series can all + be broken down per client in logs and metrics, enabling capacity planning and chargeback at the + client level rather than only at the tenant level. +- **Out-of-order sample detection per client.** The distributor already detects and discards + out-of-order samples; with `X-User-ID` present, those rejections can be attributed to the + specific client sending them, making it straightforward to identify a misconfigured or misbehaving + writer without manual log correlation across requests. +- **Per-client cardinality tracking.** Today cardinality can only be attributed to a tenant as a + whole. With client identity available on the write path, high-cardinality contributors can be + identified by source, not just by label patterns. +- **Audit and debugging.** A client identity on every write request makes it possible to answer + questions like "which client started sending this metric series?" or "which client's deployment + caused this ingestion spike?" directly from logs, without cross-referencing external systems. +- **Future per-client quota enforcement.** Once client identity is a reliable, trusted dimension, + per-client quotas (not just rate limits) — for example, a cap on active series or total samples + per day per client — become implementable without any additional identity plumbing. + +None of these require changes to this proposal; they are downstream opportunities that justify +establishing a clean, trusted identity header now rather than bolting it on later. + ## Rollout Plan Introduced as an **experimental** feature, consistent with how Cortex introduces most new limits (disabled by default, documented as experimental, graduated to stable after operational experience). Rollout in two phases: -**Phase 1**: Ship with global enforcement (see "Global enforcement" above) behind the -`client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` (disabled) so existing -deployments see no change. Validate with a small number of opted-in tenants. +**Phase 1**: Ship with distributed enforcement (see "Distributed enforcement via the distributor ring" +above) behind the `client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` +(disabled) so existing deployments see no change. Validate with a small number of opted-in tenants. **Phase 2**: Based on operational feedback, consider graduating the feature out of experimental status. @@ -315,8 +351,8 @@ status. same `X-User-ID` header)? **Yes: see the companion proposal, [Client-Identity Query Rate Limiting](./client-identity-query-rate-limiting.md),** filed alongside this one and deliberately kept separate rather than expanding this proposal's scope. -- What's the right default for the tracked-clients cap, and should evictions under sustained - pressure be observable (e.g. a counter for evictions caused by hitting the cap, distinct from - normal idle-entry turnover)? A visible eviction-rate metric would let operators tell "the cap is - properly sized" apart from "we're actively being hit with identity churn," which matters for - diagnosing the tradeoff described above. +- What's the right default for the per-tenant tracked-clients cap? The cap is configurable per + tenant via `client_identity_ingestion_tracked_clients_limit` in the overrides; evictions caused + by hitting the cap are observable via a dedicated eviction counter, letting operators distinguish + "cap is properly sized" from "we're actively being hit with identity churn" without needing to + inspect memory profiles. From 35d17419a6d09994dcb9187bf3e5a882fa764e6e Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Sat, 11 Jul 2026 22:29:16 -0700 Subject: [PATCH 5/5] proposals: note agent fan-out as a noisy-neighbor example Add many parallel agent instances fanning out under one shared tenant to the list of scenarios that motivate client-identity rate limiting, alongside the existing bad-config/retry-storm/deployment examples. Signed-off-by: Krunal Jain --- docs/proposals/client-identity-rate-limiting.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index bb045f2c6ff..cecf10e372e 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -20,9 +20,9 @@ teams, or clusters frequently share one `X-Scope-OrgID` (splitting tenants furth operational cost: more ring shards, more per-tenant limits to tune, more override entries to maintain). When that happens, today's rate limiter cannot distinguish a well-behaved writer from a noisy or misbehaving one sharing the same tenant. A single client scaling up unexpectedly (bad -config, retry storm, new deployment) can exhaust the *entire tenant's* ingestion budget and start -throttling every other legitimate writer sharing that org ID, even though none of the throttled -traffic was the cause. +config, retry storm, new deployment, or many parallel agent instances fanning out under one shared +tenant) can exhaust the *entire tenant's* ingestion budget and start throttling every other +legitimate writer sharing that org ID, even though none of the throttled traffic was the cause. Cortex's trust model already has precedent for exactly this shape of problem at the tenant level: `X-Scope-OrgID` itself is a plain, unauthenticated-by-Cortex header. Cortex trusts it entirely