fix(health): /readyz must not withdraw an idle gateway - #841
Conversation
/readyz returned 503 once the config watch had gone 300s without an event. That reads as "the watch is wedged", but the signal is elapsed time since the last config *event*, and an environment whose resources are not changing produces none — which for most deployments is the steady state. A healthy gateway therefore reported itself unready five minutes after its environment went quiet. Under Kubernetes that is an outage, not a degraded signal: every replica crosses the threshold at the same moment, so the Service loses all of its endpoints. Observed on a real cluster — /livez 200, /status/config reporting the etcd watch connected, /readyz 503, pods 0/1, and the autoscaler adding replicas that could not become Ready either. Readiness is also the wrong lever for the case the threshold was aimed at. A genuinely wedged watch is a property of the config source and hits every replica at once, so withdrawing them all turns "serving the last accepted configuration" into a total outage with nowhere to shift traffic. Config freshness stays observable and alertable on /admin/v1/health, /status/config, and the aisix_config_* metrics — none of which are touched here. Readiness now gates on what it can actually answer: not draining, and configuration applied at least once. This matches LiteLLM, whose /health/readiness checks shutdown state and dependency reachability and has no event-recency condition. Both listeners share config_readiness_block, so the admin listener's /readyz is fixed with the proxy's; the /readyz OpenAPI descriptions are updated to match. Tests: an idle-environment case on the helper and a router-level case asserting /readyz stays 200 with a two-hour apply age. Both fail with the threshold restored. They sit at the unit and router level rather than in the e2e suite because the threshold is a compile-time constant — an end-to-end proof would have to idle for five real minutes.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
Comment |
The idle-withdrawal behaviour is a bug being fixed in api7/aisix#841, not something inherent to /readyz — say so, and say when this probe moves back.
…EDA (#327) Adds charts/aisix, the public chart for the AISIX data plane. Until now the only Kubernetes install path was a hand-rolled Deployment manifest generated by the console, with no autoscaling, disruption budget, or scrape target. The chart runs the gateway in control-plane-managed mode: it takes the data-plane manager endpoint plus the gateway certificate bundle issued from the console (inline PEMs, or an existing Secret so the private key stays out of the values file), and everything else arrives from the control plane. Autoscaling: `autoscaling.*` renders a v2 HorizontalPodAutoscaler with CPU and memory targets, free-form behavior, and extraMetrics; `keda.*` renders a KEDA ScaledObject for request-driven signals. The two are mutually exclusive and enabling both fails the render, rather than letting two controllers fight over spec.replicas — which the Deployment omits whenever either is on, so helm upgrade cannot reset the count the autoscaler chose. Measured on kind from a settled 2-replica baseline: HPA decided at t=42s and new replicas were serving at t=50s; scale-down stepped 5 -> 3 -> 2. Also included: a metrics Service and optional ServiceMonitor, a PodDisruptionBudget, and a first-class rateLimit.backend=redis switch — counters are per-replica otherwise, so every replica an autoscaler adds silently multiplies each configured limit. Three pod-spec settings are load-bearing, each confirmed by breaking it: enableServiceLinks: false (service links from any aisix* Service in the namespace crash the gateway at boot), capabilities drop ALL + add NET_BIND_SERVICE (the binary's file capability makes a bare drop-ALL fail exec with exit 126), and readiness on /status/ready rather than /readyz (which withdraws an idle-but-healthy gateway; api7/aisix#841 fixes that, and this probe moves back once appVersion carries the fix). Refs api7/AISIX-Cloud#1086
1 participant
Problem
/readyzreturned 503 once the configuration watch had gone 300 seconds without an event. That condition reads as "the watch is wedged", but the signal it measures is elapsed time since the last configuration event — and an environment whose resources are not changing produces none. For most deployments that is the steady state, so a perfectly healthy gateway reported itself unready five minutes after its environment went quiet.Under Kubernetes that is an outage rather than a degraded signal: every replica crosses the threshold at the same moment, so the Service loses all of its endpoints at once.
Observed on a real cluster while validating the new data-plane Helm chart (api7/api7-helm-chart#327):
Change
Readiness now gates on what it can actually answer: not draining, and configuration applied at least once. The event-recency condition is gone, along with
READYZ_STALE_AFTER— its only use.Readiness is the wrong lever for the case the threshold was aimed at, independently of the idle-environment problem. A genuinely wedged watch is a property of the configuration source, so it hits every replica simultaneously; withdrawing them all converts "serving the last accepted configuration" into a total outage with no healthy instance to shift traffic to. That is a page-the-operator signal, not a load-balancer signal.
Nothing about configuration freshness becomes unobservable.
/admin/v1/health,/status/config, and theaisix_config_*metrics all keep their staleness view and are untouched.Both listeners share
config_readiness_block, so the admin listener's/readyzis fixed alongside the proxy's. The/readyzOpenAPI descriptions are updated to match.Baseline
LiteLLM's
/health/readinesschecks shutdown state and dependency reachability, with no event-recency condition — this converges on that behavior rather than diverging from it.Tests
an_idle_environment_stays_ready— the helper stays unblocked at 299s, 301s, 1h, and 7d.readyz_stays_200_when_no_config_event_has_arrived_in_hours— router level, with a probe reporting a two-hour apply age.Both fail with the threshold restored (verified by patching it back). They sit at the unit and router level rather than in the e2e suite because the threshold was a compile-time constant, so an end-to-end proof would have to idle for five real minutes.
Follow-ups
GET /status/readybecause it pinsappVersion 0.6.0, which still has this bug. It moves to/readyzat the first release carrying this fix.