feat(aisix): Helm chart for the AI gateway data plane, with HPA and KEDA - #327
Conversation
Adds charts/aisix, the public chart for the AISIX data plane. Until now the
only Kubernetes install path was a hand-rolled Deployment manifest, which
carries no autoscaling, no disruption budget, and no scrape target.
The chart runs the gateway in control-plane-managed mode: it takes the
data-plane manager endpoint plus the gateway certificate bundle issued from
the console (inline PEMs, or an existing Secret so the private key stays out
of the values file), and everything else arrives from the control plane.
Autoscaling:
- `autoscaling.*` renders a v2 HorizontalPodAutoscaler with CPU and memory
targets, free-form `behavior`, and `extraMetrics` for custom / external
metrics.
- `keda.*` renders a KEDA ScaledObject for request-driven signals such as a
Prometheus query over the gateway's own metrics.
- The two are mutually exclusive; enabling both fails the render instead of
letting two controllers fight over spec.replicas. The Deployment omits
spec.replicas whenever either is on, so `helm upgrade` cannot reset the
replica count the autoscaler chose.
Scaling events are safe in both directions without any extra configuration.
A new replica stays out of the Service until `/readyz` passes, which happens
only after it has applied configuration from the control plane. A terminating
one leaves the Service first, then drains: `terminationGracePeriodSeconds`
defaults to 120 because a streaming response outlives the Kubernetes default
of 30, and a preStop pause covers the window where endpoint removal has not
yet reached every kube-proxy.
Also included: a metrics Service and optional ServiceMonitor (scraping never
rides the possibly-public proxy Service), a PodDisruptionBudget, and a
first-class `rateLimit.backend=redis` switch — counters are per-replica
otherwise, so every replica an autoscaler adds silently multiplies each
configured limit. NOTES.txt warns when that combination is deployed.
Two pod-spec settings are load-bearing and verified against a real cluster
rather than assumed:
- `enableServiceLinks: false`. The gateway reads every AISIX_-prefixed
environment variable as configuration and rejects unknown keys, so service
links from any Service named `aisix*` in the namespace crash it at boot
("unknown field `cp_ui_service_port_http`").
- `capabilities: drop [ALL], add [NET_BIND_SERVICE]`. The binary carries
`cap_net_bind_service=+ep`; with that capability's effective bit set,
dropping ALL without adding it back fails exec outright (exit 126,
"Operation not permitted") rather than merely losing the privilege.
Refs api7/AISIX-Cloud#1086
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (20)
Comment |
/readyz withdraws an instance whose configuration watch has gone five minutes without an *event*, and an idle environment produces no events — so every replica reported itself unready five minutes after install and Kubernetes emptied the Service. Observed on a real cluster: /livez 200, /status/config reporting the etcd watch connected, /readyz 503, pods 0/1 and the autoscaler adding replicas that could never become Ready either. /status/ready on the metrics listener is the configuration-source gate the product documents for exactly this use: 503 until the first valid configuration is applied, 200 from then on. Same protection against serving before config arrives, without the idle-timeout cliff.
…ut in it With an existingSecret and no podAnnotations the template emitted a bare `annotations:` key followed by a comment about a checksum it had just skipped.
The idle-withdrawal behaviour is a bug being fixed in api7/aisix#841, not something inherent to /readyz — say so, and say when this probe moves back.
2 participants
Adds
charts/aisix, the public chart for the AISIX data plane. Until now the only Kubernetes install path was a hand-rolled Deployment manifest generated by the console, which carries no autoscaling, no disruption budget, and no scrape target.The chart runs the gateway in control-plane-managed mode: it takes the data-plane manager endpoint plus the gateway certificate bundle issued from the console — inline PEMs, or an existing Secret so the private key stays out of the values file — and everything else arrives from the control plane.
Autoscaling
autoscaling.*renders a v2HorizontalPodAutoscalerwith CPU and memory targets, free-formbehavior, andextraMetricsfor custom / external metrics.keda.*renders a KEDAScaledObjectfor request-driven signals, such as a Prometheus query over the gateway's own metrics.spec.replicas. The Deployment omitsspec.replicaswhenever either is on, sohelm upgradecannot reset the replica count the autoscaler chose.Scaling events are safe in both directions without extra configuration. A new replica stays out of the Service until its readiness probe passes, which happens only after it has applied configuration from the control plane. A terminating one leaves the Service first, then drains:
terminationGracePeriodSecondsdefaults to 120 because a streaming response outlives the Kubernetes default of 30, and apreStoppause covers the window where endpoint removal has not yet reached every kube-proxy.Also included
A metrics Service and optional
ServiceMonitor(so scraping never rides the possibly-public proxy Service), aPodDisruptionBudget, and a first-classrateLimit.backend=redisswitch — counters are per-replica otherwise, so every replica an autoscaler adds silently multiplies each configured limit.NOTES.txtwarns when that combination is deployed.Verification on a real cluster
Installed on kind against a real
aisix-cprelease, with a certificate issued through the control-plane API.Autoscaling, measured end to end from a settled baseline of 2 Ready replicas:
50 seconds from burst to a new Ready gateway pod. Scale-down stepped 5 → 3 → 2 once the load stopped, one HPA decision at a time.
Both replicas reached Ready under
drop: [ALL]+readOnlyRootFilesystem+runAsNonRoot, and registered with the control plane as two separate instances sharing one certificate.Three load-bearing settings, each confirmed by breaking it
enableServiceLinks: false— the gateway reads everyAISIX_-prefixed environment variable as configuration and rejects unknown keys. Patching it totruecrash-loops the pod withunknown field `cp_ui_service_port_http`— service links from any Service namedaisix*in the namespace, including a control plane installed alongside it.capabilities: drop [ALL], add [NET_BIND_SERVICE]— the binary carriescap_net_bind_service=+ep. Dropping ALL without adding it back failsexecoutright (exit 126,Operation not permitted) rather than merely losing the privilege.GET /status/readyon the metrics listener, not the proxy's/readyz./readyzwithdraws an instance whose configuration watch has gone five minutes without an event, and an idle environment produces none — every replica reported itself unready five minutes after install and Kubernetes emptied the Service, while/livezreturned 200 and/status/configreported the etcd watch connected. That is a data-plane bug, fixed in fix(health): /readyz must not withdraw an idle gateway aisix#841; this chart pinsappVersion 0.6.0, which still has it, so it uses the configuration-source gate instead — 503 until the first valid configuration is applied, 200 from then on. Confirmed stable across the 300 s mark that broke the first attempt. The probe moves to/readyzat the first release carrying the fix.Notes
ct lintbut notct install: the gateway needs a control plane to become ready, so an install test would only prove that a pod cannot start on its own.ci/default-values.yamlsupplies the endpoint and certificate placeholders lint needs, since the chart deliberately refuses to render without them.ct lintverified locally in the CI container image.aisixandaisix-cpnow release as a pair on the AISIX version axis — documented inAGENTS.md.Paired with api7/AISIX-Cloud#1188 (console Kubernetes (Helm) tab) and api7/docs#1968.
Fixes api7/AISIX-Cloud#1086