chore(helm): configurable lifecycle, terminationGracePeriodSeconds and autoscaling behavior - #261
Open
kylegato-dutchie wants to merge 1 commit into
Conversation
…d autoscaling behavior
Adds three pass-through values so the chart can express graceful pod shutdown and
HPA scaling behavior. All three default to empty, so rendered output is unchanged
for existing users (verified by diffing `helm template` against the previous
chart -- the only difference is the `helm.sh/chart` version label).
lifecycle: {} -> container `lifecycle`
terminationGracePeriodSeconds: '' -> pod `terminationGracePeriodSeconds`
autoscaling.behavior: {} -> HPA `spec.behavior`
Motivation: without a `preStop` hook there is no way to drain a pod gracefully,
and HPA scale-down deletes pods directly rather than through the eviction API, so
a PodDisruptionBudget does not apply to it. On a deployment of this chart behind
an ingress controller that produced HTTP 504s on every scale-down -- the container
received SIGTERM while the ingress was still routing to it, because Service
endpoint removal is asynchronous. A `preStop` sleep keeps the pod serving until
that propagates, and `autoscaling.behavior` lets operators damp the scale-down
oscillation that triggers a drain in the first place.
These were the only knobs missing to solve it from values; everything else needed
was already configurable.
Example:
terminationGracePeriodSeconds: 60
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 30"]
autoscaling:
enabled: true
behavior:
scaleDown:
stabilizationWindowSeconds: 900
policies:
- type: Pods
value: 1
periodSeconds: 300
Note `terminationGracePeriodSeconds` should exceed any `preStop` sleep, or the
kubelet SIGKILLs the container mid-hook.
Verified: `helm lint` clean; default render byte-identical to the previous chart
apart from the version label; the three values render into the expected fields;
and the rendered objects are accepted by a Kubernetes 1.35 API server via
`kubectl apply --dry-run=server`.
Chart version 1.2.0 -> 1.3.0 (additive features, no breaking change).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds three pass-through values so the chart can express graceful pod shutdown and HPA scaling behavior:
lifecyclelifecycle{}terminationGracePeriodSecondsterminationGracePeriodSeconds''autoscaling.behaviorspec.behavior{}All three default to empty, so rendered output is unchanged for existing users.
Motivation
Without a
preStophook there's no way to drain a pod gracefully. That matters more than it might seem, because HPA scale-down deletes pods directly rather than through the eviction API, so aPodDisruptionBudgetdoesn't apply to it.On our deployment of this chart behind an ingress controller, that produced HTTP 504s on every scale-down: the container got SIGTERM while the ingress was still routing to it, because Service endpoint removal is asynchronous. Correlation was unambiguous — the HPA oscillated
2→4→2, and 504 bursts landed only in the scale-down windows, never on scale-up, with no OOMKills, no restarts and clean application logs.A
preStopsleep keeps the pod serving until endpoint removal propagates, andautoscaling.behaviorlets operators damp the oscillation that triggers a drain in the first place. These were the only knobs missing to fix it from values — everything else needed was already configurable.Example
terminationGracePeriodSecondsshould exceed anypreStopsleep, or the kubelet SIGKILLs the container mid-hook. Both notes are in the values comments.Verification
helm lintclean.helm templatewith default values, diffed against the previous chart — identical apart from thehelm.sh/chartversion label.kubectl apply --dry-run=server.Chart version
1.2.0→1.3.0(additive, no breaking change), following the convention from #258.Happy to adjust naming or placement — I put
lifecycleandterminationGracePeriodSecondsnext topodDisruptionBudgetsince they're all pod-lifecycle concerns, andbehaviorinside the existingautoscalingblock.