fix: use interruptionQueue Helm value for Karpenter >= v0.33.0 - #8844
Open
warren830 wants to merge 1 commit into
Open
fix: use interruptionQueue Helm value for Karpenter >= v0.33.0#8844warren830 wants to merge 1 commit into
warren830 wants to merge 1 commit into
Conversation
The Karpenter chart renamed its interruption queue Helm value when it flattened `settings`. Charts before v0.33.0 read `settings.aws.interruptionQueueName`; the flattened layout reads `settings.interruptionQueue`. eksctl built a single `settings` map and re-nested it under `aws` for older charts, so both version branches shared the `interruptionQueueName` spelling. On charts >= v0.33.0 Helm silently ignores that unknown key, so `INTERRUPTION_QUEUE` is never set on the Karpenter pod and spot interruption handling is disabled with no error surfaced to the user. Instances are terminated without Karpenter draining them first. Select the queue key per version branch instead of sharing one map: the `< 0.33.0` path keeps `settings.aws.interruptionQueueName` unchanged, and the `>= 0.33.0` path now sends `settings.interruptionQueue`. No API, flag or documentation change; `withSpotInterruptionQueue` is untouched. The existing `>= 0.33.0` unit expectation encoded the wrong key, so it is corrected to the key the chart actually reads. The two `settings.aws.interruptionQueueName` specs are left as-is and act as the regression guard for the legacy contract. Signed-off-by: warren <warren.chen830@gmail.com>
Contributor
|
Hello warren830 👋 Thank you for opening a Pull Request in |
warren830
marked this pull request as ready for review
August 31, 2026 02:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Karpenter chart renamed its interruption queue Helm value when it flattened
settings, andeksctlstill sends the old spelling to new charts.Symptom. With
withSpotInterruptionQueue: trueand a Karpenter version >=0.33.0, spot instances are terminated without Karpenter draining them first.INTERRUPTION_QUEUEis absent from the Karpenter pod's environment and nothing is logged — the misconfiguration is completely silent.Root cause.
pkg/karpenter/karpenter.gobuilt a singlesettingsmap and then re-nested it underawsfor older charts:Because both branches shared that one map, both sent
interruptionQueueName. That is correct only for the pre-flattening charts. The flattened chart readssettings.interruptionQueue, so the key eksctl sends is unknown to it and Helm drops it without complaint. The chart gates the env var on that exact value:charts/karpenter/templates/deployment.yaml, unchanged from v0.33.0 through v1.2.1Chart key by version, read from
charts/karpenter/values.yamlat upstream tags:settings.aws.interruptionQueueNamesettings.interruptionQueue(plus a deprecated emptyaws: {})settings.interruptionQueueThe flattened key has never been spelled
interruptionQueueName.Change. Select the queue key per version branch instead of sharing one map. The
< 0.33.0path keepssettings.aws.interruptionQueueNamebyte-for-byte as before; only the>= 0.33.0path changes, tosettings.interruptionQueue. Deliberately minimal: no API, flag, CloudFormation or documentation change, andwithSpotInterruptionQueueis untouched.Verified behaviour across every supported version (min is
v0.20.0persupportedKarpenterVersion)karpenter.version0.20.0,v0.20.0,0.28.0,0.31.0settings.aws{…}interruptionQueueName— unchanged0.32.0,0.32.9settings.aws{…}interruptionQueueName— unchanged (see note below)0.33.0,v0.33.0settings{…}interruptionQueue— fixed0.34.0,0.37.0,1.0.0,1.2.1,v1.2.1settings{…}interruptionQueue— fixed0.33.0-rc.1settings.aws{…}interruptionQueueName— pre-release sorts below0.33.0; pre-existing boundary semantics, unchangedsettings{…}interruptionQueue— same partition as before (rejected earlier by validation anyway)The
vprefix is handled identically on both sides of the boundary. Hoisting the map into a local also removes the pre-existing aliasing where the same map object was reachable at bothsettingsandsettings.aws; eachInstallnow builds a fresh map, verified to not carry state across calls.Two adjacent questions left out of this PR on purpose — both pre-existing, neither introduced here, happy to file separate issues if you'd like them tracked:
0.32.xboundary. Upstream flattenedsettingsat chart v0.32.0, but eksctl pivots at0.33.0, so0.32.xstill receives the deprecatedsettings.aws.*shape. Independent of the key rename.pkg/karpenternever consultsWithSpotInterruptionQueue(zero references), so the cluster name is passed as the queue name even when the user setwithSpotInterruptionQueue: false— whilepkg/cfn/builder/karpenter.goonly creates the SQS queue when it is enabled. On the legacy path this has always been delivered (v0.31.0 flattenssettingsinto thekarpenter-global-settingsConfigMap). On>= 0.33.0the typo was masking it, so this fix makes that pre-existing behaviour effective there: a user on>= 0.33.0withwithSpotInterruptionQueue: falsewill now getINTERRUPTION_QUEUEset to a queue that was never created, and Karpenter will log SQS polling errors. That is the same behaviour the< 0.33.0path already has, so this PR brings the two branches to parity rather than diverging them — but gating the value onWithSpotInterruptionQueueis a real behaviour change affecting both branches, so I have not made that call here. Flagging it explicitly as a maintainer decision.Checklist
README.md, or theuserdocsdirectory)area/nodegroup) and kind (e.g.kind/improvement)Documentation: not applicable. The bug is entirely in the Helm value name eksctl sends internally; no user-facing field, flag, or default changes, so
userdocs/src/usage/eksctl-karpenter.mdstays accurate as written.Manual testing: not done — I have no AWS account able to stand up an EKS cluster with spot capacity, so I could not observe a real interruption end to end. What I did verify, entirely offline with no AWS/EKS/cluster/Docker/credentials:
>= 0.33.0unit expectation encoded the wrong key. Correcting it to the key the chart reads fails against unmodified production and passes with this change — a genuine RED → GREEN, not a test written to fit the code:karpenter.goreproduces the failure (3 passed / 1 failed); restoring it returnsok.go test -tags=release ./pkg/karpenter/→ok../pkg/actions/karpenter/,./pkg/cfn/builder/,./pkg/apis/eksctl.io/v1alpha5/,./pkg/utils/→ allok../pkg/...→ 85 packages ok, 2 failing:pkg/karpenter/providers/helmandpkg/iam/oidc. Both fail identically on unmodifiedmainin this environment (they reachpublic.ecr.awsand need a Docker credential helper), so they are environmental and unrelated to this diff.gofmtclean,go vet -tags=release ./pkg/karpenter/clean,go build -tags=release ./...clean, integration tests compile (go test -tags integration -run=^$ ./integration/...).go mod tidyproduces nogo.mod/go.sumdrift; regeneratingassets/schema.jsonproduces a byte-identical file, socheck-gomodandcheck-schemaare unaffected.charts/karpenter/values.yamlandtemplates/deployment.yamlat upstream tags v0.31.0, v0.32.0, v0.32.9, v0.33.0, v0.34.0, v0.37.0, v1.0.0 and v1.2.1 — not from memory.Verified against
mainat99984adb6164ece593c0728523b7f497ea61d60e.BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯
The version branch now owns the key choice rather than mutating a shared map after the fact, which is what allowed one spelling to serve two incompatible chart layouts in the first place.
Fixes #7697