Document GPU node groups on Talos workers: kernelModules, per-node-group schematicID, and the Blackwell open-modules requirement

[mattia-eleuteri]

What is missing

Nothing on the site documents how a GPU node group is meant to work on the Talos workers that Cozystack 1.6.0 introduced for tenant Kubernetes clusters. Two specific gaps, both of which cost real debugging time in a production 1.6.0 environment (upstream cozystack/cozystack#3563):

1. Kernel modules have to be declared. A Talos system extension installs a kernel module but does not load it — that is machine.kernel.modules' job. Until the upstream PR linked below there was no values surface for it at all; now there is one, and it needs documenting on the Kubernetes and Kubernetes Nodes reference pages:

• Kubernetes: nodeGroups.<name>.kernelModules
• Kubernetes Nodes: kernelModules at the root, next to gpus

The reference tables regenerate from each package's README.md, so the field descriptions arrive on their own for the apps hardcoded in the website Makefile. What does not arrive is the behaviour a prose page has to state: leaving the field unset makes the chart load nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset on any node group holding an nvidia.com/* GPU, a non-empty list replaces that set outright, and an explicit [] opts out entirely. That three-state behaviour is not derivable from a generated parameter table.

2. The Blackwell schematic constraint is documented nowhere, and the failure is silent. On Blackwell (GB202, PCI ID 10de:2bb5) the worker schematic must carry siderolabs/nvidia-open-gpu-kernel-modules-production. With the proprietary siderolabs/nonfree-kmod-nvidia-production extension the module loads cleanly, /dev/nvidia0 appears, and nvidia-smi -L on the host then reports No devices found — no error in dmesg, in the kubelet log, or anywhere else. Neither the Cozystack docs nor the Talos docs mention this, and there is nothing in the observable state to suggest the extension is the problem. This cost us about an hour; a single sentence on the page would have cost nothing.

Related: two debugging notes worth stating wherever the GPU worker path is described, both of which are non-obvious on Talos:

• Check the host first with chroot /host /usr/local/bin/nvidia-smi -L. Talos has no /bin/sh in the host root, so binaries have to be invoked directly rather than through a shell.
• CDI injection happens at pod admission, so tenant pods started before a node is fixed keep their old (GPU-less) environment and must be recreated.

Where it belongs

content/en/docs/next/ — the reference pages for the two apps, plus whichever GPU/node-group prose page is the right home for the schematic constraint. #561 already asks for GPU passthrough documentation for tenant Kubernetes clusters, but it predates the Talos worker rollover and covers the gpu=on auto-label and NvLinkDisable default rather than anything above; the kernel-module surface and the Blackwell extension constraint are new ground. Merge this into #561 if a maintainer prefers to keep the GPU documentation in one place.

Worth considering for backport to the released v1.6 documentation directory as well: the Blackwell constraint applies to 1.6.0 as shipped, independently of the upstream PR, since it is a property of the extension rather than of the chart.

Upstream

The kernel-modules values surface: see the linked upstream PR. Acknowledged by Aleksei Sviridkin (@lexfrei) as an undocumented behaviour. Filed as the downstream follow-up required by the upstream PR checklist.

[mattia-eleuteri]

Upstream PR: cozystack/cozystack#3571

[mattia-eleuteri]

The upstream PR grew a second field, and this one needs documenting more than the first, because the failure it fixes is invisible and the diagnosis is counter-intuitive.

schematicID per node group (nodeGroups.<name>.schematicID on the Kubernetes app, schematicID at the root on Kubernetes Nodes) overrides the cluster-wide talos.schematicID for one node group, applying to both its boot disk image and its Talos installer image.

What should be written down, beyond the generated parameter table:

A Talos schematic is a fixed set of system extensions baked into one image, and Talos refuses to finish the boot sequence when an extension service in it cannot start. ext-nvidia-persistenced and ext-nvidia-cdi-gen both require an NVIDIA card. So a node with no GPU that boots an NVIDIA schematic fails startAllServices and reboots — roughly every 70 minutes, indefinitely. Since talos.schematicID was cluster-wide, any cluster mixing GPU and non-GPU node groups had no correct value: the GPU group needs the extensions and every other group is broken by them. The guidance is therefore to set the NVIDIA schematic on the GPU node group only, never cluster-wide, and that sentence is the one worth having on the page.

The diagnosis notes matter as much as the guidance, because every obvious signal points the wrong way:

• The node reports Ready for the whole cycle. kubelet starts before the failing phase, so kubectl get nodes shows a healthy cluster.
• Every pod on the node dies at once (SandboxChanged, TaintManagerEviction), so all pods on one node carry an identical restart count. That symmetry is the tell.
• kubectl top nodes can show 96% memory just before the reboot, which reads as an OOM. It is page cache; min_over_time(node_memory_MemAvailable_bytes[30h]) never drops near the limit. The discriminant is regularity — several nodes rebooting within seconds of the same uptime is a timer, never memory pressure.
• The boot failure is only visible on the serial console (virtctl console, staying attached across a reboot), and the console is silent between boots.

A useful cross-check to document: compare dataVolumeTemplates[].spec.source.http.url on each node group's KubevirtMachineTemplate against the install.image of its TalosConfigTemplate. If the schematic differs between the two for a group, that group is booting one extension set and would upgrade into another.

Related and distinct, worth a cross-reference so readers do not conflate them: a ~70 minute worker reboot cycle can also come from apid never obtaining its certificate when port 50001 is missing from the control plane Service. Check that port before concluding it is the schematic.

[mattia-eleuteri]

Second upstream PR for this: cozystack/cozystack#3585, which changes the GPU Operator addon default for tenant clusters to the OS-provided-driver shape (driver and toolkit disabled, driver validator pointed at the host root, device plugin switched to CDI).

It changes behaviour the docs describe, so it belongs in the same documentation pass as the kernel modules and the schematic:

• Tenant GPU pods get their CUDA libraries through CDI, not through a default nvidia containerd runtime. Worth stating plainly, because it explains the one operational step that surprises people: CDI injection is decided at pod admission, so pods created before a node is working never receive anything and must be recreated.
• Operators who worked around the old default by hand, through addons.gpuOperator.valuesOverride, can drop those overrides once the release carrying #3585 lands.
• One override footgun worth a sentence: mergeOverwrite replaces a list rather than merging it, so an override that adds a single variable to devicePlugin.env silently drops the CDI defaults and GPUs stop working. The full list has to be restated.
• HAMi is explicitly not covered by #3585: with addons.hami.enabled the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring on Talos. If the docs describe HAMi on GPU node groups, that page should not imply the Talos path works yet.