[3.0] Soft lockups / node hang in cgroup writeback (inode_switch_wbs)

[tsolodov]

https://bugzilla.kernel.org/show_bug.cgi?id=217965

Environment

• Azure Linux version and/or image name: Microsoft Azure Linux 3.0 (AKS node image)
• Kernel version: 6.6.141.1-1.azl3
• Architecture: x86_64
• Affected package(s) and version(s), if applicable: kernel 6.6.141.1-1.azl3 (no specific userspace package; containerd/kubelet are victims, not causes)
• Custom image or package modifications, if applicable: none - stock AKS node image

Deployment / platform
Where are you running Azure Linux? Check all that apply and include details:

• Azure VM
• AKS
• Azure Linux container image
• Other (please describe)

Details: AKS cluster, kubelet v1.34.7, VM size Standard_D96ds_v5 (96 vCPU, 378 GiB), region eastus2. The workload is ClickHouse Cloud database pods.

Describe the bug

Deleting Kubernetes pods that hold millions of small files (8KB-5MB, hundreds of GB up to 1TB+ per pod) on the k8s ephemeral storage disk causes a kernel-side storm in the cgroup writeback code. When a pod is deleted, kubelet mass-unlinks the files and destroys the pod's cgroup; with cgroup-aware writeback (CONFIG_CGROUP_WRITEBACK) every inode charged to the dying cgroup's writeback context must be switched to the parent context (inode_switch_wbs). With millions of inodes this floods kernel workqueues, the workers contend on spinlocks, and the node soft-locks.

We believe this is the known 6.x cgroup writeback bug analyzed publicly here: https://dasl.cc/2025/01/01/debugging-our-new-linux-kernel/

We previously hit what we believe is the same bug on another cloud provider (their node OS ran a 6.6.113+ kernel). That provider reproduced it with our steps, confirmed the kernel bug as root cause, and resolved it by shipping a patched kernel in newer node OS images. The Azure Linux kernel 6.6.141.1 appears to still be affected.

Steps to reproduce

It is not reproducible in 100% cases.

1. On a large AKS node (we used 96-128 vCPU sizes) whose temporary storage disk backs kubelet's root dir / emptyDir volumes, run several pods (we used 4) that each write millions of small files (8KB-5MB) into an emptyDir volume until the disk is largely full.
2. Delete the pods.
3. Observe: pod deletion hangs with KillPodSandboxError: context deadline exceeded, load average climbs into the hundreds with near-zero user CPU, top shows hundreds of kworker/*+inode_switch_wbs threads at 100% CPU, kernel logs soft lockups, containerd stops responding, and the node becomes unreachable until rebooted.

Expected behavior

Pod deletion should not kill AKS node.

Actual behavior

The node soft-locks in the writeback path and never recovers on its own:

• kernel watchdog: BUG: soft lockup - CPU#10 stuck for 29s! [kworker/u194:22:3749522]
• containerd down: containerd service is not running, exit code: 1 (repeatedly, over ~40 minutes)
• pods stuck Terminating/Pending, new containers fail to start
• SSH to the node impossible; kubectl drain does not complete; only a VM reboot recovers the node
• after the (hard) reboot: EXT4-fs error (device sdb1): ext4_orphan_get:1430: comm mount: bad orphan inode 124781025 on the temporary storage filesystem

Screenshots or terminal output

top captured during our reproduction of the same bug on a 128-core node on the other provider (minutes after deleting the test pods, with no pod activity at all):

top - 05:55:12 up  1:35,  0 user,  load average: **339.68, 353.00, 514.53**
Tasks: 1965 total, 263 running, 1702 sleeping, 0 stopped, 0 zombie
%Cpu(s):  0.3 us, 29.6 sy,  0.0 ni, 66.5 id,  3.7 wa

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
42994 root      20   0       0      0      0 R 100.0   0.0   0:21.97 kworker/40:10+inode_switch_wbs
43240 root      20   0       0      0      0 R 100.0   0.0   0:05.26 kworker/39:9+inode_switch_wbs
 1081 root      20   0       0      0      0 R 100.0   0.0   0:25.53 kworker/98:1+inode_switch_wbs
 1093 root      20   0       0      0      0 R 100.0   0.0   0:37.26 kworker/108:1+inode_switch_wbs
 3638 root      20   0       0      0      0 R 100.0   0.0   0:40.98 kworker/56:2+inode_switch_wbs
(dozens more identical inode_switch_wbs kworker entries)

Logs

Occurrence on Azure Linux, 2026-08-13, (all times UTC):

Kubernetes node events (node-problem-detector):

Warning  SoftLockup              kernel-monitor   watchdog: BUG: soft lockup - CPU#10 stuck for 29s! [kworker/u194:22:3749522]
Warning  KernelBug               kernel-monitor   watchdog: BUG: soft lockup - CPU#10 stuck for 29s! [kworker/u194:22:3749522]
Warning  ContainerRuntimeIsDown  container-runtime-custom-plugin-monitor  containerd service is not running, exit code: 1   (x7 over 37m)

Stall stack traces from syslog (~22:37 UTC), all in the writeback path:

__pv_queued_spin_unlock_slowpath+0xa0/0xd0
__writeback_single_inode+0x44/0x360
writeback_sb_inodes+0x209/0x500
__writeback_inodes_wb+0x54/0xf0
wb_writeback+0x2af/0x320
wb_workfn+0x390/0x500
? wb_update_bandwidth+0x53/0x70
process_one_work+0x18e/0x3a0
worker_thread+0x19f/0x330
kthread+0xf6/0x130
ret_from_fork+0x41/0x60

Kubelet-side failure while the node was wedged:

Warning  FailedKillPod  kubelet  error killing pod: failed to "KillPodSandbox" for "..." with
KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"

After reboot:

Warning  Ext4HasError  kernel-monitor  Node condition Ext4Error is now: True, reason: Ext4HasError,
message: "EXT4-fs error (device sdb1): ext4_orphan_get:1430: comm mount: bad orphan inode 124781025"

[eric-desrochers]

Thanks for bringing this bug to our attention. I filed an internal ADO bug with our dev team about this. (Reference: 63624146 )