systemd: lock disks while running filesystem checks by PawelWMS · Pull Request #18653 · microsoft/azurelinux

Pawel Winogrodzki (PawelWMS) · 2026-08-28T23:56:34Z

Alternative implementation

Important

This change is a systemd-owned alternative to native e2fsprogs whole-disk locking.

Do not ship both implementations. systemd-fsck retains its exclusive lock while waiting for child fsck; native e2fsprogs locking would block while acquiring a second independently opened lock on the same disk and deadlock boot.

What

Add an Azure Linux patch that acquires an exclusive whole-disk BSD lock inside systemd-fsck.
Retain the lock in the parent process while child fsck checks the filesystem.
Release the lock after the child exits.
Wait at most 60 seconds for the lock; warn and continue unlocked on timeout.
Keep systemd and systemd-boot-signed releases synchronized at 255-35.

Why

The root filesystem check can update ext4 metadata while systemd-udevd probes the same disk. If udev observes the primary superblock between its payload and checksum writes, libblkid can temporarily reject the filesystem and remove its filesystem UUID link. The generated root mount then cannot resolve the device and boot stops in the initramfs.

systemd-udevd already takes a shared whole-disk lock while probing. Holding an exclusive lock across the filesystem check makes udev defer and retry its event after filesystem metadata is consistent.

How it works

All systemd-managed filesystem checks converge in systemd-fsck. After resolving the exact block device and verifying that an fsck implementation exists, the patch:

Resolves the partition's parent whole disk.
Opens that disk node read-only with close-on-exec.
Revalidates that the opened node still has the expected block-device number.
Waits up to 60 seconds for a blocking LOCK_EX BSD lock.
On timeout, warns and continues without the lock to preserve previous behavior.
Otherwise, forks child fsck while the parent retains the lock descriptor.
Waits for child completion and then releases the lock.

Relationship to #18695

#18695 is a clean alternative based directly on 3.0-dev. It routes udev and systemd-fsck through shared block-device lock mechanics. This PR keeps the smaller private-helper implementation. Do not merge both.

Scope

This affects every filesystem check launched through systemd-fsck, not only initrd root and /usr units. Checks for partitions on the same physical disk now serialize on one whole-disk lock.

Risk

A whole-disk resolution, open, stat, or non-timeout lock failure fails the systemd-fsck unit.
Multiple filesystem checks on one disk are serialized, including non-rotating storage where util-linux's private fsck -l lock may not previously have serialized them.
If the lock remains unavailable for 60 seconds, the filesystem check proceeds unlocked after a warning, preserving previous behavior but reopening the original race.
This implementation must not be combined with native e2fsprogs whole-disk locking.

Verification

Final head: 836fa3cab7d3d44f36026c3734d9e6b71ae4ac5d (signed).
Applied the patch to pristine systemd-stable v255 sources with GNU patch --dry-run -p1 without fuzz or offsets.
Passed repository manifest, package metadata, source-signature, spec-entanglement, duplicate-SRPM, license, static, and merge-conflict checks.
Built systemd packages successfully on x86_64 and aarch64.
Passed all selected systemd package tests: 1,216 on x86_64 and 1,217 on aarch64, with zero failures.
The only failing package test is unrelated dasel on both architectures; it fails outside this package's scope while systemd builds and tests pass.
Rebuilt the exact head independently with the matching x86_64 CI toolchain and retained the resulting RPMs.
Installed the complete seven-package exact-head set, including systemd-rpm-macros, at 255-35 in one RPM transaction on an Azure Linux 3 VM running kernel 6.6.143.1-1.azl3.
Verified that the RPM transaction regenerated initramfs without a manual dracut step.
Verified identical host/initramfs systemd-fsck SHA-256: a6cb9409af4c464e51cc7dd0f31cf877080faf4e9243dc17fd1e276fd7167248.
Rebooted to a new boot ID; system state reached running, no units failed, and emergency.target remained inactive.
Verified uncontended lock acquisition and checker completion in 8 ms.
Held the whole-disk lock for longer than 60 seconds and verified that systemd-fsck warned after 60.010 seconds, continued to the checker while contention remained, and completed successfully.
Captured managed Boot Diagnostics serial output for the successful reboot.

Acquire the exclusive whole-disk lock in systemd-fsck before launching its child fsck process, and retain it until that process exits. This is a systemd-owned alternative to native e2fsprogs locking. The two implementations must not be combined because nested locks would deadlock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb0e6716-4886-4e95-9efe-71dcde8687d6

Ziwei Mao (ZzzMao) · 2026-09-01T20:42:58Z

+                                       "Path '%s' no longer refers to block device %u:%u.",
+                                       whole_disk, major(devno), minor(devno));
+
+        r = lock_generic(fd, LOCK_BSD, LOCK_EX);


This code might introduce a dead lock. The default timeout for systemd-fsck@ service is infinity. This might cause the fsck service to wait this lock forever.

This would not be an issue in udev. It uses LOCK_NB for non-blocking lock acquiring.

Dan Streetman (ddstreetmicrosoft) · 2026-09-02T16:46:15Z

This of course would be better if it was upstreamed first, as it may require changes based on upstream feedback.

The added function is mostly identical to udev's lock_device, and it might be better to move that into a common area instead of mostly duplicating it in fsck (upstream can provide feedback on this).

Also as Ziwei Mao (@ZzzMao) pointed out, leaving the LOCK_NB off might not be a good idea due to a deadlock, though using an arbitrary hardcoded timeout with retries doesn't seem much better. This really would be better with some feedback from upstream.

Otherwise, it looks fine to me.

Keep the package spec focused on patch registration and document the mutual-exclusion requirement at the lock acquisition site. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb0e6716-4886-4e95-9efe-71dcde8687d6

Keep downstream compatibility guidance in the patch description. Do not add it to the patched source. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb0e6716-4886-4e95-9efe-71dcde8687d6

Wait up to 60 seconds for the exclusive whole-disk lock. If the wait times out, emit a warning and run the filesystem checker without the lock to preserve previous boot behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb0e6716-4886-4e95-9efe-71dcde8687d6

Pawel Winogrodzki (PawelWMS) · 2026-09-03T23:40:34Z

-+        if (lock_fd < 0)
+        if (lock_fd == -ETIMEDOUT) {
+                log_warning(
+                                "Timed out waiting 60 seconds for whole block device lock for '%s'; proceeding without lock.",


Use a constant for the timeout, so we don't repeat "60" in two places in the code and potentially have them go out of sync.

microsoft-github-policy-service Bot added Packaging 3.0-dev PRs Destined for AzureLinux 3.0 labels Aug 28, 2026

Pawel Winogrodzki (PawelWMS) force-pushed the pawelwi/systemd-fsck-whole-disk-lock branch from 8f32f47 to bc45e8f Compare August 29, 2026 00:05

Pawel Winogrodzki (PawelWMS) force-pushed the pawelwi/systemd-fsck-whole-disk-lock branch from bc45e8f to aee4066 Compare August 31, 2026 14:27

Pawel Winogrodzki (PawelWMS) commented Sep 1, 2026

View reviewed changes

Comment thread SPECS/systemd/systemd-fsck-lock-whole-disk.patch Outdated

Ziwei Mao (ZzzMao) reviewed Sep 1, 2026

View reviewed changes

Dan Streetman (ddstreetmicrosoft) reviewed Sep 2, 2026

View reviewed changes

Comment thread SPECS/systemd/systemd.spec Outdated

Pawel Winogrodzki (PawelWMS) mentioned this pull request Sep 2, 2026

systemd: share whole-disk locking with filesystem checks #18695

Draft

Pawel Winogrodzki (PawelWMS) added 2 commits September 2, 2026 16:37

docs(systemd): move lock warning to patch header

c286104

Keep downstream compatibility guidance in the patch description. Do not add it to the patched source. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb0e6716-4886-4e95-9efe-71dcde8687d6

Pawel Winogrodzki (PawelWMS) commented Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

systemd: lock disks while running filesystem checks - #18653

systemd: lock disks while running filesystem checks#18653
Pawel Winogrodzki (PawelWMS) wants to merge 4 commits into
microsoft:3.0-devfrom
PawelWMS:pawelwi/systemd-fsck-whole-disk-lock

Pawel Winogrodzki (PawelWMS) commented Aug 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Ziwei Mao (ZzzMao) Sep 1, 2026

Uh oh!

Uh oh!

Dan Streetman (ddstreetmicrosoft) commented Sep 2, 2026

Uh oh!

Pawel Winogrodzki (PawelWMS) Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Pawel Winogrodzki (PawelWMS) commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Alternative implementation

What

Why

How it works

Relationship to #18695

Scope

Risk

Verification

Uh oh!

Uh oh!

Ziwei Mao (ZzzMao) Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Dan Streetman (ddstreetmicrosoft) commented Sep 2, 2026

Uh oh!

Pawel Winogrodzki (PawelWMS) Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pawel Winogrodzki (PawelWMS) commented Aug 28, 2026 •

edited

Loading