bug: configure an OS whose /etc is assembled at boot without breaking it

[Paco Huelsz (frhuelsz)]

🔍 Description

bug. Two fixes for configuring an OS whose /etc is assembled at boot rather than shipped in the image. Azure Container Linux does this; configuring a single user was enough to make an installed system unbootable.

Stacked on #754, which is this PR's base, so the diff shows only the two commits below. It is stacked because ACL cannot be installed at all without #754, so these fixes are untestable without it. Merge #754 first; this will then retarget to main cleanly.

🤔 Rationale

ACL ships no /etc on its root filesystem. Its initrd assembles one at boot as an overlay: the factory contents on the sealed /usr as the lower layer, the root filesystem's own /etc as the upper.

Servicing runs before any of that. A chroot into the new root sees an empty /etc, so anything that rewrites a file the image provides writes a replacement rather than an edit. At boot that replacement becomes the upper layer and hides the factory file.

Configuring one user left the installed system with a 43-byte /etc/passwd containing only that user — no root, no core:

matrix:!:1000:1000::/home/matrix:/bin/bash

and the boot ended in an emergency shell:

systemd-tmpfiles: Failed to resolve user 'core': No such process
[FAILED] Failed to start initrd-setup-root.service - Root filesystem setup.
[DEPEND] Dependency failed for initrd.target

1. Refuse when there is no user database

useradd creates /etc/passwd from scratch when none exists. Without an existing database there is nothing meaningful to add users to, so refuse rather than produce one that cannot boot. This does not make the feature work; it stops it destroying the system.

Worth keeping even with the overlay fix below: it is distro-agnostic, and it turns any future recurrence into a clear message rather than an unbootable machine.

2. Give servicing the same /etc the system boots with

Mount the same overlay before entering the chroot, so servicing sees what the booted system will.

Why this is lower-risk than it looks: the overlay's upper layer is the root filesystem's /etc — exactly where Trident already writes. So the on-disk result is unchanged for everything that already worked. hostname, netplan, modules, services and additionalFiles only add files, and overlayfs merges directories. What changes is that modifying a file the image provides now copies it up from the factory version first, so it stays complete.

It also fixes more than users: any configuration rewriting a factory /etc file was subject to the same hazard, /etc/selinux/config among them.

Applied to both paths that chroot — clean install and A/B update. Images not using this layout are unaffected: the overlay is mounted only for ACL, and only when the factory directory is present.

📝 Checks

• Check dev-docs/manual-validation.md

Verified on a real ACL image via make run-netlaunch, installing and booting in QEMU/KVM.

Before — install fails, or bricks:

usermod: group 'wheel' does not exist          # factory /etc/group not visible
/etc/passwd: 1 line, 43 bytes                  # factory passwd replaced
boot -> emergency shell

After — install succeeds and the system boots:

/etc/passwd: 24 lines
root     root:x:0:0:root:/root:/bin/bash
core     core:x:500:500:Inert Identity:/home/core:/bin/bash
matrix   matrix:x:1000:1000::/home/matrix:/bin/bash
wheel:x:28:matrix
id matrix -> uid=1000(matrix) groups=1000(matrix),28(wheel)

Confirmed the overlay is unmounted and its work directory removed; the /.etc-work present at runtime is ACL's own, recreated by the initrd (findmnt shows the live /etc overlay using it).

Automated: 22 test suites pass, cargo fmt --check clean, cargo clippy introduces no new warnings.

📌 Follow-ups

TODO:

• A user created with no password is locked (matrix:! in shadow), so key-based SSH is refused by PAM. That is deliberate — Trident locks accounts with no password configured — but it means sshPublicKeys alone does not yield a usable login. Worth documenting rather than changing.

🗒️ Notes

Opened as a draft pending #754.

Found while surveying which os features work on ACL. The same survey shows hostname, netplan, modules, services and additionalFiles already work, and kernelCommandLine fails cleanly because it is implemented via /etc/default/grub, which a systemd-boot/UKI image does not have.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.