engineering: make Azure Container Linux images installable (inline dm-verity, ESP staging) - #754
Draft
Paco Huelsz (frhuelsz) wants to merge 9 commits into
Draft
engineering: make Azure Container Linux images installable (inline dm-verity, ESP staging)#754Paco Huelsz (frhuelsz) wants to merge 9 commits into
Paco Huelsz (frhuelsz) wants to merge 9 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
Paco Huelsz (frhuelsz)
changed the title
…ions
Azure Container Linux images use *inline* dm-verity: the hash tree lives
inside the same partition as the data, at a byte offset, rather than in a
dedicated hash partition. `trident grpc-client stream-disk` could not
install such an image.
Derivation resolved the verity hash partition purely by image path. With
inline verity the verity entry points at the same image as the filesystem,
so it produced a VerityDevice whose data and hash devices were the same
partition, which the storage graph then rejected:
Derived Host Configuration is invalid: Referrer 'verity-1' of kind
'verity-device' references target 'partition-2' more than once
Trident also had nowhere to record where the hash tree began, and never
passed an offset to `veritysetup open`.
Changes:
- Read `hashOffset` from COSI verity metadata. Image Customizer already
emits this field for inline layouts (see its `cosiapi` package), so this
only consumes what the producer declares; the COSI spec and schema are
unchanged.
- Carry the offset on the OS image next to the root hash. Like the root
hash, it is a property of the image rather than of the Host
Configuration, so it is read from image metadata at servicing time and
does not appear in the Host Configuration or Host Status.
- Allow a verity device to name the same partition as both its data and
hash device. `hashDeviceId` remains required, so the Host Configuration
schema is unchanged apart from documentation. The storage graph models
the inline case as the single device it is, rather than as a second
reference to the same node, which leaves the duplicate-target and
referrer-sharing invariants untouched. Verity referrer cardinality
becomes 1..=2, and the hash partition type cross-check is skipped when
there is no distinct hash partition.
- Pass `--hash-offset` to `veritysetup open`, so an inline-verity device is
actually opened and verified during servicing.
Derivation fails with a clear error naming `hashOffset` when an image uses
inline verity but does not say where the hash tree starts, rather than
producing a device that cannot be opened.
Tested against a real ACL image end to end: derivation, graph construction
and full Host Configuration validation pass; `stream-disk` writes all five
partitions; verity activates and reports `verified`; and the installed
system boots with systemd activating dm-verity on /usr from the UKI command
line.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inline dm-verity is not an officially supported configuration, so the public Host Configuration reference should not describe it. Reverts the `hashDeviceId` doc comment to its original wording. The generated `VerityDevice.md` and `host-config-schema.json` are now byte-identical to main, so this change makes no alteration to Trident's documented public API surface. Support itself is unaffected: inline verity is still derived, opened with `--hash-offset`, and verified. The behaviour is simply not advertised. The only remaining generated-doc difference is the verity-device row of the referrer cardinality table in Storage-Rules.md (2..2 becomes 1..2). That table is generated from the cardinality rule itself and states a reference count without mentioning inline verity; suppressing it would mean reverting the rule and reintroducing the graph invariant exemptions this design deliberately avoids. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Image Customizer has emitted `hashOffset` since azure-linux-image-tools#702
("Add support for inline verity"), where it is declared in its `cosiapi`
package and written only for images whose verity hash tree is stored inside
the data partition. COSI 1.2 documents describe the field nowhere, so the
spec has been behind its producer.
Document it: add `hashOffset` to the `VerityConfig` table, note that the
`image` field refers to the same image as the filesystem's own for inline
verity, add a sample, and record the field in the 1.2 changelog.
The addition is descriptive rather than a format change. `hashOffset` is
optional and absent for the ordinary separate-hash-partition layout, so
every existing COSI remains valid and consumers that ignore the field remain
correct.
Extends the schema gate with an inline-verity sample under
tests/cosi/metadata_samples/v1.2/valid/, plus a negative sample asserting
the offset cannot be negative. Verified with check-jsonschema, as the CI
workflow does; also confirmed the metadata of a real inline-verity image
validates against the updated schema.
Note this documents the COSI format Trident consumes. It does not present
inline verity as a supported Host Configuration: the Host Configuration
reference and its schema are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`trident install` could not install an Azure Container Linux image. After
partitioning the disk and writing the usr, oem and root filesystems, it
failed in Provision:
Failed to perform file-based deployment of ESP images
Failed to create a temporary file
File-based ESP deployment stages the image through
`<newroot>/var/tmp`, whose comment claims the location "is generally
guaranteed to be writable and backed by a real block device". That does not
hold for an immutable image: ACL's root filesystem ships only `usr`, `oem`,
`boot` and `lost+found` plus a few symlinks, with no `/var` at all, and
leaves systemd-tmpfiles to populate it on first boot. `NamedTempFile` then
has no directory to create into, and servicing fails after the disk has
already been repartitioned.
Create the directory when it is missing, with the sticky, world-writable
mode a conventional `/var/tmp` has, so systemd-tmpfiles finds what it
expects on first boot. An existing directory is left untouched, so images
that do ship one keep whatever mode they chose. The path computation and
creation move into `ensure_esp_extraction_dir` so both cases can be tested.
`stream-disk` was unaffected, as it writes the ESP partition image directly
rather than staging it through a file.
Also reword the neighbouring space check. It reported "Failed to check if
there is enough space available" whenever the backing device size was
unknown, which reads like a malfunction; the usual cause is simply a
partition configured to grow, whose size is not known before servicing. Say
that instead, and note the check was skipped rather than passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Installing any currently published Azure Container Linux image failed
during Provision:
Verity addon template 'verity-a.addon.efi' not found in
'.../acl/uki-addons' — cannot activate VolumeA
`activate_verity_addon_for_target_volume` decided whether an image uses
PARTUUID-based verity addons by testing whether `acl/uki-addons/` exists.
That directory is shared, though: it also holds the first boot, fips and
kdump addons, as its own doc comment says. Published ACL images ship those
three and no verity templates, so the check passed and the function went on
to demand a template the image never had.
Key the decision on the templates themselves instead. If the selected slot's
template is absent, the image only fails when the *other* slot's template is
present, since that is what shows the image really does use per-slot verity
addons and that booting would otherwise pick up the wrong slot's PARTUUIDs.
When neither is present the image simply does not use them and there is
nothing to activate, which is the behaviour the doc comment already
described.
The safety property is unchanged, and its test with it: shipping verity-a
while installing to slot B is still an error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Follow-up to 8da7273, which keyed the no-op on neither per-slot verity addon template being present. That test is a proxy: it infers an image's intent from a missing file, so an image that should ship per-slot addons but shipped none by mistake looks exactly like one that legitimately does not use them, and would be activated silently rather than rejected. Gate on the structural reason the exemption is safe instead. A per-slot addon exists to carry its slot's verity data/hash PARTUUID pair; an image whose verity is inline has no such pair, since the hash tree lives in the data partition and both refer to one device. There is nothing for an addon to select, so activation does not apply. That is a property of the image that can be established positively rather than read out of an absence. With a separate hash partition the check stays strict, including when the image ships no templates at all: such an image is meant to carry per-slot PARTUUIDs, and shipping none is a packaging fault rather than an exemption. That is the case ACL is moving towards, and where the error should keep firing. Inline-ness comes from `VerityDevice::is_inline`, the same predicate the rest of the inline verity support keys on, so the two stay consistent. An image that both uses inline verity and ships per-slot templates is contradictory; it warns and continues, since there is still nothing to activate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Installing an Azure Container Linux image failed in Configure:
Failed to generate fstab at path '/etc/fstab'
Trident already skips storage configuration for UKI images using
root-verity, on the reasoning that such an image takes its mount topology
from the signed kernel command line rather than from fstab. ACL is the same
case with the verity on /usr instead of /, but the guard tested only
root-verity, so it did not apply.
Writing the fstab would have been wrong even where it succeeded. ACL
assembles /etc as an overlay, with the factory /usr/share/distro/etc as the
lower layer and the ROOT partition's /etc as the upper, so a file written
there wins over the image's own. The factory fstab is deliberately empty,
every mount ACL needs already comes from the command line (/usr and /) or a
shipped unit (/boot and /oem), and systemd-fstab-generator writes into
/run/systemd/generator, which takes precedence over /usr/lib/systemd/system.
A generated fstab would therefore have shadowed the units the image ships
rather than merely duplicating them.
Skip fstab generation for these images. Only fstab is skipped, not the whole
step as in the root-verity case: a usr-verity image still has a writable
root, so encryption and RAID configuration remain applicable.
The mechanism was confirmed against the shipping ACL image, whose initrd
mounts the overlay in dracut module 99setup-root and creates the ROOT /etc
and workdir immediately beforehand. The absence of /etc in the image is
therefore expected rather than a packaging fault, and not something for
Trident to repair.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A hash offset only means anything when the hash tree shares the data image. Producers emit the two together — Image Customizer sets `hashOffset` only for inline verity, and gives a distinct image path otherwise — so an offset accompanying a distinct hash image is contradictory. Derivation ignored the offset in that case, which would have set the device up as though it had never been given, and failed later at `veritysetup open` with nothing pointing at the cause. Warn instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
An Azure Container Linux system installed with `trident install` did not boot. Its initrd waits for `/dev/disk/by-label/EFI-SYSTEM`, which never appears, and the boot ends in emergency mode. Every other filesystem from the OS image is written to its partition verbatim, so its label arrives with the rest of the bits. The ESP is the exception: it is deployed file by file onto a filesystem Trident creates itself, and `mkfs` gives that filesystem no label, so the image's label is lost. `stream-disk` is unaffected, as it writes the ESP partition image whole. Copy the label from the extracted ESP image onto the filesystem, which leaves the ESP as it would have been had it been written wholesale. The label is read from the image itself rather than from COSI metadata, which carries a filesystem UUID but no label. An image whose ESP is unlabelled leaves the new filesystem unlabelled too, which is equally faithful. `mkfs.vfat` can only set a label at creation time, and the label is not known then: the ESP image is not extracted until deployment, well after filesystems are created. So it is applied afterwards with `fatlabel`, which writes the boot sector and may be used on a mounted filesystem, the label surviving the unmount. Adds `fatlabel` to the dependency registry, an `osutils::fatlabel` wrapper that also enforces FAT's 11 character limit, and a filesystem-label reader alongside the existing partition-label one in `osutils::blkid`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Paco Huelsz (frhuelsz)
force-pushed
the
user/frhuelsz/inline-verity
branch
from
September 2, 2026 00:33
78ffc25 to
d1cfc98
Compare
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔍 Description
engineering + bug. Makes Azure Container Linux (ACL) images installable. Two defects, both found installing the same image:
trident grpc-client stream-diskcannot install ACL. Inline verity stores the hash tree inside the same partition as the data, at a byte offset, rather than in a dedicated hash partition.<newroot>/var/tmpexists, sotrident installcannot install ACL either — failing after the disk has already been repartitioned.trident installrejects every currently published ACL image.They are separate faults, uncovered one behind the other as each was fixed.
stream-diskonly hits (1);installhits all three. None alone makes ACL installable.🤔 Rationale
Streaming a real ACL image failed during derivation:
Derivation resolved the verity hash partition purely by image path. With inline verity the verity entry points at the same image as the filesystem, so it emitted a
VerityDevicewhose data and hash devices were the same partition, which the storage graph rejected. Trident also had nowhere to record where the hash tree began, and never passed an offset toveritysetup open.This was not a regression — inline verity was never supported.
git log --all --grep=inlineis empty and nothing referencedhashOffset. #673 made ACL work for A/B update, but that path never derives a Host Configuration; it reads the root hash fromusrhash=on the UKI cmdline and mounts/usrdirectly, so it never had to model the layout.Inline dm-verity (1)
hashOffsetfrom COSI verity metadata. Image Customizer already emits this field for inline layouts — it is declared in itscosiapipackage and was added upstream by azure-linux-image-tools#702 ("Add support for inline verity"), gated so it appears only when inline, and covered by an IC test. The encoding is intentional and stable on the producer side, confirmed against IC's source:HashOffsetis set only when the hash tree is inline (cosicommon.go:199-209), and the separate-hash layout instead yields a distinct image path with the field omitted. So the two shapes are cleanly discriminable, andverity.image.path == image.pathtogether with a presenthashOffsetis precisely how IC expresses inline verity. It has behaved this way in every release since v1.4.0 (April 2026) with no COSI version bump, so upgrading IC does not avoid this — a consumer that wants to install inline-verity images has to read the field.This change consumes what the producer already declares, and brings Trident's COSI reference up to date with it:
hashOffsetis now documented inComposable-OS-Image.md, the v1.2 JSON Schema, and the changelog. The addition is descriptive and backward compatible — the field is optional and absent for the ordinary separate-hash-partition layout, so every existing COSI stays valid.hashDeviceIdstays required, so the Host Configuration schema is completely unchanged —host-config-schema.jsonand the generatedVerityDevice.mdare byte-identical tomain. The graph models the inline case as the single device it is rather than as a second reference to the same node, which leaves the duplicate-target and referrer-sharing invariants untouched. Verity referrer cardinality becomes1..=2, and the hash partition-type cross-check is skipped when there is no distinct hash partition.--hash-offsettoveritysetup open, so an inline-verity device is actually opened and verified during servicing rather than skipped.Two documentation sets are treated differently on purpose. The COSI reference describes the format Trident consumes, so it documents
hashOffsetas a factual property of images IC already produces. The Host Configuration reference describes what users may configure, and inline dm-verity is not an officially supported configuration, so it is deliberately not documented there —host-config-schema.jsonand the generatedVerityDevice.mdare byte-identical tomain. The only generated-doc change is the verity-device row of the referrer cardinality table inStorage-Rules.md(2..2→1..2), which states a reference count without mentioning inline verity.Derivation fails with a clear error naming
hashOffsetif an image uses inline verity but does not say where the hash tree starts, instead of producing a device that cannot be opened.ESP staging directory (2)
deploy_espstages the ESP image through<newroot>/var/tmp, on the comment's reasoning that the location "is generally guaranteed to be writable and backed by a real block device" — deliberately avoiding holding a potentially large image in RAM. That assumption does not hold for an immutable image. ACL's root filesystem ships onlyusr,oem,bootandlost+foundplus symlinks, with no/varat all, leaving systemd-tmpfiles to populate it on first boot:NamedTempFile::new_inthen has nothing to create into and servicing fails in Provision, after repartitioning.The directory is now created when missing, with the sticky, world-writable mode a conventional
/var/tmphas, so systemd-tmpfiles finds what it expects on first boot. An existing directory is left untouched, so images that do ship one keep whatever mode they chose.The neighbouring space check is also reworded: it reported
Failed to check if there is enough space availablewhenever the backing device size was unknown, which reads like a malfunction. The usual cause is a partition configured togrow, whose size is not knowable before servicing — including on plain Azure Linux. It now says so, and makes clear the check was skipped rather than passed.Verity addon guard (3)
activate_verity_addon_for_target_volumedecided whether an image uses PARTUUID-based verity addons by testing whetheracl/uki-addons/exists, then hard-errored if the selected slot's template was missing. That directory is shared — its own doc comment says it holds "per-slot verity addons (verity-a/verity-b) and the first-boot addon" — and published ACL images ship first-boot, fips and kdump there while shipping no verity templates:So the guard passed and the function demanded a template the image never had, failing installation of every published ACL image. Per-slot verity addons are ACL base work that has not landed yet.
The decision now keys on whether the image's verity is inline. A per-slot addon exists to carry its slot's verity data/hash PARTUUID pair; inline verity has no such pair, since the hash tree lives in the data partition and both refer to one device, so activation has nothing to select and is skipped.
Gating on inline-ness rather than on "no templates present" matters: the latter infers intent from a missing file, so an image that should ship per-slot addons but shipped none by mistake would be indistinguishable from one legitimately exempt, and would be activated silently instead of rejected. Inline-ness is a property that can be established positively.
With a separate hash partition the check therefore stays strict — including when no templates are shipped at all, which is a packaging fault rather than an exemption. That is the direction ACL is heading, and where the error should keep firing. The pre-existing safety test (ships
verity-a, installs to slot B → error) is unchanged and still passes.Inline-ness comes from
VerityDevice::is_inline(), the same predicate defect (1) keys on, so the two features stay consistent by construction.fstab on UKI usr-verity images (4)
Trident already skips storage configuration for UKI images using root-verity, on the reasoning that such an image takes its mount topology from the signed kernel command line rather than from fstab. ACL is that same case with the verity on
/usr, but the guard tested only root-verity, so it did not apply andgenerate_fstabran — failing, because ACL's ROOT image ships no/etc.Creating the directory would have been the wrong fix. ACL assembles
/etcas an overlay — factory/usr/share/distro/etcas lower, the ROOT partition's/etcas upper — so anything written there wins over the image's own. Confirmed against the shipping image:/usr/share/distro/etc/fstabis deliberately empty (0 bytes);/usr,/) or a shipped unit (/boot,/oem);systemd-fstab-generatoris present, and emits into/run/systemd/generator, which takes precedence over/usr/lib/systemd/system.A generated fstab would therefore have shadowed the units the image ships, not merely duplicated them. The absence of
/etcis likewise expected rather than a fault: ACL's initrd creates it immediately before mounting the overlay, in dracut module99setup-root.Only fstab is skipped, not the whole step as in the root-verity case: a usr-verity image still has a writable root, so encryption and RAID configuration remain applicable.
📝 Checks
Both defects were found and re-validated on real ACL images. Manually validated end to end on a real ACL image (
acl-base-3.0.20260706), installed viastream-diskfrom an Azure Linux installer ISO built with RPMs from this branch, in QEMU/KVM:HostConfiguration::validate()passesServicing completed successfully; rebootverified6.6.143.1, with systemd activating dm-verity on/usrfrom the UKI cmdline (sysusr-usr.mount,/dev/mapper/usr, zero mount failures), 38 targets reachedFirst boot then stops at
emergency.targetbecause Ignition cannot reach Azure IMDS (169.254.169.254) — expected off-Azure for an image carryingflatcar.oem.id=azure, and downstream of everything this PR touches (verity was already mounted and verified before Ignition ran).Before writing any code, the underlying
veritysetupinvocation was confirmed against the real disk image:Automated: 22 test suites pass workspace-wide,
cargo fmt --checkclean,cargo clippyintroduces no new warnings. Addstest_derive_host_configuration_inner_inline_verity(pinning ACL's realusrpartition type GUID, which exercises partition-type rules a generic Linux GUID would mask) and..._inline_verity_missing_offset.📌 Follow-ups
TODO:
osPackagesto be an array, but real ACL COSI files set it tonulland Trident already tolerates that (acl COSI files do not contain ospackages #707). Validating an ACL image's metadata against the checked-in schema fails onosPackagesalone — a pre-existing divergence, unrelated to this PR and left alone here.🗒️ Notes
Opened as a draft: functionally validated end to end, but wanted review on the graph modelling choice before marking ready.
The main design question is how inline verity is represented. Two alternatives were built and discarded:
/usrread-only straight off the partition and leaving verity to systemd at boot. Smallest change and it validates, but Trident never verifies the image it just wrote. It also has a trap: the non-verity path drops theromount option, andresizeis skipped only whenis_read_only() || !fs_type.is_ext(). ACL survives on thebtrfsclause alone; an ext4 inline-verity image would be resized and have its appended hash tree silently destroyed.hash_device_id: Option<BlockDeviceId>. Cleaner conceptually, but it is a Host Status wire-format change that rippled through ~30 files.The approach here keeps the Host Configuration and Host Status untouched while still verifying at install time.