feat(ci): attest CDN artifact uploads with build provenance by basvandijk · Pull Request #11323 · dfinity/ic

basvandijk · 2026-08-26T07:51:32Z

First producer-side step for security finding 3618194 (MEDIUM, CWE-494): the release pipeline has no integrity anchor independent of the serving CDN — tag-release.yml and ledger-suite-release.yml publish checksums computed from their own CDN downloads, and mainnet_revisions.py records CDN-served hashes into mainnet-icos-revisions.json.

ci/src/artifacts/upload.sh already hashes every file it uploads (from the local bytes, never from anything served back by the CDN) and prints one manifest line per file — but the upload-artifacts action discarded that output into the step summary. This PR switches the output to sha256sum format (<hex> <path>, path relative to the CDN root) and turns it into a durable, attested anchor.

Changes

.github/actions/upload-artifacts: converts the upload manifest to sha256sum format, validates its shape (fail closed on empty/malformed lines), exposes its own sha256 as an output, and uploads it as an Actions artifact (manifest-name input).
ci-main.yml: both upload jobs (bazel-test-all, upload-external-artifacts) produce named manifests; a new unprivileged upload-manifests job re-checks each one against the producing job's output sha256, merges them into a single upload-manifest artifact, and anchors it in its own job output. That the uploads happen in two jobs stays an implementation detail of ci-main.yml: callers see only the release-build and manifest-sha256 workflow_call outputs. The hand-assembled binaries/{arm64-linux,arm64-darwin,x86_64-darwin}/ directories now also get the SHA256SUMS file every other CDN directory already gets from the artifact_bundle rule. No permission changes (the merge job runs no build or test code, and same-run artifact transfer uses the runtime token, so it drops the workflow-level actions/id-token grants).
bazel-test-arm64 → upload-external-artifacts binding: the arm64/macOS binaries transit Actions artifacts between the Namespace builders and the CDN-upload job; each bazel-test-arm64 leg now records the sha256sum lines of every file it uploads in its job outputs — immediately after the builds, keeping the whole-codebase build and test steps outside the trust chain — and upload-external-artifacts re-checks each downloaded artifact against them (rejecting mismatches, empty outputs, and files beyond the recorded set) before bundling. Closes the same artifact-swap hole for the binaries that the manifest checks close for the manifests.
New attest-uploads job in ci-kickoff.yml (master builds) and release-testing.yml (rc--*/hotfix-* builds): re-checks the merged manifest artifact against ci-main's manifest-sha256 output (an Actions artifact can be replaced mid-run by anything holding an actions: write token; job outputs cannot — the same check anchors every hop of the merge) and runs actions/attest (pinned, v4.2.2; the attest-build-provenance wrapper's README directs new implementations to it — same SLSA provenance predicate) with subject-checksums — one attestation covering every uploaded file including each directory's SHA256SUMS.

Why the attest job lives in the callers, not in ci-main.yml

GitHub validates a called workflow's permission requests against the caller's grant statically, even for jobs whose if: would skip them (community discussion #155062). An attest job inside ci-main.yml would therefore force attestations: write into ci-kickoff-manual.yml, which runs CI on fork PRs. With this layout no job that executes build or test code ever holds attestations: write, and ci-kickoff-manual.yml is untouched.

How consumers will verify (follow-up PRs)

gh attestation verify SHA256SUMS --repo dfinity/ic \
  --signer-workflow dfinity/ic/.github/workflows/<ci-kickoff|release-testing>.yml \
  --source-digest <commit>

then sha256sum --check each downloaded artifact against the verified sums. The --source-digest binding prevents replaying another commit's legitimately-attested objects. Follow-ups: ledger-suite-release.yml, tag-release.yml, mainnet_revisions.py.

Validation

Tested end-to-end pre-merge via a workflow_dispatch of ci-kickoff.yml on this branch with release-build=true (the upload-artifacts environment temporarily allowed this branch via a custom deployment-branch policy, removed after the test): run 33004648043 on fa328bb033 — all jobs green, including both bazel-test-arm64 legs (checksum recording), upload-external-artifacts (download verification + extras guard), upload-manifests, and Attest Uploaded Artifacts (one attestation, 180 subjects, predicateType: https://slsa.dev/provenance/v1, see: https://github.com/dfinity/ic/attestations/43234804).

Workstation verification against that run's artifacts:

Positive: curl -fsSLO https://download.dfinity.systems/ic/fa328bb033…/binaries/x86_64-linux/SHA256SUMS && gh attestation verify SHA256SUMS --repo dfinity/ic --signer-workflow dfinity/ic/.github/workflows/ci-kickoff.yml --source-digest fa328bb033… → exit 0. The follow-up PR's ci/scripts/fetch-attested-sums.sh also ran end-to-end against this attestation (exit 0, fetched file byte-identical), confirming the --format json subject structure (.[].verificationResult.statement.subject[]) and the per-directory subject-name binding on a live attestation.
Negative tests (per F-012 house style), all correctly failing: one flipped hex char in the file → exit 1; --source-digest of a different commit → exit 1; --signer-workflow …/release-testing.yml (wrong pipeline) → exit 1.
All changed workflow/action files parse as YAML; the manifest-shape check and the sha256 cross-checks fail closed by construction.

Scope / residual risk

The attestation certifies what this CI built and uploaded, not that the build is reproducible — build-determinism and repro-check remain the complementary controls. A run that uploads but dies before attesting leaves that commit unattested until re-run (the re-upload is checked byte-for-byte by rclone --immutable --checksum). Builds in ic-private are not attested; their commits gain attestations when the branch is pushed to dfinity/ic and rebuilt there.

🤖 Generated with Claude Code

Copilot

Pull request overview

Adds build-provenance attestations for CDN uploads, creating an integrity anchor independent of the CDN.

Changes:

Persists and validates upload checksum manifests.
Exposes manifest hashes and adds missing platform SHA256SUMS.
Attests verified manifests for release builds.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`.github/actions/upload-artifacts/action.yaml`	Creates and uploads validated manifests.
`.github/actions/attest-uploads/action.yaml`	Verifies, merges, and attests manifests.
`.github/workflows/ci-main.yml`	Produces manifests and workflow outputs.
`.github/workflows/ci-kickoff.yml`	Attests public master uploads.
`.github/workflows/release-testing.yml`	Attests public release-branch uploads.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

That the CDN uploads happen in two jobs (bazel-test-all and upload-external-artifacts) is an implementation detail of ci-main.yml which previously leaked to both callers and the attest-uploads action through the manifest-sha256-{main,external} output pair. A new unprivileged merge-upload-manifests job now verifies each per-job manifest against the producing job's output, merges them into a single upload-manifest artifact, and anchors it in its own job output — so ci-main.yml exposes one manifest-sha256 output, the attest-uploads action takes one input, and a future change to the set of upload jobs stays confined to ci-main.yml. The trust chain is unchanged in kind, one verified hop longer: every artifact transfer is still re-checked against a tamper-proof job output, and the merge job executes no build or test code and drops the workflow-level actions/id-token grants. Addresses the review suggestion in #11323 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

The arm64 and macOS binaries transit GitHub Actions artifacts between the bazel-test-arm64 legs (Namespace runners) and the upload-external-artifacts job, and nothing re-verified that transfer: anything in the run holding an `actions: write` token could replace an artifact and have the substitute hashed, uploaded, and attested. Each bazel-test-arm64 leg now records the sha256sum lines of every file it uploads in leg-specific job outputs (the runner drops empty outputs when merging matrix legs, and an empty output fails the check loudly), immediately after the builds so the whole-codebase build and test steps stay outside the trust chain, and upload-external-artifacts re-checks every download against them — also rejecting artifacts that carry files beyond the recorded ones — before anything is bundled for the CDN. Addresses #11323 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/ci-main.yml:550

ls omits dotfiles, so an artifact containing an extra hidden entry (for example, .payload) passes this set-equality check even though the step is intended to reject every unrecorded file. Enumerate all directory entries with find so hidden additions also fail closed.

            diff <(echo "$2" | awk '{print $2}' | LC_ALL=C sort) <(ls "$1" | LC_ALL=C sort)

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

`ls` omits dotfiles, so a replaced Actions artifact could smuggle hidden entries (or hidden directories with nested files) past the exhaustive-set check. `find . -mindepth 1` counts every entry — hidden files, directories, and nested content alike — so any unrecorded extra now fails the diff. Addresses #11323 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Fourth step for security finding 3618194 (MEDIUM, CWE-494), building on the producer-side attestations of #11323 and fetch-attested-sums.sh of #11324. mainnet_revisions.py recorded several hashes into mainnet-icos-revisions.json straight from unauthenticated CDN data: the dev update-image hash was the hash of the CDN download itself, and the setup-os and binaries hashes were read from CDN-served SHA256SUMS. The recording PR is auto-approved and auto-merged, and bazel/mainnet-icos-{images,binaries}.bzl then "verify" downloads against those same values -- self-consistency, not authenticity. All CDN-derived hashes now flow through VersionArtifactSums, which verifies each directory's SHA256SUMS via ci/scripts/fetch-attested-sums.sh (`gh attestation verify --repo dfinity/ic --signer-workflow dfinity/ic/.github/workflows/release-testing.yml --source-digest <version>` -- elected versions are always rc/hotfix builds) before any value is read from it: * update_img_hash_dev now comes from the verified {guest,host}-os/update-img-dev SHA256SUMS instead of hashing the multi-hundred-MB image the cron used to download every 2 hours. * setupos hashes and all 7 MAINNET_BINARIES hashes come from verified SHA256SUMS. * launch-measurements.json (dev and the prod fallback) is byte-verified against the attested SHA256SUMS entry before parsing. * New cross-check: the NNS-elected update-img hash from the proposal must equal the build-time hash in the verified SHA256SUMS -- a mismatch means the CDN does not serve what the NNS elected, and nothing is recorded. Cutover policy (VersionArtifactSums docstring): a PUBLIC commit without a verifiable attestation hard-fails -- no PR is created, the cron retries; backfill by re-running release-testing.yml on the version's branch (the rebuild is checked byte-for-byte against the CDN by rclone --immutable --checksum before attest-uploads runs). A commit that is NOT public (undisclosed security patch, built in ic-private and not attested here) falls back to the pre-fix CDN behavior with a loud warning, time-bounded until disclosure; CDN write access cannot remove a commit from the public repository, so the fallback is not attacker-selectable within the finding's threat model. The update_saved_* functions now check is_record_up_to_date BEFORE collecting the version's artifact info, so the 2-hourly cron stays cheap and green while recorded versions predate the attestation rollout; verification only runs when a new version is recorded. The updater workflow gets the pinned gh CLI install (gh attestation verify --source-digest needs gh >= 2.61; same step as update-mainnet-canister-revisions.yaml) and its pull_request dry-run trigger now also fires on changes to the updater script and fetch-attested-sums.sh. Verification, all run against the live APIs: * `mainnet_revisions.py --dry-run icos` passes end-to-end (all records up-to-date, no attestation calls needed). * get_replica_version_info() for the currently-elected 79c0105 fails closed: attestation lookup 404s (none exist until #11323 merges), the commit is public, and the "Refusing to record CDN-served hashes" exception is raised. * 9 new unit tests cover: attested path, missing-entry, hard-fail for public commits, warning fallback for private commits, no-fallback once attested, verified_json tamper rejection (flipped byte), elected-vs- build hash mismatch, get_binary_hashes completeness, and single-space SHA256SUMS parsing. 25/25 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Fourth step for security finding 3618194 (MEDIUM, CWE-494), building on the producer-side attestations of #11323 and fetch-attested-sums.sh of #11324. mainnet_revisions.py recorded several hashes into mainnet-icos-revisions.json straight from unauthenticated CDN data: the dev update-image hash was the hash of the CDN download itself, and the setup-os and binaries hashes were read from CDN-served SHA256SUMS. The recording PR is auto-approved and auto-merged, and bazel/mainnet-icos-{images,binaries}.bzl then "verify" downloads against those same values -- self-consistency, not authenticity. All CDN-derived hashes now flow through VersionArtifactSums, which verifies each directory's SHA256SUMS via ci/scripts/fetch-attested-sums.sh (`gh attestation verify --repo dfinity/ic --signer-workflow dfinity/ic/.github/workflows/release-testing.yml --source-digest <version>` -- elected versions are always rc/hotfix builds) before any value is read from it: * update_img_hash_dev now comes from the verified {guest,host}-os/update-img-dev SHA256SUMS instead of hashing the multi-hundred-MB image the cron used to download every 2 hours. * setupos hashes and all 7 MAINNET_BINARIES hashes come from verified SHA256SUMS. * launch-measurements.json (dev and the prod fallback) is byte-verified against the attested SHA256SUMS entry before parsing. * New cross-check: the NNS-elected update-img hash from the proposal must equal the build-time hash in the verified SHA256SUMS -- a mismatch means the CDN does not serve what the NNS elected, and nothing is recorded. Cutover policy (VersionArtifactSums docstring): a PUBLIC commit without a verifiable attestation hard-fails -- no PR is created, the cron retries; backfill by re-running release-testing.yml on the version's branch (the rebuild is checked byte-for-byte against the CDN by rclone --immutable --checksum before attest-uploads runs). A commit that is NOT public (undisclosed security patch, built in ic-private and not attested here) falls back to the pre-fix CDN behavior with a loud warning, time-bounded until disclosure; CDN write access cannot remove a commit from the public repository, so the fallback is not attacker-selectable within the finding's threat model. The update_saved_* functions now check is_record_up_to_date BEFORE collecting the version's artifact info, so the 2-hourly cron stays cheap and green while recorded versions predate the attestation rollout; verification only runs when a new version is recorded. The updater workflow gets the pinned gh CLI install (gh attestation verify --source-digest needs gh >= 2.61; same step as update-mainnet-canister-revisions.yaml) and its pull_request dry-run trigger now also fires on changes to the updater script and fetch-attested-sums.sh. Verification, all run against the live APIs: * `mainnet_revisions.py --dry-run icos` passes end-to-end (all records up-to-date, no attestation calls needed). * get_replica_version_info() for the currently-elected 79c0105 fails closed: attestation lookup 404s (none exist until #11323 merges), the commit is public, and the "Refusing to record CDN-served hashes" exception is raised. * 9 new unit tests cover: attested path, missing-entry, hard-fail for public commits, warning fallback for private commits, no-fallback once attested, verified_json tamper rejection (flipped byte), elected-vs- build hash mismatch, get_binary_hashes completeness, and single-space SHA256SUMS parsing. 25/25 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

First producer-side step for security finding 3618194 (MEDIUM, CWE-494): the release pipeline has no integrity anchor independent of the CDN. ci/src/artifacts/upload.sh already hashes every file it uploads (from the local bytes, before upload) and prints "<sha256>,<url>" per file, but the upload-artifacts action discarded that manifest into the step summary. With this change: * .github/actions/upload-artifacts converts the manifest to sha256sum format, validates its shape (fail closed), exposes its sha256 as an output and uploads it as an Actions artifact. * ci-main.yml passes manifest names for both upload jobs (bazel-test-all and upload-external-artifacts), exposes the manifest sha256s and release-build as workflow_call outputs, and gives the hand-assembled arm64/darwin binary directories the SHA256SUMS file every other CDN directory already gets from the artifact_bundle rule. * a new attest-uploads job in ci-kickoff.yml (master builds) and release-testing.yml (rc--*/hotfix-* builds) verifies the manifest artifacts against the job-output sha256s (artifacts can be swapped mid-run by an actions:write token; job outputs cannot), merges them and runs actions/attest-build-provenance with subject-checksums, covering every uploaded file INCLUDING each directory's SHA256SUMS. Consumers (tag-release.yml, ledger-suite-release.yml, ci/src/mainnet_revisions/mainnet_revisions.py) will verify their CDN downloads against these attestations in follow-up PRs: gh attestation verify SHA256SUMS --repo dfinity/ic \ --signer-workflow dfinity/ic/.github/workflows/<pipeline>.yml \ --source-digest <commit> The attest job lives in the callers, not in ci-main.yml: GitHub validates a called workflow's permission requests against the caller's grant statically, even for jobs whose 'if:' would skip them, and ci-kickoff-manual.yml (fork-PR CI) must never be granted attestations:write. No job that executes build or test code holds attestations:write, and ci-main's permissions are unchanged. Scope / residual risk: the attestation certifies what this CI built and uploaded, not that the build is reproducible -- build-determinism and repro-check remain the complementary controls. A run that uploads but fails before attesting leaves the commit unattested until re-run (the re-upload is checked byte-for-byte by rclone --immutable --checksum). Builds in ic-private are not attested; their commits gain attestations when the branch is pushed to dfinity/ic and rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review follow-up: upload.sh computes each sha256 after the rclone copies, so "before upload" was wrong. The security-relevant property is the hash source, not the timing: the uploader hashes its local copy of the file, never anything served back by the CDN. Say that instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

That the CDN uploads happen in two jobs (bazel-test-all and upload-external-artifacts) is an implementation detail of ci-main.yml which previously leaked to both callers and the attest-uploads action through the manifest-sha256-{main,external} output pair. A new unprivileged merge-upload-manifests job now verifies each per-job manifest against the producing job's output, merges them into a single upload-manifest artifact, and anchors it in its own job output — so ci-main.yml exposes one manifest-sha256 output, the attest-uploads action takes one input, and a future change to the set of upload jobs stays confined to ci-main.yml. The trust chain is unchanged in kind, one verified hop longer: every artifact transfer is still re-checked against a tamper-proof job output, and the merge job executes no build or test code and drops the workflow-level actions/id-token grants. Addresses the review suggestion in #11323 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The arm64 and macOS binaries transit GitHub Actions artifacts between the bazel-test-arm64 legs (Namespace runners) and the upload-external-artifacts job, and nothing re-verified that transfer: anything in the run holding an `actions: write` token could replace an artifact and have the substitute hashed, uploaded, and attested. Each bazel-test-arm64 leg now records the sha256sum lines of every file it uploads in leg-specific job outputs (the runner drops empty outputs when merging matrix legs, and an empty output fails the check loudly), immediately after the builds so the whole-codebase build and test steps stay outside the trust chain, and upload-external-artifacts re-checks every download against them — also rejecting artifacts that carry files beyond the recorded ones — before anything is bundled for the CDN. Addresses #11323 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The number resolves to nothing a future reader can look up; the comments already explain the property that matters (an integrity anchor independent of the CDN). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

actions/attest-build-provenance v4 is only a wrapper around actions/attest, and its README directs new implementations to the latter. Behavior is identical for this use: with no predicate input, actions/attest emits the same SLSA build-provenance attestation (the predicate type `gh attestation verify` expects by default), the subject-checksums input is unchanged, and no new permissions are needed — `artifact-metadata: write` only matters for storage records, which require push-to-registry and stay disabled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ci/src/artifacts/upload.sh now prints "<hex> <bucket path>" lines (sha256sum format, path relative to https://download.dfinity.systems/) instead of "<hex>,<url>", so the upload-artifacts action persists the manifest as-is rather than rewriting the lines with sed. The action's fail-closed format gate is unchanged and a dry run of the uploader was checked against it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Redirect the uploader's stdout straight to $RUNNER_TEMP/upload-manifest.sha256sums instead of going through a mktemp file that was cp'd there afterwards. A failed upload command can leave a partial file behind, but the manifest-consuming steps only run when this step succeeded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

`ls` omits dotfiles, so a replaced Actions artifact could smuggle hidden entries (or hidden directories with nested files) past the exhaustive-set check. `find . -mindepth 1` counts every entry — hidden files, directories, and nested content alike — so any unrecorded extra now fails the diff. Addresses #11323 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The "Verify downloaded artifacts" step failed with a dash syntax error: in container jobs a `run:` step without `shell:` can resolve to plain sh, which cannot parse the process substitutions in the extras guard. Declare bash on it and on the arm64 "Record artifact checksums" step (which uses [[ ]] and only got bash by accident of running outside a container); the merge job's step already did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

post-required-check-status-ci-main only reflected ci-main's result, so a failed attestation could not fail the required status. It now needs attest-uploads and posts failure when that job failed; a skipped attest-uploads (any non-release build, i.e. every current PR and merge queue run) passes ci-main's result through unchanged. The gate checks ci-main's result explicitly because success()/failure() are both false when a needed job was skipped, which would have left the required status unposted on every PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…fails" This reverts commit 0c88537.

Attest-job comments now name the hotfix-* disclosure route explicitly, and slack-workflow-run.yml alerts on failed hotfix-* runs (disclosure backfill runs were previously silent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…erify gh attestation verify authenticates the digest only; this attestation covers every uploaded directory, so the documented consumer procedure now names the exact-subject-name requirement that prevents one directory's attested SHA256SUMS from standing in for another's. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ions Second step for security finding 3618194 (MEDIUM, CWE-494), building on the producer-side attestations of #11323. ledger-suite-release.yml downloaded canister wasms plus the CDN's own SHA256SUMS and republished a pruned copy of that file as the release's verification data. No downloaded byte was ever hashed and compared: the only check was that each expected filename had a SHA256SUMS line, so whoever can write to the CDN buckets could get their artifacts published as an official release with matching checksums. This introduces the shared verification primitive the remaining consumers (tag-release.yml, mainnet_revisions.py) will reuse: * ci/scripts/fetch-attested-sums.sh downloads a directory's SHA256SUMS from the CDN and verifies it with `gh attestation verify --repo dfinity/ic --signer-workflow <pipeline> --source-digest <commit>` before anything reads it. --source-digest pins the attestation to the exact commit, so replaying another commit's legitimately-attested SHA256SUMS also fails. * .github/actions/fetch-verified-artifacts wraps it for workflows: pinned gh CLI install (house pattern), one attested-SHA256SUMS fetch per directory, then per file curl + `sha256sum --check` against the verified sums, failing closed on any missing entry or mismatch. It emits a SHA256SUMS of the verified hashes under the files' local names, so published checksums are the verified build-time values rather than re-hashes of the downloads. * ledger-suite-release.yml replaces its fetch()-based download step with the action (signer-workflow: ci-kickoff.yml, since ledger-suite tags point at master commits). The release body, changelog and softprops steps are unchanged apart from reading the action's out-dir. Verification: the failure paths were exercised directly. A malformed commit id and '..' path segments are rejected; fetching the real canisters/SHA256SUMS of master commit 79c0105 and verifying it fails with a 404 from the attestations API (no attestations exist until #11323 merges) and a non-zero exit -- the workflow fails closed rather than publishing unverified artifacts. Positive-path validation happens with the first ledger-suite-*.rc1 tag on a post-#11323 master commit. Note the enforcement/coverage propagation: a tag push runs the workflow file at the tag's commit, so tags on commits containing both #11323 and this change are automatically attested AND verified, while older tags keep the old behavior. Tags on commits that predate the attestation rollout can no longer be released with this workflow (documented in the header). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Third step for security finding 3618194 (MEDIUM, CWE-494), building on the producer-side attestations of #11323 and the fetch-verified-artifacts action of #11324. tag-release.yml downloaded 9 tool binaries, 2 OS images and 19 canister wasm/did pairs from the CDN with plain curl and published sha256sums.txt computed FROM THE DOWNLOADS THEMSELVES, so whoever can write to the CDN buckets could get their artifacts republished as an official GitHub release with matching checksums. Every download now goes through .github/actions/fetch-verified-artifacts: the per-directory SHA256SUMS is verified with `gh attestation verify --repo dfinity/ic --signer-workflow dfinity/ic/.github/workflows/release-testing.yml --source-digest <commit>` and each artifact is checked against the verified sums, failing the release on any mismatch. Pinning the release-testing.yml signer also enforces that a release-* tag can only ship commits that went through the release-qualification pipeline. sha256sums.txt is now the verified build-time hashes (plus the locally-assembled canisters.tar hash), not a re-hash of the downloads. Bugs fixed in passing: * The OS images had silently never been attached: since 9bd0a40 (2024-08) the download step saved setup-os-img.tar.zst / update-os-img.tar.zst while the publish step listed *.tar.gz, and softprops v0.1.15 only warned about the unmatched patterns. The image downloads are now dropped entirely (preserving the two-year de-facto release contents; the canonical image channel is the CDN + NNS election proposal hash, verified by ci/scripts/repro-check), and fail_on_unmatched_files makes any repeat of this bug fail loudly. * sha256sums.txt claimed to cover the release but only ever hashed *.gz and *.tar; it now covers exactly what ships. * On the workflow_dispatch path the softprops action defaulted tag_name to the dispatched ref, not the requested tag; tag_name is now passed explicitly. * softprops/action-gh-release upgraded from the v0.1.15 pin to the same v2.4.2 pin used by ledger-suite-release.yml. Verification: the workflow_dispatch dry-run path exercises checkout, download and verification without publishing; run it with a tag-name whose commit was built by release-testing.yml after #11323 merged. The verification failure paths (missing attestation, flipped hash, wrong --source-digest commit) were exercised in #11324 and fail closed. Note that a push-tag trigger runs the workflow file at the tag's commit, so this protects release-* tags on rc branches cut after this change reaches master; the arm64/darwin binary directories additionally need the SHA256SUMS files introduced by #11323. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fourth step for security finding 3618194 (MEDIUM, CWE-494), building on the producer-side attestations of #11323 and fetch-attested-sums.sh of #11324. mainnet_revisions.py recorded several hashes into mainnet-icos-revisions.json straight from unauthenticated CDN data: the dev update-image hash was the hash of the CDN download itself, and the setup-os and binaries hashes were read from CDN-served SHA256SUMS. The recording PR is auto-approved and auto-merged, and bazel/mainnet-icos-{images,binaries}.bzl then "verify" downloads against those same values -- self-consistency, not authenticity. All CDN-derived hashes now flow through VersionArtifactSums, which verifies each directory's SHA256SUMS via ci/scripts/fetch-attested-sums.sh (`gh attestation verify --repo dfinity/ic --signer-workflow dfinity/ic/.github/workflows/release-testing.yml --source-digest <version>` -- elected versions are always rc/hotfix builds) before any value is read from it: * update_img_hash_dev now comes from the verified {guest,host}-os/update-img-dev SHA256SUMS instead of hashing the multi-hundred-MB image the cron used to download every 2 hours. * setupos hashes and all 7 MAINNET_BINARIES hashes come from verified SHA256SUMS. * launch-measurements.json (dev and the prod fallback) is byte-verified against the attested SHA256SUMS entry before parsing. * New cross-check: the NNS-elected update-img hash from the proposal must equal the build-time hash in the verified SHA256SUMS -- a mismatch means the CDN does not serve what the NNS elected, and nothing is recorded. Cutover policy (VersionArtifactSums docstring): a PUBLIC commit without a verifiable attestation hard-fails -- no PR is created, the cron retries; backfill by re-running release-testing.yml on the version's branch (the rebuild is checked byte-for-byte against the CDN by rclone --immutable --checksum before attest-uploads runs). A commit that is NOT public (undisclosed security patch, built in ic-private and not attested here) falls back to the pre-fix CDN behavior with a loud warning, time-bounded until disclosure; CDN write access cannot remove a commit from the public repository, so the fallback is not attacker-selectable within the finding's threat model. The update_saved_* functions now check is_record_up_to_date BEFORE collecting the version's artifact info, so the 2-hourly cron stays cheap and green while recorded versions predate the attestation rollout; verification only runs when a new version is recorded. The updater workflow gets the pinned gh CLI install (gh attestation verify --source-digest needs gh >= 2.61; same step as update-mainnet-canister-revisions.yaml) and its pull_request dry-run trigger now also fires on changes to the updater script and fetch-attested-sums.sh. Verification, all run against the live APIs: * `mainnet_revisions.py --dry-run icos` passes end-to-end (all records up-to-date, no attestation calls needed). * get_replica_version_info() for the currently-elected 79c0105 fails closed: attestation lookup 404s (none exist until #11323 merges), the commit is public, and the "Refusing to record CDN-served hashes" exception is raised. * 9 new unit tests cover: attested path, missing-entry, hard-fail for public commits, warning fallback for private commits, no-fallback once attested, verified_json tamper rejection (flipped byte), elected-vs- build hash mismatch, get_binary_hashes completeness, and single-space SHA256SUMS parsing. 25/25 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

basvandijk requested a balanced review from Copilot August 26, 2026 07:51

github-actions Bot added the feat label Aug 26, 2026

Copilot started reviewing on behalf of basvandijk August 26, 2026 07:52 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread .github/actions/upload-artifacts/action.yaml Outdated

This was referenced Aug 26, 2026

fix(ci): verify ledger-suite release artifacts against build attestations #11324

Open

fix(ci): verify tag-release artifacts against build attestations #11325

Open

fix(ci): verify mainnet-icos-revisions hashes against build attestations #11326

Draft

basvandijk commented Aug 26, 2026

View reviewed changes

Comment thread .github/actions/attest-uploads/action.yaml Outdated

basvandijk requested a balanced review from Copilot August 26, 2026 10:25

Copilot started reviewing on behalf of basvandijk August 26, 2026 10:25 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread .github/workflows/ci-main.yml

basvandijk requested a balanced review from Copilot August 26, 2026 11:10

Copilot started reviewing on behalf of basvandijk August 26, 2026 11:11 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

basvandijk requested a balanced review from Copilot August 26, 2026 12:09

Copilot started reviewing on behalf of basvandijk August 26, 2026 12:09 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

basvandijk requested a balanced review from Copilot August 26, 2026 12:40

Copilot started reviewing on behalf of basvandijk August 26, 2026 12:40 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

basvandijk requested a balanced review from Copilot August 26, 2026 12:50

Copilot started reviewing on behalf of basvandijk August 26, 2026 12:51 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

Comment thread .github/workflows/ci-main.yml Outdated

basvandijk requested a balanced review from Copilot August 26, 2026 13:01

Copilot started reviewing on behalf of basvandijk August 26, 2026 13:01 View session

Copilot AI reviewed Aug 26, 2026

View reviewed changes

basvandijk had a problem deploying to upload-artifacts August 26, 2026 14:31 — with GitHub Actions Error

basvandijk requested a review from a team as a code owner August 27, 2026 10:32

github-actions Bot added the @infra label Aug 27, 2026

basvandijk mentioned this pull request Aug 27, 2026

feat(ci): Disclose Hotfix workflow #11361

Open

basvandijk requested a balanced review from Copilot August 27, 2026 20:30

Copilot started reviewing on behalf of basvandijk August 27, 2026 20:30 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Comment thread .github/actions/attest-uploads/action.yaml Outdated

basvandijk requested a balanced review from Copilot August 27, 2026 22:04

Copilot started reviewing on behalf of basvandijk August 27, 2026 22:04 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

basvandijk and others added 15 commits August 28, 2026 13:24

docs(ci): drop the internal finding number from code comments

a44c385

The number resolves to nothing a future reader can look up; the comments already explain the property that matters (an integrity anchor independent of the CDN). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Revert "fix(ci): fail the ci-main required check when attest-uploads …

dbc8b00

…fails" This reverts commit 0c88537.

renames and removed some slop

48f4243

basvandijk force-pushed the bas/attest-cdn-uploads branch from 9fd15da to f3d1c0a Compare August 28, 2026 11:24

Navigation Menu

Uh oh!

feat(ci): attest CDN artifact uploads with build provenance - #11323

feat(ci): attest CDN artifact uploads with build provenance#11323
basvandijk wants to merge 15 commits into
masterfrom
bas/attest-cdn-uploads

basvandijk commented Aug 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

basvandijk commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why the attest job lives in the callers, not in ci-main.yml

How consumers will verify (follow-up PRs)

Validation

Scope / residual risk

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

basvandijk commented Aug 26, 2026 •

edited

Loading