ci: add backfill workflow for manifest regeneration#24
Open
adilhusain-s wants to merge 3 commits intoIBM:mainfrom
Open
ci: add backfill workflow for manifest regeneration#24adilhusain-s wants to merge 3 commits intoIBM:mainfrom
adilhusain-s wants to merge 3 commits intoIBM:mainfrom
Conversation
- Add retry logic (8 attempts, 5s delay) with exponential backoff to dotnet-install.py - Handle transient HTTP errors (500, 502, 503, 504) and network failures automatically - Upgrade Trivy to v0.68.2 for improved vulnerability detection - Enable strict security gates: fail build on HIGH/CRITICAL vulnerabilities These changes improve pipeline reliability by handling network flakiness and enforcing stricter security standards during builds. Signed-off-by: Adilhusain Shaikh <Adilhusain.Shaikh@ibm.com>
Add new tooling to support decoupled manifest updates in the CI pipeline: Scripts: - generate_partial_manifest.py: Generates architecture-scoped partial manifests from GitHub release assets - Validates download URLs and filters out non-release artifacts (SBOM, Trivy scans, logs) - Parses filenames to extract version, arch, platform metadata - Handles both tagged and untagged release URLs - apply_partial_manifests.py: Merges partial manifests into versioned manifest files - Discovers and loads partial manifest JSON files - Validates required fields (version, filename, arch, platform, download_url) - Updates version-specific manifests atomically Tests: - test_generate_partial_manifest.py: Comprehensive tests for filename parsing, URL validation, and manifest generation - test_apply_partial_manifests.py: Tests for discovering, loading, validating, and applying partial manifests This tooling enables the CI workflow refactor that eliminates race conditions by decoupling artifact generation from git operations. Signed-off-by: Adilhusain Shaikh <Adilhusain.Shaikh@ibm.com>
Add workflow to regenerate or fix manifests from existing GitHub releases without rebuilding binaries. This is useful for: - Backfilling missing manifests - Fixing manifest data errors - Recovering from partial workflow failures - Migrating to new manifest formats Workflow Features: - Triggered manually (workflow_dispatch) or on schedule - Discovers existing Python releases from GitHub - Generates partial manifests from release assets per version - Applies partial manifests atomically in single aggregation step - Uses tooling from PR IBM#23 (generate_partial_manifest.py, apply_partial_manifests.py) Jobs: 1. get-tags: Discovers Python releases needing manifest updates 2. generate-partial-manifest (matrix): Creates partial manifests per version 3. apply-manifests: Downloads all partials and commits to main Benefits: - No rebuilds required - works with existing releases - Idempotent - can be run multiple times safely - Uses same tooling as main release workflow for consistency - Concurrency control prevents conflicts Signed-off-by: Adilhusain Shaikh <Adilhusain.Shaikh@ibm.com>
adilhusain-s
requested review from
anup-kodlekere,
mtarsel and
rahulssv-ibm
as code owners
This was referenced Jan 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Missing or incorrect manifests for existing GitHub releases require manual intervention or rebuilding binaries.
Solution
Add a backfill workflow that regenerates manifests from existing release assets without rebuilding binaries.
What This Provides
.github/workflows/backfill-manifests.ymlA manual workflow that:
generate_partial_manifest.py(PR feat(tooling): add partial manifest generation and application scripts #23)apply_partial_manifests.py(PR feat(tooling): add partial manifest generation and application scripts #23)Key features:
Use Cases
Example
If manifests are missing for Python 3.13.x releases:
Files Changed
.github/workflows/backfill-manifests.yml(184 lines, NEW)Dependencies
Related