feat(ci): reap stale CI workspaces and template versions on staging#172
Merged
Conversation
cancel-in-progress only frees a GitHub runner; the workspaces and template versions that integration-test runs create on staging-coder.ddev.com live on, because the per-run `coder delete` / `archive` steps never reach the server when a run is cancelled, killed, or its workspace lands in Failed. Over many PR pushes staging backs up. Add cleanup decoupled from the test job's fate, in three layers: - Scheduled janitor (ci-reap.yml + scripts/ci-reap-staging.sh): every 15 min (and on demand) reap ci-bot workspaces that are failed/stopped/canceled (any age) or running past a threshold (default 20m), then archive unused template versions for all four templates. - Reap-on-start (scripts/ci-reap-family.sh): each active test job deletes its own name-family predecessors before creating its workspace, so a new commit to a PR clears the cancelled run's leftovers immediately. - Lighter PR matrix: pull_request runs a reduced smoke set (drupal-core D12 + fork; drupal-contrib one module); push-to-main and nightly keep the full matrix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cancel-in-progressonly frees a GitHub runner — it does nothing to staging-coder.ddev.com. Each integration-test run pushes a template version and creates a workspace (owned byci-bot), and the per-runcoder delete/archivecleanup lives inif: always()steps inside the test job. When a run is cancelled (new push to a PR), force-killed, or its workspace lands inFailed, those steps never reach the server. Over many PR pushes, staging backs up with orphaned running workspaces (e.g. the same test from runs 151/152/154 all alive at once) and accumulated template versions (60+ per template).Fix
Cleanup decoupled from any individual test job's fate, in three layers:
Scheduled janitor — new
.github/workflows/ci-reap.yml+scripts/ci-reap-staging.sh. Every 15 min (and on demand viaworkflow_dispatchwithage_minutes/dry_runinputs), authenticating asci-bot, it deletes anyci-botworkspace that isfailed/stopped/canceled(any age) orrunningpast a threshold (default 20m), skipping transitional states, then runscoder templates archive <t> --allon all four templates. This is the safety net — it also handles merged/closed/abandoned PRs where no replacement run ever comes.Reap-on-start —
scripts/ci-reap-family.sh, wired into every active test job beforecoder create. The replacement run deletes its own name-family predecessors (e.g.gc-pathauto-d11-*) left behind by the cancelled run, giving immediate "new commit stops the old deployment."Lighter PR matrix — on
pull_request, drupal-core runs D12 only (+ the single fork job) and drupal-contrib runs one module; push-to-main and nightly keep the full matrix.Notes
schedule:trigger only begins firing onceci-reap.ymlis onmain. Until then it can be run from the Actions tab or locally via./scripts/ci-reap-staging.sh --force.date -d, which theubuntu-latestrunner provides (verified againstubuntu:24.04).ci-botexclusively owns CI workspaces (real users are notci-bot), soowner == ci-botis a safe selector.Add zellij to brew) that was already present before these changes.Validation
failed/stoppedworkspaces, correctly kept 29 in-progress ones.🤖 Generated with Claude Code