Skip to content

feat(ci): reap stale CI workspaces and template versions on staging#172

Merged
rfay merged 2 commits into
mainfrom
20260629_rfay_stop_deployments
Jun 29, 2026
Merged

feat(ci): reap stale CI workspaces and template versions on staging#172
rfay merged 2 commits into
mainfrom
20260629_rfay_stop_deployments

Conversation

@rfay

@rfay rfay commented Jun 29, 2026

Copy link
Copy Markdown
Member

Problem

cancel-in-progress only frees a GitHub runner — it does nothing to staging-coder.ddev.com. Each integration-test run pushes a template version and creates a workspace (owned by ci-bot), and the per-run coder delete / archive cleanup lives in if: always() steps inside the test job. When a run is cancelled (new push to a PR), force-killed, or its workspace lands in Failed, those steps never reach the server. Over many PR pushes, staging backs up with orphaned running workspaces (e.g. the same test from runs 151/152/154 all alive at once) and accumulated template versions (60+ per template).

Fix

Cleanup decoupled from any individual test job's fate, in three layers:

  1. Scheduled janitor — new .github/workflows/ci-reap.yml + scripts/ci-reap-staging.sh. Every 15 min (and on demand via workflow_dispatch with age_minutes/dry_run inputs), authenticating as ci-bot, it deletes any ci-bot workspace that is failed/stopped/canceled (any age) or running past a threshold (default 20m), skipping transitional states, then runs coder templates archive <t> --all on all four templates. This is the safety net — it also handles merged/closed/abandoned PRs where no replacement run ever comes.

  2. Reap-on-startscripts/ci-reap-family.sh, wired into every active test job before coder create. The replacement run deletes its own name-family predecessors (e.g. gc-pathauto-d11-*) left behind by the cancelled run, giving immediate "new commit stops the old deployment."

  3. Lighter PR matrix — on pull_request, drupal-core runs D12 only (+ the single fork job) and drupal-contrib runs one module; push-to-main and nightly keep the full matrix.

Notes

  • The janitor's schedule: trigger only begins firing once ci-reap.yml is on main. Until then it can be run from the Actions tab or locally via ./scripts/ci-reap-staging.sh --force.
  • Age logic relies on GNU date -d, which the ubuntu-latest runner provides (verified against ubuntu:24.04).
  • ci-bot exclusively owns CI workspaces (real users are not ci-bot), so owner == ci-bot is a safe selector.
  • This branch also includes a pre-existing commit (Add zellij to brew) that was already present before these changes.

Validation

  • bash syntax + YAML parse on all four workflows
  • Read-only dry run against staging: found 9 immediately-reapable failed/stopped workspaces, correctly kept 29 in-progress ones.

🤖 Generated with Claude Code

rfay and others added 2 commits June 29, 2026 15:45
cancel-in-progress only frees a GitHub runner; the workspaces and template
versions that integration-test runs create on staging-coder.ddev.com live on,
because the per-run `coder delete` / `archive` steps never reach the server
when a run is cancelled, killed, or its workspace lands in Failed. Over many
PR pushes staging backs up.

Add cleanup decoupled from the test job's fate, in three layers:

- Scheduled janitor (ci-reap.yml + scripts/ci-reap-staging.sh): every 15 min
  (and on demand) reap ci-bot workspaces that are failed/stopped/canceled (any
  age) or running past a threshold (default 20m), then archive unused template
  versions for all four templates.
- Reap-on-start (scripts/ci-reap-family.sh): each active test job deletes its
  own name-family predecessors before creating its workspace, so a new commit
  to a PR clears the cancelled run's leftovers immediately.
- Lighter PR matrix: pull_request runs a reduced smoke set (drupal-core D12 +
  fork; drupal-contrib one module); push-to-main and nightly keep the full
  matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-06-29 21:55 UTC

@rfay rfay merged commit 09d70fc into main Jun 29, 2026
13 of 19 checks passed
@rfay rfay deleted the 20260629_rfay_stop_deployments branch June 29, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant