Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions .github/workflows/workspace-lifecycle-cleanup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
name: Workspace lifecycle cleanup

# Daily janitor for coder.ddev.com (production). Notifies owners of idle
# workspaces, then deletes them if they stay idle after the notice period.
# See scripts/workspace-lifecycle-cleanup.sh for the full policy.
#
# Deleting a workspace via `coder delete` also frees its dind-cache Docker
# volume on the host (the destroy provisioner in each template removes it),
# which is what actually reclaims disk space on /data.
#
# Requires:
# Repository variable: PROD_CODER_URL - https://coder.ddev.com
# Repository secret: OP_SERVICE_ACCOUNT_TOKEN - 1Password service account
# 1Password items (adjust vault/item names to match your 1Password setup):
# op://coder-ddev-secrets/PROD_CODER_SESSION_TOKEN/credential
# op://coder-ddev-secrets/MAILGUN_API_KEY/credential
# op://coder-ddev-secrets/MAILGUN_DOMAIN/credential
#
# PROD_CODER_SESSION_TOKEN must belong to a user with the `owner` role —
# this deployment has no Premium license, so `owner` is the only built-in
# role that can list every user's workspaces and delete workspaces it
# doesn't own. Provision a dedicated service account for this rather than
# reusing a personal token; see "Provisioning the PROD_CODER_SESSION_TOKEN
# credential" in docs/admin/operations-guide.md for the exact commands.
#
# The state file at scripts/state/workspace-lifecycle-state.json is
# committed back to the branch after every --force run so the notify/delete
# grace period survives across ephemeral runner instances.

env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true

concurrency:
group: workspace-lifecycle-cleanup
cancel-in-progress: false

permissions:
contents: write

on:
schedule:
- cron: '17 6 * * *'
workflow_dispatch:
inputs:
dry_run:
description: 'Dry run (list only, notify/delete nothing)'
type: boolean
required: false
default: true
notify_days:
description: 'Idle days before the first notice'
type: string
required: false
default: '7'
delete_after_days:
description: 'Days after notice before deletion'
type: string
required: false
default: '7'

jobs:
cleanup:
name: Notify and reap idle workspaces
runs-on: ubuntu-latest
defaults:
run:
shell: bash -euo pipefail {0}
steps:
- uses: actions/checkout@v6

- name: Load 1Password secrets
uses: 1password/load-secrets-action@v4
with:
export-env: true
env:
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }}
PROD_CODER_SESSION_TOKEN: 'op://coder-ddev-secrets/PROD_CODER_SESSION_TOKEN/credential'
MAILGUN_API_KEY: 'op://coder-ddev-secrets/MAILGUN_API_KEY/credential'
MAILGUN_DOMAIN: 'op://coder-ddev-secrets/MAILGUN_DOMAIN/credential'

- uses: coder/setup-action@v1
with:
access_url: ${{ vars.PROD_CODER_URL }}
coder_session_token: ${{ env.PROD_CODER_SESSION_TOKEN }}

- name: Notify and reap idle workspaces
env:
DRY_RUN: ${{ github.event.inputs.dry_run || 'false' }}
NOTIFY_DAYS: ${{ github.event.inputs.notify_days || '7' }}
DELETE_AFTER_DAYS: ${{ github.event.inputs.delete_after_days || '7' }}
run: ./scripts/workspace-lifecycle-cleanup.sh $([[ "$DRY_RUN" == "true" ]] || echo --force)

- name: Commit updated state file
run: |
if ! git diff --quiet -- scripts/state/workspace-lifecycle-state.json; then
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add scripts/state/workspace-lifecycle-state.json
git commit -m "chore(workspace-cleanup): update lifecycle state [skip ci]"
git push
else
echo "No state changes to commit."
fi
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -358,6 +358,7 @@ Additional logs in workspace:
- `image/scripts/.ddev/global_config.yaml` - DDEV defaults copied into workspaces
- `scripts/coder-delete-workspace-dir.sh` - Sudo wrapper for workspace host dir cleanup (must be installed on server)
- `scripts/cleanup-deleted-workspaces.sh` - Manual cleanup for orphaned workspace dirs/volumes
- `scripts/workspace-lifecycle-cleanup.sh` - Notifies then deletes idle workspaces on coder.ddev.com (see `.github/workflows/workspace-lifecycle-cleanup.yml`)
- `VERSION` - Image version used by all templates (read automatically by Makefile)
- `openspec/project.md` - Project conventions and constraints
- `openspec/AGENTS.md` - OpenSpec workflow instructions
50 changes: 50 additions & 0 deletions docs/admin/operations-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,56 @@ docker system df

---

### Automated Idle Workspace Cleanup

A scheduled GitHub Actions workflow (`.github/workflows/workspace-lifecycle-cleanup.yml`) runs `scripts/workspace-lifecycle-cleanup.sh` daily against coder.ddev.com to keep idle workspaces from accumulating and slowly filling `/data` with `*-dind-cache` Docker volumes (each workspace keeps its full Docker-in-Docker cache volume until it's deleted, even while stopped).

Policy per workspace, based on `last_used_at`:

- **7 days idle** — owner gets a notice email (via Mailgun) explaining the workspace will be deleted, and pointing to the auth/access announcement if they can no longer log in.
- **7 more days idle after the notice** — the workspace is deleted with `coder delete --yes`.
- If the owner uses the workspace again before deletion, the pending notice is cleared automatically.

State (which workspaces have been notified and when) is tracked in `scripts/state/workspace-lifecycle-state.json`, which the workflow commits back to the repo after every run so the grace period survives across runs.

```bash
# Dry run — shows what would be notified/deleted, sends no email, deletes nothing
./scripts/workspace-lifecycle-cleanup.sh

# Actually send notices, delete workspaces past their grace period, and persist state
./scripts/workspace-lifecycle-cleanup.sh --force
```

**Prerequisites:** `coder` CLI authenticated against coder.ddev.com as a user with the `owner` role (see below); `MAILGUN_API_KEY` and `MAILGUN_DOMAIN` set for `--force` runs. See the script header for all environment overrides (`NOTIFY_DAYS`, `DELETE_AFTER_DAYS`, `EXCLUDE_OWNERS`, etc.).

The workflow can also be triggered manually (`workflow_dispatch`) with a `dry_run` input for testing.

#### Provisioning the `PROD_CODER_SESSION_TOKEN` credential

The script needs to list *every* user's workspaces (`coder list --all`) and delete workspaces it doesn't own. On this deployment (no Premium license, so no custom RBAC roles), `owner` is the only built-in role that can do both — there's no narrower "workspace admin" role available. That makes this token effectively full site-admin on production, so it's provisioned as a dedicated non-human account rather than a personal token:

```bash
# 1. Create a machine identity — no GitHub OAuth login required
coder users create --username workspace-janitor \
--email workspace-janitor@ddev.com \
--full-name "Workspace Lifecycle Janitor" \
--login-type none

# 2. Grant it owner — the only built-in role that covers list-all + delete-any-workspace
coder users edit-roles workspace-janitor --roles owner --yes

# 3. Mint a long-lived token for it (run as an existing owner/admin, e.g. your own account)
coder tokens create -u workspace-janitor --name workspace-lifecycle-ci --lifetime 8760h
```

Store the resulting token as the `PROD_CODER_SESSION_TOKEN` secret referenced by the workflow. Using a dedicated account rather than a personal token keeps deletions attributable to the bot (not an individual) in the audit log, and means the janitor doesn't break if the admin's own account is later deactivated or re-authenticated.

Check the server's configured max token lifetime before choosing `--lifetime` — if it's capped below a year, use the max allowed and set a reminder to rotate the token before it expires, since an expired token fails the workflow silently until someone notices.

`--login-type none` is deprecated in favor of `--service-account` (Premium-only) but remains functional for this purpose.

---

### Orphaned Workspace Cleanup

When a workspace is deleted, the destroy provisioner automatically removes the host directory at `/coder-workspaces/<owner>-<workspace>`. Directories can still be orphaned if the provisioner fails or for workspaces deleted before the provisioner was added. Run the cleanup script to reclaim disk space in those cases.
Expand Down
Loading
Loading