reuse:stop warm restart: wait loop sees runner online (elapsed_s=0) but no runner ever serves the label — job queued forever, stop-runner never fires

[dm1tr1yvovk]

Summary

On a reuse: stop warm restart, mode: start (v4.0.1, 65fbe4f) declared the runner registered and exited successfully, but no runner for the new label ever picked up work. The dependent job sat in queued for 3+ hours (GitHub waits up to 24h for a self-hosted label), which also means the consumer workflow's stop job — gated on the acceptance job reaching a terminal state — never fired: the pooled instance was left running and holding the whitelisted sandbox EIP until manual intervention.

Evidence

Failing run (terraform-provider-namecheap CI, pull_request): https://github.com/namecheap/terraform-provider-namecheap/actions/runs/29019773728

• Start self-hosted EC2 runner (job 86124175784) — success, 2026-07-09T13:02–13:05 UTC:

  13:04:37 Pre-attached eipalloc-1796f61b to stopped warm-pool instance i-049890a74e3bf0c4d.
  13:05:25 Checking every 10s for bootstrap failures and runner registration
  13:05:26 {"step":"wait_for_runner","mode":"start","outcome":"online","elapsed_s":0}
  13:05:26 {"step":"start","mode":"start","label":"xsvha","instance_ids":["i-049890a74e3bf0c4d"],"instance_type":"t3.medium","subnet_id":"subnet-01c4ff5a","outcome":"registered"}
  

• Acceptance test (job 86124838100, runs-on: xsvha) — queued at 13:06:20, never picked up (3h+ at time of filing).
• GET /repos/namecheap/terraform-provider-namecheap/actions/runners at ~15:50 UTC returned an empty list — no runner with label xsvha was registered at all by then. Either the registration the wait loop observed vanished within the first minute, or it observed something stale.

Contrast: successful run ~45 min earlier, same instance: https://github.com/namecheap/terraform-provider-namecheap/actions/runs/29017428944

• Identical warm-restart path, label 3xo8q, same i-049890a74e3bf0c4d, same {"step":"wait_for_runner","outcome":"online","elapsed_s":0} — but there the runner did take the job (12:18–12:20). Its mode: stop then logged:

  {"step":"remove_runner","mode":"stop","label":"3xo8q","skipped":true,"reason":"not_found"}
  

  i.e. the ephemeral runner had already deregistered after its single job — expected, but it shows the wait loop's success signal is indistinguishable between a healthy run and this failure.

Analysis

src/gh.js isOnline(label) returns true on a single instantaneous runner.status === 'online' check, and the wait loop in src/wait.js treats the first such observation as terminal success. On a warm restart that leaves two gaps:

1. No freshness guarantee. Success at elapsed_s: 0 means the very first poll matched. Nothing ties the observed registration to this boot of the instance — a registration that immediately drops (runner process exits after config.sh/on first connect, EIP/network hiccup, stale record not yet purged by GitHub) still counts as success.
2. No liveness follow-through. The action exits green the same second; if the runner dies before the dependent job is scheduled (~54s later here), nothing detects it. The consumer is left with a forever-queued job and — because typical stop jobs need the runner-consuming job — a leaked running instance holding the EIP.

This is the same general area as the v4.0.0/v4.0.1 reuse: stop registration fixes (#62, #64, #65, #66), so it looks like a remaining race rather than a regression.

Suggested fixes

1. Require a fresh registration: record the runner set (or GET /actions/runners ids) for the label before StartInstances, and only accept a runner whose id/registration is new relative to that snapshot — or require the poll loop to observe the registration appear (absent → online), never accept a first-poll instant match on warm restarts.
2. Confirm stability, not an instant: require N consecutive online polls (e.g. 2–3 over 20–30s) before declaring success, so a registration that flaps offline within seconds fails the start step instead of the downstream job queueing forever.
3. Diagnostics on this path: when mode: start succeeded but the runner disappears, there is currently zero instance-side evidence in the workflow logs. Extending the Bootstrap diagnostics: phone-home status tags, fast-fail, console-output capture, cleanup on failed start #41-style bootstrap diagnostics (console-output capture / phone-home tag) to cover the warm-restart re-registration would make the next occurrence debuggable.
4. Docs: recommend consumers add a post-start verification (poll /actions/runners for the label from the start job) and/or a reaper rule for runs whose label job stays queued with no online runner, since a leaked running instance also keeps holding the EIP that the next run's pre-attach step expects to find on a stopped instance.

Environment

• Action: namecheap/ec2-github-runner@65fbe4f0c1181c60d4df2bd3b33b8893e21fbb43 (v4.0.1)
• Consumer workflow: terraform-provider-namecheap .github/workflows/ci.yml (reuse: stop, reuse-pool-tag: sandbox-acceptance, t3.medium, EIP pre-attach before start)
• Instance: i-049890a74e3bf0c4d (warm pool, stopped at 12:20 UTC by the prior run, restarted 13:04 UTC)