Skip to content

Make systemd restart bounded so a self-restart cannot hang nerve down#185

Open
constkolesnyak wants to merge 1 commit into
ClickHouse:mainfrom
constkolesnyak:fix/bounded-systemd-restart
Open

Make systemd restart bounded so a self-restart cannot hang nerve down#185
constkolesnyak wants to merge 1 commit into
ClickHouse:mainfrom
constkolesnyak:fix/bounded-systemd-restart

Conversation

@constkolesnyak

Copy link
Copy Markdown
Contributor

A bare SIGTERM in the systemd restart branch can hang the service indefinitely.

uvicorn's graceful shutdown waits for in-flight connections to drain. When nerve restart is triggered from inside a live session — the Telegram bot, or an agent turn editing nerve's own source — that session's stream keeps a connection open, so the drain never completes. systemd's TimeoutStopSec→SIGKILL only escalates for a systemd-initiated stop, not for a process shutting itself down after a signal, so nothing forces the issue.

The result is the worst kind of outage: the gateway stops listening, Application shutdown complete never arrives, the process stays alive so Restart=always never fires, and systemctl keeps reporting the unit active while it serves nothing.

We hit this in production on 2026-07-18: an agent edited nerve/cron/service.py and restarted nerve from within its own turn. The scheduler kept ticking in the logs while nothing answered on the gateway port.

Fix

Spawn a detached helper (own session, survives the caller's death) that sends SIGTERM, waits up to 15s for a clean drain, then SIGKILLs. Once the old PID is gone, Restart=always brings up a fresh instance. Bounded, and no privileges or systemctl access needed.

Verified on the deployment where the hang originally occurred: restart now completes in ~3s with a clean Application shutdown complete. 29 CLI tests pass.

A bare SIGTERM in the systemd restart branch could hang the service
indefinitely: uvicorn's graceful shutdown waits for in-flight
connections to drain, and when restart is triggered from inside a live
session, that session's own stream keeps a connection open, so the drain
never completes. systemd's TimeoutStopSec->SIGKILL only escalates for a
systemd-initiated stop, not for a process shutting itself down after a
signal, so nothing forced the issue and the service stayed down.

Spawn a detached helper (own session, survives the caller's death) that
sends SIGTERM, waits up to 15s for a clean drain, then SIGKILLs. Once the
old PID is gone, Restart=always brings up a fresh instance. Bounded, no
privileges needed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant