Make systemd restart bounded so a self-restart cannot hang nerve down#185
Open
constkolesnyak wants to merge 1 commit into
Open
Make systemd restart bounded so a self-restart cannot hang nerve down#185constkolesnyak wants to merge 1 commit into
constkolesnyak wants to merge 1 commit into
Conversation
A bare SIGTERM in the systemd restart branch could hang the service indefinitely: uvicorn's graceful shutdown waits for in-flight connections to drain, and when restart is triggered from inside a live session, that session's own stream keeps a connection open, so the drain never completes. systemd's TimeoutStopSec->SIGKILL only escalates for a systemd-initiated stop, not for a process shutting itself down after a signal, so nothing forced the issue and the service stayed down. Spawn a detached helper (own session, survives the caller's death) that sends SIGTERM, waits up to 15s for a clean drain, then SIGKILLs. Once the old PID is gone, Restart=always brings up a fresh instance. Bounded, no privileges needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A bare SIGTERM in the systemd restart branch can hang the service indefinitely.
uvicorn's graceful shutdown waits for in-flight connections to drain. When
nerve restartis triggered from inside a live session — the Telegram bot, or an agent turn editing nerve's own source — that session's stream keeps a connection open, so the drain never completes. systemd's TimeoutStopSec→SIGKILL only escalates for a systemd-initiated stop, not for a process shutting itself down after a signal, so nothing forces the issue.The result is the worst kind of outage: the gateway stops listening,
Application shutdown completenever arrives, the process stays alive soRestart=alwaysnever fires, andsystemctlkeeps reporting the unit active while it serves nothing.We hit this in production on 2026-07-18: an agent edited
nerve/cron/service.pyand restarted nerve from within its own turn. The scheduler kept ticking in the logs while nothing answered on the gateway port.Fix
Spawn a detached helper (own session, survives the caller's death) that sends SIGTERM, waits up to 15s for a clean drain, then SIGKILLs. Once the old PID is gone,
Restart=alwaysbrings up a fresh instance. Bounded, and no privileges orsystemctlaccess needed.Verified on the deployment where the hang originally occurred: restart now completes in ~3s with a clean
Application shutdown complete. 29 CLI tests pass.