Skip to content

fix: #175 BidiStream SIGTERM causes CPU hot loop and pod stuck in Terminating#176

Merged
igalshilman merged 6 commits intorestatedev:mainfrom
introspection-org:julian/fix-disconnect-hotloop
Feb 26, 2026
Merged

fix: #175 BidiStream SIGTERM causes CPU hot loop and pod stuck in Terminating#176
igalshilman merged 6 commits intorestatedev:mainfrom
introspection-org:julian/fix-disconnect-hotloop

Conversation

@brightsparc
Copy link
Contributor

fixes: #175

When a pod receives SIGTERM during an active BidiStream invocation, two bugs caused the worker to spin at ~82% CPU and never exit:

  1. ReceiveChannel.__call__() blocked forever on an empty queue after the disconnect event was consumed
  2. create_poll_or_cancel_coroutine() fed empty body frames (b'') to the VM, creating a tight loop with no useful await points

Changes

  • server_types.py: Return synthetic http.disconnect when queue is drained and channel is disconnected
  • server_context.py: Skip notify_input() for empty body frames; add 30s timeout to block_until_http_input_closed() in leave()
  • tests/disconnect_hotloop.py: Regression tests for both fixes

@github-actions
Copy link

github-actions bot commented Feb 23, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@brightsparc
Copy link
Contributor Author

I have read the CLA Document and I hereby sign the CLA

@tillrohrmann
Copy link
Contributor

Thanks a lot for creating this fix @brightsparc. cc @igalshilman for review.

@brightsparc
Copy link
Contributor Author

Thanks a lot for creating this fix @brightsparc. cc @igalshilman for review.

Cool, looking forward to getting this landed to reduce the hot loop when draining pods.

assert isinstance(body, bytes)
# Skip empty body frames to avoid hot loop (see #175)
body = chunk.get("body", None)
if body is not None and len(body) > 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good one!

Copy link
Contributor

@igalshilman igalshilman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! Thank you @brightsparc

@igalshilman igalshilman merged commit 9437df9 into restatedev:main Feb 26, 2026
7 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Feb 26, 2026
@igalshilman
Copy link
Contributor

note to self: run verification tests once the docker image is built.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: BidiStream SIGTERM causes CPU hot loop and pod stuck in Terminating

3 participants