pool: don't lose connections handed to a cancelled waiter - #1085
Open
Fizzadar wants to merge 1 commit into
Open
Conversation
DC.acquire's third case waits for a free connection and, if its context is cancelled, deletes its request and does a non-blocking receive on the request channel. reqMap.transfer removes the request key, unlocks, and only then sends the connection, so a waiter that cancels inside that window misses it: the connection stays in the buffered channel nobody reads, never returns to c.free, and is still counted in c.total. It also never dies, so c.stuck never fires and the pool cannot recover. Sub-DC pools are created with max=1, so a single lost connection stops all media downloads on that datacenter until the process restarts. Seen on a bridge where every direct download failed with "acquire connection: context canceled" for 17 hours while the connection itself kept running. Make reqMap.delete report whether the request was still pending, and when transfer already claimed it, wait for the connection and hand it back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fizzadar
marked this pull request as ready for review
July 27, 2026 12:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DC.acquire's third case (wait for a free connection) can permanently lose a live connection.reqMap.transferremoves the waiting request's key from the map, unlocksr.mux, and only then sends the connection on the request's buffered channel. A waiter whose context is cancelled inside that window callsreqMap.delete(now a no-op, the key is gone) and then does a non-blocking receive, which finds the channel still empty. The connection is then sent into a channel nobody will ever read:c.free,c.total,c.dead/c.stuckand the pool cannot recover.Sub-DC pools are created with
max = 1(Client.invokeSub), so one lost connection wedges every subsequentacquireon that datacenter until the process restarts.