fix(exchange): apply ExchangeTimeout to key exchange steps 5 and 7 - #1838
Open
pylakey wants to merge 2 commits into
Open
fix(exchange): apply ExchangeTimeout to key exchange steps 5 and 7#1838pylakey wants to merge 2 commits into
pylakey wants to merge 2 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1838 +/- ##
==========================================
+ Coverage 71.35% 71.46% +0.10%
==========================================
Files 509 509
Lines 24137 24137
==========================================
+ Hits 17224 17249 +25
+ Misses 5658 5649 -9
+ Partials 1255 1239 -16 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens
A peer that completes the TCP handshake, answers key exchange steps 1-4, and then stops responding parks the client forever. No timeout fires, and cancelling the context has no effect.
Production goroutine dump (v0.159.0, PFS enabled), two goroutines parked 27 and 21 minutes:
There is no
unencryptedWriterframe betweenClientExchange.Runandconnection.Recv, which localizes both to a directconn.Recvcall — steps 5 or 7 — and rules out step 2, which carries theExchangeTimeoutbound. One goroutine was the primary connection (telegram.Client.Run's errgroup), the other a sub-DC pool connection (pool→tdsync.Supervisor). Both block their owners indefinitely:telegram.Client.Runnever returns, andpool.DC.Closeiscancel()+grp.Wait()with no timeout, so the deferred sub-connection teardown cannot finish either. The connection is never recreated.The caller cancelled its run context every 55 seconds — thousands of times — with no effect, because cancellation cannot interrupt a parked read on a deadline-less socket.
Why
Exchanger.WithTimeoutdocuments itself as setting the deadline of "every exchange request" (exchange/flow.go:62), andmtproto.Options.ExchangeTimeoutas the timeout of "every key exchange request" (mtproto/options.go:50-51). That bound is applied byunencryptedWriter.tryRead(exchange/proto.go:44).The server flow uses it for all three of its reads (
server_flow.go:130,:166,:267). The client flow uses it for every write and for the step 2 read (client_flow.go:34), then reads steps 5 and 7 through a bareconn.Recv(client_flow.go:146,:241). Those are the only two non-testconn.Recvcalls in the module outsidetryReaditself and the steady-state read loop inmtproto/read.go.Below the exchange,
transport.(*connection).Recvderives its socket deadline solely fromctx.Deadline()(transport/connection.go:63) and clears any previous one at:60. With no deadline it sets none, andcodec.Readparks.Nothing else can rescue the socket during this window:
handleClose— the mechanism that closes the connection on context cancellation — is started atmtproto/conn.go:227, afterconnect()returns at:221. So during the key exchange there is neither a deadline nor a closer.Likely cause of the original slip:
readUnencrypteddecodes into abin.Decoder, while steps 5 and 7 need the raw buffer — soconn.Recvwas reached for whentryRead, which reads raw and applies the timeout, was the right one of the two.Why PFS makes it fatal
mtproto.Conn.connectchooses its timeout policy by mode:connectCtxcarriesDialTimeoutacross dial and exchange (connect.go:17-22), so the bare reads inherit a deadline by accident and cannot park forever.DialTimeoutis scoped to the dial alone andconnectPFS(ctx)receives the caller's raw context (connect.go:26-34,:47-48).Two conditions compose; remove either and the wedge disappears. The PFS reasoning is sound — temporary plus permanent key generation legitimately takes longer than a dial — so this PR does not touch it. It makes the documented per-read bound actually apply, after which both modes are bounded by the same mechanism and the PFS policy is harmless.
EnablePFSis opt-in and off by default, which is why this has gone unnoticed.The change
Two lines:
c.conn.Recv→c.tryReadat both sites, using the package's existing unexported helper.go doc -all ./exchangeis identical before and after, 36 declarations).errors.Ischeck: in the affected configuration no error was previously observable — the call hung.DialTimeout.connect's existingmultierr.AppendInto(&rErr, conn.Close())discards the socket, so there is no risk of reusing a connection with a half-consumed frame.Test
exchange/client_flow_timeout_test.godrives the real server flow overtransport.Intermediate.Pipe()with a wrapper whoseSendsucceeds normally and whoseRecvgoes silent after N replies — reproducing a peer that accepts our writes and then stops answering. Allowing 1 reply pins the step 5 read, allowing 2 pins step 7, so a regression at either site fails its own case.On
mainboth subtests hang to their 30s guard; with this change both pass in about a second. The two commits are ordered so this is visible in the PR's own history: the test lands first and fails, the fix follows.The caller context is deliberately
context.Background()— the point is that a context with no deadline of its own must still be bounded by the configuredExchangeTimeout.Note on the error shape: the test's fake conn returns
ctx.Err(), so the assertion iscontext.DeadlineExceeded. Over a real socket the deadline surfaces as a*net.OpErrorwrappingos.ErrDeadlineExceededinstead. Either waytelegram.Client.isPermanentErrordoes not match it, so the reconnect loop retries — the desired behaviour.Not in this PR
Deliberately excluded, each a separate concern: a cancellation watcher for the connect phase, an idle watchdog for connections whose peer goes silent in steady state, context-aware transport locks, and rpc acknowledgement semantics.
This supersedes #1830, which bundled all of the above and which I am closing. Its blast radius — +2840/-105 across 29 files, six independent changes, rewriting
transport/connection.goandmtproto/conn.go— made it unreviewable as a unit, with no partial-accept path. That was my mistake in packaging. The exchange timeout is the only part required to stop the permanent wedge, so it goes first and alone.