[Bug]: Conversation stuck busy-hang on the second message (regression from #2372)

[1688mengdie]

Summary

Summary

After the first turn completes, sending a second message in the same conversation makes the session permanently stuck in a "busy" state: the input is blocked, the pending queue never drains, and the only workaround is creating a new conversation. This is a regression introduced by PR #2372 and is still present on the latest main (2ef884e, which includes 405c1c7). A brand-new conversation works fine for its first message; the bug reliably reproduces on the second message of any conversation.

Area

Agent runtime / core

Reproduction or evidence

Environment: official upstream main at 2ef884e (includes PR #2372 merge 405c1c7), Windows 11, Desktop app.

Steps:

Open a new conversation.
Send a message and wait for the first turn to complete normally.
Send a second message in the same conversation.
The session stays busy forever: message never dispatched, subsequent sends blocked, pending queue never drains. Creating a new conversation is the only recovery.
Root cause chain (references on the 405c1c7 tree):

Frontend state machine settlement depends solely on DialogTurnCompleted + a 500 ms timer. finalizeTurnCompletionState is the only path that issues FINISHING_SETTLED -> IDLE (EventHandlerModule.ts:1110-1207, transitions.ts:26/47). The backend does not emit SessionStateChanged(Idle) on normal completion (only recovered-processing and cancel paths, coordinator.rs:7036/7526), so the frontend resets 500 ms after DialogTurnCompleted — regardless of whether the backend finished persisting.
PR #2372 made the backend completion path hold the session mutation lock for disk I/O. complete_dialog_turn now calls acquire_session_mutation(session_id) (session_manager.rs:7482) and, while holding it, runs load_dialog_turn + merge + save_dialog_turn (the same keyed lock also serializes dialog-turn starts — comment at session_manager.rs:7475-7481). The PR itself acknowledges this can hold session coordination for seconds with large snapshot indexes (PR #2372 Motivation). The backend SessionState::Idle reset (coordinator.rs:2990-3007) therefore happens noticeably later than the frontend's fixed 500 ms reset.
The widened frontend-IDLE / backend-Processing window makes the second submission hit the backend rejection. The frontend busy gate only consults the local state machine (MessageModule.ts:243-246), so once the frontend is IDLE the user can submit. LocalSessionDriver.startTurn transitions START -> PROCESSING before the RPC (LocalSessionDriver.ts:365 -> :452), then the backend rejects because the session is still Processing (coordinator.rs:5761-5787: Err("Session state does not allow starting new dialog")), or silently queues (scheduler.rs:1561-1577, DialogSubmitOutcome::Queued with no event).
The official catch path does not reliably reset the machine on this rejection. The reset condition is ownsProcessingTurn = createdLocalTurnId === activeDialogTurnId (MessageModule.ts:417-429), which does not cover every rejected/queued path. When the machine stays in PROCESSING, every subsequent send is blocked by the busy gate and the pending queue never drains (drain gated on IDLE, MessageModule.ts:632-641) -> permanent busy hang.
Additionally, eventOwnsLatestSessionTurn (EventHandlerModule.ts:183-192) can suppress settlement entirely when the previous turn's DialogTurnCompleted arrives after a newer turn was appended (multi-turn conversations): BACKEND_STREAM_COMPLETED and beginTurnCompletion are both skipped (:2648/:2658) and FINISHING_SETTLED is never issued, leaving the machine in PROCESSING permanently.
Why the first message is unaffected: on first submission the backend is guaranteed IDLE (fresh session), so the normal event flow runs end to end.

Regression attribution: git log 405c1c7..upstream/main contains only 5 unrelated harmonyos/IME commits — no fix has landed since. Before #2372 the same frontend interception logic existed but the backend completion path was lock-free and fast, so the race window was practically unreachable.

Environment, if relevant

BitFun commit: 2ef884e (upstream main, includes PR #2372 405c1c7)
OS: Windows 11
Mode: Desktop app, agentic session

Area

Desktop app

Reproduction or evidence

Environment: official upstream main at 2ef884e (includes PR #2372 merge 405c1c7), Windows 11, Desktop app.

Steps:

1. Open a new conversation.
2. Send a message and wait for the first turn to complete normally.
3. Send a second message in the same conversation.
4. The session stays busy forever: message never dispatched, subsequent sends blocked, pending queue never drains. Creating a new conversation is the only recovery.

Root cause chain (references on the 405c1c7 tree):

1. Frontend state machine settlement depends solely on DialogTurnCompleted + a 500 ms timer. finalizeTurnCompletionState is the only path that issues FINISHING_SETTLED -> IDLE (EventHandlerModule.ts:1110-1207, transitions.ts:26/47). The backend does not emit SessionStateChanged(Idle) on normal completion (only recovered-processing and cancel paths, coordinator.rs:7036/7526), so the frontend resets 500 ms after DialogTurnCompleted — regardless of whether the backend finished persisting.

2. PR fix(sessions): stabilize durable history reconciliation #2372 made the backend completion path hold the session mutation lock for disk I/O. complete_dialog_turn now calls acquire_session_mutation(session_id) (session_manager.rs:7482) and, while holding it, runs load_dialog_turn + merge + save_dialog_turn (the same keyed lock also serializes dialog-turn starts — comment at session_manager.rs:7475-7481). The PR itself acknowledges this can hold session coordination for seconds with large snapshot indexes (PR fix(sessions): stabilize durable history reconciliation #2372 Motivation). The backend SessionState::Idle reset (coordinator.rs:2990-3007) therefore happens noticeably later than the frontend's fixed 500 ms reset.

3. The widened frontend-IDLE / backend-Processing window makes the second submission hit the backend rejection. The frontend busy gate only consults the local state machine (MessageModule.ts:243-246), so once the frontend is IDLE the user can submit. LocalSessionDriver.startTurn transitions START -> PROCESSING before the RPC (LocalSessionDriver.ts:365 -> :452), then the backend rejects because the session is still Processing (coordinator.rs:5761-5787: Err("Session state does not allow starting new dialog")), or silently queues (scheduler.rs:1561-1577, DialogSubmitOutcome::Queued with no event).

4. The official catch path does not reliably reset the machine on this rejection. The reset condition is ownsProcessingTurn = createdLocalTurnId === activeDialogTurnId (MessageModule.ts:417-429), which does not cover every rejected/queued path. When the machine stays in PROCESSING, every subsequent send is blocked by the busy gate and the pending queue never drains (drain gated on IDLE, MessageModule.ts:632-641) -> permanent busy hang.

5. Additionally, eventOwnsLatestSessionTurn (EventHandlerModule.ts:183-192) can suppress settlement entirely when the previous turn's DialogTurnCompleted arrives after a newer turn was appended (multi-turn conversations): BACKEND_STREAM_COMPLETED and beginTurnCompletion are both skipped (:2648/:2658) and FINISHING_SETTLED is never issued, leaving the machine in PROCESSING permanently.

Why the first message is unaffected: on first submission the backend is guaranteed IDLE (fresh session), so the normal event flow runs end to end.

Regression attribution: git log 405c1c7..upstream/main contains only 5 unrelated harmonyos/IME commits — no fix has landed since. Before #2372 the same frontend interception logic existed but the backend completion path was lock-free and fast, so the race window was practically unreachable.

Environment, if relevant

No response

[bobleer]

Thanks for the very detailed report — the symptom and the repro are exactly right, and the "only a new conversation recovers" detail was the most useful clue. The root cause turned out to be somewhere else than the analysis in the report, so here is what the runtime evidence showed.

What actually happens

I instrumented the whole chain — the emit site in the execution engine, the host delivery site, the webview surface gate, the SessionStream admission point, and the settlement handler — and reproduced it on macOS Desktop.

For every completed turn the backend emits once and the host delivers once, but the webview surface gate saw the event 3 times, and SessionStream returned 1 apply + 2 drop.

14c4b5c73 (#2325) moved the stateful runtime-session cursor routing into every logical Tauri listener. SessionStream is a per-(surface, session) ordering gate, and its cursor dedup exists to suppress a genuinely re-delivered event (snapshot and live stream overlapping), so it has to decide about a delivery. With one native listener per logical subscriber, a single delivery was pushed through that gate once per subscriber: the first advanced the applied cursor, and every later subscriber was refused as not-ahead and silently dropped.

agentic://dialog-turn-completed has several independent subscribers — turn settlement in AgenticEventListener, plus useAssistantBootstrap, useDialogCompletionNotify, useAcpPlan — so which one actually received it depended on registration order. On the reproducing run the settlement subscriber was one of the drops, so handleDialogTurnComplete never ran, the state machine stayed in PROCESSING, and every later message sat in the pending queue forever.

The logs contain a clean control group for this: agentic://dialog-turn-started and agentic://dialog-turn-interrupted have exactly one subscriber each, arrived exactly once, and always got apply. Only the terminal event, with its multiple subscribers, was affected.

On the #2372 attribution

The report attributes this to #2372 holding the session mutation lock across disk I/O and widening a frontend-IDLE / backend-Processing race. Instrumentation on that path rejected it:

• complete_dialog_turn held the mutation lock for about 4 ms, not seconds.
• The backend reset SessionState to Idle correctly and the scheduler observed a normal Completed outcome.
• The second message was never submitted to the backend at all — no rejection and no silent queue on the backend side. The frontend busy gate blocked it locally because the machine was still PROCESSING, and the backend queue was empty every time it was checked.

So the widened-race and backend-rejection steps in the chain did not occur. The regression source is #2325, not #2372. Steps 1 and 5 of your analysis are also not what fired here — eventOwnsLatestSessionTurn returned true and alreadyHandledTerminal was false on every settlement, so the event was not being suppressed by ownership; it never reached the handler in the first place.

Worth noting separately: your point that a missed terminal event has no recovery path is correct and independent of this bug. Settlement is purely edge-triggered on DialogTurnCompleted, handleSessionStateChanged only repairs FINISHING + backend Idle (never PROCESSING + Idle), and machineContext.backendSyncedAt is written but never read. That is why a single lost event is unrecoverable rather than self-healing. It is not the cause here, but it is what turned a dropped event into a permanent hang.

Fix

1.0.0-explore already carried the fix for this (5bb566c14 by @wsp), which reached the same diagnosis independently. #2384 cherry-picks it onto main unchanged so the branches do not diverge: each native Tauri listener is shared across adapter instances, surface and cursor routing run once per delivery, and logical subscribers are fanned out afterwards. It also closes the cross-adapter variant that Peer Device Mode can hit while several transport adapters are alive.

Post-fix verification across 13 consecutive turns in 5 sessions: 1 arrival, 1 apply, 0 drop per completed turn, settlement reached finalizeTurnCompletionState 13/13, and 3 messages queued while busy all drained normally. This included turns with 13/20/28/31/40 tool rounds and session switches mid-turn.

#2384 will close this issue when it merges.