fix(voice): avoid dropping turns and resuming early while a speech is paused - #2204
Open
rosetta-livekit-bot[bot] wants to merge 4 commits into
Open
fix(voice): avoid dropping turns and resuming early while a speech is paused#2204rosetta-livekit-bot[bot] wants to merge 4 commits into
rosetta-livekit-bot[bot] wants to merge 4 commits into
Conversation
🦋 Changeset detectedLatest commit: 75a80ab The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
… on the promise Port review of #2204 found the settlement disposition in onTurnSettled diverged from Python in both directions. A cancelled end-of-turn bounce resolves rather than rejects (the body observes the abort and returns), so the resume fired on a decision that was torn down instead of decided; and a bounce that rejected was swallowed, leaving nothing armed and the agent's audio output paused indefinitely. Add an explicit Task.cancelled accessor and check it, then call the resume synchronously as Python does. Also clear overlapOpen before the sentinel-write early return, guard the teardown end-of-agent-speech report against rejection, and record the known _rt_turn_detection_enabled divergence at the cancel gate. Strengthens the two "keeps the resume armed" tests to drive the real timer to expiry and assert the pause is released, with a cancelling counterpart proving they are not vacuous. Co-authored-by: Cursor <cursoragent@cursor.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ng the live signal Task.cancelled consulted the controller's current abort state, so the value was not a property of how the task finished. Aborting afterwards — including through a shared controller, or via runEOUDetection's unconditional cancel of the previous bounce — retroactively reclassified an already-completed task as cancelled, which made onTurnSettled suppress the resume for a decision that had completed normally and left the paused speech wedged. Capture the disposition once, immediately before the result future settles, and have the getter return that snapshot. On the rejection path a failure unrelated to the abort is recorded as an exceptional completion rather than a cancellation, matching asyncio. An ignored abort deliberately still reports cancelled: the EOU bounce observes the signal and returns normally, and treating that as a plain completion would reinstate the original bug. Documents the exact guarantee, the ignored-abort rule, and the shared-controller limitation on the accessor itself. Also asserts the synthetic agent-ended overlap sentinel at the real end of agent speech, closing the coverage gap that let the incorrect overlapOpen hoist pass. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # agents/src/voice/agent_activity.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports livekit/agents#6662 to keep adaptive interruption inference alive while speech is paused, commit unjudged realtime overlaps, and defer false-interruption resume until the active turn decision settles.
Adds the corresponding paused-speech, overlap-lifecycle, teardown, skipped-reply, and server-side turn-taking regression coverage plus a patch changeset for
@livekit/agents.Source diff coverage
livekit-agents/livekit/agents/voice/agent_activity.py: ported toagents/src/voice/agent_activity.ts. Asyncio timers/tasks are translated toNodeJS.TimeoutandTask; timeout values remain in the target framework’s millisecond convention. Includes turn-gate, deferred resume ownership, cancellation consolidation, skipped/server-side reply exceptions, and pause interruption teardown.livekit-agents/livekit/agents/voice/audio_recognition.py: ported toagents/src/voice/audio_recognition.ts. Preserves detector streams across pause/resume, tracks the currently open overlap independently from turn history, closes overlaps exactly once, and tears them down on a real agent-speech end.tests/test_false_interruption_resume.py: adapted to the co-located Vitest counterpartagents/src/voice/agent_activity.test.ts. All eight source scenarios are covered with target-native fake timers,Task/Future,SpeechHandle, and activity stubs.tests/test_realtime_adaptive_interruption.py: adapted to the existing JS counterpartagents/src/voice/realtime_adaptive_interruption.test.ts. Ports the modified unjudged/confirmed turn-gate assertions and all six added detector-stream lifecycle assertions.agents/src/utils.tsandagents/src/utils.test.ts: target-only addition, no Python counterpart. Acancelledgetter is added toTask(agents/src/utils.ts:664-665) backed by a disposition snapshot taken at settlement (:501,:565,:573), standing in for Python'sasyncio.Task.cancelled(), which has no JS equivalent. It is marked@internal. Two non-guarantees are deliberate and documented at the getter: an abort that arrives after the task already settled reportstrue, and aborting a controller shared with sibling tasks marks those siblings cancelled. Both preserve the Python behavior this port depends on; neither is a general-purpose cancellation predicate.Validation
pnpm exec vitest run agents: 111 files, 1571 passed, 5 skippedpnpm --filter @livekit/agents typecheckpnpm --filter @livekit/agents lintpnpm --filter @livekit/agents buildpnpm buildpnpm --filter @livekit/agents api:checkwas additionally attempted, but the repository’s current API Extractor rejects the existingexport * assyntax before API comparison.Ported from livekit/agents#6662
Original PR description
Three fixes around a paused agent speech: the turn gate, the overlap the barge-in verdict comes from, and the resume timer.
Turns silently dropped (realtime without STT)
a. The gate treated a paused speech as still deciding.
on_end_of_turndropped a turn whenever the agent had a live, uninterrupted speech and no interruption had been flagged, on the assumption that a verdict might still arrive. Once the speech is paused that assumption is permanently false — pausing silences the agent and ends the overlap, so no verdict is coming — and the user was ignored for a whole turn while the agent resumed mid-sentence.Only a confirmed verdict suppresses a turn now. An unjudged overlap commits, so a barge-in verdict that lands late can interrupt the agent on what may have been a backchannel — an unwanted interruption is audible, so the user hears the agent stop and carries on, while a discarded turn is invisible and leaves them repeating themselves to an agent that never registered them.
b. Pausing tore down the overlap that verdict comes from. Pausing signalled end-of-agent-speech to the adaptive detector, which reset its stream and closed the overlap window. The verdict is emitted when the user's speech ends, so with the window gone none ever arrived: no late detection could land, and every paused backchannel would interrupt the agent.
A pause no longer ends the detector's overlap and a resume no longer restarts its stream. The user's speech ending closes it instead, which emits the verdict and stops inference.
Resume racing the turn decision
The resume timer counts
false_interruption_timeoutfrom the VADEND_OF_SPEECHevent, while the turn commits atlast_speaking_time + endpointing_delay— measured from before the VAD silence window. When the detector reads the pause as mid-utterance that delay becomesmax_delay, so with the shipped defaults the resume lands 0.25s before the commit and the agent is cut off right after resuming.An open turn decision now owns the paused speech: the timer defers to it and resumes only once the turn is dropped, while a committed turn cancels the resume synchronously from
on_end_of_turn. The elapsed timeout counts toward that wait, so a resume with no decision pending keeps its current latency.Cue voice E2E
Evidence provenance: the Cue runs below were captured at
cb5422a9, which is no longer the current head. The head is now merge commit75a80aba, which mergesorigin/main(including #2209) into this branch to clear a conflict. That merge resolved a single test-file import union and changed none of this PR's source logic —git diff cb5422a9 75a80aba -- agents/src/voice/agent_activity.tscontains no change to any resume, settlement, false-interruption or end-of-turn-task line. The behavior these runs exercise is therefore unchanged at the current head.Verified exact head
cb5422a99e6dc1575d8e2dcb93cac049884abc8bagainstorigin/main(8260fa1e) in voice mode with real providers:inference.VAD(silero),deepgram/nova-3STT,openai/gpt-4.1-miniLLM, andcartesia/sonic-3TTS. No framework source was modified for the run.The scenario is the resume/turn-decision race: a barge-in over live agent speech pauses that speech, and the question is whether the false-interruption resume fires before the turn decision settles. One two-turn script and one predicate were used for both arms:
cb5422a9: sessionsid_c61ffc33c374, agentcue-2204-head, predicate resolved in 22907 ms. Decisive event at offset 32:{"marker":"resume_deferred","eouTaskOpen":true,"pausedStillSet":true,"armOrdinal":1.0,"userTurnsCommitted":0.0}. Nofalse_interruptionevent occurs anywhere in the session, and the commit recordsresumedBefore: 0.0— the resume waited for the decision.8260fa1e: sessionsid_bf98ff262ceb, agentcue-2204-base, predicate failed. Decisive event at offset 33:{"marker":"false_interruption","resumesTotal":1.0,"armOrdinal":1.0,"resumed":true,"eouTaskOpen":true,"userTurnsCommitted":0.0}— it resumed 297 ms after the barge-in with the turn decision still open, and the commit recordsresumedBefore: 1.0.Shared precondition. Both arms reached the identical state before diverging, so the negative arm is meaningful rather than a setup failure. Each logged
pause_armedat offset 31 witheouTaskOpen: true,agentState: "listening",ordinal: 1,timeoutMs: 300,userTurnsCommitted: 0; the timer then expired 307 ms later on head and 297 ms on baseline. The arms diverge only in what happens at expiry.Arm attribution. Each arm ran under a distinct
agentNamewith exactly one worker registered for that name, and each session is self-identifying: abuild_identityevent reads off the loaded prototypes of the code under test, so the artifacts prove their own provenance rather than relying on run bookkeeping. Head reportshasCancelFalseInterruptionTimer: true,hasEndOfTurnTaskGetter: true,startFalseInterruptionTimerbody hash9aaf14da0aec(length 2434); baseline reportsfalse,false, hash4b797e597651(length 1480). Each also carries its owngitShaand anagentsModulePathpointing into its own worktree'sdist.Limitations.
agent_false_interruptionis not in theAgentSessionEventproto oneof, so it is relayed through_emitDebugMessagefrom the framework's own emit site.pause_armedandresume_deferredcome from a 10 ms poller, so their timestamps carry ±10 ms.endpointing.minDelay = maxDelay = 20s, Deepgramendpointing = 1500ms, VADminSilenceDuration = 100ms,interruption.mode = 'vad',minDuration = 300ms,falseInterruptionTimeout = 300ms. The same ordering bug exists at production defaults but with a much narrower window.Merge-order note (resolved)
#2209 has landed on
mainasd96cd050, and this branch has been merged up to it. The predicted conflict inagents/src/voice/agent_activity.test.tswas a strict import union and is resolved in merge commit75a80aba: the test file now importsAgentActivity, onEnterStorage, transcriptsEquivalentalongside the typesEndOfTurnInfo, PreemptiveGenerationInfo. No other file conflicted, and none of this PR's resume or settlement logic changed in the merge. The agents suite is green at 112 files, 1594 passed, 5 skipped — up from 1571 pre-merge, which is #2209's tests joining the tree.