Audio routing freeze mid-call (1.6.9)

[ullaimlu]

Description

Audio routing stops working mid-call. user_state stays stuck on "listening", no user_state_changed events are emitted, STT produces no output, and the call hangs up after inactivity prompts.

User continues speaking throughout (confirmed via separate audio stream subscription), but the audio doesn't reach STT/VAD/turn detection components.

Happens randomly during multi-turn conversations - not always on the same turn. Can't reproduce reliably but occurs frequently enough to block production deployment.

Expected vs Actual

Expected: Turn detection works every turn, user_state transitions normally, STT produces transcripts

Actual: Audio routing freezes, user_state stuck on "listening", no user_state_changed events, no STT output

Environment

• Version: livekit-agents 1.6.9, livekit 1.1.14
• Deployment: Self-hosted, outbound SIP
• Models: Deepgram nova-3 (STT), gpt-4.6-luna (LLM), Deepgram aura-2 (TTS)

Comprehensive Testing

Tested multiple configurations to isolate the issue. All configurations exhibit the same freeze behavior.

Configuration 1: Silero VAD + Turn Detector v1-mini

vad=inference.VAD(model="silero", min_speech_duration=0.05, min_silence_duration=0.25, activation_threshold=0.4)
turn_handling=TurnHandlingOptions(
    turn_detection=inference.TurnDetector(version="v1-mini"),
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={"mode": "vad", "min_words": 3},
)

Result: Freezes mid-call

Configuration 2: No VAD + Turn Detector v1-mini

vad=None  # Explicitly disabled
turn_handling=TurnHandlingOptions(
    turn_detection=inference.TurnDetector(version="v1-mini"),
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={"mode": "vad", "min_words": 3},
)

Result: Freezes mid-call
Evidence: Logs confirm Session VAD: None

Configuration 3: No VAD + STT Endpointing

vad=None
turn_handling=TurnHandlingOptions(
    turn_detection="stt",  # Deepgram handles endpointing
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={"mode": "vad", "min_words": 3},
)

Result: Freezes mid-call

What This Proves

The bug is NOT caused by:

• Silero VAD specifically (tested with vad=None)
• Turn Detector v1-mini (tested with turn_detection="stt")
• Corrupted VAD/STT instance state (tested with fresh instances per turn)
• Network/SIP issues (confirmed audio frames arrive via separate subscription)

The bug appears to be a framework-level issue in:

• Audio routing in AudioRecognition
• Audio frame delivery to STT/VAD/turn detection components
• Affects all turn detection methods equally

Additional Testing and Workarounds (All Failed)

Configuration changes tested:

• Removed allow_interruptions=False from all session.say() / generate_reply() calls, replaced with temporary elevation of interruption.min_words threshold - no change
• Switched from deprecated silero.VAD.load() to inference.VAD(model="silero") - no change
• Switched from text-based turn_detector.multilingual.MultilingualModel to audio-based inference.TurnDetector(version="v1-mini") - no change
• Created fresh VAD and STT instances per session (not shared across sessions) - still freezes

Runtime workarounds tested:

1. Runtime VAD refresh - agent.update_options(vad=fresh_vad) when freeze detected - still freezes
2. Override _update_vad() - Force fresh VAD instances on every activity - still freezes, logging shows VAD channel exists (closed=False) but emits no events
3. Disable STT pipeline reuse - Force fresh STT WebSocket each time - still freezes

Evidence During Freeze

When freeze occurs:

• user_state = "listening" (stuck)
• agent_state = "listening"
• No user_state_changed events emitted
• No STT output/transcripts
• Audio frames arrive at room (verified via rtc.AudioStream.from_participant())
• VAD channel exists and closed=False (when VAD is used)

Example Timeline

Successful turns followed by freeze (common pattern)

16:04:51 - Agent speaks (turn 1)
16:04:53 - User response detected successfully
16:04:57 - Agent speaks (turn 2)
16:05:00 - User response detected successfully
16:05:03 - Agent speaks (turn 3)
16:05:06 - User response detected successfully
16:05:09 - Agent speaks (turn 4)
16:05:34 - Agent waits for user response
          [FREEZE OCCURS - audio routing stops]
          User continues speaking (confirmed via separate audio stream)
          No user_state_changed events emitted
          No STT transcription output
16:05:39 - WARNING: No user_state events for 5.0s agent_state=listening user_state=listening
16:05:41 - Agent: inactivity prompt 1
16:05:46 - WARNING: No user_state events for 5.0s
16:05:48 - Agent: inactivity prompt 2
16:05:53 - WARNING: No user_state events for 5.0s
16:05:55 - Agent: inactivity prompt 3
16:06:00 - WARNING: No user_state events for 5.0s
16:06:09 - Agent: farewell message and hangup

Note: The freeze can occur after any number of successful turns (sometimes turn 2, sometimes turn 5+). No consistent pattern to when it occurs.

Impact

Blocking production deployment. High call failure rate, poor user experience. Affects self-hosted deployments with outbound SIP.

Root Cause Analysis Needed

Audio frames reach the room but don't reach STT/VAD/turn detection. Need framework investigation of:

1. AudioRecognition audio routing - How frames are routed to detection components during multi-turn conversations
2. STT stream lifecycle - Why STT stops consuming frames mid-call
3. Activity transitions - Whether audio routing breaks during activity handoffs

The fact that all turn detection methods fail identically suggests the issue is in the shared audio routing infrastructure, not in any specific detection component.

[longcw]

Thanks for the report. The rtc.AudioStream.from_participant() subscription you used is a separate stream from the one RoomIO reads, so it cannot tell us whether those frames reached the session.

Please record the audio that RoomIO delivers, then reproduce the freeze:

await session.start(..., record={"audio": True})

The file lands in job_ctx.session_directory / "audio.ogg". It contains only the frames that RoomIO passed to the session, with no padding over gaps.

If the user audio stops at the freeze, the fault is in the track subscription. If the user audio continues through the freeze, the fault is downstream, and we will take it from there. Please attach the file and the DEBUG logs for the same call.

[ullaimlu]

Thanks for the report. The rtc.AudioStream.from_participant() subscription you used is a separate stream from the one RoomIO reads, so it cannot tell us whether those frames reached the session.

Please record the audio that RoomIO delivers, then reproduce the freeze:

await session.start(..., record={"audio": True})
The file lands in job_ctx.session_directory / "audio.ogg". It contains only the frames that RoomIO passed to the session, with no padding over gaps.

If the user audio stops at the freeze, the fault is in the track subscription. If the user audio continues through the freeze, the fault is downstream, and we will take it from there. Please attach the file and the DEBUG logs for the same call.

Audio freeze recording results

debug_test.log
room-audio.MP3
server-recording.MP3

Thanks for your answer, i reproduced the bug and recorded the session as requested. Found that user audio stops reaching the session.

The audio.ogg file from session.start(record={"audio": True}) shows no user audio after the freeze occurs, even though I have a separate server recording where you can clearly hear me speaking. So the audio is reaching the room but not the session.

First two user responses worked fine:

• 12:42:40 - User: "Sí, señora." (transcribed ok)
• 12:42:47 - User: "Sí." (transcribed ok)

After agent third message at 12:43:26, everything stops:

• 12:43:31 - First warning: "No user_state events for 5.0s"
• More warnings every 5 seconds
• Agent repeats "no te escucho" a few times then hangs up

I was speaking the whole time but nothing got transcribed.

I also added monitoring to check if it was a Deepgram STT issue, but no.

Files

Attaching:

1. room-audio.MP3 (audio.ogg) - session recording (no user audio after freeze)
2. debug_test.log - full DEBUG logs
3. server-recording.MP3 server recording if needed - has user audio during freeze

Thanks for your help.

[ullaimlu]

Hey @longcw! just wanted to check if you’ve had a chance to take a look at this, i’d really appreciate any feedback when you get a chance. Thank you!

[longcw]

Thank you for the recording. It shows that RoomIO stopped receiving your audio: the user channel goes to digital silence right after the "Sí." at 12:42:47, while the server recording still has you speaking in four later places.

The session logs no stream closed and no error at that moment, so the track subscription stayed open and the frames simply stopped arriving or only silent frames arrived.

Could you share an example that can reproduce this issue?

[ullaimlu]

Hi @longcw, thanks for the response.

We investigated further on the infrastructure side and found that part of the problem was related to our SIP setup — in the original recording I shared, audio wasn't reaching the room at all. After applying some configuration changes, we confirmed that this was no longer the case.

We then implemented the recording approach you suggested (session.start(..., record={"audio": True})) in our production environment. After reviewing several calls, we confirmed that user audio is now reaching the room correctly — unlike before, the audio.ogg recordings clearly capture the user speaking.

However, the freeze still occurs. We can hear the user speaking in the room recording while the agent is stuck in our user inactivity loop. So the audio arrives at the room but stops reaching STT/VAD internally.

Regarding a reproducible example: this occurs under high concurrency in our production telephony environment and we haven't been able to reproduce it in isolation.

We noticed PR #7005 seems to address this scenario. We'll continue testing on our side and try to gather more evidence using that PR as reference.