Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/quiet-clouds-warm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents': patch
---

Disable the default AEC warmup for outbound SIP calls while preserving explicit settings.
55 changes: 55 additions & 0 deletions agents/src/voice/agent_session.test.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,72 @@
// SPDX-FileCopyrightText: 2026 LiveKit, Inc.
//
// SPDX-License-Identifier: Apache-2.0
import { ParticipantKind, type RemoteParticipant } from '@livekit/rtc-node';
import { describe, expect, it, vi } from 'vitest';
import { AgentSession, resolveRecordingOptions } from './agent_session.js';
import { AgentSessionEventTypes, createUserInputTranscribedEvent } from './events.js';
import { SpeechHandle } from './speech_handle.js';

type AgentSessionInternals = AgentSession & {
_agentState: string;
_aecWarmupTimer: NodeJS.Timeout | null;
_userState: string;
_setUserAwayTimer: () => void;
};

describe('AgentSession AEC warmup', () => {
it.each([
[ParticipantKind.SIP, {}, null],
[ParticipantKind.SIP, { 'sip.ruleID': 'SDR_inbound' }, 3000],
[ParticipantKind.STANDARD, {}, 3000],
] as const)(
'uses the call type default for participant kind %s with attributes %o',
(kind, attributes, expectedDuration) => {
const session = new AgentSession({ vad: null });
const participant = { info: { kind }, attributes } as RemoteParticipant;

session._onRoomIOParticipantLinked(participant);

expect(session.sessionOptions.aecWarmupDuration).toBe(expectedDuration);
expect(session._aecWarmupRemaining).toBe(expectedDuration ?? 0);
},
);

it.each([null, 0, 1500] as const)(
'preserves an explicit AEC warmup duration of %s for outbound SIP',
(duration) => {
const session = new AgentSession({ vad: null, aecWarmupDuration: duration });
const participant = {
info: { kind: ParticipantKind.SIP },
attributes: {},
} as RemoteParticipant;

session._onRoomIOParticipantLinked(participant);

expect(session.sessionOptions.aecWarmupDuration).toBe(duration);
expect(session._aecWarmupRemaining).toBe(duration ?? 0);
},
);

it('cancels AEC warmup that already started for outbound SIP', () => {
const session = new AgentSession({ vad: null });
const internals = session as AgentSessionInternals;
const timer = setTimeout(() => {}, 10_000);
internals._aecWarmupTimer = timer;
const clearTimeoutSpy = vi.spyOn(globalThis, 'clearTimeout');
const participant = {
info: { kind: ParticipantKind.SIP },
attributes: {},
} as RemoteParticipant;

session._onRoomIOParticipantLinked(participant);

expect(clearTimeoutSpy).toHaveBeenCalledWith(timer);
expect(internals._aecWarmupTimer).toBeNull();
clearTimeoutSpy.mockRestore();
});
});

describe('AgentSession.run', () => {
it('forwards inputModality to generateReply', async () => {
const session = new AgentSession();
Expand Down
31 changes: 29 additions & 2 deletions agents/src/voice/agent_session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@
import { type JsonObject, Struct } from '@bufbuild/protobuf';
import { Mutex } from '@livekit/mutex';
import { AgentSession as pb } from '@livekit/protocol';
import type { AudioFrame, Room } from '@livekit/rtc-node';
import {
type AudioFrame,
ParticipantKind,
type RemoteParticipant,
type Room,
} from '@livekit/rtc-node';
import { ThrowsPromise } from '@livekit/throws-transformer/throws';
import type { TypedEventEmitter as TypedEmitter } from '@livekit/typed-emitter';
import type { Context, Span } from '@opentelemetry/api';
Expand Down Expand Up @@ -120,6 +125,9 @@ import type {
import { migrateLegacyOptions, stripUndefined } from './turn_config/utils.js';
import { setParticipantSpanAttributes } from './utils.js';

const SIP_RULE_ID_ATTR = 'sip.ruleID';
const DEFAULT_AEC_WARMUP_DURATION = 3000;

export interface AgentSessionUsage {
/** List of usage summaries, one per model/provider combination. */
modelUsage: Array<Partial<ModelUsage>>;
Expand Down Expand Up @@ -291,7 +299,7 @@ export type AgentSessionOptions<UserData = UnknownUserData> = {
/**
* Duration in milliseconds for AEC (Acoustic Echo Cancellation) warmup, during which
* interruptions from audio activity are suppressed. Set to `null` to disable.
* @defaultValue 3000
* Defaults to 3000, or `null` for outbound SIP calls.
*/
aecWarmupDuration?: number | null;

Expand Down Expand Up @@ -417,6 +425,7 @@ export class AgentSession<
private idleReleased = new Event();

private _aecWarmupTimer: NodeJS.Timeout | null = null;
private readonly _aecWarmupDurationExplicit: boolean;

// Connection options for STT, LLM, and TTS
private _connOptions: ResolvedSessionConnectOptions;
Expand Down Expand Up @@ -510,6 +519,7 @@ export class AgentSession<
constructor(options: AgentSessionOptions<UserData> = {}) {
super();

this._aecWarmupDurationExplicit = options.aecWarmupDuration !== undefined;
const { agentSessionOptions: opts, legacyVoiceOptions } =
migrateLegacyOptions<UserData>(options);

Expand Down Expand Up @@ -1665,6 +1675,23 @@ export class AgentSession<
}
}

/** @internal */
_onRoomIOParticipantLinked(participant: RemoteParticipant): void {
if (this._aecWarmupDurationExplicit) {
return;
}

const isOutboundSip =
participant.info.kind === ParticipantKind.SIP && !participant.attributes[SIP_RULE_ID_ATTR];
this.sessionOptions.aecWarmupDuration = isOutboundSip ? null : DEFAULT_AEC_WARMUP_DURATION;
this._aecWarmupRemaining = this.sessionOptions.aecWarmupDuration ?? 0;

if (isOutboundSip && this._aecWarmupTimer !== null) {
clearTimeout(this._aecWarmupTimer);
this._aecWarmupTimer = null;
}
}

private _onUserInputTranscribed(ev: UserInputTranscribedEvent): void {
if (ev.isFinal && this._userState !== 'speaking') {
if (this._userState === 'away') {
Expand Down
1 change: 1 addition & 0 deletions agents/src/voice/room_io/room_io.ts
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,7 @@ export class RoomIO {
}

this.participantAvailableFuture.resolve(participant);
this.agentSession._onRoomIOParticipantLinked(participant);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Echo-cancellation warmup is not disabled for outbound phone calls when the caller is already in the room

The session is only told about the linked caller from the connect path (this.agentSession._onRoomIOParticipantLinked(participant) at agents/src/voice/room_io/room_io.ts:262) and not from the path that switches to a caller that has already joined, so those outbound phone calls keep the 3-second interruption block.
Impact: For outbound calls where the callee is already present when the agent focuses on them, early speech from the callee is still ignored during the greeting.

Two code paths resolve the linked participant, only one notifies the session

RoomIO.setParticipant() also resolves participantAvailableFuture directly when the target participant is already in room.remoteParticipants (agents/src/voice/room_io/room_io.ts:473-483) and never calls _onRoomIOParticipantLinked. This public method is the documented way to focus the session on a SIP callee (see examples/src/telephony_amd.ts:75). If the SIP participant is created before setParticipant is called, the linked-participant hook never runs and sessionOptions.aecWarmupDuration stays at the 3000 ms default even for an outbound SIP call, which is exactly the case this PR aims to fix.

Prompt for agents
RoomIO has two places where the linked participant future gets resolved: onParticipantConnected (agents/src/voice/room_io/room_io.ts:261) and setParticipant (around agents/src/voice/room_io/room_io.ts:473-483, when the target participant is already connected). The PR only notifies AgentSession from the first one, so outbound SIP detection is skipped when the session is switched onto an already-connected SIP participant (a supported flow, see examples/src/telephony_amd.ts). Consider extracting a single helper that resolves the future and notifies the session, and use it in both places.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is unclear why you would have another participant before a SIP call and have the callee waiting on the line? In that case, it is almost like an inbound call.

};

private onParticipantDisconnected = (participant: RemoteParticipant) => {
Expand Down
Loading