feat(ultravox): implement say() via ForcedAgentMessage - #7001
feat(ultravox): implement say() via ForcedAgentMessage#7001ssgutierrez42 wants to merge 15 commits into
Conversation
…mption Deployment-only plumbing (do NOT include in the upstream PR). When the 11x livekit-agent worker pulls livekit-plugins-ultravox from this fork via a uv git source, uv discovers this repo's workspace and resolves livekit-agents (plus its silero/etc. extras) from the git checkout as file URLs, which uv rejects as transitive URL deps. Excluding the ultravox plugin from the workspace makes it resolve standalone so its livekit-agents>=1.6.9 requirement comes from PyPI, while only the forked ultravox plugin is sourced from git.
…-ai/livekit-agents into deploy/ultravox-external-voice
Add supports_say=True and implement RealtimeSession.say() using
Ultravox's ForcedAgentMessage wire protocol with uninterruptible=true.
This enables session.say() to speak verbatim text server-side without
LLM processing, with server-enforced uninterruptibility — matching the
pattern used by the xAI plugin's force_message primitive.
The existing _handle_state_event("speaking") handler auto-resolves the
pending generation future, so no new lifecycle plumbing is needed.
…thub.com/11x-ai/livekit-agents into deploy/ultravox-forced-agent-message
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| @@ -70,7 +69,6 @@ | |||
| livekit-plugins-telnyx = { workspace = true } | |||
| livekit-plugins-trugen = { workspace = true } | |||
| livekit-plugins-turn-detector = { workspace = true } | |||
There was a problem hiding this comment.
is there a reason for this move?
There was a problem hiding this comment.
No - fixing this.
|
|
||
| return fut | ||
|
|
||
| def say( |
There was a problem hiding this comment.
@ livekit team, let me know what you think of this approach --
It allows the use of Ultravox's ForcedAgentMessage https://docs.ultravox.ai/apps/datamessages, which can force the underlying TTS engine to "say" anything.
Here we expand on the say() LiveKit functionality
https://docs.livekit.io/agents/multimodality/audio/#session-say
However, given existing documentation, I am curious on the best home for this functionality -- one of:
say()- current PRgenerate_replywhenallow_interruptions=False- A new function altogether.
This update is influenced by the xAI plugin, which also overrides say with a forced_message approach: https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.py
There was a problem hiding this comment.
|
|
||
| return fut | ||
|
|
||
| def say( |
There was a problem hiding this comment.
For a future PR, we may evaluate how-to pass allow_interruptions=False to this function. It maps to ForcedAgentMessageEvent.uninterruptible
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
2 participants
Summary
say()method toRealtimeSessionthat speaks verbatim text via Ultravox'sForcedAgentMessageprotocolForcedAgentMessageEventtype to the events module and wire it into theUltravoxEventTypeunionsupports_say=Truein Ultravox model capabilities so the framework knowssay()is availableUserTextMessageEventwith empty text), a pattern seen throughout the plugin.Why
LiveKit's
session.say()lets agents speak exact text without LLM processing. This is useful for scripted content.The upstream Ultravox plugin didn't implement it because the Ultravox WebSocket API uses its own
ForcedAgentMessagedata message type rather than other patterns in the project (e.g.response.create).This PR translates
say()calls into UltravoxForcedAgentMessageevents withuninterruptible: true.References
session.say()docs: https://docs.livekit.io/agents/multimodality/audio/#session-say