Skip to content

feat: add provider tool call lifecycle events - #6973

Draft
TheCodingCvrlo wants to merge 1 commit into
livekit:mainfrom
TheCodingCvrlo:carlo/provider-tool-call-lifecycle
Draft

feat: add provider tool call lifecycle events#6973
TheCodingCvrlo wants to merge 1 commit into
livekit:mainfrom
TheCodingCvrlo:carlo/provider-tool-call-lifecycle

Conversation

@TheCodingCvrlo

@TheCodingCvrlo TheCodingCvrlo commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Tracking issue: #6984

Summary

Surfaces provider-executed (server-side) tool calls — e.g. Mistral web search, and applicable to any provider with built-in tools — as a first-class lifecycle on AgentSession, parallel to tool_execution_updated for locally-executed tools.

  • Core (llm): new ProviderToolCall payload + a provider_tool_call event on the base LLM EventEmitter (the same seam plugins already use for metrics_collected). No change to ChatChunk.
  • Core (voice): new provider_tool_execution_updated session event carrying a discriminated ProviderToolCallStarted | ProviderToolCallEnded. AgentActivity bridges the LLM event onto the session, wired right next to metrics_collected.
  • Plugin (mistralai): emits provider_tool_call by translating the Conversations stream's ToolExecutionStarted / ToolExecutionDone. No bespoke events on the LLM.
Mistral stream (ToolExecutionStarted / …Done)
  → plugin:    llm.emit("provider_tool_call", ProviderToolCall(phase=...))
  → framework: bridge in AgentActivity (like metrics_collected)
  → session:   emit("provider_tool_execution_updated", ProviderToolCallStarted | ProviderToolCallEnded)
  → app:       session.on("provider_tool_execution_updated", ...)

Motivation

Context: Related discussion on #5869 with @tinalenguyen and @jeanprbt . #5869 adds a single terminal function_tools_called event (once per turn). This PR gives a higher abstraction by adding the lifecycle needed to modify state while provider tools are executed, e.g. playing a "thinking"/background sound for exactly the duration of a web search.

The design mirrors the existing local-tool lifecycle (ToolCallStarted / ToolCallEnded via tool_execution_updated) so consumers get a consistent, provider-agnostic API and subscribe once.

Design notes

  • Plugins only translate (provider-specific → generic ProviderToolCall) and emit on the LLM; the framework owns the app-facing provider_tool_execution_updated contract and the bridge, so any future provider plugin gets the session event for free.
  • ProviderToolCallEnded carries result (the tool output) when the provider returns one.

Tests

  • tests/test_plugin_mistralai_llm.py — the plugin's _parse_event emits provider_tool_call (started → delta accumulates → done with full args + result; done-without-start is safe).
  • tests/test_provider_tool_events.py — end-to-end through a real AgentSession: provider_tool_execution_updated fires start→end per tool, in order.
  • make check clean locally (ruff + mypy strict).

Open questions for maintainers

  • Relationship to (core): add function_tools_called event #5869 — happy to align or converge (e.g. keep function_tools_called as the per-turn summary and this as the lifecycle, or unify naming).
  • Scope — this wires the pipeline path; the realtime path and the remote_session protobuf relay (which would need a livekit/protocol message) are natural follow-ups.
  • Naming / shape — event names, and whether result should be structured rather than str.

Opening as a draft for discussion.

Surface provider-executed (server-side) tool calls (e.g. Mistral web
search) as a session lifecycle, parallel to tool_execution_updated for
locally-run tools:

- llm.ProviderToolCall + a "provider_tool_call" event on the base LLM
  EventEmitter (same seam as metrics_collected)
- AgentSession "provider_tool_execution_updated" event with
  ProviderToolCallStarted / ProviderToolCallEnded, bridged from the LLM
  event in AgentActivity
- mistralai plugin emits it from ToolExecutionStarted/Done

Gives an early "started" edge (to drive real-time UX such as a background
sound while a server-side tool runs) plus an "ended" edge with the
result, without touching ChatChunk.

Relates to livekit#5869.
@TheCodingCvrlo
TheCodingCvrlo marked this pull request as ready for review August 26, 2026 13:08
@TheCodingCvrlo
TheCodingCvrlo requested a review from a team as a code owner August 26, 2026 13:08
@TheCodingCvrlo
TheCodingCvrlo marked this pull request as draft August 26, 2026 13:08

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant