Skip to content

fix(elevenlabs): populate SpeechData.confidence from Scribe logprobs - #2203

Merged
davidzhao merged 1 commit into
mainfrom
combats-solitude-fining
Aug 2, 2026
Merged

fix(elevenlabs): populate SpeechData.confidence from Scribe logprobs#2203
davidzhao merged 1 commit into
mainfrom
combats-solitude-fining

Conversation

@rosetta-livekit-bot

@rosetta-livekit-bot rosetta-livekit-bot Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Ports livekit/agents#6461 to populate ElevenLabs Scribe transcription confidence from spoken-word log probabilities for both batch and realtime STT responses.

Adds a patch changeset for @livekit/agents-plugin-elevenlabs.

Validation

  • pnpm build
  • pnpm exec vitest run plugins/elevenlabs (19 passed, 2 credential-gated tests skipped)
  • pnpm lint
  • pnpm format:check
Source diff coverage
  • livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py: Adapted to plugins/elevenlabs/src/stt.ts. The Python helper is translated to an internal TypeScript geometric-mean calculation, the target response parser is extended to retain Scribe's type and logprob fields, and confidence is wired into both batch and realtime SpeechData construction.
  • tests/test_plugin_elevenlabs_stt.py: Adapted to plugins/elevenlabs/src/stt.test.ts. All four source test cases are preserved in Vitest form; the three helper cases run through batch recognition to avoid exposing an internal helper as public API, and the committed-transcript case runs through the realtime WebSocket path.
  • Not applicable: None. Every source file and behavior has a target counterpart, with no infrastructure gaps.

Ported from livekit/agents#6461

Original PR description

What / Why

The ElevenLabs STT plugin never populates SpeechData.confidence, so it always stays at its 0.0 default — even though Scribe returns a per-word logprob in every response. Consumers that use confidence (e.g. to tell a clean transcript from a low-confidence / garbled one) get no signal from ElevenLabs.

Verified live: both scribe_v1/scribe_v2 (REST) and scribe_v2_realtime return a natural-log logprob per words[] token, but _process_stream_event / _transcription_to_speech_event read only text + word start/end, so it is discarded and confidence stays 0.0.

Change

Add _speech_confidence(words): average the spoken-word (type == "word") logprobs and exponentiate → a [0, 1] confidence (geometric mean of the token probabilities). Returns 0.0 when per-word logprobs aren't available (e.g. a non-timestamped commit), so behaviour is unchanged there.

Wired into both the REST and WebSocket SpeechData construction.

Note: this deliberately does not use Scribe's language_probability — that is language-identification confidence, a different quantity from transcription confidence, so folding it into SpeechData.confidence would be misleading.

Same class of fix as #5829 / #5830 (a sibling STT plugin not threading its confidence into SpeechData).

Tests

Unit tests for the helper (confident vs low-confidence logprobs, no-logprob → 0.0) and an end-to-end check that a committed transcript sets confidence. ruff check / ruff format clean.

🤖 Generated with Claude Code

@rosetta-livekit-bot
rosetta-livekit-bot Bot requested a review from a team as a code owner August 2, 2026 07:31
@changeset-bot

changeset-bot Bot commented Aug 2, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0675840

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-azure Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-krisp Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-protoface Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-soniox Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugins-test Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@rosetta-livekit-bot
rosetta-livekit-bot Bot requested a review from davidzhao August 2, 2026 07:32

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@davidzhao
davidzhao merged commit 8a92b61 into main Aug 2, 2026
4 checks passed
@davidzhao
davidzhao deleted the combats-solitude-fining branch August 2, 2026 18:04
@github-actions github-actions Bot mentioned this pull request Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant