fix(elevenlabs): populate SpeechData.confidence from Scribe logprobs - #2203
Merged
Conversation
🦋 Changeset detectedLatest commit: 0675840 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
davidzhao
approved these changes
Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports livekit/agents#6461 to populate ElevenLabs Scribe transcription confidence from spoken-word log probabilities for both batch and realtime STT responses.
Adds a patch changeset for
@livekit/agents-plugin-elevenlabs.Validation
pnpm buildpnpm exec vitest run plugins/elevenlabs(19 passed, 2 credential-gated tests skipped)pnpm lintpnpm format:checkSource diff coverage
livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py: Adapted toplugins/elevenlabs/src/stt.ts. The Python helper is translated to an internal TypeScript geometric-mean calculation, the target response parser is extended to retain Scribe'stypeandlogprobfields, and confidence is wired into both batch and realtimeSpeechDataconstruction.tests/test_plugin_elevenlabs_stt.py: Adapted toplugins/elevenlabs/src/stt.test.ts. All four source test cases are preserved in Vitest form; the three helper cases run through batch recognition to avoid exposing an internal helper as public API, and the committed-transcript case runs through the realtime WebSocket path.Ported from livekit/agents#6461
Original PR description
What / Why
The ElevenLabs STT plugin never populates
SpeechData.confidence, so it always stays at its0.0default — even though Scribe returns a per-wordlogprobin every response. Consumers that useconfidence(e.g. to tell a clean transcript from a low-confidence / garbled one) get no signal from ElevenLabs.Verified live: both
scribe_v1/scribe_v2(REST) andscribe_v2_realtimereturn a natural-loglogprobperwords[]token, but_process_stream_event/_transcription_to_speech_eventread onlytext+ wordstart/end, so it is discarded andconfidencestays0.0.Change
Add
_speech_confidence(words): average the spoken-word (type == "word") logprobs and exponentiate → a[0, 1]confidence (geometric mean of the token probabilities). Returns0.0when per-word logprobs aren't available (e.g. a non-timestamped commit), so behaviour is unchanged there.Wired into both the REST and WebSocket
SpeechDataconstruction.Note: this deliberately does not use Scribe's
language_probability— that is language-identification confidence, a different quantity from transcription confidence, so folding it intoSpeechData.confidencewould be misleading.Same class of fix as #5829 / #5830 (a sibling STT plugin not threading its confidence into
SpeechData).Tests
Unit tests for the helper (confident vs low-confidence logprobs, no-logprob →
0.0) and an end-to-end check that a committed transcript setsconfidence.ruff check/ruff formatclean.🤖 Generated with Claude Code