Python: [Bug]: Anthropic streaming double-counts token usage (message_start usage is summed with the cumulative message_delta usage)

[ebarkhordar]

Description

What happened: on the Anthropic streaming path, _process_stream_event emits a usage Content on both message_start and message_delta. ChatResponse.from_updates sums every usage Content it sees (_types.py:1972-1976 calls add_usage_details, which sums all numeric values). But Anthropic's message_delta usage is a cumulative total for the message, not a per-delta increment, so summing it onto the message_start seed inflates the final usage_details.

Anthropic's streaming docs state this directly:

The token counts shown in the usage field of the message_delta event are cumulative.

(https://docs.anthropic.com/en/docs/build-with-claude/streaming, "Event types")

What I expected: response.usage_details after a streaming call equals the cumulative totals the API reported, matching what the non-streaming path produces.

Two cases, and I want to be precise about how sure I am of each.

Case B, the unconditional one. message_delta carries only output_tokens (the docs' basic example is "usage": {"output_tokens": 15}). The message_start seed (output_tokens: 1) is added to the delta's cumulative total, so output_token_count is always over by the seed. Measured: 26 when the truth is 25. This holds no matter which fields the API populates.

Case A, the amplified one. message_delta also carries prompt-side fields. Anthropic's docs show exactly this shape in their web search example:

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":10682,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":510,"server_tool_use":{"web_search_requests":1}}}

When the delta looks like that, the prompt-side counts double as well. Measured: input_token_count 200 vs 100, cache_read_input_token_count 100 vs 50, cache_creation_input_token_count 20 vs 10.

I did not call the live API, so I am not claiming a specific real-traffic number. Case A rests on Anthropic's published example and the cumulative note above, not on my own wire capture. Case B is true regardless.

Why it seems to have survived: the exact-value usage assertions cover the non-streaming path (test_process_message_basic) and _parse_usage_from_anthropic in isolation (test_parse_usage_from_anthropic_basic, test_parse_usage_with_cache_tokens). The only streaming usage assertion I found is test_anthropic_client_integration_basic_chat, which is @pytest.mark.integration and asserts usage_details is not None. Nothing drives message_start + message_delta through from_updates and checks the summed result. The non-streaming path (:1072, single message.usage) is correct, so this is streaming-only.

The Gemini client already guards the identical hazard, attaching usage only on the final chunk (gemini/_chat_client.py:1063-1065, "Attach usage to the final chunk only"), which is a useful reference point for the fix shape.

I have not bisected this, so I am not calling it a regression.

Code Sample

Offline, no live API calls. Feeds real Anthropic SDK event objects through the real path.

from agent_framework import ChatResponse
from agent_framework_anthropic import AnthropicClient
from anthropic.types.beta import BetaMessage, BetaRawMessageDeltaEvent, BetaRawMessageStartEvent, BetaUsage
from anthropic.types.beta.beta_raw_message_delta_event import Delta as BetaDelta
from anthropic.types.beta.beta_message_delta_usage import BetaMessageDeltaUsage

client = AnthropicClient(api_key="sk-ant-offline-dummy", model="claude-opus-4-8")

start = BetaRawMessageStartEvent(
    type="message_start",
    message=BetaMessage(
        id="msg_1", type="message", role="assistant", content=[],
        model="claude-opus-4-8", stop_reason=None, stop_sequence=None,
        usage=BetaUsage(input_tokens=100, output_tokens=1,
                        cache_read_input_tokens=50, cache_creation_input_tokens=10),
    ),
)
# Case B: the delta reports the cumulative output total (25) and nothing else.
delta = BetaRawMessageDeltaEvent(
    type="message_delta",
    delta=BetaDelta(stop_reason="end_turn", stop_sequence=None),
    usage=BetaMessageDeltaUsage(output_tokens=25),
)

updates = [u for u in (client._process_stream_event(e) for e in (start, delta)) if u]
print(ChatResponse.from_updates(updates).usage_details["output_token_count"])

Prints 26. The cumulative truth reported by the API is 25.

Swapping the delta for BetaMessageDeltaUsage(output_tokens=25, input_tokens=100, cache_read_input_tokens=50, cache_creation_input_tokens=10) (case A) additionally gives input_token_count=200 against a truth of 100.

Error Messages / Stack Traces

No error. The counts are silently wrong, which is the reason I filed this rather than a crash report.

--- B: message_delta usage = {output_tokens: 25}
    OK  input_token_count: aggregated=100  cumulative-truth=100
    BUG output_token_count: aggregated=26  cumulative-truth=25
    OK  cache_read_input_token_count: aggregated=50  cumulative-truth=50
    OK  cache_creation_input_token_count: aggregated=10  cumulative-truth=10

--- A: message_delta usage carries prompt-side fields too
    BUG input_token_count: aggregated=200  cumulative-truth=100
    BUG output_token_count: aggregated=26  cumulative-truth=25
    BUG cache_read_input_token_count: aggregated=100  cumulative-truth=50
    BUG cache_creation_input_token_count: aggregated=20  cumulative-truth=10

Package Versions

agent-framework-core and agent-framework-anthropic built from source at 85c00fc55b6f8cef8b88e83f364b527a0d5cd275 (main), anthropic SDK 0.116.0 (resolved inside the package's own anthropic>=0.80.0,<0.117.0 pin)

Python Version

Python 3.12.13

Additional Context

Where it lands: usage_details flows into every await stream.get_final_response(), into the tool loop's cross-iteration usage total (_tools.py:2642/2723), and into the OpenTelemetry gen_ai token metrics (observability.py:349-350). Anyone doing cost accounting, token budgeting, or cache-hit analysis on streaming Anthropic calls gets numbers that are off by at least the output seed.

Environment: python:3.12-slim container, no network, both packages pip-installed from the tree at the SHA above. The packages/anthropic suite is green there (148 passed), so this is an additive gap rather than a broken checkout.

One note on the fix, since it may not be as simple as it looks. Copying the Gemini approach exactly (drop usage on message_start, keep it only on the final event) is clean for case A, but in case B the delta carries no input_tokens at all, so the prompt-side counts would be lost entirely rather than merely inflated. An approach that holds up under both is to track the last cumulative usage seen within the per-request _stream() generator (_chat_client.py:546-551, not on the client instance, which is shared) and yield only the increment. That keeps _process_stream_event and _parse_usage_from_anthropic signatures untouched. I would rather surface the tradeoff than assume which way you want it.

Happy to send a PR with Fixes #N and unit tests covering both delta shapes if you would like it, or leave it to you if the seam is one you would rather own. Verified with Claude Code assistance; I ran the repro myself and the analysis above is my own.