Skip to content

Python: fix(copilot): propagate usage details, finish_reason, and model from SDK events#6928

Open
droideronline wants to merge 2 commits into
microsoft:mainfrom
droideronline:fix/copilot-agent-response-metadata
Open

Python: fix(copilot): propagate usage details, finish_reason, and model from SDK events#6928
droideronline wants to merge 2 commits into
microsoft:mainfrom
droideronline:fix/copilot-agent-response-metadata

Conversation

@droideronline

@droideronline droideronline commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Closes #6930

Summary

The Copilot SDK emits an ASSISTANT_USAGE event after every model turn that carries full input/output token counts, cache tokens, reasoning tokens, finish reason (reason), and model name. Previously all of this metadata was discarded and AgentResponse.usage_details, AgentResponse.finish_reason, and model info were always missing.

Problem

When calling agent.run() with the GitHub Copilot agent, AgentResponse.usage_details and AgentResponse.finish_reason are always None.

Direct SDK invocation emits:

SessionEvent(type=ASSISTANT_USAGE, data=AssistantUsageData(
    model="claude-sonnet-4-5",
    input_tokens=120, output_tokens=40,
    cache_read_tokens=5, reason="stop"
))

Agent Framework before this fix:

response.usage_details  # None ❌
response.finish_reason  # None ❌

Agent Framework after this fix:

response.usage_details  # {"input_token_count": 120, "output_token_count": 40, "total_token_count": 160, "cache_read_input_token_count": 5} ✅
response.finish_reason  # "stop" ✅
response.additional_properties["model"]  # "claude-sonnet-4-5" ✅

Changes

python/packages/github_copilot/agent_framework_github_copilot/_agent.py

  1. Adds UsageDetails to imports from agent_framework.
  2. Adds _parse_copilot_usage() — converts the SDK event data payload to UsageDetails using getattr for forward compatibility (works with the Data shim in SDK v1.0.2 and with AssistantUsageData in newer SDK releases).
  3. Non-streaming (_run_impl) — subscribes an extra event handler before send_and_wait to capture ASSISTANT_USAGE events. After send_and_wait completes, the collected data populates AgentResponse.usage_details, finish_reason, and model name in additional_properties. Falls back to output_tokens on ASSISTANT_MESSAGE data when no dedicated usage event arrives (older CLI versions).
  4. Streaming (_stream_updates) — handles SessionEventType.ASSISTANT_USAGE in the event handler, yielding an AgentResponseUpdate with Content.from_usage() payload so _process_update() aggregates it into AgentResponse.usage_details, and propagates finish_reason.

python/packages/github_copilot/tests/test_github_copilot_agent.py

Adds 8 tests in TestGitHubCopilotAgentUsageMetadata:

  • test_run_propagates_usage_details_from_assistant_usage_event
  • test_run_propagates_finish_reason_from_assistant_usage_event
  • test_run_fallback_to_output_tokens_when_no_usage_event
  • test_run_no_usage_when_no_usage_event_and_no_output_tokens
  • test_run_propagates_model_name_in_additional_properties
  • test_run_streaming_propagates_usage_details
  • test_run_streaming_propagates_finish_reason
  • test_run_streaming_no_usage_event_no_crash

Testing

All 128 existing tests pass. 8 new tests added (136 total).

…SDK events

The Copilot SDK emits an ASSISTANT_USAGE event after every model turn that
carries full input/output token counts, cache tokens, reasoning tokens,
finish_reason (reason), and model name.  Previously all of this metadata
was discarded and AgentResponse.usage_details was always None.

Changes:
* Adds _parse_copilot_usage() to convert the SDK event data payload (a
  Data shim in v1.0.2, or AssistantUsageData in newer releases) to the
  framework's UsageDetails TypedDict using getattr for forward compat.
* Non-streaming (_run_impl): subscribes an extra event handler before
  send_and_wait to capture the ASSISTANT_USAGE event; the collected data
  populates AgentResponse.usage_details, finish_reason, and the model
  name via additional_properties. Falls back to output_tokens on the
  ASSISTANT_MESSAGE data when no dedicated usage event arrives.
* Streaming (_stream_updates): handles SessionEventType.ASSISTANT_USAGE
  in the event handler, yielding an AgentResponseUpdate with a
  Content.from_usage() payload so _process_update() aggregates it into
  AgentResponse.usage_details, and propagates finish_reason.
* Adds 8 tests covering non-streaming and streaming usage propagation,
  finish_reason, model name, fallback, and the no-metadata case.
Copilot AI review requested due to automatic review settings July 6, 2026 07:45
@giles17 giles17 added the python Usage: [Issues, PRs], Target: Python label Jul 6, 2026
@github-actions github-actions Bot changed the title fix(copilot): propagate usage details, finish_reason, and model from SDK events Python: fix(copilot): propagate usage details, finish_reason, and model from SDK events Jul 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes missing response metadata in the Python GitHub Copilot agent by propagating SDK-emitted usage and completion information into AgentResponse / streaming updates, aligning Copilot behavior with other chat clients in the framework.

Changes:

  • Add parsing for ASSISTANT_USAGE event payloads into UsageDetails (token counts, cache tokens, reasoning tokens).
  • Populate non-streaming AgentResponse.usage_details, finish_reason, and model name (via additional_properties).
  • In streaming mode, emit usage updates so standard _process_update() aggregation populates final usage/finish metadata, and add targeted tests for both modes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
python/packages/github_copilot/agent_framework_github_copilot/_agent.py Capture Copilot SDK usage events and map them into Agent Framework response metadata for streaming and non-streaming runs.
python/packages/github_copilot/tests/test_github_copilot_agent.py Add test coverage ensuring usage details, finish reason, and model metadata are propagated correctly (including fallback and no-crash cases).

Comment thread python/packages/github_copilot/agent_framework_github_copilot/_agent.py Outdated
…on decoupling

Review feedback addressed:

1. Import add_usage_details — required to accumulate per-turn token counts
   across multiple ASSISTANT_USAGE events.

2. Collect all ASSISTANT_USAGE events (non-streaming) — the handler now
   appends to a list instead of overwriting a single variable, so tool-call
   turns that produce multiple ASSISTANT_USAGE events are all captured.

3. Sum per-turn usage into per-run total (non-streaming) — iterate the
   collected events and call add_usage_details() to produce the same
   cumulative total as the streaming path; take finish_reason and model
   from the last event.

4. Decouple streaming finish_reason from usage presence — emit an
   AgentResponseUpdate when either usage details OR finish_reason is
   present, so finish_reason is never dropped for events that carry a
   reason string but no input/output token counts.

Also adds 2 regression tests:
- test_run_accumulates_usage_across_multiple_usage_events
- test_run_streaming_emits_finish_reason_without_token_counts
@droideronline

Copy link
Copy Markdown
Contributor Author

@moonbox3 @eavanvalkenburg - kindly take a look when you get a chance. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: bug(copilot): AgentResponse.usage_details and finish_reason always None when using GitHubCopilotAgent

3 participants