Python: fix(copilot): propagate usage details, finish_reason, and model from SDK events#6928
Open
droideronline wants to merge 2 commits into
Open
Conversation
…SDK events The Copilot SDK emits an ASSISTANT_USAGE event after every model turn that carries full input/output token counts, cache tokens, reasoning tokens, finish_reason (reason), and model name. Previously all of this metadata was discarded and AgentResponse.usage_details was always None. Changes: * Adds _parse_copilot_usage() to convert the SDK event data payload (a Data shim in v1.0.2, or AssistantUsageData in newer releases) to the framework's UsageDetails TypedDict using getattr for forward compat. * Non-streaming (_run_impl): subscribes an extra event handler before send_and_wait to capture the ASSISTANT_USAGE event; the collected data populates AgentResponse.usage_details, finish_reason, and the model name via additional_properties. Falls back to output_tokens on the ASSISTANT_MESSAGE data when no dedicated usage event arrives. * Streaming (_stream_updates): handles SessionEventType.ASSISTANT_USAGE in the event handler, yielding an AgentResponseUpdate with a Content.from_usage() payload so _process_update() aggregates it into AgentResponse.usage_details, and propagates finish_reason. * Adds 8 tests covering non-streaming and streaming usage propagation, finish_reason, model name, fallback, and the no-metadata case.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes missing response metadata in the Python GitHub Copilot agent by propagating SDK-emitted usage and completion information into AgentResponse / streaming updates, aligning Copilot behavior with other chat clients in the framework.
Changes:
- Add parsing for
ASSISTANT_USAGEevent payloads intoUsageDetails(token counts, cache tokens, reasoning tokens). - Populate non-streaming
AgentResponse.usage_details,finish_reason, and model name (viaadditional_properties). - In streaming mode, emit
usageupdates so standard_process_update()aggregation populates final usage/finish metadata, and add targeted tests for both modes.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| python/packages/github_copilot/agent_framework_github_copilot/_agent.py | Capture Copilot SDK usage events and map them into Agent Framework response metadata for streaming and non-streaming runs. |
| python/packages/github_copilot/tests/test_github_copilot_agent.py | Add test coverage ensuring usage details, finish reason, and model metadata are propagated correctly (including fallback and no-crash cases). |
…on decoupling Review feedback addressed: 1. Import add_usage_details — required to accumulate per-turn token counts across multiple ASSISTANT_USAGE events. 2. Collect all ASSISTANT_USAGE events (non-streaming) — the handler now appends to a list instead of overwriting a single variable, so tool-call turns that produce multiple ASSISTANT_USAGE events are all captured. 3. Sum per-turn usage into per-run total (non-streaming) — iterate the collected events and call add_usage_details() to produce the same cumulative total as the streaming path; take finish_reason and model from the last event. 4. Decouple streaming finish_reason from usage presence — emit an AgentResponseUpdate when either usage details OR finish_reason is present, so finish_reason is never dropped for events that carry a reason string but no input/output token counts. Also adds 2 regression tests: - test_run_accumulates_usage_across_multiple_usage_events - test_run_streaming_emits_finish_reason_without_token_counts
Contributor
Author
|
@moonbox3 @eavanvalkenburg - kindly take a look when you get a chance. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6930
Summary
The Copilot SDK emits an
ASSISTANT_USAGEevent after every model turn that carries full input/output token counts, cache tokens, reasoning tokens, finish reason (reason), and model name. Previously all of this metadata was discarded andAgentResponse.usage_details,AgentResponse.finish_reason, and model info were always missing.Problem
When calling
agent.run()with the GitHub Copilot agent,AgentResponse.usage_detailsandAgentResponse.finish_reasonare alwaysNone.Direct SDK invocation emits:
Agent Framework before this fix:
Agent Framework after this fix:
Changes
python/packages/github_copilot/agent_framework_github_copilot/_agent.pyUsageDetailsto imports fromagent_framework._parse_copilot_usage()— converts the SDK event data payload toUsageDetailsusinggetattrfor forward compatibility (works with theDatashim in SDK v1.0.2 and withAssistantUsageDatain newer SDK releases)._run_impl) — subscribes an extra event handler beforesend_and_waitto captureASSISTANT_USAGEevents. Aftersend_and_waitcompletes, the collected data populatesAgentResponse.usage_details,finish_reason, and model name inadditional_properties. Falls back tooutput_tokensonASSISTANT_MESSAGEdata when no dedicated usage event arrives (older CLI versions)._stream_updates) — handlesSessionEventType.ASSISTANT_USAGEin the event handler, yielding anAgentResponseUpdatewithContent.from_usage()payload so_process_update()aggregates it intoAgentResponse.usage_details, and propagatesfinish_reason.python/packages/github_copilot/tests/test_github_copilot_agent.pyAdds 8 tests in
TestGitHubCopilotAgentUsageMetadata:test_run_propagates_usage_details_from_assistant_usage_eventtest_run_propagates_finish_reason_from_assistant_usage_eventtest_run_fallback_to_output_tokens_when_no_usage_eventtest_run_no_usage_when_no_usage_event_and_no_output_tokenstest_run_propagates_model_name_in_additional_propertiestest_run_streaming_propagates_usage_detailstest_run_streaming_propagates_finish_reasontest_run_streaming_no_usage_event_no_crashTesting
All 128 existing tests pass. 8 new tests added (136 total).