Skip to content

Add GitHub Copilot parser and fix incremental timestamps for resumed sessions - #41

Open
zzzLi-56 wants to merge 2 commits into
uber:mainfrom
lx39214:add-copilot-parser-and-fix-incremental-timestamps
Open

Add GitHub Copilot parser and fix incremental timestamps for resumed sessions#41
zzzLi-56 wants to merge 2 commits into
uber:mainfrom
lx39214:add-copilot-parser-and-fix-incremental-timestamps

Conversation

@zzzLi-56

Copy link
Copy Markdown

This PR does two things:

  1. Adds support for parsing GitHub Copilot session-state logs
  2. Fixes incremental export behavior for resumed sessions in existing parsers

Changes

GitHub Copilot support

Added a new `CopilotParser` that reads session data from:

  • `~/.copilot/session-state//events.jsonl`
  • `workspace.yaml`
  • `vscode.metadata.json`

It extracts:

  • user / assistant chat history
  • tool requests and tool execution results
  • permissions
  • skill invocations
  • subagent activity
  • session metadata / context

The parser is registered in:

  • `adr_sensor/parsers/init.py`
  • `adr_sensor/observer.py`

and exposed through the CLI via:

```bash
adr-sensor --source copilot
```

Incremental export fixes

`--save-sessions` is documented as incremental, but resumed sessions were not always re-exported because some parsers emitted a session-start timestamp instead of a last-activity timestamp.

This specifically fixes the case where additional activity is appended to an existing Codex session after an earlier incremental export.

The same incremental timestamp issue also affected Claude Code sessions because the parser retained the earliest event timestamp instead of the latest one.

This PR fixes that by updating:

  • `CodexParser` to use the latest event timestamp when available
  • `ClaudeParser` to retain the latest session timestamp instead of the earliest one

This makes resumed sessions behave correctly with:

```bash
adr-sensor --save-sessions --all-history
```

Testing

Added targeted tests for:

  • latest timestamp behavior in resumed Claude sessions
  • latest event timestamp behavior in resumed Codex sessions
  • GitHub Copilot session parsing
  • Copilot registration in `AgentObserver`

Validated with focused pytest runs:

```bash
python -m pytest tests/test_parsers.py -q -k "resumed_session or parse_session_dir"
python -m pytest tests/test_observer.py -q -k "test_init_default"
```

Notes

While testing on Windows, some unrelated pre-existing cross-platform path tests failed due to platform-specific path expectations. The targeted tests for the changes in this PR passed successfully."

@CLAassistant

CLAassistant commented Aug 12, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@pengyuzhang pengyuzhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second-pass review at head 2e889da, focused on the three highest-impact findings; posted inline below. Summary of where they land:

  1. copilot_parser.py:134 — the new parser's timestamp precedence reintroduces the resumed-session staleness bug this PR fixes elsewhere, and the new test asserts the wrong precedence. Verified by execution.
  2. claude_parser.py:130 — the min→max change interacts with the filename-based exporter: snapshots accumulate unboundedly per active session, nothing prunes.
  3. codex_parser.py:98 — the same change causes a one-time fleet-wide mass re-export on deploy, plus a same-second truncation edge that permanently drops a session's tail messages.

The timestamp fixes themselves are correct in isolation, and the parser wiring (--source copilot, dispatch, schema) is sound — the issues are in the new Copilot parser's own timestamp handling and in unhandled interactions with the incremental exporter, not in the fix's intent.

Comment on lines +134 to +143
timestamp = (
self._normalize_optional_timestamp(workspace_meta.get("updated_at"))
or self._normalize_optional_timestamp(vscode_meta.get("modified"))
or self._normalize_optional_timestamp(session_data["last_event_at"])
or session_data["timestamp"]
or self._normalize_optional_timestamp(workspace_meta.get("created_at"))
or self._normalize_optional_timestamp(vscode_meta.get("created"))
or self._normalize_optional_timestamp(session_data["first_event_at"])
or datetime.now(timezone.utc)
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timestamp fallback chain prefers stale sidecar metadata over the newest event — reintroducing for Copilot the exact resumed-session bug this PR fixes for Claude and Codex.

workspace.yaml:updated_at and vscode.metadata.json:modified sit ahead of last_event_at, so a session whose events.jsonl is newer than its side-channel metadata gets the older timestamp. That's precisely the resumed-session case: events append at T2 while the sidecar files still read T1 (vscode.metadata.json written once at creation, or workspace.yaml not rewritten on resume).

Verified by executing the parser with stale metadata (modified=2026-07-31, events through 2026-08-10): the emitted AgentEvent.timestamp was 2026-07-31. filter_entries_by_existing_files (observer.py:320) only re-exports when entry_ts > existing_ts, so all post-resume activity — including any malicious tool use ADR exists to catch — is silently never exported. Same defect, third parser.

Suggested change
timestamp = (
self._normalize_optional_timestamp(workspace_meta.get("updated_at"))
or self._normalize_optional_timestamp(vscode_meta.get("modified"))
or self._normalize_optional_timestamp(session_data["last_event_at"])
or session_data["timestamp"]
or self._normalize_optional_timestamp(workspace_meta.get("created_at"))
or self._normalize_optional_timestamp(vscode_meta.get("created"))
or self._normalize_optional_timestamp(session_data["first_event_at"])
or datetime.now(timezone.utc)
)
timestamp = (
self._normalize_optional_timestamp(session_data["last_event_at"])
or self._normalize_optional_timestamp(workspace_meta.get("updated_at"))
or self._normalize_optional_timestamp(vscode_meta.get("modified"))
or session_data["timestamp"]
or self._normalize_optional_timestamp(workspace_meta.get("created_at"))
or self._normalize_optional_timestamp(vscode_meta.get("created"))
or self._normalize_optional_timestamp(session_data["first_event_at"])
or datetime.now(timezone.utc)
)

Note the new test in test_parsers.py asserts the current (wrong) precedence — it expects the updated_at value — so it locks the bug in and needs its expected value updated along with this.

try:
ts = normalize_timestamp(obj["timestamp"])
if sessions[session_id]["timestamp"] is None or ts < sessions[session_id]["timestamp"]:
if sessions[session_id]["timestamp"] is None or ts > sessions[session_id]["timestamp"]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With latest-event timestamps in the export filename, every active session now accumulates one full-history snapshot per run — and nothing prunes the old ones. (Applies equally to the same change in codex_parser.py:98; the mechanism lives in observer.py:289.)

Before this change the embedded timestamp was the stable session start, so one session mapped to one file and each run overwrote it in place. Now the timestamp moves forward whenever the session has activity, save_sessions_to_individual_files opens the new path with 'w', and no code deletes prior snapshots. A session used daily leaves one full-chat-history file per day:

adr.sess-abc.20260810_174501.json   ← Monday's full history
adr.sess-abc.20260811_183212.json   ← Tuesday's (superset of Monday's)
adr.sess-abc.20260812_091544.json   ← ...

The output dir grows without bound, and any consumer watching it re-ingests the session's entire history each time — every earlier message duplicated downstream unless the consumer dedups by session_id.

Together with the upgrade-time re-export (comment on codex_parser.py:98), this suggests the moving timestamp shouldn't be part of the file's identity: either keep a stable filename per session and track the latest-exported-event time in the file contents or a sidecar index, or prune a session's older snapshots when writing the new one.


return AgentEvent(
timestamp=session_data["timestamp"] or datetime.now(timezone.utc),
timestamp=session_data["last_event_timestamp"] or session_data["timestamp"] or datetime.now(timezone.utc),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One-time mass re-export on upgrade: the first run after deploying this change re-exports essentially every session in the retention window, even with zero new activity. (Same for the claude_parser.py:130 min→max flip; the comparison is observer.py:320.)

The incremental filter compares the freshly parsed timestamp against the one embedded in the existing file's name, and skips only when entry_ts > existing_ts fails. On the first post-deploy run, every existing file still carries the old start-semantics timestamp while every freshly parsed entry carries its latest-event timestamp — and latest > start for any session longer than a second. So the check passes for all of them, and every host re-exports a duplicate full-history snapshot of every session at once: a fleet-wide burst of duplicates into the pipeline on deploy day.

Related edge in the same comparison: both sides are truncated to whole seconds (%Y%m%d_%H%M%S filenames; .replace(microsecond=0)) with a strict >, so a session whose final events land within the already-exported second is permanently skipped — for an ended session, those tail messages are never exported. A >= won't fix that (it would re-export forever); it needs sub-second precision or a content-based change check.

Worth handling the transition explicitly (e.g. migrate existing filenames once, or dedup by session_id downstream) rather than letting the burst happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants