Skip to content

Add GitHub Copilot parser and fix incremental timestamps for resumed sessions #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zzzLi-56 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add GitHub Copilot parser and fix incremental timestamps for resumed sessions #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f04cfa
Add GitHub Copilot parser and fix incremental timestamps for resumed …
zzzLi-56 Aug 12, 2026
2e889da
Add GitHub Copilot parser and fix incremental timestamps for resumed …
zzzLi-56 Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Sensor/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ ADR Sensor is a Python library that collects telemetry from AI coding agents to
| **Cline (Claude Dev)** | `cline` | JSON task files | macOS, Linux, Windows |
| **Claude Desktop** | `claude_desktop` | JSONL audit logs | macOS, Windows |
| **OpenAI Codex CLI** | `codex` | JSONL (`~/.codex/sessions/`) | macOS, Linux, Windows |
| **GitHub Copilot** | `copilot` | JSONL (`~/.copilot/session-state/`) | macOS, Linux, Windows |
| **Warp Terminal** | `warp` | SQLite (`warp.sqlite`) | macOS, Windows |
| **opencode** | `opencode` | SQLite (`opencode.db`) or JSON tree | macOS, Linux |

Expand Down Expand Up @@ -110,6 +111,7 @@ adr-sensor
adr-sensor --source claude
adr-sensor --source cursor
adr-sensor --source codex
adr-sensor --source copilot
adr-sensor --source claude_desktop
adr-sensor --source opencode

Expand Down Expand Up @@ -349,6 +351,7 @@ adr-sensor/
│ │ ├── cline_parser.py
│ │ ├── claude_desktop_parser.py
│ │ ├── codex_parser.py
│ │ ├── copilot_parser.py
│ │ ├── opencode_parser.py
│ │ └── warp_parser.py
│ ├── schemas/
Expand Down
5 changes: 4 additions & 1 deletion Sensor/adr_sensor/observer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from .parsers.claude_desktop_parser import ClaudeDesktopParser
from .parsers.claude_parser import ClaudeParser
from .parsers.cline_parser import ClineParser
from .parsers.copilot_parser import CopilotParser
from .parsers.codex_parser import CodexParser
from .parsers.cursor_parser import CursorParser
from .parsers.opencode_parser import OpencodeParser
Expand Down Expand Up @@ -50,6 +51,7 @@ class AgentObserver:
("cline", "Cline"),
("warp", "Warp Terminal"),
("codex", "Codex"),
("copilot", "GitHub Copilot"),
("opencode", "opencode"),
)

Expand All @@ -72,6 +74,7 @@ def __init__(self, output_dir: Optional[Path] = None, max_age_days: Optional[int
ClaudeDesktopParser(max_age_days=max_age_days) if max_age_days is not None else ClaudeDesktopParser()
)
self.codex_parser = CodexParser()
self.copilot_parser = CopilotParser()
self.cline_parser = ClineParser()
self.warp_parser = WarpParser(max_age_days=max_age_days) if max_age_days is not None else WarpParser()
self.opencode_parser = (
Expand Down Expand Up @@ -114,7 +117,7 @@ def ingest_all(

Args:
source_filter: Which source to ingest. One of 'all', 'claude', 'cursor',
'claude_desktop', 'cline', 'warp', 'codex', 'opencode'.
'claude_desktop', 'cline', 'warp', 'codex', 'copilot', 'opencode'.

Returns:
Tuple of (agent_events, system_configs).
Expand Down
2 changes: 2 additions & 0 deletions Sensor/adr_sensor/parsers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from .claude_desktop_parser import ClaudeDesktopParser
from .claude_parser import ClaudeParser
from .cline_parser import ClineParser
from .copilot_parser import CopilotParser
from .codex_parser import CodexParser
from .cursor_parser import CursorParser
from .opencode_parser import OpencodeParser
Expand All @@ -18,6 +19,7 @@
"ClaudeDesktopParser",
"ClaudeParser",
"ClineParser",
"CopilotParser",
"CodexParser",
"CursorParser",
"OpencodeParser",
Expand Down
2 changes: 1 addition & 1 deletion Sensor/adr_sensor/parsers/claude_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ def parse_jsonl_file(self, file_path: Path) -> List[AgentEvent]:
if "timestamp" in obj:
try:
ts = normalize_timestamp(obj["timestamp"])
if sessions[session_id]["timestamp"] is None or ts < sessions[session_id]["timestamp"]:
if sessions[session_id]["timestamp"] is None or ts > sessions[session_id]["timestamp"]:

@pengyuzhang pengyuzhang Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With latest-event timestamps in the export filename, every active session now accumulates one full-history snapshot per run — and nothing prunes the old ones. (Applies equally to the same change in codex_parser.py:98; the mechanism lives in observer.py:289.)

Before this change the embedded timestamp was the stable session start, so one session mapped to one file and each run overwrote it in place. Now the timestamp moves forward whenever the session has activity, save_sessions_to_individual_files opens the new path with 'w', and no code deletes prior snapshots. A session used daily leaves one full-chat-history file per day:

adr.sess-abc.20260810_174501.json   ← Monday's full history
adr.sess-abc.20260811_183212.json   ← Tuesday's (superset of Monday's)
adr.sess-abc.20260812_091544.json   ← ...

The output dir grows without bound, and any consumer watching it re-ingests the session's entire history each time — every earlier message duplicated downstream unless the consumer dedups by session_id.

Together with the upgrade-time re-export (comment on codex_parser.py:98), this suggests the moving timestamp shouldn't be part of the file's identity: either keep a stable filename per session and track the latest-exported-event time in the file contents or a sidecar index, or prune a session's older snapshots when writing the new one.

sessions[session_id]["timestamp"] = ts
except Exception:
pass
Expand Down
13 changes: 12 additions & 1 deletion Sensor/adr_sensor/parsers/codex_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ def parse_jsonl_file(self, file_path: Path) -> Optional[AgentEvent]:
session_data: Dict[str, Any] = {
"id": None,
"timestamp": None,
"last_event_timestamp": None,
"cwd": None,
"model": None,
"messages": [],
Expand Down Expand Up @@ -94,7 +95,7 @@ def parse_jsonl_file(self, file_path: Path) -> Optional[AgentEvent]:
)

return AgentEvent(
timestamp=session_data["timestamp"] or datetime.now(timezone.utc),
timestamp=session_data["last_event_timestamp"] or session_data["timestamp"] or datetime.now(timezone.utc),

@pengyuzhang pengyuzhang Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One-time mass re-export on upgrade: the first run after deploying this change re-exports essentially every session in the retention window, even with zero new activity. (Same for the claude_parser.py:130 min→max flip; the comparison is observer.py:320.)

The incremental filter compares the freshly parsed timestamp against the one embedded in the existing file's name, and skips only when entry_ts > existing_ts fails. On the first post-deploy run, every existing file still carries the old start-semantics timestamp while every freshly parsed entry carries its latest-event timestamp — and latest > start for any session longer than a second. So the check passes for all of them, and every host re-exports a duplicate full-history snapshot of every session at once: a fleet-wide burst of duplicates into the pipeline on deploy day.

Related edge in the same comparison: both sides are truncated to whole seconds (%Y%m%d_%H%M%S filenames; .replace(microsecond=0)) with a strict >, so a session whose final events land within the already-exported second is permanently skipped — for an ended session, those tail messages are never exported. A >= won't fix that (it would re-export forever); it needs sub-second precision or a content-based change check.

Worth handling the transition explicitly (e.g. migrate existing filenames once, or dedup by session_id downstream) rather than letting the burst happen.

source="codex",
session_id=f"codex_{session_data['id']}",
project_path=session_data["cwd"],
Expand Down Expand Up @@ -174,6 +175,16 @@ def _process_event(self, event: Dict[str, Any], session_data: Dict[str, Any]):
evt_type = event.get("type")
payload = event.get("payload", {})

event_timestamp = event.get("timestamp")
if event_timestamp:
try:
normalized = normalize_timestamp(event_timestamp)
current = session_data.get("last_event_timestamp")
if current is None or normalized > current:
session_data["last_event_timestamp"] = normalized
except Exception:
pass

if evt_type == "session_meta":
session_data["id"] = payload.get("id")
if payload.get("timestamp"):
Expand Down
Loading