Make command rows compaction-aware

CHANGELOG.md

@@ -3,8 +3,19 @@
 ## Unreleased
 
 - Expand queued prompt templates and Agent Skills at delivery, with arguments, images, short aliases such as `/bro`, and full-batch restoration if expansion fails.
+- Make command rows compaction-aware: idle `/compact` uses Pi's public compaction API, `/reload` waits for direct or automatic compaction to settle, and ordinary messages stay in Pi's native compaction queue.
+- Preserve stable row IDs, lanes, attachments and pause state when committed rows cross a `/reload` runtime swap, including rows added after reload scheduling and repeated queued reloads.
+- Keep image-bearing command text as a normal queued message so attachments are never discarded.
+- Hold queued follow-ups while Pi decides whether an error, length stop or context overflow needs retry or automatic compaction.
+- Restore and pause `/compact` when compaction cannot start, and restore only an unsent all-mode tail after a synchronous partial handoff failure.
+- Leave RPC, JSON and print-mode input unchanged; queue ownership is TUI-only.
+- Rebind editor guards across runtime reloads and capture command rows even while slash autocomplete is visible.
+- Normalize native post-compaction input classification so whitespace and immediate hidden built-ins cannot strand queued rows.
+- Add deterministic AgentSession retry coverage and a reproducible real-TUI evidence harness for manual/overflow compaction, abort recovery, native ordering, repeated reloads, resources and all-mode delivery.
+- Keep Pi package ranges unpinned so compatibility validation follows current Pi releases.
+
 - Add command rows: `/compact [instructions]` and `/reload` queue in FIFO position and execute only once the agent is idle, so rows behind them wait — e.g. a queued `continue` delivers after compaction completes.
-- Queue a mid-run `Enter` on `/reload` instead of surfacing Pi's built-in "wait until the agent finishes" warning; mid-run `Enter` on `/compact` keeps Pi's built-in behaviour.
+- Queue a mid-run `Enter` on `/reload` instead of surfacing Pi's built-in "wait until the agent finishes" warning; mid-run `Enter` on `/compact` uses Pi's public compaction API and holds queued rows until compaction settles.
 - Restore rows queued behind a `/reload` after the runtime swap.
 - Execute idle `Option+Enter` command submissions instead of letting them reach the LLM as text.
 

README.md

@@ -67,7 +67,7 @@ The extension keeps Pi’s 2 delivery classes:
 - each lane keeps its own first-in, first-out order
 - Pi’s `one-at-a-time` and `all` settings apply independently at active-run delivery boundaries
 
-The extension hands messages back to Pi’s native queues only when their delivery boundary arrives. They remain visible and editable before that point. Pi records delivered rows as normal user messages.
+The extension hands messages back to Pi’s native queues only when their delivery boundary arrives. They remain visible and editable before that point. Pi records delivered rows as normal user messages. Queue ownership is TUI-only; RPC, JSON and print-mode input pass through unchanged.
 
 ## Prompt templates and Agent Skills
 
@@ -77,12 +77,15 @@ Pi cannot invoke arbitrary commands through its public extension API. `/compact`
 
 ## Command rows
 
-Rows whose text is exactly `/compact`, `/compact <instructions>` or `/reload` are command rows. They execute the Pi command instead of becoming an LLM message:
+Text-only rows whose text is exactly `/compact`, `/compact <instructions>` or `/reload` are command rows. A row with image attachments remains a normal message even if its text matches a command, so attachments are never discarded. Command rows execute the Pi command instead of becoming an LLM message:
 
 - `Option+Enter` while the agent works queues the command in follow-up order
 - a command row executes only once the agent is idle; rows behind it wait — so `/compact` followed by `continue` compacts first and delivers `continue` after compaction completes
-- `/reload` runs Pi’s built-in reload; rows queued behind it are restored after the runtime swap
-- `Enter` on `/reload` while the agent works queues it too, replacing Pi’s built-in “wait until the agent finishes” warning; `Enter` on `/compact` keeps Pi’s built-in immediate behaviour
+- `/reload` runs Pi’s built-in reload; committed rows queued behind it retain their IDs, lanes, attachments and pause state across the runtime swap
+- idle `/compact` uses Pi’s public compaction API so queued rows resume when compaction finishes; a start failure restores and pauses the command row
+- `/reload` submitted while the agent works or tracked compaction runs stays queued instead of showing Pi’s built-in wait warning
+- `Enter` on `/compact` while the agent works uses Pi’s public compaction API and holds visible rows until compaction settles
+- ordinary messages submitted during compaction remain in Pi’s native queue
 - `Option+Enter` on a command while the agent is idle executes it immediately instead of sending the text to the model
 - command rows show a `⚙` marker and pause, resume and edit like any other row; editing a row into or out of command form just works
 
@@ -106,13 +109,17 @@ A touched head row is pinned until you save or cancel. In `one-at-a-time` mode,
 
 Aborting a run pauses both visible lanes. This prevents a follow-up from starting immediately after the abort.
 
-Press `Enter` on the empty composer to resume. A failed handoff returns the affected batch to the front of its lane.
+Press `Enter` on the empty composer to resume. A synchronous handoff or preflight failure returns the affected batch to the front of its lane.
 
-Queue state, pause state and edit drafts are session-local. They never enter the Pi transcript.
+Queue state, pause state and edit drafts are session-local. They never enter the Pi transcript or persistent session data. A `/reload` runtime swap carries committed rows and pause state through a short in-process handoff; unsaved edit drafts do not cross the swap.
 
-## Proof limitation
+## Public API limits
 
-If an `all`-mode lane stays pinned until the agent settles, saving from idle restarts the run with that lane’s head. Pi receives the remaining rows at the next native boundary. Exact single-batch restart after this edge case remains open before release.
+Pi’s public `sendUserMessage` API is fire-and-forget. The extension restores synchronous dispatch failures and preflight/expansion failures without reordering, but Pi does not expose later asynchronous input rejection to extensions. Inferring rejection from queue timing could duplicate a delayed successful handoff, so the extension does not do that.
+
+Pi also exposes queued `/reload` only through the TUI editor’s `void` submit callback. The extension prevents known busy and compaction conflicts and restores trailing rows on a successful runtime swap, but Pi cannot acknowledge or reject that submit back to the extension.
+
+If an `all`-mode lane stays pinned until the agent settles, saving from idle starts the new run with the lane head, then delivers the remaining rows in FIFO order at the next native boundary. The public API has no atomic idle-to-native-queue batch operation, so this restart cannot be one native batch.
 
 ## Editor composition
 
@@ -127,12 +134,13 @@ The extension composes with custom editors including raw-paste and pi-session-hu
 ```bash
 npm install
 npm run ci
+./test/tui-evidence.sh /tmp/pi-queue-tui-evidence
 pi -e ./index.ts
 ```
 
-The automated suite covers delivery, editing, command rows, resource expansion, recovery, images and editor composition. Check TUI changes in a real Pi session as well.
+The automated suite covers delivery, editing, command rows, resource expansion, recovery, images, editor composition, repeated reloads, real retry ordering, real manual compaction success/failure and real automatic overflow compaction. The tmux harness exercises the same paths through Pi's real TUI, including actual runtime reloads and native post-compaction input.
 
-Automated against Pi 0.80.9 and smoke-tested interactively with Pi 0.84.1.
+The Pi package ranges are intentionally unpinned. The full suite and real-TUI harness are verified against the current resolved Pi release; see [the validation record](docs/validation.md) for exact commands and evidence.
 
 ## Security
 

docs/validation.md

@@ -0,0 +1,93 @@
+# Compaction and reload validation
+
+This document records the deterministic validation matrix for compaction-aware command rows. The implementation remains extension-only and uses public Pi extension APIs.
+
+## Automated suite
+
+The Pi package ranges are intentionally unpinned. The lockfile records the versions used for a reproducible checkout, but the package manifest does not declare an artificial Pi compatibility target.
+
+Run the resolved dependency set:
+
+```bash
+npm ci --ignore-scripts
+npm run ci
+```
+
+Refresh to the current Pi packages before compatibility review:
+
+```bash
+npm update --ignore-scripts \
+  @earendil-works/pi-ai \
+  @earendil-works/pi-coding-agent \
+  @earendil-works/pi-tui
+npm run ci
+```
+
+The suite covers queue/edit invariants, command classification, images, one-at-a-time and all-mode delivery, synchronous partial handoff restoration, non-TUI pass-through, prompt and Skill expansion, manual compaction success/failure, automatic overflow compaction, retry ordering, repeated reload restoration, and compaction/native-input ordering.
+
+Latest result with Pi 0.84.1: 81 tests passed.
+
+## Real TUI evidence
+
+`test/tui-evidence.sh` starts the real Pi 0.84.1 TUI under tmux with a deterministic faux provider. It uses actual terminal key sequences, public compaction lifecycle events, public provider registration, actual runtime reloads, and Pi's real native compaction queue.
+
+Run:
+
+```bash
+./test/tui-evidence.sh /tmp/pi-queue-tui-evidence
+```
+
+The output directory contains plain terminal captures, provider-call logs, lifecycle-event logs, and runtime-initialization logs. Run it immediately before review so `summary.txt` records the exact Pi version, commit and working-tree state under test. A release evidence run should report `working tree: clean`.
+
+The latest complete run reported:
+
+```text
+pi: 0.84.1
+commit: <reviewed commit>
+working tree: clean
+manual events: {"event":"session_before_compact","reason":"manual"} {"event":"session_before_compact","reason":"manual"}
+overflow events: {"event":"session_before_compact","reason":"overflow"} {"event":"session_before_compact","reason":"threshold"}
+runtime initializations across two queued reloads: 3
+captures: abort-paused, manual-reload-resources, native-before-command, automatic-overflow, all-mode
+```
+
+The three runtime initializations are the initial load plus two queued `/reload` rows. The final queued message ran after both reloads.
+
+The semantic capture excerpts were:
+
+```text
+[compaction]
+Compacted from 798 tokens
+FAUX RESPONSE: after manual compaction
+
+Error: Compaction failed: Summarization failed: synthetic TUI summary failure
+FAUX RESPONSE: after failed compaction
+
+Operation aborted
+follow-ups (1) · paused
+enter resume · option+up edit · escape keep paused
+FAUX RESPONSE: after abort resume
+
+Reloaded keybindings, extensions, skills, prompts, themes, and context files
+FAUX RESPONSE: after repeated reload
+
+PROMPT EXPANDED: first=alpha all=alpha beta default=fallback
+[skill] bro
+FAUX RESPONSE: <skill name="bro" ...>
+```
+
+The native post-compaction ordering capture showed the ordinary native message finishing before the extension-owned command, and `/reload` never reached the model:
+
+```text
+[compaction]
+Compacted from 785 tokens
+ordinary native during compaction
+FAUX RESPONSE: ordinary native during compaction
+Reloaded keybindings, extensions, skills, prompts, themes, and context files
+```
+
+The overflow event log recorded `reason: "overflow"`, the TUI rendered a compaction entry, and `overflow-provider-calls.jsonl` proved the queued follow-up completed exactly once. `all-mode-provider-calls.jsonl` proved all three rows reached Pi exactly once in FIFO order; the all-mode capture rendered them together before the final response.
+
+## Public API boundary
+
+`ExtensionAPI.sendUserMessage` and the TUI editor submit callback return `void`. The extension can restore synchronous handoff failures and preflight/expansion failures, but it cannot prove every later asynchronous acceptance or rejection without risking duplicate delivery. Queued `/reload` likewise has no result channel. These limits are documented in the README and are not hidden by timing heuristics.