Skip to content

test: add tvOS platform command coverage manifest - #1919

Merged
thymikee merged 3 commits into
mainfrom
codex/1426-tvos-coverage-manifest
Aug 20, 2026
Merged

test: add tvOS platform command coverage manifest#1919
thymikee merged 3 commits into
mainfrom
codex/1426-tvos-coverage-manifest

Conversation

@thymikee

@thymikee thymikee commented Aug 20, 2026

Copy link
Copy Markdown
Member

Summary

Validation

Evidence gathered at commit 8611dcfde:

  • Fresh worktree install/build, pnpm check:quick, and the full pnpm check:affected --run completed successfully; all runnable checks passed.
  • The new tvOS static gate passed all 7 tests, and the existing provider-backed tvOS remote scenario passed.
  • Planted-red proof: omitting only the artifacts row failed catalog completeness and changed the report to 53 total / 38 gaps; restoring it returned all 7 gate tests to green.
  • Follow-up fix: host-dependent audio is now a command-contract row backed by the existing exhaustive tvOS-inclusive capability parity test. The gate restores full oracle parity and projects the declared host-dependent admission; the pinned report is 3 denials / 13 contracts / 38 gaps.
  • GitHub-authoritative native/device/coverage lanes remain for CI. The follow-up has been pushed and those checks are rerunning; this PR is published and reported, not merge-ready on live evidence.

Follow-up

The coverage summary builder remains local to each platform manifest to keep this tvOS change platform-scoped and avoid introducing a shared row type. The macOS coverage-manifest PR should factor the pure summary builder into test/integration/support/ before adding the fourth copy; this PR's count assertion keeps tvOS command additions explicit.

Touched files: 2, tvOS-only. Docs/help/skills were unchanged because this is a test-coverage manifest and no command behavior changed.

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.34 MB 2.34 MB 0 B
JS gzip 769.4 kB 769.4 kB 0 B
npm tarball 894.5 kB 894.5 kB 0 B
npm unpacked 3.12 MB 3.12 MB 0 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 18.6 ms 20.0 ms +1.4 ms
CLI --help 47.0 ms 47.5 ms +0.5 ms

Top changed chunks: no changes in the largest emitted chunks.

@thymikee

Copy link
Copy Markdown
Member Author

Review — approve with two asks (neither changes the classification)

Read the full manifest and gate against the #1426 ground rules; also verified the wiring and the grouped gap issue independently.

What's right, and verified:

  • Wired into a PR lane by construction — the gate sits at test/integration/smoke-tvos-platform-coverage.test.ts, which the test:integration:node glob (test/integration/*.test.ts) picks up, same as the ios/android/web gates. Not a silent-green file. ✓
  • Honest 0-live — no tvOS CI lane exists, and the manifest says so instead of dressing the provider scenario up as live. That's the point of the exercise. ✓
  • Self-verifying evidence, three ways — contract rows are checked for file-exists + contains-named-test at gate time; the provider-scenario rows are cross-checked against actual callCommand('<cmd>' invocations in the scenario source; and capability-denial rows are mechanically diffed against isCommandSupportedOnDevice(TVOS_SIMULATOR) rather than hand-listed. That last test is stronger than what the iOS manifest does — a capability-matrix change and this manifest can't drift apart silently. ✓
  • One grouped gap issue (test: close tvOS platform command-coverage gaps from #1426 #1914, exists, open) with a gate test pinning it — exactly the rescope's rule. ✓
  • Counts reconcile: 3 denial + 12 contract + 39 gap + 0 live = 54 = PUBLIC_COMMANDS. ✓
  • Hand-rolled row union, no speculative shared type module — per the rescope. ✓

Ask 1 — the summary builder just became a third copy. buildCoverageClassificationSummary is now duplicated verbatim in android-emulator-e2e/coverage-manifest.ts, web-e2e/coverage-manifest.ts, and here. The rescope said "factor when a fourth copy makes it painful" — macOS and Linux manifests are both queued, so copies four and five are weeks away at most. Either factor the builder (it's a pure ~25-line function; test/integration/support/ is the natural home) in this PR, or leave an explicit note assigning the factoring to the macOS PR so it doesn't become five copies by drift. Same #1892-uniqueStrings logic. The row type unions should stay hand-rolled per platform — this is only about the counter.

Ask 2 — planted-red proof, one sentence. The exhaustiveness deepEqual + satisfies make a dropped row a structural failure, but the #1426 ground rule asks each gate to demonstrate it once. Run it with one row commented out, paste the one-line failure into the PR body, done.

One nit, no action needed: the hardcoded counts assertion (3/12/39/0/54) doubles as the "coverage report" acceptance artifact and will need a bump whenever a command is added or a gap closes — that friction looks intentional (it forces the manifest to be revisited), just confirming it's understood rather than accidental.

@thymikee

Copy link
Copy Markdown
Member Author

Addressed the summary-builder ask in the PR body: this tvOS PR stays platform-scoped, and the macOS coverage-manifest PR is explicitly assigned to factor the pure builder into test/integration/support/ before adding the fourth copy. The planted-red proof is also recorded in the Validation section.

@thymikee

Copy link
Copy Markdown
Member Author

The required Integration Tests lane is red because the manifest records audio as a known-gap while the authoritative tvOS capability matrix denies it. Classify C.audio as capability-denial and update the pinned summary, or change the capability source if audio is meant to be admitted; the manifest must not disagree with its mechanical oracle. Then rerun the integration lane.

@thymikee

Copy link
Copy Markdown
Member Author

Fixed in follow-up commit 44e92a4 and pushed. The hosted Integration Tests failure showed that audio is denied on CI but host-dependent on Darwin; the manifest now records audio as capability-denial, pins 4 denials / 38 gaps, and the gate asserts Darwin admission separately. Focused tvOS gate and pnpm check:affected --run pass locally. Issue #1914 was updated to remove audio from its gap list.

@thymikee

thymikee commented Aug 20, 2026

Copy link
Copy Markdown
Member Author

[P1] Keep the capability oracle total. On Darwin, audio is admitted for the tvOS simulator, so this is not a capability denial; filtering it from both sides only hides the mismatch. Restore full parity and model host-dependent audio as a contract/known gap with executable tvOS evidence, rather than exempting it from the matrix.

@thymikee

Copy link
Copy Markdown
Member Author

Addressed the P1 in follow-up commit 8611dcf and pushed. Audio is now a host-dependent command-contract row backed by the existing tvOS-inclusive capability parity test; the gate compares the full oracle and projects the declared host-dependent admission instead of filtering audio. Counts are now 3 denials / 13 contracts / 38 gaps. Local focused and affected gates pass.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 20, 2026
@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed 8611dcf: the capability comparison is total again, and host-dependent audio is a command-contract row backed by the executable matrix-parity oracle. No code findings. All completed checks are green; iOS Smoke is still in progress, so this is published and reviewed rather than merge-ready.

@thymikee

Copy link
Copy Markdown
Member Author

Follow-up: iOS Smoke completed successfully on 8611dcf; all checks are now green. #1919 is merge-ready.

@thymikee
thymikee merged commit 96afa9d into main Aug 20, 2026
28 checks passed
@thymikee
thymikee deleted the codex/1426-tvos-coverage-manifest branch August 20, 2026 19:47
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-20 19:48 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant