Design: usage instrumentation → Loki → AI flow-coverage analysis#896
Open
edwh wants to merge 3 commits into
Open
Design: usage instrumentation → Loki → AI flow-coverage analysis#896edwh wants to merge 3 commits into
edwh wants to merge 3 commits into
Conversation
Approved design for capturing real user flows (client autocapture + API middleware), shipping them to a private Loki via the NDJSON+Alloy pattern, and mining them into a ranked test-gap report that an AI consumes to write missing golden-flow tests. Produced from research into the Freegle client-to-Loki pipeline, a full codebase inventory, and current practice for behavioural capture and trace-based test generation, then hardened by a four-lens adversarial review (privacy/GDPR, signal quality, operations, frontend feasibility). Implementation follows in phased PRs per the design's Phasing section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzY1phBjyT3HogpeS53zon
Round 2 attacked the roll-our-own premise and the flow->test matching feasibility with a worked example from the real codebase. Result: - Client capture now rides the already-deployed Matomo tracker (which is Matomo CLOUD, not self-hosted as v1 claimed): tagged data-flow events and always-on discovery descriptors both go via trackEvent, consent is fixed with Matomo's native consent API, and the bespoke ingestion endpoint + hardening + batching are deleted. No off-the-shelf product replaces the server-side stream or the mining/matching layer (verified across 11 candidates), so those stay custom. - The matching layer gains binding correctness rules the paper simulation proved necessary: ambient-route denylist (naive route overlap scored Jaccard ~0.17 against the flow's own covering test), dual wire/effective HTTP methods (Laravel method override), flow segmentation and variant normalization rules, a tested PHPUnit disambiguation rule (58 identical helper call sites), Playwright page.evaluate rewrites, and a round-trip validation gate: flowmine must correctly attribute the flows generated by the existing 6 Playwright specs before production output is trusted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzY1phBjyT3HogpeS53zon
Runs a prototype of the flowmine matcher against simulated capture data, with the manifest extracted from the real Playwright specs. Demonstrates both halves of the fitness claim: group-create is matched to group.test.js with evidence (duplicate avoided) and account registration is detected as uncovered, yielding a generated candidate spec verified against the actual registration view. The first run mis-attributed group-create to device.test.js because that spec performs the same createGroup() helper as scaffolding - flat overlap scoring cannot distinguish subject from setup. The fix (helper evidence weighted 0.25 unless a test title shares the flow's domain tokens) plus a coarser variant key (flow page + action-route set) are now binding in the design's manifest section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzY1phBjyT3HogpeS53zon
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design document for instrumenting real usage (client + API) into a self-hosted Loki and mining it for test-coverage gaps — the plan is to capture what users actually do, compare against the existing Playwright/PHPUnit/Jest estate, and generate the missing golden-flow tests from evidence rather than guesswork.
The document records the research base (Freegle's client→Loki pipeline, full codebase inventory, current-practice review), the adversarial-review findings that shaped it (consent gating that actually works — including a pre-existing bug where Matomo ignores the consent cookie; data-flow attributes as stable element identity with a CI lint against decay; ingestion hardening at nginx; k-anonymity floor on anything exported for AI analysis), the decisions taken, and a four-phase implementation plan where each phase is its own PR.
Merging this PR adopts the design; phase 1 (capture + ship) follows separately.