docs(factories): document measurement and improvement - #522
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![oz-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR adds the Factories measurement and improvement documentation, including metrics, scorers, benchmark suites, autofix, and an improvement-loop workflow. The content is generally aligned with the docs style guide, and I did not find security concerns or spec-drift issues from the provided context.
Concerns
- The new page links readers to Factories pages that are not present in the attached diff, which risks publishing broken internal links.
Verdict
Found: 1 critical, 0 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
| * [**Factory agents**](./factory-agents) - Choose responsibilities, models, harnesses, skills, and tools for specialized roles. | ||
| * [**Factory definitions as code**](./factory-as-code) - Review changes to agent prompts, skills, and factory configuration. |
There was a problem hiding this comment.
🚨 [CRITICAL] These related-page links point to ./factory-agents and ./factory-as-code, but neither target page appears in this diff; add those pages before publishing or link to existing Factories pages so the docs link check does not fail.
3acee02 to
020e7cc
Compare
020e7cc to
848d581
Compare
848d581 to
9e6562a
Compare
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the placeholder Factories measurement page with a complete guide to dashboard metrics, Scorers, benchmarks, Self-improvement, and a practical evidence-driven improvement loop.
Concerns
- No blocking concerns found in the attached diff.
Verdict
Found: 0 critical, 0 important, 0 suggestions
Approve
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
Co-Authored-By: Warp Agent <agent@warp.dev>
9e6562a to
aaecb28
Compare
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR replaces the factories measurement placeholder with a full page covering dashboard metrics, Scorers, benchmarks, Self-improvement, and a practical improvement loop. The content is scoped to the changed page, and the security/spec passes did not identify material issues.
Concerns
- Two wording issues make otherwise useful explanations harder to parse; both have inline suggestions.
Verdict
Found: 0 critical, 0 important, 2 suggestions
Approve with nits
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
|
|
||
| ## Configure Scorers | ||
|
|
||
| A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable. |
There was a problem hiding this comment.
💡 [SUGGESTION] Rewrite this clause so it says Correctness runs automatically in benchmarks, rather than that it "benchmarks run automatically."
| A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable. | |
| A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that runs automatically in benchmarks. Define one user-defined Scorer for one decision so its failures remain actionable. |
|
|
||
| ## Compare configurations with benchmarks | ||
|
|
||
| A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test: |
There was a problem hiding this comment.
💡 [SUGGESTION] Use the possessive form to make this sentence easier to parse.
| A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test: | |
| A benchmark suite fixes one agent and compares its configurations on the same tasks. Define the suite, then launch it with the configurations you want to test: |
1 participant
Summary
Organizes dashboard metrics, scorers, benchmarks, and autofix around the questions they answer and their limitations. Manual/periodic scoring and a six-step evidence-driven improvement loop form the operational path.
Final size: 883 prose words. Across the section, the senior editorial pass reduced prose from about 14,600 to 7,649 words while preserving verified behavior and security caveats.
Dependency
Depends on #513, which provides the Factories section scaffold and targets
hyc/factory-launch. Until #513 merges, its shared commits appear in this PR; afterward the diff reduces to this feature's content. Branch-local CI can report missing sibling-page links (and #519 can report #513's removed hub slug) until the dependent content and shared IA branch land; the full nine-page integration build is green.Validation
npm run typecheck: passednpm run build: 377 pages built successfullyLatest source refresh
Adds current Dashboard metrics, Scorer terminology, Self-improvement surfaces, pr_facts limitations, Time saved/Autonomy/latency/run breakdown, and redesigned Benchmarks.
Verified against Warp
3d4ee7236363and warp-server2c864b0b8404. Broken, placeholder, partial, and spec-only surfaces remain excluded.Proposed reviewers
For planning only; no review requests have been sent.
@szgupta@Legoben@vkodithalaScreenshots
Not included. The revision uses sourced tables, Mermaid diagrams, and verified code/config examples; no safe approved Factory UI assets exist yet.
Unverified claims
None — all UI labels, defaults, eligibility claims, diagrams, and configuration details were verified against source or deliberately omitted.
Follow-ups
A dependent shared IA PR will add the grouped nine-page sidebar, desktop/mobile Early Access badge, and legacy generic-page cleanup after the content pages land.
Conversation: https://staging.warp.dev/conversation/5ff89820-2d80-4518-981e-178845029de1
Plan: https://staging.warp.dev/drive/notebook/7ZPKWz7hM5I59o4Gg2ptYi
Co-Authored-By: Warp Agent agent@warp.dev