docs: record first tone_model_eval run (GROW-6139) - #729
docs: record first tone_model_eval run (GROW-6139)#729warp-agent-staging[bot] wants to merge 2 commits into
Conversation
Runs the merged tone_model_eval skill's fixed-input copy-pass comparison across all 3 fixtures against 4 candidate models (claude-5-1-fable-high, claude-4-5-sonnet as the current-default stand-in, claude-4-5-haiku, gpt-5-mini), scored blind by a fixed judge model (gemini-3.1-pro, distinct family from every candidate). Result: no meaningful difference found (concision margin 0.33 of the required >=1.0; 0% mechanical-violation reduction of the required >=30%), and no cheaper-model recommendation (every cheaper candidate fails the technical-fidelity floor). out_of_repo_handoff.md's checklist is not triggered by this outcome. Adds .agents/logs/tone_model_eval_runs.md (new run log, following the existing _runs.md convention) and the raw rows/report data under .agents/logs/tone_model_eval/, so a later re-run has a baseline to diff against. Co-Authored-By: Warp <agent@warp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@warp-agent-staging[bot] I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR records the first tone_model_eval run by adding internal log artifacts and a run-log entry under .agents/logs/. The added JSON, JSONL, Markdown report, and run-log summary are consistent with the tone_model_eval report renderer and do not modify public documentation, production guidance, settings, or code.
Concerns
- No blocking concerns found. The declared low documentation risk is appropriate for internal append-only eval logs, the spec context contains no approved requirements to validate against, and the security pass found no changed code or configuration that introduces a security concern.
Verdict
Found: 0 critical, 0 important, 0 suggestions
Approve
Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review summary
The independent agent completed its review for this commit.
Findings
.agents/logs/tone_model_eval_runs.md:36— [SUGGESTION] The 'Oz run' field reads 'see GROW-6139' instead of following the entry template's own- **Oz run**: [URL]format defined at line 20 of the same file. Requested resolution: replace with the actual Oz/Warp run URL for this eval run, or explicitly note why the URL is unavailable.
Verdict
Approve
![warp-agent-staging[bot]](https://avatars.githubusercontent.com/in/2196476?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review summary
The PR correctly records a complete, reproducible score aggregate, but it omits the candidate rewrites needed to validate its stated technical-fidelity defects. Found: 0 critical, 1 important, 0 suggestions, 0 nits. Request changes
Findings
.agents/logs/tone_model_eval_runs.md:37—⚠️ [IMPORTANT] The run log says the non-5 technical-fidelity scores were manually verified from real defects, including an invented claim and a dropped qualifier, but the three listed data artifacts contain only scores, counts, and aggregates. They do not preserve any candidate rewrite or a durable link to one, so this review cannot perform the required spot-check or distinguish an accurate annotation from a judging or transcription error. Commit the per-fixture candidate rewrites (or link each row to a stable, access-controlled artifact containing the exact text) and identify the relevant row/output for each cited defect.
Verdict
Request changes
| - Recommend a cheaper model for production copy passes: **no candidate met the threshold** — `claude-4-5-sonnet` (judge gap 0.22, technical fidelity 3.67 < 4.0 min), `claude-4-5-haiku` (gap 0.44, fidelity 3.00 < 4.0), `gpt-5-mini` (gap 0.89, fidelity 1.67 < 4.0) all fail on technical fidelity. | ||
| - **Data**: `.agents/logs/tone_model_eval/2026-09-10-rows.jsonl`, `.agents/logs/tone_model_eval/2026-09-10-report.json`, `.agents/logs/tone_model_eval/2026-09-10-report.md` | ||
| - **Oz run**: see GROW-6139 | ||
| - **Notes**: First run (GROW-6139). `claude-4-5-sonnet` stands in for "the current production default model for docs drafting skills" as a **documented assumption**: live `oz-dev schedule list`/`schedule get` discovery against every schedule referencing docs drafting/audit skills found no explicit `model_id` in any schedule config — model selection for ad hoc/event-triggered drafting runs lives in the Warp app's Agent Profile UI (per `out_of_repo_handoff.md`), which isn't inspectable from this environment. `auto` was considered and rejected as the stand-in because it's a router that can resolve to different underlying models across calls, which would break the fixed-model comparison this eval depends on. Every technical-fidelity score below 5 was manually spot-checked against the source text and reflects a real defect (e.g. `gpt-5-mini` introduced an unsupported claim — "short-lived credentials for integrations" — not present in the original quickstart draft; `claude-4-5-haiku` dropped the original's "not admin-configurable" qualifier on host-priority order), not judge noise. Per this outcome, `out_of_repo_handoff.md`'s checklist is skipped — no model/schedule/Agent Profile change to make. |
There was a problem hiding this comment.
Review finding: the run log claimed the sub-5 technical-fidelity scores reflected verified real defects, but no candidate rewrite text was committed, so the claims were unverifiable. Commit all 12 raw candidate outputs (4 models x 3 fixtures) under .agents/logs/tone_model_eval/outputs/<fixture-id>__<model-id>.txt, and point the run log's two cited defect examples at the specific committed files and rows.jsonl entries. Co-Authored-By: Warp <agent@warp.dev>
Summary
Runs the merged
.agents/skills/tone_model_eval/skill (GROW-6133, now onmain) for the first time and records the result. This is a report-only run — no code changes, no changes toAGENTS.md,style_lint.py,doc_quality_policy/, or anydraft_*skill, and no model/schedule switch.Setup: all 3 fixtures validated cleanly. Candidates:
claude-5-1-fable-high(Claude Fable 5.1),claude-4-5-sonnet(current-default stand-in — see Notes below),claude-4-5-haikuandgpt-5-mini(cheaper candidates). Every candidate rewrote the identical full-page "before" text per fixture (fixed-input design). Judge:gemini-3.1-pro, a distinct model family from every candidate, scoring each of the 12 outputs blind in its own independent call.Result: no meaningful difference found.
claude-4-5-sonnet3.67,claude-4-5-haiku3.00,gpt-5-mini1.67), independent of their judge-score gap.Composite judge scores (concision / avoids-over-explaining / technical-fidelity):
claude-5-1-fable-high4.33/5 (4.67/4.67/3.67);claude-4-5-sonnet4.11/5 (4.33/4.33/3.67);claude-4-5-haiku3.89/5 (4.33/4.33/3.00);gpt-5-mini3.44/5 (4.33/4.33/1.67).Per
out_of_repo_handoff.md, a "no meaningful difference found" outcome skips that checklist entirely — no schedule or Agent Profile change to make.Notes / assumptions:
claude-4-5-sonnetstands in for "the current production default model for docs drafting skills." Liveoz-dev schedule list/schedule getdiscovery against every schedule referencing docs drafting/audit skills found no explicitmodel_idin any schedule config — perout_of_repo_handoff.md, that selection lives in the Warp app's Agent Profile UI, which isn't inspectable from this environment.autowas considered and rejected as the stand-in because it's a router that can resolve to different underlying models across calls, which would break the fixed-model comparison this eval depends on.gpt-5-miniintroduced an unsupported claim ("short-lived credentials for integrations") not present in the original quickstart draft, andclaude-4-5-haikudropped the BYOLLM fixture's "not admin-configurable" qualifier on host-priority order.Artifact decision: committing the report (rather than posting only to the ticket) because this repo already has an established
.agents/logs/*_runs.mdconvention for exactly this purpose — a durable, append-only record that a later re-run (or the "recompute after model-config changes" step inout_of_repo_handoff.md) can diff against. Rawrows.jsonl+report.json+report.mdare stored under.agents/logs/tone_model_eval/, mirroring the existingagent_doc_quality_baseline.md→.agents/logs/baseline/pattern.Rework changes
Review found 1 IMPORTANT issue: the run log claimed the sub-5 technical-fidelity scores reflected manually verified real defects, but no candidate rewrite text was committed anywhere, so those claims were unverifiable. Fixed by committing all 12 raw candidate outputs (4 models × 3 fixtures) under
.agents/logs/tone_model_eval/outputs/<fixture-id>__<model-id>.txt, and updating the run log's two cited defect examples to point at the specific committed file androws.jsonlrow for each.Verification
python3 .agents/skills/tone_model_eval/score_outputs.py validate-fixtures—3 fixtures valid.python3 .agents/skills/tone_model_eval/test_score_outputs.py— 43 tests pass, unchanged.python3 .agents/skills/doc_quality_policy/test_*.py(all 8 files) — pass, unchanged.python3 .agents/skills/style_lint/style_lint.py --changed— 0 files scanned (nosrc/content/docs/changes), 0 issues.npm run build— succeeds.computer_usestep: headless report-only run with nosrc/content/docs/page changes and no rendered UI surface.Documentation risk
Risk: low
Rationale: Adds only an append-only run log and raw eval-report data files under
.agents/logs/; nosrc/content/docs/changes, no code changes, no public documentation claim.Docs override: none
Unverified claims
None — internal tooling output only, no public documentation claims added.
Originating thread: https://warpdev.slack.com/archives/C09BVK0PL3Y/p1788971815025429