fix(docs): the quickstart's first run is one task — 19m37s to 5m52s, re-measured (#1171) - #1179
Conversation
…log (#1171) The README told a new user to promote every generated task and then run `cf work batch run --all-ready`. On the 0.9.3 cold-start run that was 25 agent runs, serially: the harness cut it off at 925s and the walkthrough totalled 19m37s against a 15-minute budget. Every other step in that run sums to ~315s, so the whole overrun is that one step. The fix is not to make 25 serial agent runs faster — nothing makes them fit — it is to stop telling a first-time user to build their entire project before they have seen the loop work once. Step 5 now promotes one task by the 8-char ID `cf tasks list` renders; Step 6 runs `cf work start <task-id> --execute`. The full backlog stays documented, in a collapsed aside that says it is long-running and points at `cf work batch follow` — the honest version of the third option in the issue. `walkthrough.sh` follows the same path it measures, so it no longer has an unbounded batch step. `docs/QUICKSTART.md` gets the single-task run as Option A for the same reason. `tests/ci/test_quickstart_budget_guard_1171.py` pins this in both places. The regression here is a docs edit, which nothing else in the suite can see. Refs #1171
…1171) The cold-start harness ran again against published 0.9.3 with the single-task quickstart: **352s, no TIMEOUT step**, down from 1177s. Artifacts in `artifacts-1171/`; the demo document grows a Run C built from them, so the README's timing claim points at a measurement rather than an estimate. Third-party review (codex) found four things worth fixing, all mine: - `walkthrough.sh` ran `cf tasks set status '' READY` when ID extraction failed, skipped the agent run, still reached `cf proof run`, and reported a fast total measuring a path nobody walked. `set -uo pipefail` has no `-e`, so nothing stopped it. Both steps now sit inside the non-empty guard. - The README said "to a PR in under six minutes". The harness stops at `cf proof run` and never runs `cf pr create`, which needs a GitHub remote. The claim now says what was actually measured. - `cf work batch follow` was documented bare in three places, but `batch_id` is a required Argument — following it literally is an argument error. `status` is the optional one. CLAUDE.md said all four were optional; only one is. - The guards did not cover `docs/QUICKSTART.md`, which could have regressed to recommending the whole backlog with everything else still green. The two new guards are mutation-checked: moving the promote step out of the TASK_ID branch, and dropping the long-running warning, each turn one red. Refs #1171
315s is the pypi-0.9.3 run's number; Run B in this document is the source-614 run. The claim is the same either way — the batch step is the entire overrun — but the arithmetic should match the table directly above it. Also drops a 'cf pr create is fast' aside the harness never measured. Refs #1171
GLM precision review — PR #1179
|
…1171) Post-PR review caught that `else_branch < promote < work_start` is satisfied by if [ -z "$TASK_ID" ]; then note ...; else :; fi step "5-promote-one" ... step "6-work-start" ... which is precisely the regression the test claims to prevent. It now slices the block between `else` and its column-0 `fi` and asserts both steps are inside it. Mutation-checked against that exact bypass: valid bash, and now red. Also drops an overclaim in the Run C prose. Run B was a source install and Run C is published 0.9.3, with 25 generated tasks against 21 — the commands are not the only variable that changed. Neither of the others is worth fourteen minutes, which is the point, so the prose now says that instead. Refs #1171
Third-party review —
|
| Finding | Fix |
|---|---|
walkthrough.sh ran cf tasks set status '' READY on failed ID extraction, skipped the agent run, still reached cf proof run, and reported a fast total measuring a path nobody walked (set -uo pipefail, no -e) |
Both steps moved inside the non-empty guard; critical finding filed on the other branch |
README claimed "empty directory to a PR in under six minutes" — the harness stops at cf proof run and never runs cf pr create |
Claim narrowed to what was measured |
cf work batch follow documented bare in README, docs/QUICKSTART.md and CLAUDE.md, but batch_id is a required typer.Argument |
Fixed in all three; status is the optional one |
Guards did not cover docs/QUICKSTART.md, which could regress alone with everything else green |
Added, handling its codeframe spelling |
Local gate: 6495 passed, 19 skipped, 0 failed (10m24s). ruff check . clean.
ReviewVerified this against the actual CLI ( Confirmed accurate:
Minor, non-blocking observations:
No bugs found in the logic changes ( |
GLM precision review — PR #1179
|
ReviewIndependent pass over the final diff ( Verified, nothing new to add on correctness:
No new issues found. This matches the prior review's conclusion — I looked for a reason to disagree given three rounds of review already passed on this PR, and didn't find one. The one thing worth restating because it's easy to lose in a long thread: the two previously-noted minors (grepping 8-hex chars from the whole Good to merge from my read. |
Demo verification — every acceptance criterion, with outcome evidenceThe demo for this issue is the cold-start harness itself: a throwaway container
The run
The two non-zero exits, deliberately not fixed hereNeither costs time, and neither is #1171:
The transcript also still shows the |
claude-review's leftover from the restructure: 'While the batch runs, check status in another terminal' still framed monitoring around a batch, but Option A is now a single synchronous run that streams to the terminal you started it in. Says which is which, and adds the `follow <batch-id>` form that Step 5 already points at. Refs #1171
GLM precision review — PR #1179
|
ReviewFourth pass on this PR (after two codex rounds, a GLM automated pass, and two prior claude reviews already in the thread) -- independently re-verified the concrete claims against the current source rather than re-reading the prior conclusions, since that is where a doc PR like this actually fails. Verified against codeframe/cli/app.py directly:
Read walkthrough.sh end to end: the TASK_ID guard genuinely wraps both step '5-promote-one' and step '6-work-start' inside the else branch, 6-proof-run runs unconditionally afterward (matches what the test suite and the Run C transcript both show), and the interpolation of $TASK_ID into the nested bash -c command is safe because TASK_ID is constrained by the extraction regex to 8 hex characters before it ever reaches the shell -- no injection surface even from adversarial task titles. Doc consistency: README's Detailsblock is well-formed and closes correctly; QUICKSTART's Step 6 ('Monitor Progress') no longer frames monitoring purely around a batch -- it explicitly separates Option A (synchronous, no monitoring needed) from Options B/C (batch, needs a second terminal), which addresses the sequencing leftover an earlier review round flagged in this thread.No new bugs, and nothing I checked contradicts the prior rounds' conclusions -- the numeric claims (352s/5m52s, the 825s delta, the 276s Run B baseline) all trace back to the committed artifacts-1171/timings.tsv and total.txt correctly. This is a well-scoped, thoroughly self-checked docs fix. Good to merge from my read as well. |
1 participant
Closes #1171.
The README's Quick Start could not be completed in fifteen minutes, and the
reason was not that anything was slow. Step 5 told a new user to promote every
generated task to
READYand Step 6 told them to runcf work batch run --all-ready— for a small todo API that is 21–25 agent runs, serially. Thecold-start harness cut the batch off at 900s and the walkthrough totalled
19m37s.
Nothing makes twenty-five serial agent runs fit in fifteen minutes. So the fix
is in the documentation: the quickstart's first run is one task.
What changed
README Steps 5 and 6. Step 5 promotes a single task by the 8-character ID
cf tasks listrenders (cf tasks set status <task-id> READY—cf tasks settakes a partial ID, so that prefix is enough). Step 6 runs
cf work start <task-id> --execute. The full backlog is still documented, in a collapsedaside that says plainly that it is long-running and points at
cf work batch status/cf work batch follow <batch-id>— the honest version of the issue'sthird option, for the path that really is long.
docs/QUICKSTART.mdgets the single-task run as Option A for the samereason, and the same long-running warning on the batch.
scripts/quickstart-cleanroom/walkthrough.shfollows the path it measures,so it no longer has an unbounded batch step.
Re-measured, not estimated
Third acceptance criterion. Same harness, same container, published
0.9.3—the path a new user actually takes. Artifacts in
scripts/quickstart-cleanroom/artifacts-1171/, and the demo document grows aRun C built from them.
6-batch-run6-work-startAll three acceptance criteria met. The whole 825-second difference is the one
step that is gone: every other step in the old run summed to 276 seconds.
What
head -1picked is worth noting — priority 0,Depsempty.cf tasks generateorders its output foundational-first, so following the docs literallylands on an unblocked task. The docs don't rely on that:
cf work startdoesnot gate on dependencies, so any ID gives the same run.
Guard
tests/ci/test_quickstart_budget_guard_1171.py— six assertions acrossREADME.md,docs/QUICKSTART.mdandwalkthrough.sh. The regression here is adocs edit, which nothing else in the suite can see. Mutation-checked: moving the
promote step out of the
TASK_IDguard, dropping the long-running warning, andputting the batch back in front of
cf work starteach turn one red.Review findings fixed (codex, pre-PR)
walkthrough.shrancf tasks set status '' READYwhen ID extraction failed,skipped the agent run, still reached
cf proof run, and reported a fast totalmeasuring a path nobody walked —
set -uo pipefailhas no-e, so nothingstopped it. Both steps now sit inside the non-empty guard, with a critical
finding on the other branch.
stops at
cf proof runand never runscf pr create. Claim now matches themeasurement.
cf work batch followwas written bare in the README,docs/QUICKSTART.mdand
CLAUDE.md, butbatch_idis a requiredtyper.Argument— following itliterally is an argument error.
statusis the optional one;stop,resumeand
followare not. Fixed in all three.docs/QUICKSTART.md, which could have regressed onits own with everything else green.
Known limitations
cf work startends by filing a blocker (exit 1) rather than guessing at adecision — by design, and the README now says so and points at
cf blocker answer.cf proof runexits 2 on an empty ledger — that is [P2.34] The quickstart's final step always fails — cf proof run exits 2 on a fresh workspace and the README never says to capture a requirement #1173, unchangedand out of scope here.
0.9.3, which predates theCallType.VERIFICATION_FIXfix, so the transcript still shows that traceback ([P1.43] CallType has no 'verification_fix' member — every self-correction call loses its token usage and prints a traceback #1172, already fixed on
mainbut not yet published — the issue is still open and should be closed).cf prd generate(76s) and the agent run (179s) are LLM-bound and will vary; the margin to the
budget is large enough that this does not seem worth pinning in CI.