ci: smoke-check a local preview of the packaged Worker before merge - #278
Merged
Conversation
The 09-04 outage was detected in production. #269 built detection and recovery around the deploy, but the only environment in which a rendering defect is DETECTED is still production, and the chain `deploy succeeds -> smoke fails -> rollback fires` has never executed end to end. Runs `.github/scripts/smoke-docs.mjs` — the same script `deploy-docs.yml` runs against the live site, unmodified — with `--base` pointing at a local `opennextjs-cloudflare preview` of the `.open-next` package the `build` job already produces on every pull request since #262. No second build, no Cloudflare credentials. Readiness is asserted from wrangler's own `Ready on` line before anything is judged, because "no findings" from a server that never started is indistinguishable from a rendered site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChPQM8jamxLUfUAxwFpJ8S
…ASURED A crashed preview and an exhausted boot budget call for different fixes — a broken bundle versus a slow runner — and the log printed underneath is the same either way. Both demonstrated: with the budget cut to 5s the step reports the budget ran out; with the compiled OpenNext config removed it reports the process exited. Both exit 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChPQM8jamxLUfUAxwFpJ8S
`setsid` only moves the child into its own group once the forked child has exec'd it, so reading the group with `ps` immediately after `&` is a race that can still return the step's own group — and the guard that stops the step from signalling itself then falls back to signalling the pnpm wrapper alone, which ignores SIGTERM. Measured on run 34250422860: the step went green and the runner's own orphan sweeper had to terminate esbuild and two workerd processes after the job. The identical code cleaned up completely in this repo's container on every run, which is how a race presents. Resolving the group inside the cleanup function removes the window. A preview that genuinely shares the step's group now says so as a warning instead of leaving a silent orphan, and the ready line records both group ids. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChPQM8jamxLUfUAxwFpJ8S
os-bill
marked this pull request as ready for review
September 8, 2026 16:33
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #274
A rendering defect currently has exactly one detector: production. #269 built the
recovery around it — the deploy is gated on CI, the published artifact is the one CI
tested, the live site is smoke-checked after deploying, a bad deploy auto-rolls back —
but the chain
deploy succeeds -> smoke fails -> rollback fireshas never once executedend to end. A check that fires before merge costs a red pull request; the same check
firing after merge costs a live outage plus a recovery path nobody has seen run.
This adds one step to the
buildjob:.github/scripts/smoke-docs.mjs— the samescript
deploy-docs.ymlruns againsthttps://docs.objectos.ai, unmodified — pointedwith
--baseat a localopennextjs-cloudflare previewof the Worker this job alreadypackages.
The card was cheaper than it was written, and this is why
It was filed when the build job packaged a Worker only on a push to
main. #262 removedthat condition, so on every pull request a
.open-nextpackage already exists by thetime this step runs.
opennextjs-cloudflare previewdoes not build: it populates theincremental cache and runs
wrangler devon the package that is already there. Noopennextjs-cloudflare build, no secondnext build, no Cloudflare credentials — theWorker is served locally under real workerd.
Measured on a GitHub runner, this step start to finish:
Ready onFor scale, in the same job
turbo run buildtook 69 s and packaging the Worker 16 s. Thewhole
buildjob went from about 100 s to 131 s.Why the step runs last, after the artifact upload
previewcopies.open-next/cacheinto.open-next/assets/cdn-cgi. Measured on thistree,
.open-nextgoes from 386 MB to 653 MB the moment the preview boots.opennextjs-cloudflare deploymakes that same copy in the deploy job from the.open-next/cachethe artifact already carries, so booting the preview before the uploadwould add 268 MB to every
mainartifact, both ways across the wire, to ship a copy thedeploy remakes anyway. Ordered last, the uploaded bundle is byte-for-byte what it was
before this step existed, and the gating is unchanged:
deploy-docsneeds the wholebuildjob, so a red here keeps a bad render off production exactly as a red anywhereabove it does.
The gate is demonstrated red, three ways
1. The 09-04 defect, reproduced offline. Ablated
apps/docs/open-next.config.tsbyremoving the
incrementalCacheoverride — the configuration that caused the outage:dummy cache plus
dynamicParams = falsemeans every page is answered by the prerendered_not-foundroute. Mutation confirmed on disk (git hash-objectmoved from315674e7tod70daefc, injected marker present once, removed line absent) and confirmedto have reached the artifact the preview serves (the compiled
.open-next/.build/open-next.config.mjslost itsstatic-assets-incremental-cachereference), then re-packaged before judging anything. The step exited 1 with
all four targets 404,
[status] [final-path] [error-shell] [too-little-text] [h1-mismatch] [few-links]— the same 21 the card recorded formainatd6f5dda.Restored by
git checkout HEADon the absolute path, verified byte-for-byte (hash back to315674e7,git diff HEADempty), re-packaged, and the ablation marker confirmed absentfrom the compiled config again.
2. A preview that never becomes ready is NOT MEASURED, not a pass. With the boot
budget cut to 5 s against a healthy preview, the step exits 1: "the 5s boot budget ran
out with the preview still starting". This is the failure shape the card warns about —
"no findings" from a dead server is indistinguishable from a rendered site, and this lane
logged four probes of that shape in one day.
3. A preview that cannot start at all. With the compiled OpenNext config removed the
preview dies in about two seconds; the step exits 1 with "the preview process exited
before it was ready" and prints the preview log (
ERROR Could not find compiled Open Next config) into the step summary.Green in CI, step 15 of run 34250966221, its own record:
The negative control is not decoration and is carried along unchanged: every run also
fetches a
/docs/slug no page claims and fails onnegative-control-passedif it comesback clean.
What the first CI run caught, and the fix
Run 34250422860 was green and still left evidence of a defect in this step: the runner's
own post-job sweeper reported
Terminate orphan processforesbuildand twoworkerdprocesses. The preview is a chain of six processes — pnpm, node, sh, pnpm, wrangler,
workerd — and a SIGTERM to the pnpm wrapper at the top does not reach the rest, so it is
started with
setsidand the cleanup signals the whole process group.setsidonly moves the child into its own group once the forked child has exec'd it, soreading that group with
psimmediately after&is a race that can still return thestep's own group — and the guard that stops the step from signalling itself then falls
back to the pnpm wrapper alone. The same code cleaned up completely in this repo's
container on every run, which is how a race presents. The group is now resolved inside the
cleanup function, at kill time. Run 34250966221 prints
preview pid 3100 in process group 3100, step in 2034and itsCleaning up orphan processesline is followed by nothing atall. A preview that genuinely shares the step's group now emits a warning instead of
leaving a silent orphan.
Scope
.github/workflows/ci.ymlonly..github/scripts/smoke-docs.mjsis untouched — it neededno change to accept a local base,
--basewas already there.deploy-docs.ymlandrollback-docs.ymlare untouched. No changeset: nothing is published from a package here.Verification on
815e881, the head of this branch: CI run 34250966221 green with step 15The docs site renders — smoke-check a local previewrecordedsuccess; the step bodyextracted from the committed YAML and executed verbatim locally, green plus both NOT
MEASURED shapes red;
node .github/scripts/smoke-docs.mjs --self-testexit 0 (11 responsecases, 2 run cases, all 12 rules demonstrated able to fail); the workflow parses to 14
buildsteps and the file carries no control bytes.🤖 Generated with Claude Code
https://claude.ai/code/session_01ChPQM8jamxLUfUAxwFpJ8S