Bound the Turnstile retry loop and prove the production secret

[adewale]

Summary

I compared the runner's Turnstile protection with the canonical Siteverify handler in Cloudflare's Turnstile Spin skill. The comparison turned up a runaway loop and a blind spot, and each makes the other worse.

The loop. runner.js re-challenged every time a response carried the challenge marker, including right after it had just sent a token. So if Siteverify kept rejecting tokens (wrong secret, hostname, or action), one Run click became an endless cycle of widget solves, Worker POSTs, and Siteverify calls. With the real widget, main made 12 POSTs and 11 solves in 20 seconds from a single click, and was still going. Each Run now gets at most one challenge. If the retried token is rejected, the run ends with "Turnstile verification failed. Press Run to try again."

The blind spot. Deployment smoke skips Turnstile with the bypass header, and the wide event recorded both "challenge issued" and "token rejected" as fail. So a wrong, rotated, or test secret would pass smoke while every new session's first run failed.

What changed:

• Runner: one challenge per Run. The browser check drives a stub server that always rejects. It fails on the old runner (20 POSTs, 19 solves) and passes on the new one (2 POSTs, 1 solve).
• Siteverify hardening:
  • a 10-second AbortSignal.timeout
  • tokens over the documented 2048 characters are rejected without calling Siteverify
  • a null or non-string hostname fails closed instead of raising AttributeError (a 500)
  • the call is one helper, shared by learner verification and the new probe
• Observability: turnstile.outcome now separates challenged from fail. Failures carry a closed-vocabulary reason plus allowlisted Siteverify error_codes. Live Siteverify returns invalid-input-secret and missing-input-secret with HTTP 400, so error codes are read from non-2xx bodies; a non-2xx response still never passes. scripts/learner_report.py breaks failures down by reason and code, and flags configuration alerts (bad or missing secret, missing site key).
• Deploy-time secret probe: POST /__smoke/turnstile is gated by the existing smoke header and returns 404 without it. It sends the Worker's secret to Siteverify with Cloudflare's documented dummy token (production secrets reject it) and returns valid / invalid / testing_key / unverified / unexpected / absent. The secret is never included. With PBE_SMOKE_BYPASS_SECRET, scripts/smoke_deployment.py fails unless the secret works and a site key is configured, or Turnstile is off.
• Test-secret detection: Cloudflare's always-fail test secret answers the dummy token with the same invalid-input-response as a working production secret. Only metadata.result_with_testing_key tells them apart, and the probe checks it.
• Docs: updates to the runner-protection spec, observability spec, learner-analytics guide, and changelog, plus four lessons-learned entries. The fourth notes that make deploy runs the project-pinned Wrangler with account credentials, now that Dependabot proposes weekly npm updates.

Two things the probe cannot prove: that the secret and site key belong to the same widget, and that the widget allows the production hostname. A browser run still covers those. A production deploy with no secret at all reports Turnstile as off and passes smoke.

Verification

• make verify: the CI verify check passed on head 544395a with Google Chrome, including browser-layout-test. Locally, every step also passed except two focus-indicator checks in browser-layout-test ("Search focus indicator is too weak", "CodeMirror focus indicator is too weak"). They fail the same way on main (3f300af) in the sandbox's Playwright Chromium, as Upgrade Wrangler to 4.136.3 #14 also saw.
• scripts/format_examples.py --check
• make verify-python-version VERSION=3.13 (109 examples)
• git diff --check
• Added or updated regression tests and verified they fail when the fix is reverted:
  • the rejection fixture fails on the old runner.js
  • the new Siteverify tests fail against the original main.py, including the AttributeError on a null hostname
• Manual verification, on local Workers using Cloudflare's public test keys and live Siteverify:
  • Rejection loop: the before/after captures below. The before run shows the loop continuing.
  • Bogus secret: logged as rejected + invalid-input-secret, and learner_report.py raised the configuration alert.
  • Always-pass test secret, runner path: failed closed on hostname_mismatch, with no cookie and no code run.
  • Probe: reported invalid for a bogus secret, testing_key for both test secrets, and absent with no secret. A missing or wrong header got a 404, and the response is JSON with no-store.
  • Full smoke script: passes locally and prints SKIP for the probe when no bypass secret is set.

The probe's valid path is covered only by unit tests using Cloudflare's documented reply, because no production secret is available here. The first production smoke with PBE_SMOKE_BYPASS_SECRET will confirm it:

PBE_SMOKE_BYPASS_SECRET=... scripts/smoke_deployment.py https://www.pythonbyexample.dev

Visual evidence (required for UI changes)

Before — one Run click keeps re-challenging and stays on "Verification required…" (12 POSTs and 11 real solves after 20 s, still growing):

Open the before capture

After — one solve, two POSTs, then the server's message and a free Run button (still two POSTs after 20 s):

Open the after capture

Reproduction: uv run --group workers pywrangler dev --port <port> --var TURNSTILE_SECRET_KEY:2x0000000000000000000000000000000AA --var TURNSTILE_SITE_KEY:1x00000000000000000000BB, then click Run once on /examples/values and capture .runner-grid after six seconds (base: 3f300af; head: eae0dc3; viewport: 1200×900, light scheme; real invisible test widget with the always-fail test secret). Details and SHA-256 hashes are in docs/pr-evidence/README.md.

Example changes

Not applicable: no files under src/example_sources/ changed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PTpdP6DbuVU4VPLRzjNYGu