Measure rendered styles in the browser, smoke-test deploys, warn before waivers expire

[adewale]

Summary

This PR implements the pythonbyexample recommendations from the September 2026 verification audit (https://github.com/adewale/testing-best-practices/blob/claude/github-testing-practices-review-n5749j/research/PORTFOLIO_VERIFICATION_AUDIT_2026-09.md, pythonbyexample row). There are three commits.

1. P1: rendered-style contracts replace CSS/JS source-text tests.

tests/test_app.py read public/site.css 17 times and read the JS sources repeatedly. It asserted literal text such as "transform: scale(0.96)", "text-wrap: balance" or "execution: 'execute'". docs/lessons-learned.md records these tests breaking on the harmless has-figure change, and they still pass when the rendered page regresses.

scripts/check_browser_layout.mjs already drives a real Worker and Chrome in CI. It now measures the same intentions on the page:

• Pressed states: computed transforms under a forced :active (via CSS.forcePseudoState) for Run, Copy link and the copy button (0.96), and for home cards (0.99).
• Sizing and type: touch targets of at least 40px for runner buttons and nav links. Also underlined nav links, balanced h1, antialiased body text, tabular execution time, and no element with transition: all.
• Layout:
  • Body text follows the reader's browser font size (Page.setFontSizes at 20px).
  • The skip link is off-screen until focused, then on-screen.
  • The share button sits at the toolbar's end, and the copy button is anchored to its cell.
  • Long output wraps without page overflow, and tall output grows the panel instead of being clipped.
  • The fallback textarea does not overflow its panel.
• Contrast: in both light and dark themes, body text ≥ 4.5, the Run button ≥ 4.5, and the output terminal ≥ 7. The page palette switches with the theme, and marginalia figures stay on light paper in dark mode.
• Accessibility fallbacks: the header turns solid under emulated prefers-reduced-transparency and prefers-contrast: more, and nav links get the full text colour under more contrast. The nav is visible on landing.
• Behaviour previously asserted as JS text:
  • Network errors end a run with Run failed: ….
  • An unedited Copy link copies the plain page URL.
  • The copy button falls back to execCommand('copy') without the Clipboard API. Its status is an off-screen polite live region, and its mask glyph changes.
  • Arrow keys ignore modifier keys and focused buttons. ArrowLeft walks back, and at the first example it is a no-op with no error.
  • Search results expose combobox, listbox and option state.
  • The Turnstile widget renders with execution: 'execute' and no size, and is removed and hidden afterwards.
  • Every design token on the About page resolves.
• Arrow-guard fix: the existing checks read the pathname 50 ms later, which cannot see a navigation that has started but not committed. They now count attempted navigations with the Navigation API.
• Removed: the source-text assertions, including 19 tests made only of them. HTML-rendering assertions stay.

2. P1: make deploy smoke-tests the deployed origin. Deploy now ends with a post-deploy-smoke step. It runs scripts/smoke_deployment.py against DEPLOY_URL (default https://www.pythonbyexample.dev). If any GET or POST check fails, it exits non-zero with "the new version is already live: investigate now, or roll back". SMOKE_ARGS passes extra flags explicitly.

2 (continued). P2: waiver expiry warns 30 days ahead, and a weekly run fails early.

• check_quality_scores.py warns when a waiver has 30 days or fewer left. --fail-within-days N turns that window into an error, and --as-of fixes the date.
• A new weekly Scheduled quality workflow (also on workflow_dispatch) runs make quality-checks and then make check-waiver-expiry.
• The hello-world waiver expires 2026-12-01. Before this change, its first signal would have been a failure on every PR from that day. Now it turns the scheduled run red from the first Monday in its final 30 days.

3. P2/P3: stale docs and phrase-order tests.

• docs/lessons-learned.md said "54 tests today cover 9 contract families". tests/test_marginalia_geometry.py has 31 tests in 13 contract classes. The sentence now points at the classes instead of giving a number.
• The same file cited scripts/refresh_golden_fixture.py, which was deleted in 0929c25. It now names what catches loader and parser regressions today.
• tests/test_markdown_migration_prereqs.py loses its spec-wording tests, including the "Red" < "Green" < "Refactor" order check. The README/CI command-contract test stays.

Verification

• make verify. I ran each target against a local pywrangler dev on port 9796 (to avoid clashing with other local servers) with Chrome for Testing 154:
  • make build test seo-cache-lint verify-examples quality-checks search-ranking-test lint check-generated exited 0, with 227 tests OK.
  • node scripts/check_browser_layout.mjs http://127.0.0.1:9796/examples/values exited 0.
• scripts/format_examples.py --check: exit 0.
• make verify-python-version VERSION=3.13: exit 0, "Verified 109 example(s)".
• git diff --check is clean, npm audit --audit-level=high exits 0, and git diff --exit-code -- pylock.toml shows no change.
• Regression checks fail when the behaviour is reverted. Details are below.
• Manual verification: make post-deploy-smoke DEPLOY_URL=http://127.0.0.1:9796 against the local Worker printed "Deployment smoke OK (9 GETs, 5 POSTs)". The first attempt hit a local-dev ProxyWorker … Network connection lost on POST /examples/subprocesses. Retried by itself, that POST returned 200 three times, and the rerun passed.

Red, then green.

Browser contracts. I applied 11 mutations at once, ran make build, and ran the browser test:

• site.css:
  • body font set to 16px/1.6;
  • removed .share-button { margin-left: auto; }, .button:active, .skip-link:focus, the dark figure-paper rule, the reduced-transparency backdrop-filter: none, and .cell-source { position: relative; };
  • output switched to white-space: pre.
• runner.js: size: 'invisible' instead of execution: 'execute', and the metaKey guard dropped.
• syntax-highlight.js: the execCommand result forced to false.

It exited 1 and named all 11:

Turnstile widget is not rendered in Invisible execute mode (execution=undefined, size set=true)
Source copy did not fall back to execCommand when the Clipboard API is unavailable
Copy button is not anchored to its source cell
Arrow navigation fired with a modifier key or from a focused button
Dark mode does not keep marginalia figures on light paper (rgba(0, 0, 0, 0))
Share button does not sit apart at the end of the toolbar (gap 8px, 100.859375px from the end)
Skip link does not appear on screen when focused
Long output overflows the output panel instead of wrapping ({"textPastPanel":5450.6875,"pageOverflow":5470})
button[type="submit"] does not press down to scale(0.96) when active (none)
Body text ignores the reader's browser font size (16px at a 20px default)
prefers-reduced-transparency: header stays translucent (blur(16px), rgba(0, 0, 0, 0))

After restoring and rerunning make build, the test exited 0 and git status was clean.

The first drafts of two checks missed their mutations: the modifier-key check (pathname after 50 ms) and the wrapping check (scrollWidth). I rewrote them, using the Navigation API and a text range measured against the panel edge, until they failed on the mutation.

Waiver expiry.

• The new tests fail against origin/main's script, because expiry_warning and waiver_expiry_findings do not exist there.
• The old check_expiry_date('2026-12-01', today=2026-11-05) returns None, so it stays silent.
• On this branch, against the real registry:

Invocation	Exit code	Output
no flags (today)	0	no warning
--as-of 2026-11-05	0	WARNING: quality waiver hello-world: expires on 2026-12-01 (26 days left)…
--as-of 2026-11-05 --fail-within-days 30	1
--as-of 2026-12-01	1	expired
--fail-within-days 30 (today)	0

Deploy smoke.

• make -n deploy shows pywrangler deploy followed by post-deploy-smoke.
• make post-deploy-smoke DEPLOY_URL=http://127.0.0.1:9 exited 2 with Deployment smoke FAILED … The new version is already live.
• Against the local Worker it exited 0.
• No deploy was run.

Test counts. 249 on origin/main, 227 now: 19 source-text-only tests and 5 spec-wording tests removed, 2 expiry tests added. Each intermediate commit also passes on its own (230 and 232 tests).

Not run: GitHub Actions itself. What I did check for scheduled-quality.yml:

• It parses with yaml.safe_load.
• It pins actions exactly as verify.yml does (SHA-pinned setup-uv, @v7 for the rest).
• It uses only contents: read and no secrets.
• make quality-checks and make check-waiver-expiry passed locally with only the uv sync environment.

Not in this PR

• Source-text tests of src/main.py, the git hooks, Makefile and workflows (test_turnstile_verification_is_session_gated_in_worker, test_worker_entrypoint_uses_fastapi_asgi_bridge, test_dynamic_worker_execution_uses_hash_keyed_get_cache, test_generated_drift_is_blocked_before_commit_and_merge). The recommendation targeted the CSS/JS tests, and 466f616 already added behavioural coverage for most of the main.py paths.
• Brand-colour literals and existence-only selector checks. Assertions on #FF4801, #F5F1EB, #521000, #EBD5C1, .runner-grid, --space-6 and box-shadow: were dropped rather than converted, because they pinned text, not behaviour. Legibility is covered by the contrast checks instead.
• Running make deploy. It touches production.

Notes

• Bypass secret for the deploy smoke. With Turnstile challenges enabled in production, the smoke's POST checks need PBE_SMOKE_BYPASS_SECRET exported, as the script's docstring already says. Without it, make deploy will now fail loudly after deploying. SMOKE_ARGS=--skip-post is available only as a deliberate choice.
• Browser test runtime. It now takes about 40 s, up from about 21 s. It uses the DOM/CSS domains, Emulation.setFocusEmulationEnabled, Page.setFontSizes and media-feature emulation, all of which current google-chrome on ubuntu-latest supports.
• Local-only Chrome setup, not committed. Chrome ran as root here, so it needed --no-sandbox. It also needed to trust this session's egress-proxy CA, which I did with --ignore-certificate-errors-spki-list for the proxy CA keys only. Both lived in a local CHROME_PATH wrapper; the committed script is unchanged in that respect.

🤖 Generated with Claude Code

https://claude.ai/code/session_012eisRQmQvg4TphwztxhQGJ