fix(seo): make the edge-injected AI-crawler block loud instead of silent (#82)

[eaitbrahim]

Refs #82.

What this does — and what it deliberately does not

The block itself is not fixable from this repo. public/robots.txt is already correct; Cloudflare prepends the Disallow rules at the edge, and an origin cannot override an edge-injected block. Flipping it is a zone-level dashboard action (documented below, still to be done by hand).

What this PR fixes is the silence. Nine AI crawlers have been disallowed on production while every check in this repo stayed green, because nothing here was wrong.

Changes

File	Why
scripts/check-ai-crawlers.mjs	Parses the live robots.txt (RFC 9309 group semantics) and exits non-zero when a watched answer-engine crawler is disallowed from /. Also flags a blanket User-agent: * block and echoes any Content-Signal.
.github/workflows/ai-crawler-check.yml	Runs it daily + on dispatch.
package.json	npm run check:ai-crawlers
docs/DEPLOYMENT.md	New AI crawler access section with the dashboard procedure.
docs/seo/IMPLEMENTATION-ROADMAP.md	1.5 marks llms.txt shipped, records the blocker, and swaps its falsifiability check to the script.

Why it is not in ci.yml

It tests production, not the pull request, and the remedy is a dashboard toggle rather than a code change. Wiring it into CI would block merges on something no contributor can fix in a branch. It runs daily instead, so the regression stays visible until the zone setting is corrected.

This workflow will be red until the Cloudflare setting is flipped. That is the intended behaviour, not a broken build.

Verification

Against production — correctly fails:

robots.txt: https://keeltrading.com/robots.txt (66 lines, 11 groups)
Content-Signal: search=yes,ai-train=no,use=reference
note: a Cloudflare-managed block is being injected at the edge (not from this repo)
::error::9 AI crawler(s) disallowed from /: ClaudeBot, GPTBot, Google-Extended, CCBot,
meta-externalagent, Applebot-Extended, Amazonbot, Bytespider, CloudflareBrowserRenderingCrawler
exit: 1

Against a permissive robots.txt — correctly passes:

robots.txt: https://jesse.trade/robots.txt (4 lines, 1 groups)
OK — none of the 11 watched AI crawlers are disallowed from /
exit: 0

Unreachable origin exits 1 with a distinct message. PerplexityBot and OAI-SearchBot are watched but absent from the served file, so they correctly fall through to User-agent: * Allow: / and do not trip the check.

Still to do by hand

Cloudflare dashboard → keeltrading.com zone → the AI crawler control that mentions managed robots.txt / content signals (shipped as AI Crawl Control, previously AI Audit / "Block AI bots"). Turning it off closes #82 and turns this workflow green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01STvEejFVZBywHPFsppjp2U

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE