fix(seo): make the edge-injected AI-crawler block loud instead of silent (#82)

.github/workflows/ai-crawler-check.yml

@@ -0,0 +1,38 @@
+name: AI crawler access
+
+# Roadmap 1.5 guard (issue #82). Checks the LIVE robots.txt, because the block
+# this catches is injected by Cloudflare at the edge and is invisible to the
+# build — public/robots.txt can be perfectly correct while the served file
+# disallows every answer engine.
+#
+# Deliberately NOT part of ci.yml: it tests production, not the pull request,
+# and the remedy is a Cloudflare dashboard toggle rather than a code change.
+# Blocking merges on it would punish the wrong people. It runs daily so the
+# regression stays visible until the zone setting is corrected.
+
+on:
+  schedule:
+    - cron: "17 6 * * *"
+  workflow_dispatch:
+    inputs:
+      origin:
+        description: "Origin to check"
+        required: false
+        default: "https://keeltrading.com"
+
+permissions:
+  contents: read
+
+jobs:
+  robots:
+    name: AI crawlers are not disallowed
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+
+      - name: Check served robots.txt
+        run: node scripts/check-ai-crawlers.mjs "${{ inputs.origin || 'https://keeltrading.com' }}"

docs/DEPLOYMENT.md

@@ -68,6 +68,34 @@ canonical-URL side needs no changes — the build has always emitted
 After the 60-day ICANN window (from 2026-08-19), optionally transfer the
 registrar to Cloudflare.
 
+## AI crawler access (zone setting, not code — issue #82)
+
+`public/robots.txt` is four lines and allows everything. The **served**
+`robots.txt` is not: Cloudflare prepends a `# BEGIN Cloudflare Managed content`
+block at the edge that disallows nine AI crawlers (`ClaudeBot`, `GPTBot`,
+`CCBot`, `Google-Extended`, `meta-externalagent`, `Applebot-Extended`,
+`Amazonbot`, `Bytespider`, `CloudflareBrowserRenderingCrawler`) and sets
+`Content-Signal: ai-train=no,use=reference`.
+
+The origin cannot override an edge-injected block — editing `public/robots.txt`
+does nothing. It is turned off in the dashboard: the `keeltrading.com` zone →
+the AI crawler control that mentions managed `robots.txt` or content signals.
+Cloudflare has shipped this as **AI Crawl Control**, previously **AI Audit** /
+"Block AI bots"; the label has moved across releases, so search the zone for the
+setting rather than a fixed menu path.
+
+This contradicts roadmap 1.5, which requires GPTBot/ClaudeBot/PerplexityBot to
+be crawlable — blocking them opts the site out of being cited by answer engines,
+while normal Google Search indexing is unaffected (`search=yes` and
+`User-agent: * Allow: /` still stand). `ai-train=no` is a defensible stance to
+keep; the blanket `Disallow: /` per crawler is the part that costs GEO reach.
+
+Verify with `npm run check:ai-crawlers` (any origin: `node
+scripts/check-ai-crawlers.mjs https://staging.example`). The **AI crawler
+access** workflow runs it daily against production and fails loudly while the
+block is in place. It is intentionally not in `ci.yml` — it tests production,
+not the pull request.
+
 ## Operating notes
 
 - **A build that cannot fetch a pinned engine doc fails on purpose** (FR-4).

docs/seo/IMPLEMENTATION-ROADMAP.md

@@ -11,11 +11,22 @@ item carries its dependency and its "how we know it failed" check.
 | 1.2 | GSC property + sitemap submission (both locale sitemaps via sitemap-index) | 1.1 | Coverage report shows 24/24 discovered within 14d |
 | 1.3 | ~~Title/meta rewrite on the 16 existing locale pages~~ **shipped 2026-08-19** (table below) | none — do now | Titles 30–60 chars, unique, keyword-leading |
 | 1.4 | ~~JSON-LD additions: `WebSite` (+`sameAs` on Organization: GitHub org, engine repo); extend About FAQ with the KEEL-ticker line~~ **shipped 2026-08-19** | none | Rich Results Test passes; entity panel check at 3mo |
-| 1.5 | `llms.txt` at root summarizing the site for AI crawlers; ensure GPTBot/ClaudeBot/PerplexityBot not blocked in robots.txt | none | File fetchable; AI crawlers appear in CF analytics |
+| 1.5 | ~~`llms.txt` at root summarizing the site for AI crawlers~~ **shipped 2026-08-19**; ensure GPTBot/ClaudeBot/PerplexityBot not blocked in robots.txt — **BLOCKED, see #82** | none | `npm run check:ai-crawlers` exits 0; AI crawlers appear in CF analytics |
 | 1.6 | ~~BreadcrumbList schema on docs pages; DefinedTermSet on glossary~~ **shipped 2026-08-19** | none | Rich Results Test per template |
 | 1.7 | Cloudflare Web Analytics token live (language dimension = FR/ES decision input, PRD D4) | 1.1 | Beacon firing; language report populating |
 | 1.8 | Engine README gains the keeltrading.com link (PRD §10 allows) — entity `sameAs` reinforcement | 1.1 | Reciprocal links crawlable |
 
+> **1.5 blocker (found 2026-08-23, issue #82).** `public/robots.txt` is correct,
+> but Cloudflare injects a `# BEGIN Cloudflare Managed content` block at the edge
+> that disallows nine AI crawlers — `ClaudeBot`, `GPTBot`, `CCBot`,
+> `Google-Extended`, `meta-externalagent`, `Applebot-Extended`, `Amazonbot`,
+> `Bytespider`, `CloudflareBrowserRenderingCrawler` — and sets
+> `Content-Signal: ai-train=no`. The origin cannot override it; the remedy is a
+> zone-level dashboard setting (docs/DEPLOYMENT.md § AI crawler access).
+> `llms.txt` therefore ships and serves 200 while the crawlers it exists for are
+> told not to read the pages it points at. `scripts/check-ai-crawlers.mjs` now
+> fails loudly on this, daily, so it cannot go quiet again.
+
 ### Title/meta rewrite table (current → proposed, EN shown; AR mirrors)
 
 | Page | Current title | Proposed (≤60 chars) |

package.json

@@ -10,6 +10,7 @@
     "build": "npm run fetch && astro build",
     "preview": "astro preview",
     "check": "astro check",
+    "check:ai-crawlers": "node scripts/check-ai-crawlers.mjs",
     "og": "node scripts/generate-og.mjs"
   },
   "dependencies": {

scripts/check-ai-crawlers.mjs

@@ -0,0 +1,134 @@
+#!/usr/bin/env node
+/**
+ * Roadmap 1.5 guard — "ensure GPTBot/ClaudeBot/PerplexityBot not blocked in
+ * robots.txt" (docs/seo/IMPLEMENTATION-ROADMAP.md).
+ *
+ * This checks the LIVE site, not the build. public/robots.txt can be perfectly
+ * correct while Cloudflare prepends a "# BEGIN Cloudflare Managed content"
+ * block at the edge that disallows every AI crawler — which is exactly what
+ * issue #82 found. Nothing in this repo can catch that, so nothing did.
+ *
+ * The origin cannot override the injected block; the fix is a zone-level
+ * dashboard setting (see docs/DEPLOYMENT.md). This script exists so the
+ * regression is loud instead of silent.
+ *
+ * Usage: node scripts/check-ai-crawlers.mjs [origin]
+ * Exit 0 = no AI crawler is disallowed. Exit 1 = blocked, or unreachable.
+ */
+const ORIGIN = process.argv[2] || process.env.SITE_ORIGIN || "https://keeltrading.com";
+const URL_ROBOTS = `${ORIGIN.replace(/\/$/, "")}/robots.txt`;
+
+// Answer engines and their training/grounding fetchers. Blocking these is a
+// GEO decision, so it must be a deliberate one — never a default left on.
+const WATCHED = [
+  "ClaudeBot",
+  "GPTBot",
+  "OAI-SearchBot",
+  "PerplexityBot",
+  "Google-Extended",
+  "CCBot",
+  "meta-externalagent",
+  "Applebot-Extended",
+  "Amazonbot",
+  "Bytespider",
+  "CloudflareBrowserRenderingCrawler",
+];
+
+/**
+ * Parse robots.txt into groups. Consecutive User-agent lines share one group;
+ * the first directive line closes the agent list, and the next User-agent
+ * after that starts a fresh group (RFC 9309 §2.2.1).
+ */
+function parseGroups(text) {
+  const groups = [];
+  let current = null;
+  let collectingAgents = false;
+
+  for (const raw of text.split(/\r?\n/)) {
+    const line = raw.replace(/#.*$/, "").trim();
+    if (!line) continue;
+
+    const sep = line.indexOf(":");
+    if (sep === -1) continue;
+    const field = line.slice(0, sep).trim().toLowerCase();
+    const value = line.slice(sep + 1).trim();
+
+    if (field === "user-agent") {
+      if (!current || !collectingAgents) {
+        current = { agents: [], rules: [], signals: [] };
+        groups.push(current);
+        collectingAgents = true;
+      }
+      current.agents.push(value);
+      continue;
+    }
+
+    if (!current) continue;
+    collectingAgents = false;
+    if (field === "allow" || field === "disallow") current.rules.push({ field, value });
+    if (field === "content-signal") current.signals.push(value);
+  }
+  return groups;
+}
+
+/** The group that applies to `agent`, preferring an exact match over `*`. */
+function groupFor(groups, agent) {
+  const wanted = agent.toLowerCase();
+  return (
+    groups.find((g) => g.agents.some((a) => a.toLowerCase() === wanted)) ||
+    groups.find((g) => g.agents.includes("*")) ||
+    null
+  );
+}
+
+/** True when the group blocks the whole site. */
+function blocksRoot(group) {
+  if (!group) return false;
+  const blanket = group.rules.some((r) => r.field === "disallow" && r.value === "/");
+  if (!blanket) return false;
+  // An explicit Allow: / alongside Disallow: / wins on the longest-match rule.
+  return !group.rules.some((r) => r.field === "allow" && r.value === "/");
+}
+
+let text;
+try {
+  const res = await fetch(URL_ROBOTS, {
+    headers: { "user-agent": "keeltrading.com-ai-crawler-check" },
+    signal: AbortSignal.timeout(20_000),
+  });
+  if (!res.ok) {
+    console.error(`::error::${URL_ROBOTS} returned HTTP ${res.status}`);
+    process.exit(1);
+  }
+  text = await res.text();
+} catch (err) {
+  console.error(`::error::could not fetch ${URL_ROBOTS} — ${err.message}`);
+  process.exit(1);
+}
+
+const groups = parseGroups(text);
+const blocked = WATCHED.filter((agent) => {
+  const g = groups.find((x) => x.agents.some((a) => a.toLowerCase() === agent.toLowerCase()));
+  return blocksRoot(g);
+});
+
+const wildcardBlocked = blocksRoot(groupFor(groups, "*"));
+const signals = groups.flatMap((g) => g.signals);
+
+console.log(`robots.txt: ${URL_ROBOTS} (${text.split(/\r?\n/).length} lines, ${groups.length} groups)`);
+if (signals.length) console.log(`Content-Signal: ${signals.join(" | ")}`);
+if (text.includes("Cloudflare Managed content")) {
+  console.log("note: a Cloudflare-managed block is being injected at the edge (not from this repo)");
+}
+
+if (wildcardBlocked) {
+  console.error("::error::User-agent: * is disallowed from / — the whole site is closed to crawlers");
+}
+if (blocked.length) {
+  console.error(`::error::${blocked.length} AI crawler(s) disallowed from /: ${blocked.join(", ")}`);
+  console.error("Roadmap 1.5 requires these to be crawlable. This is NOT fixable from this repo —");
+  console.error("it is a zone-level Cloudflare setting. See issue #82 and docs/DEPLOYMENT.md.");
+}
+if (blocked.length || wildcardBlocked) process.exit(1);
+
+console.log(`OK — none of the ${WATCHED.length} watched AI crawlers are disallowed from /`);