diff --git a/.github/workflows/ai-crawler-check.yml b/.github/workflows/ai-crawler-check.yml new file mode 100644 index 0000000..9745fbe --- /dev/null +++ b/.github/workflows/ai-crawler-check.yml @@ -0,0 +1,38 @@ +name: AI crawler access + +# Roadmap 1.5 guard (issue #82). Checks the LIVE robots.txt, because the block +# this catches is injected by Cloudflare at the edge and is invisible to the +# build — public/robots.txt can be perfectly correct while the served file +# disallows every answer engine. +# +# Deliberately NOT part of ci.yml: it tests production, not the pull request, +# and the remedy is a Cloudflare dashboard toggle rather than a code change. +# Blocking merges on it would punish the wrong people. It runs daily so the +# regression stays visible until the zone setting is corrected. + +on: + schedule: + - cron: "17 6 * * *" + workflow_dispatch: + inputs: + origin: + description: "Origin to check" + required: false + default: "https://keeltrading.com" + +permissions: + contents: read + +jobs: + robots: + name: AI crawlers are not disallowed + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: 22 + + - name: Check served robots.txt + run: node scripts/check-ai-crawlers.mjs "${{ inputs.origin || 'https://keeltrading.com' }}" diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 31bcb99..f46a2ef 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -68,6 +68,34 @@ canonical-URL side needs no changes — the build has always emitted After the 60-day ICANN window (from 2026-08-19), optionally transfer the registrar to Cloudflare. +## AI crawler access (zone setting, not code — issue #82) + +`public/robots.txt` is four lines and allows everything. The **served** +`robots.txt` is not: Cloudflare prepends a `# BEGIN Cloudflare Managed content` +block at the edge that disallows nine AI crawlers (`ClaudeBot`, `GPTBot`, +`CCBot`, `Google-Extended`, `meta-externalagent`, `Applebot-Extended`, +`Amazonbot`, `Bytespider`, `CloudflareBrowserRenderingCrawler`) and sets +`Content-Signal: ai-train=no,use=reference`. + +The origin cannot override an edge-injected block — editing `public/robots.txt` +does nothing. It is turned off in the dashboard: the `keeltrading.com` zone → +the AI crawler control that mentions managed `robots.txt` or content signals. +Cloudflare has shipped this as **AI Crawl Control**, previously **AI Audit** / +"Block AI bots"; the label has moved across releases, so search the zone for the +setting rather than a fixed menu path. + +This contradicts roadmap 1.5, which requires GPTBot/ClaudeBot/PerplexityBot to +be crawlable — blocking them opts the site out of being cited by answer engines, +while normal Google Search indexing is unaffected (`search=yes` and +`User-agent: * Allow: /` still stand). `ai-train=no` is a defensible stance to +keep; the blanket `Disallow: /` per crawler is the part that costs GEO reach. + +Verify with `npm run check:ai-crawlers` (any origin: `node +scripts/check-ai-crawlers.mjs https://staging.example`). The **AI crawler +access** workflow runs it daily against production and fails loudly while the +block is in place. It is intentionally not in `ci.yml` — it tests production, +not the pull request. + ## Operating notes - **A build that cannot fetch a pinned engine doc fails on purpose** (FR-4). diff --git a/docs/seo/IMPLEMENTATION-ROADMAP.md b/docs/seo/IMPLEMENTATION-ROADMAP.md index e54d95f..e5247a1 100644 --- a/docs/seo/IMPLEMENTATION-ROADMAP.md +++ b/docs/seo/IMPLEMENTATION-ROADMAP.md @@ -11,11 +11,22 @@ item carries its dependency and its "how we know it failed" check. | 1.2 | GSC property + sitemap submission (both locale sitemaps via sitemap-index) | 1.1 | Coverage report shows 24/24 discovered within 14d | | 1.3 | ~~Title/meta rewrite on the 16 existing locale pages~~ **shipped 2026-08-19** (table below) | none — do now | Titles 30–60 chars, unique, keyword-leading | | 1.4 | ~~JSON-LD additions: `WebSite` (+`sameAs` on Organization: GitHub org, engine repo); extend About FAQ with the KEEL-ticker line~~ **shipped 2026-08-19** | none | Rich Results Test passes; entity panel check at 3mo | -| 1.5 | `llms.txt` at root summarizing the site for AI crawlers; ensure GPTBot/ClaudeBot/PerplexityBot not blocked in robots.txt | none | File fetchable; AI crawlers appear in CF analytics | +| 1.5 | ~~`llms.txt` at root summarizing the site for AI crawlers~~ **shipped 2026-08-19**; ensure GPTBot/ClaudeBot/PerplexityBot not blocked in robots.txt — **BLOCKED, see #82** | none | `npm run check:ai-crawlers` exits 0; AI crawlers appear in CF analytics | | 1.6 | ~~BreadcrumbList schema on docs pages; DefinedTermSet on glossary~~ **shipped 2026-08-19** | none | Rich Results Test per template | | 1.7 | Cloudflare Web Analytics token live (language dimension = FR/ES decision input, PRD D4) | 1.1 | Beacon firing; language report populating | | 1.8 | Engine README gains the keeltrading.com link (PRD §10 allows) — entity `sameAs` reinforcement | 1.1 | Reciprocal links crawlable | +> **1.5 blocker (found 2026-08-23, issue #82).** `public/robots.txt` is correct, +> but Cloudflare injects a `# BEGIN Cloudflare Managed content` block at the edge +> that disallows nine AI crawlers — `ClaudeBot`, `GPTBot`, `CCBot`, +> `Google-Extended`, `meta-externalagent`, `Applebot-Extended`, `Amazonbot`, +> `Bytespider`, `CloudflareBrowserRenderingCrawler` — and sets +> `Content-Signal: ai-train=no`. The origin cannot override it; the remedy is a +> zone-level dashboard setting (docs/DEPLOYMENT.md § AI crawler access). +> `llms.txt` therefore ships and serves 200 while the crawlers it exists for are +> told not to read the pages it points at. `scripts/check-ai-crawlers.mjs` now +> fails loudly on this, daily, so it cannot go quiet again. + ### Title/meta rewrite table (current → proposed, EN shown; AR mirrors) | Page | Current title | Proposed (≤60 chars) | diff --git a/package.json b/package.json index 2f2ed60..d5a95c3 100644 --- a/package.json +++ b/package.json @@ -10,6 +10,7 @@ "build": "npm run fetch && astro build", "preview": "astro preview", "check": "astro check", + "check:ai-crawlers": "node scripts/check-ai-crawlers.mjs", "og": "node scripts/generate-og.mjs" }, "dependencies": { diff --git a/scripts/check-ai-crawlers.mjs b/scripts/check-ai-crawlers.mjs new file mode 100755 index 0000000..52413dc --- /dev/null +++ b/scripts/check-ai-crawlers.mjs @@ -0,0 +1,134 @@ +#!/usr/bin/env node +/** + * Roadmap 1.5 guard — "ensure GPTBot/ClaudeBot/PerplexityBot not blocked in + * robots.txt" (docs/seo/IMPLEMENTATION-ROADMAP.md). + * + * This checks the LIVE site, not the build. public/robots.txt can be perfectly + * correct while Cloudflare prepends a "# BEGIN Cloudflare Managed content" + * block at the edge that disallows every AI crawler — which is exactly what + * issue #82 found. Nothing in this repo can catch that, so nothing did. + * + * The origin cannot override the injected block; the fix is a zone-level + * dashboard setting (see docs/DEPLOYMENT.md). This script exists so the + * regression is loud instead of silent. + * + * Usage: node scripts/check-ai-crawlers.mjs [origin] + * Exit 0 = no AI crawler is disallowed. Exit 1 = blocked, or unreachable. + */ +const ORIGIN = process.argv[2] || process.env.SITE_ORIGIN || "https://keeltrading.com"; +const URL_ROBOTS = `${ORIGIN.replace(/\/$/, "")}/robots.txt`; + +// Answer engines and their training/grounding fetchers. Blocking these is a +// GEO decision, so it must be a deliberate one — never a default left on. +const WATCHED = [ + "ClaudeBot", + "GPTBot", + "OAI-SearchBot", + "PerplexityBot", + "Google-Extended", + "CCBot", + "meta-externalagent", + "Applebot-Extended", + "Amazonbot", + "Bytespider", + "CloudflareBrowserRenderingCrawler", +]; + +/** + * Parse robots.txt into groups. Consecutive User-agent lines share one group; + * the first directive line closes the agent list, and the next User-agent + * after that starts a fresh group (RFC 9309 §2.2.1). + */ +function parseGroups(text) { + const groups = []; + let current = null; + let collectingAgents = false; + + for (const raw of text.split(/\r?\n/)) { + const line = raw.replace(/#.*$/, "").trim(); + if (!line) continue; + + const sep = line.indexOf(":"); + if (sep === -1) continue; + const field = line.slice(0, sep).trim().toLowerCase(); + const value = line.slice(sep + 1).trim(); + + if (field === "user-agent") { + if (!current || !collectingAgents) { + current = { agents: [], rules: [], signals: [] }; + groups.push(current); + collectingAgents = true; + } + current.agents.push(value); + continue; + } + + if (!current) continue; + collectingAgents = false; + if (field === "allow" || field === "disallow") current.rules.push({ field, value }); + if (field === "content-signal") current.signals.push(value); + } + return groups; +} + +/** The group that applies to `agent`, preferring an exact match over `*`. */ +function groupFor(groups, agent) { + const wanted = agent.toLowerCase(); + return ( + groups.find((g) => g.agents.some((a) => a.toLowerCase() === wanted)) || + groups.find((g) => g.agents.includes("*")) || + null + ); +} + +/** True when the group blocks the whole site. */ +function blocksRoot(group) { + if (!group) return false; + const blanket = group.rules.some((r) => r.field === "disallow" && r.value === "/"); + if (!blanket) return false; + // An explicit Allow: / alongside Disallow: / wins on the longest-match rule. + return !group.rules.some((r) => r.field === "allow" && r.value === "/"); +} + +let text; +try { + const res = await fetch(URL_ROBOTS, { + headers: { "user-agent": "keeltrading.com-ai-crawler-check" }, + signal: AbortSignal.timeout(20_000), + }); + if (!res.ok) { + console.error(`::error::${URL_ROBOTS} returned HTTP ${res.status}`); + process.exit(1); + } + text = await res.text(); +} catch (err) { + console.error(`::error::could not fetch ${URL_ROBOTS} — ${err.message}`); + process.exit(1); +} + +const groups = parseGroups(text); +const blocked = WATCHED.filter((agent) => { + const g = groups.find((x) => x.agents.some((a) => a.toLowerCase() === agent.toLowerCase())); + return blocksRoot(g); +}); + +const wildcardBlocked = blocksRoot(groupFor(groups, "*")); +const signals = groups.flatMap((g) => g.signals); + +console.log(`robots.txt: ${URL_ROBOTS} (${text.split(/\r?\n/).length} lines, ${groups.length} groups)`); +if (signals.length) console.log(`Content-Signal: ${signals.join(" | ")}`); +if (text.includes("Cloudflare Managed content")) { + console.log("note: a Cloudflare-managed block is being injected at the edge (not from this repo)"); +} + +if (wildcardBlocked) { + console.error("::error::User-agent: * is disallowed from / — the whole site is closed to crawlers"); +} +if (blocked.length) { + console.error(`::error::${blocked.length} AI crawler(s) disallowed from /: ${blocked.join(", ")}`); + console.error("Roadmap 1.5 requires these to be crawlable. This is NOT fixable from this repo —"); + console.error("it is a zone-level Cloudflare setting. See issue #82 and docs/DEPLOYMENT.md."); +} +if (blocked.length || wildcardBlocked) process.exit(1); + +console.log(`OK — none of the ${WATCHED.length} watched AI crawlers are disallowed from /`);