feat(seo): IndexNow bulk submission (pnpm indexnow) + per-host key files#782
Merged
Conversation
Founder authorised a one-time bulk IndexNow push. This adds the key files and the submitter. Scope, stated plainly: IndexNow reaches Bing, Yandex, Seznam, Naver and Yep. NOT Google -- Google tested it in 2021 and never shipped support, so nothing here moves Google indexation. The reason to run it is Bing, where we are close to absent. Two key files, not one: indexnow.org's FAQ is explicit that each subdomain is a separate host needing its own key file at its own root -- a key on the apex does not cover subdomains. So www.webhook.co and docs.webhook.co each get a copy. The key is PUBLIC BY DESIGN (it is served at a URL) and carries no privilege beyond "may submit URLs for this host", which is why it is committed rather than kept in a secret store. Guards, each tested: - submitBatch verifies the key file is LIVE and serves the key before it will POST -- an `ok` response whose body is not the key (a soft-404 HTML page) counts as failure, so a host that 200s everything cannot look verified. - every URL must be on the submitted host (IndexNow answers a mismatch with a bare 422; we fail locally, naming the offender). - host must be webhook.co or a subdomain, refused before any network call, with lookalikes (evilwebhook.co, webhook.co.evil.com) covered. - batching at the protocol's 10,000-URL ceiling. apps/docs is an EXPERIMENT: Mintlify documents .txt as Enterprise-only, so the docs key file may 404. verifyKeyLive turns that into a clean refusal rather than a wasted submission, and the answer is then measured rather than assumed. 16 new tests; full lint gate green (630 script tests). Verified the key file lands in the www static export at the root with exact content.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
…two test gaps gitleaks: generic-api-key fires on the name "KEY" plus entropy alone. The value is PUBLISHED by design at https://<host>/<key>.txt -- the protocol requires it to be world-readable so engines can verify domain ownership. Suppressed inline (the repo's established pattern), with a .gitleaksignore fingerprint for this branch's frozen history. Two gaps ai-review flagged, both real: 1. parseSitemapLocs was untested -- it decides which URLs get submitted. Added cases for plain locs, whitespace, CDATA unwrapping, empty locs and a document with none. 2. Nothing kept the deployed key FILES in sync with INDEXNOW_KEY. Rotating the constant without renaming the files would make verifyKeyLive refuse every submission against a 404, and nothing in CI would notice -- the files are inert static assets. Added a guard asserting each host's file exists, is named for the key, and contains it; plus one asserting every submittable host has a key file declared. Verified the guard fails RED with a key file removed, then restored. 21 tests pass.
…KeyLive Second ai-review round, all three findings legitimate gaps in a security-adjacent validator: - submitBatch's URL-mismatch guard was untested (only the sibling host guard was). Now asserts it refuses before any network call. - Malformed / empty entries took the catch path unasserted. - Relative and non-http(s) URLs were untested. `javascript:` is the interesting one: it PARSES, with an empty hostname, so it fails the host comparison rather than the try/catch -- worth pinning explicitly. Also took the non-blocking finding: verifyKeyLive now calls assertHostAllowed. It is exported and issues an outbound GET built from `host`, so leaving it unguarded made it a request primitive that could be pointed at an arbitrary origin. submitBatch already guarded; this closes the direct-call path. 25 tests pass.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL flagged /evil\.com/ in the off-host test as js/regex/missing-regexp-anchor, HIGH: an unanchored regex tested against URL-ish text can match anywhere, so arbitrary hosts may precede or follow it. Substantively a false positive -- it reads an error message, it does not sanitise a URL -- but a test should not model the shape of a broken host check, and the anchored form is a stronger assertion anyway. Now matches the message SHAPE with an anchored regex and the offending URL with a literal String#includes. Verified the matcher discriminates: false against an unrelated error, true against the real one. 25 tests pass.
CodeQL moved from js/regex/missing-regexp-anchor to js/incomplete-url-substring-sanitization on the replacement: it treats ANY substring test against a URL literal as a host check, because a substring can match anywhere in a URL with arbitrary hosts before or after it. Both forms were the wrong shape. Now the offender is asserted by EXACT EQUALITY on the message's trailing URL, with the host kept as a bare hostname so no URL literal is ever compared. Verified the assertion discriminates: true for the real URL, false for a different one. 25 tests pass; no URL-literal substring or regex comparisons remain.
choraria
added a commit
that referenced
this pull request
Jul 23, 2026
…the key The docs key file shipped in #782 as a deliberate EXPERIMENT: Mintlify documents .txt as Enterprise-only, so whether docs could participate was unknown rather than assumed. Now measured. Deployed, then fetched: https://www.webhook.co/<key>.txt -> 200, serves the key https://docs.webhook.co/<key>.txt -> 404 "Asset not found" So docs cannot host an IndexNow key file, and the committed apps/docs/<key>.txt was dead weight. Removed. docs.webhook.co is now absent from SITEMAP_FOR_HOST and the test's KEY_FILE_DIRS, with a comment recording WHY and saying not to re-add it, so the next person does not repeat the experiment. The stale usage line in the header is gone too -- a comment that outruns the code is worse than none. Bing coverage for the docs host comes from Bing Webmaster Tools' Search Console import instead, which needs no key file. www is unaffected and already submitted: 36 URLs accepted, HTTP 202. 25 tests pass; verified the CLI refuses docs cleanly and www --dry-run still resolves 36 URLs.
choraria
added a commit
that referenced
this pull request
Jul 23, 2026
TWO fixes. The first is urgent: main is currently RED on gitleaks at 74e921e (#782), and I caused it. 1. gitleaks. In #782 I put `// gitleaks:allow` on the lines ABOVE the INDEXNOW_KEY constant. That does nothing -- the directive must sit on the SAME LINE as the finding. #782 went green only because the .gitleaksignore fingerprint covered its BRANCH commit, and a squash-merge mints a NEW commit SHA, so the suppression evaporated the moment it landed and main went red. The file's own header warns that fingerprints are commit-bound; the squash case is the same trap. Fixed by moving the directive onto the line, and adding a fingerprint for the squash commit that is now frozen in main's history. Both are needed: the directive for any future commit that touches the line, the fingerprint for the one already written. Verified with the gitleaks CLI locally, reproducing RED on the old tree first and confirming clean after -- not by watching the check flip colour. 2. docs.webhook.co dropped from IndexNow. The key file shipped in #782 as a deliberate experiment, since Mintlify documents .txt as Enterprise-only. Measured after deploy: https://www.webhook.co/<key>.txt -> 200, serves the key https://docs.webhook.co/<key>.txt -> 404 "Asset not found" So docs cannot host a key file and apps/docs/<key>.txt was dead weight. Removed, and docs dropped from SITEMAP_FOR_HOST and the test's KEY_FILE_DIRS with a comment recording why, so nobody repeats it. Bing coverage for docs comes from Bing Webmaster Tools' GSC import instead. www is unaffected and already submitted: 36 URLs, HTTP 202.
choraria
added a commit
that referenced
this pull request
Jul 23, 2026
TWO fixes. The first is urgent: main is currently RED on gitleaks at 74e921e (#782), and I caused it. 1. gitleaks. In #782 I put `// gitleaks:allow` on the lines ABOVE the INDEXNOW_KEY constant. That does nothing -- the directive must sit on the SAME LINE as the finding. #782 went green only because the .gitleaksignore fingerprint covered its BRANCH commit, and a squash-merge mints a NEW commit SHA, so the suppression evaporated the moment it landed and main went red. The file's own header warns that fingerprints are commit-bound; the squash case is the same trap. Fixed by moving the directive onto the line, and adding a fingerprint for the squash commit that is now frozen in main's history. Both are needed: the directive for any future commit that touches the line, the fingerprint for the one already written. Verified with the gitleaks CLI locally, reproducing RED on the old tree first and confirming clean after -- not by watching the check flip colour. 2. docs.webhook.co dropped from IndexNow. The key file shipped in #782 as a deliberate experiment, since Mintlify documents .txt as Enterprise-only. Measured after deploy: https://www.webhook.co/<key>.txt -> 200, serves the key https://docs.webhook.co/<key>.txt -> 404 "Asset not found" So docs cannot host a key file and apps/docs/<key>.txt was dead weight. Removed, and docs dropped from SITEMAP_FOR_HOST and the test's KEY_FILE_DIRS with a comment recording why, so nobody repeats it. Bing coverage for docs comes from Bing Webmaster Tools' GSC import instead. www is unaffected and already submitted: 36 URLs, HTTP 202.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Founder authorised a one-time bulk IndexNow push (previously held pending explicit approval, per the SEO-indexation lane brief).
Scope, stated plainly: IndexNow reaches Bing, Yandex, Seznam, Naver, Yep — not Google. Google tested it in 2021 and never shipped support, so nothing here moves Google indexation. The reason to run it is Bing, where we currently have ~2 URLs indexed.
Two key files, not one
indexnow.org's FAQ is explicit: "Each subdomain is treated as a separate host, which means you must create and manage individual key files for each one." A key on the apex does not cover subdomains — so
www.webhook.coanddocs.webhook.coeach get their own copy at their own root.The key is public by design (it is served at a URL) and carries no privilege beyond "may submit URLs for this host". That is why it is committed rather than kept in a secret store.
Guards, each tested
submitBatchverifies the key file is live and serves the key before it will POST. Anokresponse whose body is not the key (a soft-404 HTML page) counts as failure — so a host that 200s everything cannot look verified.422; we fail locally and name the offender.webhook.coor a subdomain, refused before any network call, with lookalikes (evilwebhook.co,webhook.co.evil.com) covered.apps/docsis deliberately an experimentMintlify documents
.txtas Enterprise-only, so the docs key file may 404 after deploy.verifyKeyLiveturns that into a clean refusal rather than a wasted submission — and it converts an unknown into a measured fact rather than an assumption. If it 404s, the file gets removed and docs goes to Bing via Webmaster Tools' GSC import instead (no key file needed).Test plan
out/<key>.txt, 36 bytes).deploy-wwwran, verify both key URLs live, then runpnpm indexnow <host>and report the HTTP result per host.No human UI verification needed — a CLI script and two static text files, no rendered surface.