feat(seo): Bing Webmaster read-path + opt-in URL submitter (pnpm bing)#787
Merged
Conversation
Bing is the only search engine whose submission path we can automate at all. Google's Indexing API takes JobPosting/BroadcastEvent only and Search Console's "Request indexing" is UI-only, so every Google submission is a human clicking a button. Bing exposes SubmitUrlBatch at 100/day, 800/month. `pnpm bing` reports crawl + index state, plus `sites`, `feeds`, `quota` and `submit`. Submission is a DRY RUN unless --confirm is passed, so running the command to see what it would do cannot silently write to a third party's index. It also gives us a second, independent read on the indexation diagnosis, and it agrees: Bing crawled 111 pages in nine days, has 3 URLs in its index, and reports InLinks = 0 every single day. Google let us infer "no authority" from its only referrers being spam domains; Bing states the inbound-link count as a number. Two engines disagreeing would have falsified the diagnosis. Bing also fetched docs.webhook.co/sitemap.xml successfully (150 URLs, 07-21) -- the exact file Google's stuck record refused for ten days. Independent evidence that defect was Google-side. Two security properties, both mutation-proven RED before the fix: - The API key travels in the QUERY STRING, so any logged or thrown URL is a credential in the operator's scrollback and in the session transcript. Everything printed goes through redact(); making it a no-op fails 2 tests. - assertSiteAllowed compares the PARSED hostname, not a substring. A startsWith/includes check admits https://webhook.co.evil.com/ -- the incomplete-URL-substring-sanitization defect CodeQL flags at HIGH, and which it already flagged once in this repo. Mutating it to .includes() fails 1 test. parseDotNetDate slices strings rather than matching a pattern: the natural regex trips security/detect-unsafe-regex, and a warning that is permanently "fine, ignore it" is how a real one later gets ignored too. The test fixture is a self-describing low-entropy string, not a realistic hex key. A hex fixture reads as a credential to gitleaks' generic-api-key rule, and the honest fix is a fixture that is not secret-shaped rather than an allow-comment -- suppressing the scanner in a file whose whole subject is credential hygiene teaches the wrong lesson. 20 new tests, no network (fetch is injected). scripts/**: 638 pass, 0 fail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU
… the parser
Both blocking findings were real.
1. --confirm was asserted on parseArgs only. The branch in main() that must
not write was never executed by a test, so a regression that submitted
unconditionally would have stayed green. That is the exact failure mode of
"a guard's tests must run the guard", in a file whose whole subject is not
writing to a third party's index by accident.
main() is now an injectable run(argv, {fetchImpl, env, log}). The new test
drives the real code path with a fake that serves reads and throws on any
non-GET -- because the property is "performs no WRITE", not "performs no
I/O": the dry run deliberately reads the quota so the operator sees the
budget before confirming. Mutating the gate to always-submit fails it.
2. A foreign siteUrl on the write path was unguarded by any test (the code was
correct; nothing proved it). Now covered with neverFetch.
Also took both non-blocking security notes, and the first is more than
hardening: the key rides in the QUERY STRING, so following a redirect hands
the full URL -- key included -- to whatever host the redirect names. Every
request now sets redirect:"error" and referrerPolicy:"no-referrer"; removing
them fails a test. And redact() now scrubs the registered key value wherever
it appears, not only in an apikey= parameter, so a key echoed back in an
upstream error body cannot print in full.
25 tests (was 20), scripts/**: 643 pass, 0 fail. Re-verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU
Second AI review round, both findings real. 1. The failure-redaction test was a tautology. Its fixture body never contained the key and the thrown message never included the request URL, so `!message.includes(KEY)` was true by construction -- green even with redact() stubbed to the identity. The fixture now echoes the key back the way a real upstream 4xx quoting the request would, and asserts /REDACTED/. Added the mirror case on the read path. Mutating redact() to a no-op now fails 5 tests; before this commit it failed 2, and neither of those was the one guarding an error path. 2. redirect:"error" was asserted only on the GET path. Dropping ...REQUEST_INIT from the POST alone would have stayed green while leaking the query-string key on a redirect. The POST success test now asserts redirect and referrerPolicy; mutating the POST only fails it. Both non-blocking notes were considered and left: registerSecret ignoring strings under 8 chars is deliberate (a short "secret" would redact ordinary substrings of real output), and the allowlist error embeds the siteUrl the caller passed, which is an operator-supplied URL rather than a credential and is what makes the message actionable. 26 tests, scripts/**: 644 pass, 0 fail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bing is the only search engine whose submission path we can automate at all. Google's Indexing API accepts
JobPosting/BroadcastEventonly, and Search Console's "Request indexing" is UI-only — so every Google submission is a human clicking a button. Bing exposesSubmitUrlBatchat a 100/day, 800/month quota.The founder provisioned a Bing Webmaster API key; this makes it turnkey.
What
pnpm bing [crawl|sites|feeds|quota|submit], mirroring thegsc-*readers.Submission is a DRY RUN unless
--confirmis passed — running the command to see what it would do cannot silently write to a third party's index.The second opinion it buys
Bing independently corroborates
internal/marketing/seo-indexation-diagnosis.md, and states more plainly what Google only let us infer:Bing is crawling actively (111 pages in nine days) and has 3 URLs in its index, with
InLinks = 0every single day of the window. Google made us infer "no authority" from its only reported referrers being spam domains; Bing reports the inbound-link count as an integer. Two engines disagreeing would have falsified the diagnosis. They agree.Also confirms the Google-side sitemap defect was Google-side: Bing fetched
docs.webhook.co/sitemap.xmlsuccessfully on 07-21, all 150 URLs — the exact file Google's stuck record refused to fetch for ten days.Two security properties, both mutation-proven RED first
redact(). Making it a no-op fails 2 tests.assertSiteAllowedcompares the parsed hostname, not a substring. AstartsWith/includescheck admitshttps://webhook.co.evil.com/— the incomplete-URL-substring-sanitization defect CodeQL flags at HIGH, and which it has already flagged once in this repo. Mutating it to.includes()fails 1 test.parseDotNetDateslices strings rather than matching a pattern. The natural regex — a greedy digit run followed by an optional fixed-width offset — tripssecurity/detect-unsafe-regex, and a warning that is permanently "fine, ignore it" is how a real one later gets ignored too.It also ignores the
/Date(…-0700)/offset rather than applying it: the epoch is already UTC, so honouring the offset would shift every timestamp by seven hours.Verification
fetchis injected throughout.scripts/**: 638 pass, 0 fail. Fulllintchain and 33/33 typecheck green via the pre-commit hook.crawl,sites,feeds,quota, the dry-run submit, and the allowlist refusal (not an allowed site (host evil.com)).600.🤖 Generated with Claude Code
https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU