feat(seo): Bing Webmaster read-path + opt-in URL submitter (pnpm bing)#786
feat(seo): Bing Webmaster read-path + opt-in URL submitter (pnpm bing)#786choraria wants to merge 2 commits into
Conversation
Bing is the only search engine whose submission path we can automate at all. Google's Indexing API takes JobPosting/BroadcastEvent only and Search Console's "Request indexing" is UI-only, so every Google submission is a human clicking a button. Bing exposes SubmitUrlBatch at 100/day, 800/month. `pnpm bing` reports crawl + index state, `sites`, `feeds`, `quota`, and `submit`. Submission is a DRY RUN unless --confirm is passed, so running the command to see what it would do cannot silently write to a third party's index. It also gives us a second, independent read on the indexation diagnosis, and it agrees: Bing crawled 111 pages in nine days, has 3 URLs in its index, and reports InLinks = 0 every single day. Google let us infer "no authority" from its only referrers being spam domains; Bing states the inbound-link count as a number. Two engines disagreeing would have falsified the diagnosis. Two security properties, both mutation-proven RED before the fix: - The API key travels in the QUERY STRING, so any logged or thrown URL is a credential in the operator's scrollback and in the session transcript. Everything printed goes through redact(); making it a no-op fails 2 tests. - assertSiteAllowed compares the PARSED hostname, not a substring. A startsWith/includes check admits https://webhook.co.evil.com/ -- the incomplete-URL-substring-sanitization defect CodeQL flags at HIGH, and which it already flagged once in this repo. Mutating it to .includes() fails 1 test. parseDotNetDate slices strings rather than matching a pattern: the natural regex trips security/detect-unsafe-regex, and a warning that is permanently "fine, ignore it" is how a real one later gets ignored too. 20 new tests, no network (fetch is injected). scripts/**: 638 pass, 0 fail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU
gitleaks failed CI on the test's fixture key -- a realistic 32-char hex string reads as a credential to the generic-api-key rule (entropy 4.0), even though it is an obvious dummy. Fixed by making the fixture self-describing rather than by adding a gitleaks:allow comment. Suppressing the scanner in a file whose entire subject is credential hygiene teaches exactly the wrong lesson to the next reader, and an allow-comment is indistinguishable at a glance from one covering a real secret. redact() matches on the `apikey=` query parameter, not on the value's shape, so the swap weakens nothing: mutating redact() to a no-op still fails 2 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU
🔴 AI review — VERDICT: BLOCKBlocking issues found (advisory — review before merging; add the Full reviewSecurity reviewNo blocking issues (injection/XSS, committed secrets, authz/SSRF, PII in logs). Key handling uses env/ Non-blocking notes only: prefer QA / test reviewBlocking
No skipped tests or lowered coverage thresholds. Existing coverage for VERDICT: BLOCK |
|
Superseded by #787. gitleaks scans branch history, not just the final tree — so fixing the test fixture in a follow-up commit left the original hex string in No content difference between the two. |
1 participant
Why
Bing is the only search engine whose submission path we can automate at all. Google's Indexing API accepts
JobPosting/BroadcastEventonly, and Search Console's "Request indexing" is UI-only — so every Google submission is a human clicking a button. Bing exposesSubmitUrlBatchat a 100/day, 800/month quota.The founder provisioned a Bing Webmaster API key; this makes it turnkey.
What
pnpm bing [crawl|sites|feeds|quota|submit], mirroring thegsc-*readers.Submission is a DRY RUN unless
--confirmis passed — running the command to see what it would do cannot silently write to a third party's index.The second opinion it buys
Bing independently corroborates
internal/marketing/seo-indexation-diagnosis.md, and states more plainly what Google only let us infer:Bing is crawling actively (111 pages in nine days) and has 3 URLs in its index, with
InLinks = 0every single day of the window. Google made us infer "no authority" from its only reported referrers being spam domains; Bing reports the inbound-link count as an integer. Two engines disagreeing would have falsified the diagnosis. They agree.Also confirms the Google-side sitemap defect was Google-side: Bing fetched
docs.webhook.co/sitemap.xmlsuccessfully on 07-21, all 150 URLs — the exact file Google's stuck record refused to fetch for ten days.Two security properties, both mutation-proven RED first
redact(). Making it a no-op fails 2 tests.assertSiteAllowedcompares the parsed hostname, not a substring. AstartsWith/includescheck admitshttps://webhook.co.evil.com/— the incomplete-URL-substring-sanitization defect CodeQL flags at HIGH, and which it has already flagged once in this repo. Mutating it to.includes()fails 1 test.parseDotNetDateslices strings rather than matching a pattern. The natural regex — a greedy digit run followed by an optional fixed-width offset — tripssecurity/detect-unsafe-regex, and a warning that is permanently "fine, ignore it" is how a real one later gets ignored too.It also ignores the
/Date(…-0700)/offset rather than applying it: the epoch is already UTC, so honouring the offset would shift every timestamp by seven hours.Verification
fetchis injected throughout.scripts/**: 638 pass, 0 fail. Fulllintchain and 33/33 typecheck green via the pre-commit hook.crawl,sites,feeds,quota, the dry-run submit, and the allowlist refusal (not an allowed site (host evil.com)).600.🤖 Generated with Claude Code
https://claude.ai/code/session_01TZdJCPPGQGJYj8CywaR9KU