fix(docs): pin the fumadocs-core peek omission that encodes markdown as entities - #281
Merged
Merged
Conversation
…as entities `defaultStringifier` wraps every `mdast-util-to-markdown` serializer handler through `modHandler` and does not carry the handler's `peek` onto the wrapper. `containerPhrasing` uses `Handle.peek` to read the next sibling's first character without side effects; with it missing, the lookahead invokes that sibling's full handler and leaves `state.attentionEncodeSurroundingInfo` set from node N+1, which the loop then applies to node N. The effect reaches the served text: `/llms-full.txt` and all 79 per-page `/llms.mdx` bodies carried 67 HTML numeric character references each, in files that are not HTML, and 4 of 661 markdown link targets were delivered malformed because the reference landed on the target's closing parenthesis. Two of those were also invisible to the absolute-URL rewrite in `llms-full.txt/route.ts`, whose `SITE_LINK` needs a literal closing parenthesis. Fixed in the producer rather than by decoding on the way out: a consumer-side decode would need a code-span guard, and the defect destroys inline-code closing delimiters, so that guard would key on structure the bug has already broken. The defect is present in 16.8.12, 16.15.1, 16.15.2 and 16.15.8, so upgrading is not a fix. The patch file's header carries the upstream state it tracks and the condition under which it can be deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ChPQM8jamxLUfUAxwFpJ8S
os-bill
marked this pull request as ready for review
September 8, 2026 21:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #197
Route A, as decided by the maintainer: pin the upstream one-liner with
pnpm.patchedDependenciesand fix the producer, so/llms-full.txtand all 79per-page
/llms.mdxbodies are repaired together.NOTATION.
AMPstands for one literal ampersand character, per the card'sconvention: this tracker's body sanitizer decodes HTML numeric character
references, including inside code fences.
AMP#x2A;means the six charactersthat would otherwise be typed as ampersand, hash, x, 2, A, semicolon.
What is patched
fumadocs-core,dist/mdx-plugins/stringifier.js.defaultStringifierwrapsevery
mdast-util-to-markdownserializer handler throughmodHandlerand doesnot carry the handler's
peekonto the wrapper.containerPhrasingusesHandle.peekto read the next sibling's first character cheaply and withoutside effects; with it missing the lookahead invokes that sibling's full handler,
which leaves
state.attentionEncodeSurroundingInfoset from node N+1 for theloop to apply to node N. The patch copies
.peekonto the wrapper.The cause, the isolated A/B and the first end-to-end proof are on #197 and are
not repeated here. What this PR adds is the durable form of the same change and
the evidence that it survives a real
pnpm install.Measurements — re-taken in this tree, at base
50aacb9Same harness before and after, reading the built bodies off
apps/docs/.next/server/app. Union re-run on the final tree at5ce94d3./llms-full.txt/llms.mdxbodies/llms-full.txt/llms-full.txtReference kinds before, in both outputs:
AMP#x2A;x58,AMP#x60;x5,AMP#x29;x4.Three before-figures differ from the ones the decision comment quotes, and
the newer ones are the real ones. Link targets are 661, not 650, and
absolute site targets 560, not 549 — eleven on-site links have landed in
content/docs/since the investigation read1ebd7b7. The malformed count (4)and the reference counts (67, 58/5/4) are unchanged. The absolute-target figure
still moves by exactly +2: the two
[AI Builder]links whose closingparenthesis the bug had turned into
AMP#x29;, whichSITE_LINKinllms-full.txt/route.tscould therefore never match.Membership and order untouched. The 93 top-level
#page headings arebyte-identical in sequence, all 799 heading lines are byte-identical in
sequence, and the line count is identical. Decoding the before body and diffing
against the after body leaves 21 differing lines: 19 are GFM table column
padding (a cell that loses 5 characters narrows its column, so every row in that
table is repadded) and 2 are the AI Builder links acquiring their absolute URL.
Nothing else differs.
The patch applies on a clean install
node_moduleswiped in all three workspace projects, thenpnpm install --frozen-lockfile— which is what CI runs — exit 0. pnpm 10.28.2prints no "patch applied" line on success, so the evidence is what the installer
itself wrote, read back off disk afterwards:
node_modules/.pnpm/fumadocs-core@16.8.12_patch_hash=042796c0fe0ae66ccc2f1eadd4df4b8399a43c39c49facd3a0d938_.../node_modules/fumadocs-core— the
patch_hash=segment is the installer's, not mine;grepon that file, resolved throughapps/docs' own module resolution,finds
if (handler.peek) wrapped.peek = handler.peek;at line 81, and findsthe pristine anchor line 0 times.
The patch step can fail — both ways
A patch step that cannot fail is indistinguishable from one that applied, so
both failure modes were demonstrated:
removed-context line (
node.ctxtonode.CTX, confirmed on disk byhash-object before and after),
node_moduleswiped,pnpm installexits1 with
ERR_PNPM_PATCH_FAILED Could not apply patch ....pnpm.patchedDependenciesdeleted frompackage.json,node_moduleswiped, install exit 0 — and the installed filecarries the pristine anchor and no
peekline. A full rebuild then puts thereferences back at exactly the baseline figures: 67 references, 4 of 661
malformed, 695176 bytes, matching the before build byte count.
Both legs were restored from
HEADunder anEXIT INT TERMtrap and therestore verified by
git diff HEADbeing empty, not by an exit code.Where the manifest key went
pnpm 10.28.2 put
patchedDependenciesin the rootpackage.json, under thepnpmblock that already carriesonlyBuiltDependencies— not inpnpm-workspace.yaml. That placement ispnpm patch-commit's own, left wherethe installed tool wrote it and verified to work through a wiped
--frozen-lockfileinstall.Worker size gate
Three
wrangler deploy --dry-runreadings in this container, all far under the61440 KiB budget (headroom ~2895 KiB):
Total Upload:pnpm run buildthen packagepnpm turbo run buildthen packageThe two readings of this same branch differ by 0.22 KiB, so a sub-KiB gap
between readings is not by itself evidence of a change in the bundle. No delta
is claimed here in either direction; the gate is green.
Gates run before pushing, on
5ce94d3All exit 0, verdicts read from each command's own output:
pnpm install --frozen-lockfileon a wipednode_modulesnode .github/scripts/check-node-floor.mjs --self-testand the real scannode scripts/pm/check-half-states.mjs --self-test(1551 cases)node apps/docs/scripts/gen-zh-hant.mjs --check(73 files byte-identical)pnpm turbo run type-check --continuepnpm turbo run buildnode .github/scripts/check-locale-surface.mjs— 409 sitemap URLs, 0unexpected / 0 missing, and both
llmsbodies still carry all 60 en-onlytitles and none from other locales
pnpm turbo run test --forceopennextjs-cloudflare build --skipNextBuildplus the size weigh-in abovesmoke-docs.mjsagainst a localopennextjs-cloudflare preview, 4 pages rendered, its own negative controldemonstrated red
Upstream, and the next bump
The defect is present in 16.8.12 (installed), 16.15.1, 16.15.2 and 16.15.8
(
lateston 2026-09-08), each checked by unpacking the published tarball; theanchor line is byte-identical in all four. Upgrading is not a fix. This session
cannot open an issue on
fuma-nama/fumadocs, so a ready-to-post upstream reportis filed as a comment on #197 for the maintainer to route.
#239 bumps this package to 16.15.2. The pin is keyed to
16.8.12, so on anyother version the patch is simply not applied and the defect returns silently —
which is why the counts above should be re-measured on that PR, and the patch
regenerated against the new version in the same PR. 16.15.x also reformats an
iftwo hunks above this one in the same file. The patch file's header carriesall of this, plus the condition under which the file can be deleted.
Not touched
apps/docs/lib/source.ts,apps/docs/app/llms-full.txt/route.ts,content/docs/, and.github/workflows/ci.yml(in flight on #277). No consumeredit was needed, which is the result Route A was chosen for.
🤖 Generated with Claude Code
Generated by Claude Code
Generated by Claude Code