fix: skip pruning inside <pre>/<code> in PruningContentFilter - #2111
Open
kinorai wants to merge 1 commit into
Open
fix: skip pruning inside <pre>/<code> in PruningContentFilter#2111kinorai wants to merge 1 commit into
kinorai wants to merge 1 commit into
Conversation
Syntax highlighters wrap every token in <span>, including whitespace-only spans (<span class="w"> </span>). PruningContentFilter scored those at ~0.26-0.36 against the default 0.48 threshold and decomposed them, collapsing 'import torch' to 'importtorch' in fit_markdown — the same failure unclecode#1181 fixed on the scraper path in v0.7.8. Heavily-highlighted <pre> blocks (huge tag_len, near-zero text density) could also be dropped wholesale. Mirror the scraper-side guard: never score or prune <pre>/<code> subtrees. Fixes unclecode#2110 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2110
What
PruningContentFilter._prune_treenow skips<pre>/<code>subtrees entirely — never scored, never pruned. This mirrors the scraper-side guard that #1181 added toWebScrapingStrategy.remove_empty_elements_fastin v0.7.8, which fixedraw_markdownbut leftfit_markdownunprotected.Why
Syntax highlighters wrap every token in
<span>, including whitespace-only spans (<span class="w"> </span>). Those score ~0.26–0.36 against the default 0.48 threshold (text_density 0, tag_weight span=0.3, log(1)=0) and get decomposed —import torchbecomesimporttorch,FROM golang:1.21 AS builderbecomesFROMgolang:1.21ASbuilderinfit_markdown. Heavily-highlighted<pre>blocks (hundreds of spans → huge tag_len, near-zero text density) can be dropped wholesale, which removes all code from e.g. MDN pages. Details and real-URL repros in #2110.Tests
tests/test_pruning_code_whitespace.py(4 tests, modeled ontest_pruning_preserve_whitelist_1900.py):<pre>block is not droppedimport torch,FROM golang:1.21 AS builder)Without the fix, the two whitespace tests fail with the exact corruption signature; with it, all 24 pruning-filter tests pass (including the full #1900 preserve-whitelist suite — no regressions).
preserve_tags=["pre","code"](#1904) remains a valid explicit workaround for older 0.9.x versions; this PR makes the safe behavior the default.🤖 Generated with Claude Code