Skip to content

fix(rescript): stop binary .res resources hanging the indexer (#2176) - #2352

Merged
DeusData merged 2 commits into
mainfrom
fix/issue-2176
Sep 28, 2026
Merged

DeusData merged 2 commits into
mainfrom
fix/issue-2176

Conversation

@DeusData

Copy link
Copy Markdown
Owner

A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished
indexing: .res maps to ReScript, so the resource was parsed as ReScript
source and the definitions pass spun. Three defects, each fixed:

  1. Quadratic usage walk (the hang). The ReScript let binding check
    (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript
    and Nickel) climbed from every named leaf with ts_node_parent, which
    descends from the root and scans each level's children. A binary
    file parses into a flat error tree (~170k root children per MB), so
    each climb cost O(root children) and the file O(n^2): 1 MB spent
    46 s in cbm_extract_unified, 4.7 MB never returned. The check now
    climbs the unified walk's own cursor (O(1) per hop), like the other
    occurrence classifiers; the node-based climb stays only for callers
    outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end.

  2. Quadratic lexing in error recovery. tree-sitter retries the lexer at
    every byte of an unparseable stretch with every external token
    valid; the ReScript scanner's template-string loop then ran to the
    next backtick/$/backslash/NUL (end of file if none) and discarded
    the result. That cost O(n^2) inside lexing, where the parse budget's
    progress callback never runs, so such files were dropped by the
    5 s CPU budget instead of parsed -- plain text too (100 KB of
    non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the
    template branch when NEWLINE is also valid, which only the ERROR
    state allows. Valid files parse to identical trees. Recorded in
    vendored/grammars/MANIFEST.md.

  3. Binary content parsed as source. .res is also Godot's binary
    resource format and the Windows compiled-resource format. Following
    the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the
    first 4 KB and reports a file holding a NUL byte as unsupported;
    ReScript text never contains one.

Tests (deterministic work counters, no clocks):

  • extraction::extract_rescript_let_bindings_use_the_walk_cursor --
    zero root-descending parent hops at 128 and 1024 statements
    (before: 515 / 4099).
  • complexity::complexity_rescript_error_recovery_lexing_is_linear --
    lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio
    must be linear (before 3.99, after 2.00).
  • language::lang_res_binary_resource_unsupported and
    lang_res_rescript_stays_rescript.

Fixes #2176


Known leftover (not in this PR): the same root-descending parent climb remains in other is_policy_binding branches (agda, llvm, wolfram, tlaplus, elixir, lisp, …) in extract_usages.c; a generic cursor-ancestor iterator would fix them all.

A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished
indexing: `.res` maps to ReScript, so the resource was parsed as ReScript
source and the definitions pass spun. Three defects, each fixed:

1. Quadratic usage walk (the hang). The ReScript `let` binding check
   (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript
   and Nickel) climbed from every named leaf with ts_node_parent, which
   descends from the root and scans each level's children. A binary
   file parses into a flat error tree (~170k root children per MB), so
   each climb cost O(root children) and the file O(n^2): 1 MB spent
   46 s in cbm_extract_unified, 4.7 MB never returned. The check now
   climbs the unified walk's own cursor (O(1) per hop), like the other
   occurrence classifiers; the node-based climb stays only for callers
   outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end.

2. Quadratic lexing in error recovery. tree-sitter retries the lexer at
   every byte of an unparseable stretch with every external token
   valid; the ReScript scanner's template-string loop then ran to the
   next backtick/$/backslash/NUL (end of file if none) and discarded
   the result. That cost O(n^2) inside lexing, where the parse budget's
   progress callback never runs, so such files were dropped by the
   5 s CPU budget instead of parsed -- plain text too (100 KB of
   non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the
   template branch when NEWLINE is also valid, which only the ERROR
   state allows. Valid files parse to identical trees. Recorded in
   vendored/grammars/MANIFEST.md.

3. Binary content parsed as source. `.res` is also Godot's binary
   resource format and the Windows compiled-resource format. Following
   the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the
   first 4 KB and reports a file holding a NUL byte as unsupported;
   ReScript text never contains one.

Tests (deterministic work counters, no clocks):
- extraction::extract_rescript_let_bindings_use_the_walk_cursor --
  zero root-descending parent hops at 128 and 1024 statements
  (before: 515 / 4099).
- complexity::complexity_rescript_error_recovery_lexing_is_linear --
  lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio
  must be linear (before 3.99, after 2.00).
- language::lang_res_binary_resource_unsupported and
  lang_res_rescript_stays_rescript.

Fixes #2176

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…anner patch

security-static (scripts/security-vendored.sh) failed on this branch:
the #2176 fix patches internal/cbm/vendored/grammars/rescript/scanner.c
(skip the template-string branch in the error-recovery state) and records
that patch in vendored/grammars/MANIFEST.md, but the checked-in digests in
scripts/vendored-checksums.txt still described the unpatched files.

Both changes are intentional and reviewed in the parent commit; this
records their SHA-256 via `scripts/security-vendored.sh --update`. Only
the two affected lines change, and the integrity check passes again.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Indexer livelocks in the definitions pass on a real-world binary .res file (Godot 4 resource), while synthetic blobs of equal size index fine

1 participant