Skip to content

[Pratt Parser] Add a fuzzer comparing ANTLR and Pratt parser outputs and fix uncovered discrepancies - #1231

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_982225263
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_982225263

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

[Pratt Parser] Add a fuzzer comparing ANTLR and Pratt parser outputs and fix uncovered discrepancies

Added CelPrattParserFuzzer to fuzz CEL inputs against both AntlrParser and PrattParser, asserting error parity and AST equality.

Parser differences uncovered and fixed in PrattParser / Lexer:

  1. Mixed unary operator chains (e.g., -!ll, !-x): ANTLR requires consecutive unary operators to be homogeneous (!/-) and only allows - after ! when immediately followed by an integer or floating-point literal (e.g., !-42), whereas Pratt previously allowed arbitrary mixtures of ! and - without parentheses.
  2. Vertical tab (\v, ASCII 11): ANTLR does not treat \v as whitespace, whereas Lexer and PrattParser previously skipped it.
  3. Unquoted .in field selector: ANTLR treats in as a keyword token and rejects unquoted x.in (requiring backtick-quoted x.`in`), whereas Pratt previously accepted unquoted .in after ..
  4. Chained optional select (T.?a.?a) AST positions: ANTLR records the position of the field constant in _?._ at the start of the member expression (T), whereas Pratt stopped at intermediate .?/[]/() nodes.
  5. Numeric literals immediately followed by identifier characters (e.g., 9in-x): ANTLR tokenizes numeric literals (NUM_INT, NUM_UINT, NUM_FLOAT) without rejecting trailing identifier characters so 9in-x parses as 9 in -x, whereas Pratt's Lexer previously rejected trailing identifier characters at lexing time.
  6. Invalid quoted field selectors inside has(...) (e.g., has(a.`$b`)): When normalizeIdent() rejects an invalid backtick-quoted field name, PrattParser previously still constructed a CelSelect with an empty field string, causing CelExprFactory.newSelect() to throw IllegalArgumentException during has() macro expansion instead of returning an unset error expression like AntlrParser.

Intentional parser differences ignored by CelPrattParserFuzzer (where PrattParser behavior is preferred):

  1. Raw byte string literal prefixes (rb'...', rB'...', Rb'...', RB'...'): ANTLR only accepts br/bR/Br/BR prefix order, whereas Pratt accepts both br and rb.
  2. Standalone commas in empty collection literals ([,], {,}, Msg{,}): ANTLR accepts empty collections containing only a comma, whereas Pratt requires at least one element/entry before a trailing comma.
  3. Leading-dot identifier positions (.R): Pratt records the position of leading-dot identifiers at the . token, whereas ANTLR records it at the identifier token after ..

@copybara-service
copybara-service Bot force-pushed the test_982225263 branch 2 times, most recently from 575b72b to e18d48c Compare September 16, 2026 05:40
…and fix uncovered discrepancies

Added `CelPrattParserFuzzer` to fuzz CEL inputs against both `AntlrParser` and `PrattParser`, asserting error parity and AST equality.

Parser differences uncovered and fixed in `PrattParser` / `Lexer`:
1. **Mixed unary operator chains (e.g., `-!ll`, `!-x`)**: ANTLR requires consecutive unary operators to be homogeneous (`!`/`-`) and only allows `-` after `!` when immediately followed by an integer or floating-point literal (e.g., `!-42`), whereas Pratt previously allowed arbitrary mixtures of `!` and `-` without parentheses.
2. **Vertical tab (`\v`, ASCII 11)**: ANTLR does not treat `\v` as whitespace, whereas `Lexer` and `PrattParser` previously skipped it.
3. **Unquoted `.in` field selector**: ANTLR treats `in` as a keyword token and rejects unquoted `x.in` (requiring backtick-quoted `` x.`in` ``), whereas Pratt previously accepted unquoted `.in` after `.`.
4. **Chained optional select (`T.?a.?a`) AST positions**: ANTLR records the position of the field constant in `_?._` at the start of the `member` expression (`T`), whereas Pratt stopped at intermediate `.?`/`[]`/`()` nodes.
5. **Numeric literals immediately followed by identifier characters (e.g., `9in-x`)**: ANTLR tokenizes numeric literals (`NUM_INT`, `NUM_UINT`, `NUM_FLOAT`) without rejecting trailing identifier characters so `9in-x` parses as `9 in -x`, whereas Pratt's `Lexer` previously rejected trailing identifier characters at lexing time.
6. **Invalid quoted field selectors inside `has(...)` (e.g., `` has(a.`$b`) ``)**: When `normalizeIdent()` rejects an invalid backtick-quoted field name, `PrattParser` previously still constructed a `CelSelect` with an empty field string, causing `CelExprFactory.newSelect()` to throw `IllegalArgumentException` during `has()` macro expansion instead of returning an unset error expression like `AntlrParser`.

Intentional parser differences ignored by `CelPrattParserFuzzer` (where `PrattParser` behavior is preferred):
1. **Raw byte string literal prefixes (`rb'...'`, `rB'...'`, `Rb'...'`, `RB'...'`)**: ANTLR only accepts `br`/`bR`/`Br`/`BR` prefix order, whereas Pratt accepts both `br` and `rb`.
2. **Standalone commas in empty collection literals (`[,]`, `{,}`, `Msg{,}`)**: ANTLR accepts empty collections containing only a comma, whereas Pratt requires at least one element/entry before a trailing comma.
3. **Leading-dot identifier positions (`.R`)**: Pratt records the position of leading-dot identifiers at the `.` token, whereas ANTLR records it at the identifier token after `.`.

PiperOrigin-RevId: 982225263
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant