The proposal is to give the parser a clearer contract: consuming a token stops at that token; it does not automatically scan the following token. Grammar code would inspect tokens through peek/peek2 and explicitly select specialized readers for constructs such as regexes and templates.
Expected benefits:
- Simpler grammar code with fewer dependencies on scanner position and mode-switch timing.
- Ordinary lookahead without saving and restoring the full parser state.
- Fewer repairs for characters already scanned under a different interpretation.
- Consistent rollback of comments, diagnostics, and recovery state during speculative parsing.
What changes
Currently, Parser.next advances past the current token and scans its successor. Parser.expect also does this on a successful match. The parser therefore holds a current token whose characters the scanner has already consumed.
The proposed interface separates inspection and consumption. Inspection may populate an internal cache, but it does not advance the parser’s logical position or publish diagnostics. Consumption does not request a successor.
The API names below are illustrative.
1. Simple lookahead becomes a read
Checking whether the following token is a colon currently involves speculative advancement:
Parser.lookahead p (fun p ->
Parser.next p;
p.token = Colon)
With cached token inspection:
Longer decisions, such as recognizing arrow functions or parsing attributes speculatively, would still use scoped speculation. Their state restoration would be handled centrally.
2. Regexes and templates are read from explicit boundaries
Consider:
let pattern = /.foo/g
let greeting = `hello ${name}!`For the regex, the current scanner initially recognizes /. as ForwardslashDot. Regex parsing subsequently reconstructs the leading dot with "." ^ pattern.
A dedicated regex reader would instead read the complete literal from its opening slash, even if ordinary tokens had already been inspected.
For templates, consuming a delimiter would leave the following text available to the template reader:
Parser.expect p Backtick;
let chunk = Parser.take_template_chunk p in
Likewise, after parsing an interpolation, consuming } would not scan the remaining template text as ordinary code. Any incompatible prefetched tokens would be discarded inside the cursor abstraction.
3. Angle brackets keep their interpretation local
type nested<'a>= array<option<'a>>
let shifted = value >> 1
The ordinary scanner would expose separate angle delimiters. Type parsing would consume them directly; expression parsing would recognize adjacent characters as >=, >>, >>>, or << when an operator is expected.
This removes the need to maintain a persistent Diamond mode across parsing calls.
Compatibility and validation
The intended result is simpler parser internals with existing syntax and AST behavior preserved. Validation should compare ASTs and source locations, formatting, comments, diagnostics, and recovery behavior, alongside benchmarks for both compiler and formatter parsing.
The full cursor change is not implemented yet. Any diagnostic differences or performance costs should be reviewed explicitly. The additional cursor machinery should also be judged against the special cases it removes.
The proposal is to give the parser a clearer contract: consuming a token stops at that token; it does not automatically scan the following token. Grammar code would inspect tokens through
peek/peek2and explicitly select specialized readers for constructs such as regexes and templates.Expected benefits:
What changes
Currently,
Parser.nextadvances past the current token and scans its successor.Parser.expectalso does this on a successful match. The parser therefore holds a current token whose characters the scanner has already consumed.The proposed interface separates inspection and consumption. Inspection may populate an internal cache, but it does not advance the parser’s logical position or publish diagnostics. Consumption does not request a successor.
The API names below are illustrative.
1. Simple lookahead becomes a read
Checking whether the following token is a colon currently involves speculative advancement:
With cached token inspection:
Longer decisions, such as recognizing arrow functions or parsing attributes speculatively, would still use scoped speculation. Their state restoration would be handled centrally.
2. Regexes and templates are read from explicit boundaries
Consider:
For the regex, the current scanner initially recognizes
/.asForwardslashDot. Regex parsing subsequently reconstructs the leading dot with"." ^ pattern.A dedicated regex reader would instead read the complete literal from its opening slash, even if ordinary tokens had already been inspected.
For templates, consuming a delimiter would leave the following text available to the template reader:
Likewise, after parsing an interpolation, consuming
}would not scan the remaining template text as ordinary code. Any incompatible prefetched tokens would be discarded inside the cursor abstraction.3. Angle brackets keep their interpretation local
The ordinary scanner would expose separate angle delimiters. Type parsing would consume them directly; expression parsing would recognize adjacent characters as
>=,>>,>>>, or<<when an operator is expected.This removes the need to maintain a persistent Diamond mode across parsing calls.
Compatibility and validation
The intended result is simpler parser internals with existing syntax and AST behavior preserved. Validation should compare ASTs and source locations, formatting, comments, diagnostics, and recovery behavior, alongside benchmarks for both compiler and formatter parsing.
The full cursor change is not implemented yet. Any diagnostic differences or performance costs should be reviewed explicitly. The additional cursor machinery should also be judged against the special cases it removes.