[perf] Use a flat token stream representation when lexing

[Kobzol]

View all comments

Opening for a perf experiment, nothing to see here.

[Kobzol]

@bors try @rust-timer queue

[rust-bors]

☀️ Try build successful (CI)
Build commit: 8374c7c (8374c7ca6b40db24e90f1c62568ddd867a1d48ce)
Base parent: ab4bbeb (ab4bbeb07dc7dc2d30e9f83d3e0326df981e0ee2)

[rust-timer]

Finished benchmarking commit (8374c7c): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.1%, 1.2%]	73
Regressions ❌ (secondary)	0.6%	[0.1%, 1.4%]	59
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-1.8%	[-2.4%, -1.3%]	13
All ❌✅ (primary)	0.4%	[0.1%, 1.2%]	73

Max RSS (memory usage)

Results (secondary -6.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.9%	[1.8%, 8.5%]	14
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-13.6%	[-22.8%, -2.3%]	21
All ❌✅ (primary)	-	-	0

Cycles

Results (primary 1.1%, secondary -7.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.4%, 3.0%]	3
Regressions ❌ (secondary)	3.5%	[2.1%, 5.2%]	8
Improvements ✅ (primary)	-3.4%	[-3.4%, -3.4%]	1
Improvements ✅ (secondary)	-13.6%	[-19.9%, -4.9%]	14
All ❌✅ (primary)	1.1%	[-3.4%, 3.0%]	4

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.393s -> 489.02s (-0.08%)
Artifact size: 389.94 MiB -> 389.88 MiB (-0.02%)

[Kobzol]

Seems like doing the flat representation in the lexer doesn't help much when we still have to convert it back to all the allocations required for the parser. The whole thing would have to be flattened to have the desired perf. effect.

[Kobzol]

@bors try @rust-timer queue

[rust-bors]

💔 Test for 5588d8a failed: CI. Failed job:

• try - dist-x86_64-linux-quick (web logs, enhanced plaintext logs)

[Kobzol]

@bors try

[rust-bors]

☀️ Try build successful (CI)
Build commit: 0ea5705 (0ea5705702a45a8dd68a0f34aa0c62e3d1b1ca95)
Base parent: cc05892 (cc05892c8346313865afd91ca12ee1fde6d3603c)

[Kobzol]

@rust-timer build 0948993

[rust-timer]

Finished benchmarking commit (0948993): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	1.2%	[0.1%, 6.2%]	137
Regressions ❌ (secondary)	44.8%	[0.1%, 801.3%]	105
Improvements ✅ (primary)	-0.7%	[-2.4%, -0.2%]	77
Improvements ✅ (secondary)	-1.2%	[-5.4%, -0.2%]	67
All ❌✅ (primary)	0.5%	[-2.4%, 6.2%]	214

Max RSS (memory usage)

Results (primary -2.7%, secondary -5.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.9%	[1.3%, 5.7%]	8
Improvements ✅ (primary)	-2.7%	[-5.1%, -0.8%]	100
Improvements ✅ (secondary)	-7.1%	[-19.1%, -1.0%]	44
All ❌✅ (primary)	-2.7%	[-5.1%, -0.8%]	100

Cycles

Results (primary 1.0%, secondary 121.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.8%	[1.8%, 3.7%]	15
Regressions ❌ (secondary)	227.4%	[2.3%, 775.7%]	18
Improvements ✅ (primary)	-2.6%	[-3.3%, -1.5%]	8
Improvements ✅ (secondary)	-4.8%	[-21.4%, -2.3%]	15
All ❌✅ (primary)	1.0%	[-3.3%, 3.7%]	23

Binary size

Results (primary 0.0%, secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.1%]	26
Regressions ❌ (secondary)	0.1%	[0.0%, 0.1%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[0.0%, 0.1%]	26

Bootstrap: 492.1s -> 494.004s (0.39%)
Artifact size: 406.87 MiB -> 407.62 MiB (0.19%)

[Kobzol]

@bors try parent=last @rust-timer queue

[rust-log-analyzer]

The job test-pr-check-2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[RUSTC-TIMING] tracing_error test:false 0.075
error[E0599]: no method named `len` found for struct `ArenaTokenStream` in the current scope
   --> src/tools/clippy/clippy_utils/src/ast_utils/mod.rs:992:21
    |
992 |         && l.tokens.len() == r.tokens.len()
    |                     ^^^
    |
help: there is a method `length` with a similar name
    |
992 |         && l.tokens.length() == r.tokens.len()
    |                        +++

error[E0599]: no method named `len` found for struct `ArenaTokenStream` in the current scope
   --> src/tools/clippy/clippy_utils/src/ast_utils/mod.rs:992:39
    |
992 |         && l.tokens.len() == r.tokens.len()
    |                                       ^^^
    |
help: there is a method `length` with a similar name
    |
992 |         && l.tokens.len() == r.tokens.length()
    |                                          +++

error[E0599]: no method named `iter` found for struct `ArenaTokenStream` in the current scope
   --> src/tools/clippy/clippy_utils/src/ast_utils/mod.rs:993:21
    |
993 |         && l.tokens.iter().zip(r.tokens.iter()).all(|(a, b)| a.eq_unspanned(b))
    |                     ^^^^ method not found in `ArenaTokenStream`

error[E0599]: no method named `iter` found for struct `ArenaTokenStream` in the current scope
   --> src/tools/clippy/clippy_utils/src/ast_utils/mod.rs:993:41
    |
993 |         && l.tokens.iter().zip(r.tokens.iter()).all(|(a, b)| a.eq_unspanned(b))
    |                                         ^^^^ method not found in `ArenaTokenStream`

[RUSTC-TIMING] cargo_metadata test:false 0.794
[RUSTC-TIMING] winnow test:false 1.053
[RUSTC-TIMING] rustc_hash test:false 0.048
    Checking rustc-demangle v0.1.28

[rust-bors]

☀️ Try build successful (CI)
Build commit: 4c1be54 (4c1be54e70b4d5eaffca5750dfc16b390f8c6f58)
Base parent: 98fd715 (98fd715edd3a0a5aa8f2041d7e20a3143b0fc0d3)

[Kobzol]

@bors try parent=last

[rust-timer]

Finished benchmarking commit (4c1be54): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	1.1%	[0.2%, 5.8%]	139
Regressions ❌ (secondary)	49.0%	[0.2%, 803.3%]	96
Improvements ✅ (primary)	-0.7%	[-2.2%, -0.2%]	74
Improvements ✅ (secondary)	-1.2%	[-5.0%, -0.2%]	71
All ❌✅ (primary)	0.5%	[-2.2%, 5.8%]	213

Max RSS (memory usage)

Results (primary -0.6%, secondary -5.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.4%	[0.8%, 4.3%]	35
Regressions ❌ (secondary)	2.1%	[0.6%, 3.3%]	10
Improvements ✅ (primary)	-2.4%	[-3.8%, -1.1%]	59
Improvements ✅ (secondary)	-6.6%	[-17.7%, -1.2%]	43
All ❌✅ (primary)	-0.6%	[-3.8%, 4.3%]	94

Cycles

Results (primary 0.7%, secondary 100.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.9%	[2.1%, 4.1%]	17
Regressions ❌ (secondary)	217.4%	[2.0%, 778.6%]	19
Improvements ✅ (primary)	-2.6%	[-3.8%, -2.1%]	11
Improvements ✅ (secondary)	-4.8%	[-14.2%, -2.0%]	21
All ❌✅ (primary)	0.7%	[-3.8%, 4.1%]	28

Binary size

Results (primary 0.0%, secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.1%]	23
Regressions ❌ (secondary)	0.1%	[0.0%, 0.1%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[0.0%, 0.1%]	23

Bootstrap: 492.1s -> 493.477s (0.28%)
Artifact size: 406.87 MiB -> 406.87 MiB (0.00%)

[Kobzol]

@rust-timer queue

[rust-bors]

☀️ Try build successful (CI)
Build commit: 4013206 (4013206d4b683737f5ab521e8b3fe62bef982018)
Base parent: 98fd715 (98fd715edd3a0a5aa8f2041d7e20a3143b0fc0d3)

[rust-bors]

☔ The latest upstream changes made this pull request unmergeable. Please resolve the merge conflicts by rebasing.

[rust-timer]

Finished benchmarking commit (4013206): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	1.3%	[0.2%, 6.6%]	150
Regressions ❌ (secondary)	2.9%	[0.2%, 15.6%]	99
Improvements ✅ (primary)	-0.7%	[-2.5%, -0.2%]	71
Improvements ✅ (secondary)	-1.2%	[-5.0%, -0.1%]	62
All ❌✅ (primary)	0.6%	[-2.5%, 6.6%]	221

Max RSS (memory usage)

Results (primary 0.7%, secondary -3.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.8%	[0.9%, 9.6%]	44
Regressions ❌ (secondary)	2.3%	[0.9%, 4.5%]	10
Improvements ✅ (primary)	-2.0%	[-3.1%, -1.0%]	36
Improvements ✅ (secondary)	-5.7%	[-11.4%, -1.0%]	29
All ❌✅ (primary)	0.7%	[-3.1%, 9.6%]	80

Cycles

Results (primary 1.6%, secondary 3.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.4%	[1.7%, 5.6%]	21
Regressions ❌ (secondary)	6.3%	[2.5%, 16.4%]	36
Improvements ✅ (primary)	-2.7%	[-3.4%, -2.2%]	9
Improvements ✅ (secondary)	-5.2%	[-17.3%, -2.1%]	11
All ❌✅ (primary)	1.6%	[-3.4%, 5.6%]	30

Binary size

Results (primary 0.0%, secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.1%]	26
Regressions ❌ (secondary)	0.1%	[0.0%, 0.1%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[0.0%, 0.1%]	26

Bootstrap: 492.1s -> 493.643s (0.31%)
Artifact size: 406.87 MiB -> 406.99 MiB (0.03%)