perf: specialize primitive sums for run-end arrays

[connortsui20]

Summary

Sums primitive run-end arrays by multiplying values by their run lengths, for both Sum and SumV2. Covers whole-array and grouped aggregation, with decimals using the existing fallback.

Changes

Checks validity before decoding runs and returns early for empty or all-null inputs. Whole-array aggregation lives in whole.rs, group traversal in grouped.rs, and run intersection with weighted arithmetic in runs.rs. All-valid inputs scan the end and value slices directly. Only partially valid inputs use valid-run indices.

Fixed-size groups share a forward cursor. List-view groups seek independently because their ranges can overlap or arrive out of order. Widened signed products and fused multiply-add allow cancellation without overflowing the intermediate product. Weighted floating-point sums can round differently from repeated addition.

Validation

• cargo nextest run -p vortex-runend --lib: all 127 tests passed. The added case checks advancing past null groups when all run values are valid. Existing cases cover overlapping and out-of-order ranges, runs split across consecutive groups, slices, nulls, overflow, and finite floating-point cancellation.
• The earlier combined vortex-array and vortex-runend run passed 3,577 tests with 1 skipped. These kernel changes only affect vortex-runend.
• cargo test --doc -p vortex-runend: passed.
• cargo clippy --all-targets --all-features -- -D warnings: passed.
• cargo +nightly fmt -p vortex-runend -- --check: passed. Full workspace formatting still produces four existing macro-format differences in vortex-duckdb and vortex-ffi, outside this PR.

Benchmark results

Local aarch64-apple-darwin medians with Rust 1.98.0 and 100 samples, using cargo bench -p vortex-runend --bench run_end_sum -- --sample-count 100. Both modes use 31e3f624bad64e9046c5f5adb418903e01ac64e9, with the baseline disabling the run-end kernels in its session. Each input contains 2,048 nullable i32 elements, and grouped timings include finalization to a primitive array.

Workload	Specialized	Kernels disabled
Whole array	1.062 µs	1.770 µs
2-element groups	7.603 µs	12.70 µs
128-element groups	2.312 µs	2.916 µs

These cases use 64-element runs, with every fifth run null. The benchmark also covers run lengths 4 and 1,024 and group sizes 1 and 8. Additional whole-array cases exercise AllInvalid, AllValid, and NonNullable validity.

Two additional grouped cases exercise all-valid run values. With the same benchmark fixtures applied to the previous implementation at f6685e9ba706f8df162fbb2071c94018f97c7418, local medians changed from 7.499 to 6.374 µs for 2-element groups, and from 2.082 to 2.062 µs for 128-element groups.

All 35 local benchmark cases ran below 1 ms per iteration, with a slowest median of 27.12 µs.

CodSpeed simulation reports all 35 cases below 1 ms at this head. The slowest is grouped_runend_fallback[1, 4] at 830.774 µs. The two all-valid grouped cases measure 274.698 µs for 2-element groups and 111.420 µs for 128-element groups. Both pages of the report were checked.

The benchmark workflow passed. The overall CodSpeed analysis still flags other existing walltime benchmarks: arrow_checked_add_u32_neon[16384] and dict_canonicalize_gt_u8_avx512[16000000]. Those results have not been investigated as part of this change.

[codspeed-hq]

Merging this PR will degrade performance by 11.87%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 2188 untouched benchmarks
🆕 35 new benchmarks
⏩ 218 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	dict_canonicalize_gt_u8_avx512[16000000]	6.8 ms	11.3 ms	-39.91%
❌	WallTime	arrow_checked_add_u32_neon[16384]	12.2 µs	20.3 µs	-39.87%
⚡	WallTime	dict_canonicalize_gt_u8_neon[1000000]	559.8 µs	487.3 µs	+14.89%
⚡	Simulation	allocate_drop_arrow[0]	456.9 ns	402.7 ns	+13.45%
⚡	WallTime	dict_canonicalize_gt_u8_neon[16000000]	9.4 ms	8.3 ms	+12.92%
🆕	Simulation	grouped_runend_fallback[1, 1024]	N/A	738.5 µs	N/A
🆕	Simulation	grouped_runend_fallback[1, 4]	N/A	830.8 µs	N/A
🆕	Simulation	grouped_runend_fallback[1, 64]	N/A	763.8 µs	N/A
🆕	Simulation	grouped_runend_fallback[128, 1024]	N/A	149.8 µs	N/A
🆕	Simulation	grouped_runend_fallback[128, 4]	N/A	217.6 µs	N/A
🆕	Simulation	grouped_runend_fallback[128, 64]	N/A	147.3 µs	N/A
🆕	Simulation	grouped_runend_fallback[2, 1024]	N/A	443.9 µs	N/A
🆕	Simulation	grouped_runend_fallback[2, 4]	N/A	518.4 µs	N/A
🆕	Simulation	grouped_runend_fallback[2, 64]	N/A	454.6 µs	N/A
🆕	Simulation	grouped_runend_fallback[8, 1024]	N/A	220.6 µs	N/A
🆕	Simulation	grouped_runend_fallback[8, 4]	N/A	287.9 µs	N/A
🆕	Simulation	grouped_runend_fallback[8, 64]	N/A	222.2 µs	N/A
🆕	Simulation	grouped_runend[1, 1024]	N/A	455.4 µs	N/A
🆕	Simulation	grouped_runend[1, 4]	N/A	482.3 µs	N/A
🆕	Simulation	grouped_runend[1, 64]	N/A	461.4 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/sum-runend (31e3f62) with develop (e3b8eb2)}

Footnotes

1. 218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩