Skip to content

Support wide decimals in DecimalBytePartsArray and kernels - #9809

Open
mhk197 wants to merge 8 commits into
mk/dbp-v2-featurefrom
mk/dbp-array
Open

Support wide decimals in DecimalBytePartsArray and kernels#9809
mhk197 wants to merge 8 commits into
mk/dbp-v2-featurefrom
mk/dbp-array

Conversation

@mhk197

@mhk197 mhk197 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

DecimalBytePartsArray previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability.

The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces:

Decimal storage Children
i8 / i16 / i32 / i64 Signed MSP only; shares the original value buffer
i128 i64 MSP + one u64 lower part
i256 i64 MSP + three u64 lower parts

Only the MSP carries validity. Every lower part must be a non-nullable u64 array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain ArrayRefs, so their individual encodings are independent of the decimal representation.

execute::<DecimalArray> reassembles a DecimalArray from the MSP and lower parts children.

take with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts.

@codspeed-hq

codspeed-hq Bot commented Sep 8, 2026

Copy link
Copy Markdown

Merging this PR will regress 2 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 2 regressed benchmarks
✅ 2223 untouched benchmarks
🆕 72 new benchmarks
⏩ 204 skipped benchmarks1
🗄️ 1 archived benchmark run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation random_i16[0.95] 77.2 µs 94.2 µs -18.09%
Simulation decompress[datetime_for_bp] 160.3 µs 193.6 µs -17.19%
Simulation random_i8[0.5] 91.3 µs 66.7 µs +36.93%
Simulation allocate_drop_arrow[0] 456.9 ns 402.7 ns +13.45%
Simulation chunked_bool_canonical_into[(1000, 10)] 30.7 µs 27.2 µs +12.9%
Simulation allocate_drop_bytes[0] 575.7 ns 521.6 ns +10.39%
🆕 WallTime dbp_assemble_neon[(I128, 1024)] N/A 748 ns N/A
🆕 WallTime dbp_assemble_neon[(I128, 8192)] N/A 3.7 µs N/A
🆕 WallTime dbp_assemble_neon[(I256, 1024)] N/A 2.1 µs N/A
🆕 WallTime dbp_assemble_neon[(I256, 8192)] N/A 15.1 µs N/A
🆕 WallTime dbp_assemble_neon[(I64, 1024)] N/A 105 ns N/A
🆕 WallTime dbp_assemble_neon[(I64, 8192)] N/A 110 ns N/A
🆕 WallTime dbp_split_all_null_neon[(I128, 1024)] N/A 241 ns N/A
🆕 WallTime dbp_split_all_null_neon[(I128, 8192)] N/A 244 ns N/A
🆕 WallTime dbp_split_all_null_neon[(I256, 1024)] N/A 250 ns N/A
🆕 WallTime dbp_split_all_null_neon[(I256, 8192)] N/A 253 ns N/A
🆕 WallTime dbp_split_all_null_neon[(I64, 1024)] N/A 286 ns N/A
🆕 WallTime dbp_split_all_null_neon[(I64, 8192)] N/A 291 ns N/A
🆕 WallTime dbp_split_all_valid_neon[(I128, 1024)] N/A 742 ns N/A
🆕 WallTime dbp_split_all_valid_neon[(I128, 8192)] N/A 4.3 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/dbp-array (b2e49fb) with mk/dbp-v2-feature (4edb7aa)

Open in CodSpeed

Footnotes

  1. 204 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it.

@mhk197
mhk197 force-pushed the mk/dbp-array branch 3 times, most recently from f7dfebe to a566a28 Compare September 9, 2026 02:43
@mhk197 mhk197 changed the title Support multi-part decimal arrays and kernels Support wide decimals in DecimalBytePartsArray and kernels Sep 9, 2026
@mhk197
mhk197 marked this pull request as ready for review September 9, 2026 04:17
@mhk197
mhk197 removed this pull request from stack #9811 September 9, 2026 15:09
@mhk197
mhk197 added this pull request to stack #9813 September 9, 2026 15:09
@mhk197 mhk197 added changelog/skip Do not list PR in the changelog and removed changelog/skip Do not list PR in the changelog labels Sep 9, 2026
Base automatically changed from mk/dbp-parts to mk/dbp-v2-feature September 10, 2026 15:15
Represent wide decimals with a signed high part and up to three unsigned
low parts. Add validation, execution, kernel support, property tests, and
assembly benchmarks while keeping serialization on the frozen format.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Move array helpers onto a crate-private extension trait, preserve decimal
precision and scale when replacing the MSP, and group slicing with the
other compute operations. Inline canonical execution and select scalar
storage directly from the lower-part count.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
.cast(array.msp().dtype().with_nullability(*target_nullability))?;

Ok(Some(
DecimalByteParts::try_new(new_msp, *target_decimal)?.into_array(),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this was redundant to begin with because we check above that target dtype is same as current modulo nullability

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/skip Do not list PR in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant