Support wide decimals in DecimalBytePartsArray and kernels - #9809
Conversation
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | random_i16[0.95] |
77.2 µs | 94.2 µs | -18.09% |
| ❌ | Simulation | decompress[datetime_for_bp] |
160.3 µs | 193.6 µs | -17.19% |
| ⚡ | Simulation | random_i8[0.5] |
91.3 µs | 66.7 µs | +36.93% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
| ⚡ | Simulation | chunked_bool_canonical_into[(1000, 10)] |
30.7 µs | 27.2 µs | +12.9% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
575.7 ns | 521.6 ns | +10.39% |
| 🆕 | WallTime | dbp_assemble_neon[(I128, 1024)] |
N/A | 748 ns | N/A |
| 🆕 | WallTime | dbp_assemble_neon[(I128, 8192)] |
N/A | 3.7 µs | N/A |
| 🆕 | WallTime | dbp_assemble_neon[(I256, 1024)] |
N/A | 2.1 µs | N/A |
| 🆕 | WallTime | dbp_assemble_neon[(I256, 8192)] |
N/A | 15.1 µs | N/A |
| 🆕 | WallTime | dbp_assemble_neon[(I64, 1024)] |
N/A | 105 ns | N/A |
| 🆕 | WallTime | dbp_assemble_neon[(I64, 8192)] |
N/A | 110 ns | N/A |
| 🆕 | WallTime | dbp_split_all_null_neon[(I128, 1024)] |
N/A | 241 ns | N/A |
| 🆕 | WallTime | dbp_split_all_null_neon[(I128, 8192)] |
N/A | 244 ns | N/A |
| 🆕 | WallTime | dbp_split_all_null_neon[(I256, 1024)] |
N/A | 250 ns | N/A |
| 🆕 | WallTime | dbp_split_all_null_neon[(I256, 8192)] |
N/A | 253 ns | N/A |
| 🆕 | WallTime | dbp_split_all_null_neon[(I64, 1024)] |
N/A | 286 ns | N/A |
| 🆕 | WallTime | dbp_split_all_null_neon[(I64, 8192)] |
N/A | 291 ns | N/A |
| 🆕 | WallTime | dbp_split_all_valid_neon[(I128, 1024)] |
N/A | 742 ns | N/A |
| 🆕 | WallTime | dbp_split_all_valid_neon[(I128, 8192)] |
N/A | 4.3 µs | N/A |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/dbp-array (b2e49fb) with mk/dbp-v2-feature (4edb7aa)
Footnotes
-
204 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩
f7dfebe to
a566a28
Compare
DecimalBytePartsArray and kernels
Represent wide decimals with a signed high part and up to three unsigned low parts. Add validation, execution, kernel support, property tests, and assembly benchmarks while keeping serialization on the frozen format. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Move array helpers onto a crate-private extension trait, preserve decimal precision and scale when replacing the MSP, and group slicing with the other compute operations. Inline canonical execution and select scalar storage directly from the lower-part count. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
7283d4f to
a20094e
Compare
| .cast(array.msp().dtype().with_nullability(*target_nullability))?; | ||
|
|
||
| Ok(Some( | ||
| DecimalByteParts::try_new(new_msp, *target_decimal)?.into_array(), |
There was a problem hiding this comment.
I think this was redundant to begin with because we check above that target dtype is same as current modulo nullability
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
DecimalBytePartsArraypreviously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability.The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces:
i8/i16/i32/i64i128i64MSP + oneu64lower parti256i64MSP + threeu64lower partsOnly the MSP carries validity. Every lower part must be a non-nullable
u64array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remainArrayRefs, so their individual encodings are independent of the decimal representation.execute::<DecimalArray>reassembles aDecimalArrayfrom the MSP and lower parts children.takewith nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts.