Reduce allocations when reading small-precision decimals in the Spark connector

[xiaoh1024]

Problem or motivation

VortexArrowColumnVector.DecimalAccessor currently reads every non-null Decimal128 value through:

Decimal.apply(accessor.getObject(rowId), precision, scale)

Arrow Java's getObject() reconstructs a BigDecimal through a temporary byte array and BigInteger. This conversion is also used for columns with precision 1–18, whose valid unscaled integers fit in a Java long. These per-value allocations add overhead when Spark reads decimal columns.

The relevant code is in VortexArrowColumnVector.java. I have a local prototype and would like feedback on the approach before opening a PR.

Proposed solution

Use a specialized accessor for source columns with precision 1–18:

• Read the two native 64-bit words of the Decimal128 value, accounting for native byte order.
• Check that the high word is the sign extension of the low word. If so, construct a BigDecimal with BigDecimal.valueOf(unscaled, sourceScale) and pass it to the existing Decimal.apply(value, requestedPrecision, requestedScale) conversion.
• Use Arrow's existing full-width conversion when the stored integer does not fit in a signed long. Wider-precision columns keep the existing accessor.

Spark would continue to handle requested rescaling, rounding, and precision overflow. The prototype retains Spark's expanded Decimal representation to preserve checked integer-cast behavior. Arrow's Decimal128 storage format remains unchanged.

Additional context

Preliminary JMH measurements compare the prototype with a13c246c4985e54362319e789a5a9eea50444ef9. The consumer writes the returned decimals to an UnsafeRow. Each Spark version/scenario has three baseline/candidate pairs, with one measured JVM per variant per pair, five 1-second warmup iterations and ten 1-second measurement iterations. Batches contain 4,096 non-null values with alternating signs.

Environment: Amazon Corretto 17.0.16, G1, 512 MiB heap, shared x86_64 KVM host with non-exclusive CPU affinity. Spark versions are 3.5.9/Scala 2.12 and 4.1.2/Scala 2.13.

Source → requested decimal	Spark 3.5.9, baseline → prototype (ns/value)	Spark 4.1.2, baseline → prototype (ns/value)	Allocation, both versions (B/value)
(18, 2) → (18, 2)	68.83 → 35.98	66.35 → 35.55	192 → 160
(18, 2) → (18, 1)	96.64 → 54.93	97.50 → 56.57	320 → 224
(38, 2) → (38, 2)	99.86 → 99.66	117.74 → 118.21	232 → 232

The prototype passes 26 targeted tests on each Spark version, covering precision/scale changes, rounding and overflow, malformed buffers, slices, object independence, negative scales, and checked integer casts. Both native-word layouts are exercised in tests. The same 26 tests per Spark version also pass on a big-endian s390x Temurin 17.0.16 JVM running under QEMU.

Related discussion: #8197 considered narrow Arrow Decimal exports and the JNI/Spark Decimal128 assumptions. This proposal optimizes the Java conversion while retaining Decimal128 input.

AI assistance was used to develop the prototype, tests, and this issue draft.

[robert3005]

I think that’s a fair improvement. Though iirc this models what upstream spark does

[xiaoh1024]

I think that’s a fair improvement. Though iirc this models what upstream spark does

Thanks for pointing that out. I checked upstream Spark, and ArrowColumnVector does use the same conversion path. I’ve opened SPARK-59443 and am preparing a PR to apply the same optimization there as well.