[python] Preserve high-precision DECIMAL values in the row file format - #10124
Conversation
| finally: | ||
| os.unlink(path) | ||
|
|
||
| def test_high_precision_decimal(self): |
There was a problem hiding this comment.
Nit, not a blocker
This test changes the writer and reader together, so a symmetric scaling mistake in both paths could still round-trip successfully. Since row files are shared with the Java implementation, could we add one independent assertion - either decode a hand-built signed unscaled byte sequence for DECIMAL(38, 10) , or inspect the writer’s encoded bytes and compare them with the exact expected unscaled integer? That would directly protect cross-language wire compatibility rather than only the Python writer/reader pair
Reading and writing a DECIMAL column in the row file format both used Python decimal arithmetic under the default context, whose precision is 28 significant digits. For DECIMAL(p) with p greater than 28, the unscaled value has up to 38 significant digits, so the writer's value * 10^scale and the reader's Decimal(unscaled) / Decimal(10^scale) silently rounded it to 28 digits: a DECIMAL(38, 0) value 12345678901234567890123456789012345678 round-tripped as 12345678901234567890123456790000000000. Compute the unscaled value on write and rescale it on read under a context wide enough for the column (max(precision + abs(scale), 38)) using scaleb, mirroring pypaimon/data/decimal.py. DECIMAL(p) with p <= 18 is unchanged.
… DECIMAL Decode a hand-built signed unscaled byte sequence for DECIMAL(38, 10) and assert the exact value, so a symmetric scaling mistake in the writer and reader cannot round-trip undetected and the row-file wire form stays compatible with the Java implementation.
98e78ae to
c6d02e2
Compare
|
This fixes silent high-precision DECIMAL corruption in the row format. I reviewed both write-side unscaled conversion and read-side rescaling under Local verification on the PR patch: all 27 |
Purpose
Reading and writing a
DECIMALcolumn in therowfile format both use Python decimal arithmetic under the default context (28 significant digits). ForDECIMAL(p)withp > 28, the unscaled integer has up to 38 digits, so:int(value * (10 ** scale))rounds to 28 digits before encoding, andDecimal(unscaled) / Decimal(10 ** scale)rounds to 28 digits after decoding.A
DECIMAL(38, 0)value12345678901234567890123456789012345678round-trips as12345678901234567890123456790000000000— silent data corruption.The fix computes the unscaled value (write) and rescales it (read) under a context wide enough for the column —
max(precision + abs(scale), 38)withscaleb— mirroring the existingpypaimon/data/decimal.py.DECIMAL(p <= 18)is unchanged.Tests
test_format_row_reader_writer.test_high_precision_decimalround-trips positive, negative and null values acrossDECIMAL(38,0),DECIMAL(38,10),DECIMAL(28,4)andDECIMAL(18,2)and asserts exact values. Fails on master (DECIMAL(38,0)reads back rounded to 28 digits); passes here.