[python] Cast WHERE literals for DECIMAL, DATE and TIME columns - #10119
jackylee-ch wants to merge 1 commit into
Conversation
The WHERE-clause parser cast DECIMAL literals to float and left DATE/TIME
literals as strings, so string-based reads (pypaimon table read --where,
MultimodalTable.query().where) mishandled those columns. A DECIMAL column reads
back as decimal.Decimal, so price = 99.99 compared Decimal against float, never
matched, and silently returned zero rows. DATE/TIME columns raised in the Arrow
comparison kernel ("Function 'equal' has no kernel matching input types
(date32[day], string)") instead of filtering, and the CLI only catches
ValueError.
Cast DECIMAL/NUMERIC/DEC to decimal.Decimal and DATE/TIME to
datetime.date/datetime.time, matching how the sibling predicate_json_parser
already converts them. The pushed-down Arrow filter binds the literal's own
scale, so a DECIMAL literal is rescaled to the column scale when exact (50
matches a DECIMAL(10, 2) 50.00) and kept as written otherwise (a finer literal
like 99.999 matches nothing rather than rounding into a false hit). A malformed
decimal literal is re-raised as ValueError, which parse_where_clause documents
and the CLI relies on.
TIMESTAMP is left as-is: its LOCAL TIME ZONE form reads back tz-aware and needs
dedicated normalization, so it is out of scope here.
| match = re.search(r'\(\s*\d+\s*,\s*(\d+)\s*\)', type_name) | ||
| scale = int(match.group(1)) if match else 0 | ||
| try: | ||
| rescaled = value.quantize(decimal.Decimal(1).scaleb(-scale)) |
There was a problem hiding this comment.
Python’s default decimal context has precision 28, while Paimon supports DECIMAL precision up to 38. For a DECIMAL(38,2) value such as 123456789012345678901234567890123456.00 , querying with the exact integer-form literal causes quantize to raise InvalidOperation ; this catch returns the scale-0 value, and Arrow dataset filtering silently returns zero rows. Please quantize inside a local context whose precision is at least the declared column precision, and add a DECIMAL(38,2) regression where the integer-form literal must match the stored scale-2 value.
background/reproduced: the integer-form literal returned 0 rows, while the same literal ending in .00 returned 1.
| # filtering. TIMESTAMP is intentionally left out: its LOCAL TIME ZONE form | ||
| # reads back tz-aware and needs dedicated normalization. | ||
| return datetime.date.fromisoformat(value_str) | ||
| if base_type == 'TIME': |
There was a problem hiding this comment.
date.fromisoformat and time.fromisoformat were introduced in Python 3.7, but this package still declares python_requires >=3.6 and runs a Python 3.6 CI job. On 3.6, every DATE or TIME WHERE literal now raises AttributeError . The green 3.6 job does not catch this because it runs only the curated py36 subset and excludes where_parser_test.py . Could these literals be parsed through a 3.6-compatible helper and covered by the 3.6 subset?
| # reads back tz-aware and needs dedicated normalization. | ||
| return datetime.date.fromisoformat(value_str) | ||
| if base_type == 'TIME': | ||
| return datetime.time.fromisoformat(value_str) |
There was a problem hiding this comment.
same as above; time.fromisoformat accepts values such as 12:30:00+01:00 , but Paimon TIME has no timezone. Arrow then compares only the wall-clock value, so this literal currently matches a stored 12:30:00 while silently discarding +01:00 . Please reject parsed values with non-null tzinfo (or otherwise define explicit normalization) and add a test proving invalid offset-bearing TIME input fails with ValueError
2 participants
Purpose
The WHERE parser (
table read --where,MultimodalTable.query().where(...)) cast DECIMAL literals tofloatand left DATE/TIME literals as strings:DECIMALreads back asdecimal.Decimal, soprice = 99.99comparedDecimaltofloatand silently returned zero rows.DATE/TIMEraised in the Arrowequalkernel instead of filtering._cast_literalnow casts DECIMAL todecimal.Decimaland DATE/TIME todatetime.date/time, matching the siblingpredicate_json_parser. Since the pushed-down filter binds the literal's own scale, a decimal is rescaled to the column scale when exact, else kept (50matches50.00; a finer99.999matches nothing, not rounded); malformed decimals re-raiseValueError.TIMESTAMPstays out of scope (LOCAL TIME ZONE reads back tz-aware).Tests
WhereParserScanTestwrites aDECIMAL(10,2)/DATE/TIMEtable and asserts each WHERE returns the right rows, including integer and over-precise decimals; on master the decimal case returns[]and date/time raiseArrowNotImplementedError.Written with Claude Code; verification is mine.