Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support:

Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.

There is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support.

## String to Decimal

Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior,
Expand Down Expand Up @@ -160,5 +158,3 @@ as `"1.23E+4"`).

<!--BEGIN:CAST_ANSI_TABLE-->
<!--END:CAST_ANSI_TABLE-->

See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,12 @@ under the License.

# Date/Time Expressions

- **Hour, Minute, Second**: Incorrectly apply timezone conversion to TimestampNTZ inputs. TimestampNTZ stores local
time without timezone, so no conversion should be applied. These expressions work correctly with Timestamp inputs.
[#3180](https://github.com/apache/datafusion-comet/issues/3180)
- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and
routes through the JVM codegen dispatcher by default, producing Spark-identical results. The
native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100;
see "Date and Time Functions" below) and can be enabled by setting
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled
correctly regardless of session timezone (timezone-independent truncation).
[#2649](https://github.com/apache/datafusion-comet/issues/2649)

## Date and Time Functions

Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/latest/compatibility/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ incorrect result. When any single window expression in a `WindowExec` falls back
and `sum` / `avg` on year-month or day-time interval input types. Windowed aggregates inherit the same input-type
support as the batch aggregates, so these fall back in both contexts.
- `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on
overflow instead of returning Spark's `NULL` ([#4729](https://github.com/apache/datafusion-comet/issues/4729)).
overflow instead of returning Spark's `NULL`.
- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL`
([#4834](https://github.com/apache/datafusion-comet/issues/4834)).
- `first_value` / `last_value` on a `RANGE` frame with a literal offset
Expand Down
13 changes: 1 addition & 12 deletions docs/source/user-guide/latest/compatibility/scans.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,11 @@ The following features are not supported and cause Comet to fall back to Spark:
- No support for `input_file_name()`, `input_file_block_start()`, or `input_file_block_length()` SQL functions.
Comet's Parquet scan does not use Spark's `FileScanRDD`, so these functions cannot populate their values.
- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to `true`
- Duplicate field names in case-insensitive mode (e.g., a Parquet file with both `B` and `b` columns)
are detected at read time and raise a `SparkRuntimeException` with error class `_LEGACY_ERROR_TEMP_2093`,
matching Spark's behavior.
- `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized reader opts into
Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which Comet's native reader
does not replicate. By default Comet falls back to Spark in this case. Set
`spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to running the
Comet Parquet scan regardless. See
[#4352](https://github.com/apache/datafusion-comet/issues/4352).
Comet Parquet scan regardless.

The following limitation may produce incorrect results without falling back to Spark:

Expand All @@ -68,13 +64,6 @@ The following limitations raise an error at scan time rather than falling back t
non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`. Disable Comet for the
query, or cast the column to `BINARY` before persisting, if you need to preserve non-UTF-8 bytes.
See [#4121](https://github.com/apache/datafusion-comet/issues/4121).
- Reading `TimestampLTZ` as `TimestampNTZ` on Spark 3.x. Spark raises an error per
[SPARK-36182](https://issues.apache.org/jira/browse/SPARK-36182) because LTZ encodes UTC-adjusted
instants that cannot be safely reinterpreted as timezone-free values, and Comet matches this by
rejecting the read. This applies to all LTZ physical encodings (INT96, TIMESTAMP_MICROS,
TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's
behavior. See [#4219](https://github.com/apache/datafusion-comet/issues/4219).

### Schema Mismatch Handling

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,7 @@ Spark 4.0.2 is supported with Java 17 and Scala 2.13.

### Known Limitations

- **Collation support** ([#1947](https://github.com/apache/datafusion-comet/issues/1947),
[#4051](https://github.com/apache/datafusion-comet/issues/4051)): Spark 4.0 introduced collation
- **Collation support**: Spark 4.0 introduced collation
support. Non-default collated strings are not yet supported by Comet and will fall back to Spark.

## Spark 4.1
Expand Down