diff --git a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md index db2b1dff8b..b9fea8998f 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md +++ b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md @@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support: Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting. -There is an [epic](https://github.com/apache/datafusion-comet/issues/313) where we are tracking the work to fully implement ANSI support. - ## String to Decimal Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's behavior, @@ -160,5 +158,3 @@ as `"1.23E+4"`). - -See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details. diff --git a/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md index 4c5d5c9db0..ea9eb10e37 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md +++ b/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md @@ -19,16 +19,12 @@ under the License. # Date/Time Expressions -- **Hour, Minute, Second**: Incorrectly apply timezone conversion to TimestampNTZ inputs. TimestampNTZ stores local - time without timezone, so no conversion should be applied. These expressions work correctly with Timestamp inputs. - [#3180](https://github.com/apache/datafusion-comet/issues/3180) - **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and routes through the JVM codegen dispatcher by default, producing Spark-identical results. The native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; see "Date and Time Functions" below) and can be enabled by setting `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled correctly regardless of session timezone (timezone-independent truncation). - [#2649](https://github.com/apache/datafusion-comet/issues/2649) ## Date and Time Functions diff --git a/docs/source/user-guide/latest/compatibility/operators.md b/docs/source/user-guide/latest/compatibility/operators.md index 28c0954047..eec21d283b 100644 --- a/docs/source/user-guide/latest/compatibility/operators.md +++ b/docs/source/user-guide/latest/compatibility/operators.md @@ -44,7 +44,7 @@ incorrect result. When any single window expression in a `WindowExec` falls back and `sum` / `avg` on year-month or day-time interval input types. Windowed aggregates inherit the same input-type support as the batch aggregates, so these fall back in both contexts. - `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on - overflow instead of returning Spark's `NULL` ([#4729](https://github.com/apache/datafusion-comet/issues/4729)). + overflow instead of returning Spark's `NULL`. - `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL` ([#4834](https://github.com/apache/datafusion-comet/issues/4834)). - `first_value` / `last_value` on a `RANGE` frame with a literal offset diff --git a/docs/source/user-guide/latest/compatibility/scans.md b/docs/source/user-guide/latest/compatibility/scans.md index 4f2f14a944..184ae9f06b 100644 --- a/docs/source/user-guide/latest/compatibility/scans.md +++ b/docs/source/user-guide/latest/compatibility/scans.md @@ -43,15 +43,11 @@ The following features are not supported and cause Comet to fall back to Spark: - No support for `input_file_name()`, `input_file_block_start()`, or `input_file_block_length()` SQL functions. Comet's Parquet scan does not use Spark's `FileScanRDD`, so these functions cannot populate their values. - No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to `true` -- Duplicate field names in case-insensitive mode (e.g., a Parquet file with both `B` and `b` columns) - are detected at read time and raise a `SparkRuntimeException` with error class `_LEGACY_ERROR_TEMP_2093`, - matching Spark's behavior. - `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized reader opts into Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which Comet's native reader does not replicate. By default Comet falls back to Spark in this case. Set `spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to running the - Comet Parquet scan regardless. See - [#4352](https://github.com/apache/datafusion-comet/issues/4352). + Comet Parquet scan regardless. The following limitation may produce incorrect results without falling back to Spark: @@ -68,13 +64,6 @@ The following limitations raise an error at scan time rather than falling back t non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`. Disable Comet for the query, or cast the column to `BINARY` before persisting, if you need to preserve non-UTF-8 bytes. See [#4121](https://github.com/apache/datafusion-comet/issues/4121). -- Reading `TimestampLTZ` as `TimestampNTZ` on Spark 3.x. Spark raises an error per - [SPARK-36182](https://issues.apache.org/jira/browse/SPARK-36182) because LTZ encodes UTC-adjusted - instants that cannot be safely reinterpreted as timezone-free values, and Comet matches this by - rejecting the read. This applies to all LTZ physical encodings (INT96, TIMESTAMP_MICROS, - TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted - ([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's - behavior. See [#4219](https://github.com/apache/datafusion-comet/issues/4219). ### Schema Mismatch Handling diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md index a6ad5bcf16..5715fef765 100644 --- a/docs/source/user-guide/latest/compatibility/spark-versions.md +++ b/docs/source/user-guide/latest/compatibility/spark-versions.md @@ -60,8 +60,7 @@ Spark 4.0.2 is supported with Java 17 and Scala 2.13. ### Known Limitations -- **Collation support** ([#1947](https://github.com/apache/datafusion-comet/issues/1947), - [#4051](https://github.com/apache/datafusion-comet/issues/4051)): Spark 4.0 introduced collation +- **Collation support**: Spark 4.0 introduced collation support. Non-default collated strings are not yet supported by Comet and will fall back to Spark. ## Spark 4.1