You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/29 00:59:45 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by ce84a24c198688d980b5ff3493c74fc91764215f
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bd12e9631f Publish built docs triggered by ce84a24c198688d980b5ff3493c74fc91764215f
bd12e9631f is described below
commit bd12e9631f4025af3c8e94e94154680ffb0863b8
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Jul 29 00:59:42 2026 +0000
Publish built docs triggered by ce84a24c198688d980b5ff3493c74fc91764215f
---
.../expressions/spark-3.4/datetime.md.txt | 12 +++++
.../compatibility/expressions/spark-3.4/map.md.txt | 8 +++
.../expressions/spark-3.4/math.md.txt | 12 +++++
.../expressions/spark-3.4/misc.md.txt | 4 ++
.../expressions/spark-3.4/string.md.txt | 1 +
.../expressions/spark-3.5/datetime.md.txt | 12 +++++
.../compatibility/expressions/spark-3.5/map.md.txt | 8 +++
.../expressions/spark-3.5/math.md.txt | 12 +++++
.../expressions/spark-3.5/misc.md.txt | 4 ++
.../expressions/spark-3.5/string.md.txt | 1 +
.../expressions/spark-4.0/datetime.md.txt | 10 ++++
.../compatibility/expressions/spark-4.0/map.md.txt | 8 +++
.../expressions/spark-4.0/math.md.txt | 12 +++++
.../expressions/spark-4.0/misc.md.txt | 4 ++
.../expressions/spark-4.0/string.md.txt | 1 +
.../expressions/spark-4.1/datetime.md.txt | 10 ++++
.../compatibility/expressions/spark-4.1/map.md.txt | 8 +++
.../expressions/spark-4.1/math.md.txt | 12 +++++
.../expressions/spark-4.1/misc.md.txt | 4 ++
.../expressions/spark-4.1/string.md.txt | 1 +
.../user-guide/latest/compatibility/index.md.txt | 56 ++++++++++++++++++++
.../user-guide/latest/compatibility/scans.md.txt | 14 +++--
searchindex.js | 2 +-
.../expressions/spark-3.4/datetime.html | 16 ++++++
.../compatibility/expressions/spark-3.4/map.html | 8 +++
.../compatibility/expressions/spark-3.4/math.html | 16 ++++++
.../compatibility/expressions/spark-3.4/misc.html | 4 ++
.../expressions/spark-3.4/string.html | 1 +
.../expressions/spark-3.5/datetime.html | 16 ++++++
.../compatibility/expressions/spark-3.5/map.html | 8 +++
.../compatibility/expressions/spark-3.5/math.html | 16 ++++++
.../compatibility/expressions/spark-3.5/misc.html | 4 ++
.../expressions/spark-3.5/string.html | 1 +
.../expressions/spark-4.0/datetime.html | 12 +++++
.../compatibility/expressions/spark-4.0/map.html | 8 +++
.../compatibility/expressions/spark-4.0/math.html | 16 ++++++
.../compatibility/expressions/spark-4.0/misc.html | 4 ++
.../expressions/spark-4.0/string.html | 1 +
.../expressions/spark-4.1/datetime.html | 12 +++++
.../compatibility/expressions/spark-4.1/map.html | 8 +++
.../compatibility/expressions/spark-4.1/math.html | 16 ++++++
.../compatibility/expressions/spark-4.1/misc.html | 4 ++
.../expressions/spark-4.1/string.html | 1 +
user-guide/latest/compatibility/index.html | 61 ++++++++++++++++++++++
user-guide/latest/compatibility/scans.html | 14 +++--
45 files changed, 454 insertions(+), 9 deletions(-)
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
index 5b5d157d4b..bfaeefa8c2 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
@@ -88,6 +88,18 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only `TimestampType` and `TimestampNTZType` inputs are supported.
+## MakeDate
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an out-of-range `(year, month, day)` triple surfaces as `CometNativeException` rather than Spark's `SparkDateTimeException` with error class `DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
+## NextDay
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an invalid `dayOfWeek` surfaces as `CometNativeException` rather than Spark's `SparkIllegalArgumentException` with error class `ILLEGAL_DAY_OF_WEEK`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
## PreciseTimestampConversion
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
index 59c923b8ed..ddbe8e5d18 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/map.md.txt
@@ -34,12 +34,20 @@ IEEE total ordering for floating-point, which differs from Spark's `Double.compa
## MapFromArrays
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- `BinaryType` is not supported as a map key in `map_from_entries`
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/math.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/math.md.txt
index 14487e18b8..7783506a36 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/math.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/math.md.txt
@@ -27,6 +27,12 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only integral, floating-point, and decimal types are supported
+## IntegralDivide
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input, `Long.MinValue div -1` does not overflow-check: Comet's decimal-backed path returns the wrapped value `Long.MinValue` under legacy mode and silently returns the same wrapped value under ANSI, where Spark raises `ARITHMETIC_OVERFLOW` ([#5065](https://github.com/apache/datafusion-comet/issues/5065)).
+
## Rand
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
@@ -38,4 +44,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- The `seed` argument must be a literal value
+
+## Round
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input with a negative `scale` of `-19` or lower, `10^(-scale)` overflows the native integer type and Comet returns `0` instead of Spark's overflowed value. Under ANSI mode Spark raises `ARITHMETIC_OVERFLOW` at the same input; Comet still returns `0` ([#5070](https://github.com/apache/datafusion-comet/issues/5070)).
<!--END:EXPR_COMPAT-->
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
index ed04e17891..8cb763f62f 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
@@ -35,6 +35,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
## MakeDecimal
+The following differences from Spark are always present and do not require any additional configuration:
+
+- The native implementation silently returns `NULL` on overflow regardless of the `nullOnOverflow` flag, so under ANSI mode Comet returns `NULL` where Spark raises `NUMERIC_VALUE_OUT_OF_RANGE`. `MakeDecimal` is inserted by the `DecimalAggregates` optimizer rule, so this affects `sum`/`avg` over low-precision decimals when the unscaled long overflows the target precision ([#5066](https://github.com/apache/datafusion-comet/issues/5066)).
+
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- Only `LongType` input is supported
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
index b165b2ab48..809ff65585 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
@@ -54,6 +54,7 @@ The following cases have no native implementation and always run in the JVM usin
By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- Spark allows single-quoted JSON and unescaped control characters which Comet does not support
+- For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet's native implementation returns the last occurrence ([#4947](https://github.com/apache/datafusion-comet/issues/4947))
## InitCap
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
index 5b5d157d4b..bfaeefa8c2 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
@@ -88,6 +88,18 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only `TimestampType` and `TimestampNTZType` inputs are supported.
+## MakeDate
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an out-of-range `(year, month, day)` triple surfaces as `CometNativeException` rather than Spark's `SparkDateTimeException` with error class `DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
+## NextDay
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an invalid `dayOfWeek` surfaces as `CometNativeException` rather than Spark's `SparkIllegalArgumentException` with error class `ILLEGAL_DAY_OF_WEEK`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
## PreciseTimestampConversion
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
index 59c923b8ed..ddbe8e5d18 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/map.md.txt
@@ -34,12 +34,20 @@ IEEE total ordering for floating-point, which differs from Spark's `Double.compa
## MapFromArrays
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- `BinaryType` is not supported as a map key in `map_from_entries`
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/math.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/math.md.txt
index 14487e18b8..7783506a36 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/math.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/math.md.txt
@@ -27,6 +27,12 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only integral, floating-point, and decimal types are supported
+## IntegralDivide
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input, `Long.MinValue div -1` does not overflow-check: Comet's decimal-backed path returns the wrapped value `Long.MinValue` under legacy mode and silently returns the same wrapped value under ANSI, where Spark raises `ARITHMETIC_OVERFLOW` ([#5065](https://github.com/apache/datafusion-comet/issues/5065)).
+
## Rand
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
@@ -38,4 +44,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- The `seed` argument must be a literal value
+
+## Round
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input with a negative `scale` of `-19` or lower, `10^(-scale)` overflows the native integer type and Comet returns `0` instead of Spark's overflowed value. Under ANSI mode Spark raises `ARITHMETIC_OVERFLOW` at the same input; Comet still returns `0` ([#5070](https://github.com/apache/datafusion-comet/issues/5070)).
<!--END:EXPR_COMPAT-->
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
index cf20bb6607..e62b869189 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
@@ -35,6 +35,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
## MakeDecimal
+The following differences from Spark are always present and do not require any additional configuration:
+
+- The native implementation silently returns `NULL` on overflow regardless of the `nullOnOverflow` flag, so under ANSI mode Comet returns `NULL` where Spark raises `NUMERIC_VALUE_OUT_OF_RANGE`. `MakeDecimal` is inserted by the `DecimalAggregates` optimizer rule, so this affects `sum`/`avg` over low-precision decimals when the unscaled long overflows the target precision ([#5066](https://github.com/apache/datafusion-comet/issues/5066)).
+
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- Only `LongType` input is supported
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
index b165b2ab48..809ff65585 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
@@ -54,6 +54,7 @@ The following cases have no native implementation and always run in the JVM usin
By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- Spark allows single-quoted JSON and unescaped control characters which Comet does not support
+- For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet's native implementation returns the last occurrence ([#4947](https://github.com/apache/datafusion-comet/issues/4947))
## InitCap
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
index 9cf5333c96..567f7840b9 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
@@ -90,6 +90,12 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only `TimestampType` and `TimestampNTZType` inputs are supported.
+## MakeDate
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an out-of-range `(year, month, day)` triple surfaces as `CometNativeException` rather than Spark's `SparkDateTimeException` with error class `DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
## MakeTimestamp
By default, `MakeTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
@@ -98,6 +104,10 @@ By default, `MakeTimestamp` is evaluated in the JVM using Spark's own code-gener
## NextDay
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an invalid `dayOfWeek` surfaces as `CometNativeException` rather than Spark's `SparkIllegalArgumentException` with error class `ILLEGAL_DAY_OF_WEEK`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
The following incompatibilities cause `NextDay` to fall back to Spark by default. Set `spark.comet.expression.NextDay.allowIncompatible=true` to enable Comet acceleration despite these differences.
- next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646)
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
index e5c0f79d10..85ce8af197 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/map.md.txt
@@ -34,12 +34,20 @@ IEEE total ordering for floating-point, which differs from Spark's `Double.compa
## MapFromArrays
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- `BinaryType` is not supported as a map key in `map_from_entries`
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/math.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/math.md.txt
index 14487e18b8..7783506a36 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/math.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/math.md.txt
@@ -27,6 +27,12 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only integral, floating-point, and decimal types are supported
+## IntegralDivide
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input, `Long.MinValue div -1` does not overflow-check: Comet's decimal-backed path returns the wrapped value `Long.MinValue` under legacy mode and silently returns the same wrapped value under ANSI, where Spark raises `ARITHMETIC_OVERFLOW` ([#5065](https://github.com/apache/datafusion-comet/issues/5065)).
+
## Rand
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
@@ -38,4 +44,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- The `seed` argument must be a literal value
+
+## Round
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input with a negative `scale` of `-19` or lower, `10^(-scale)` overflows the native integer type and Comet returns `0` instead of Spark's overflowed value. Under ANSI mode Spark raises `ARITHMETIC_OVERFLOW` at the same input; Comet still returns `0` ([#5070](https://github.com/apache/datafusion-comet/issues/5070)).
<!--END:EXPR_COMPAT-->
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
index cf20bb6607..e62b869189 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
@@ -35,6 +35,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
## MakeDecimal
+The following differences from Spark are always present and do not require any additional configuration:
+
+- The native implementation silently returns `NULL` on overflow regardless of the `nullOnOverflow` flag, so under ANSI mode Comet returns `NULL` where Spark raises `NUMERIC_VALUE_OUT_OF_RANGE`. `MakeDecimal` is inserted by the `DecimalAggregates` optimizer rule, so this affects `sum`/`avg` over low-precision decimals when the unscaled long overflows the target precision ([#5066](https://github.com/apache/datafusion-comet/issues/5066)).
+
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- Only `LongType` input is supported
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
index 21d1b7a90a..303c5e047d 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
@@ -54,6 +54,7 @@ The following cases have no native implementation and always run in the JVM usin
By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- Spark allows single-quoted JSON and unescaped control characters which Comet does not support
+- For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet's native implementation returns the last occurrence ([#4947](https://github.com/apache/datafusion-comet/issues/4947))
## InitCap
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
index 9cf5333c96..567f7840b9 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
@@ -90,6 +90,12 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only `TimestampType` and `TimestampNTZType` inputs are supported.
+## MakeDate
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an out-of-range `(year, month, day)` triple surfaces as `CometNativeException` rather than Spark's `SparkDateTimeException` with error class `DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
## MakeTimestamp
By default, `MakeTimestamp` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MakeTimestamp.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
@@ -98,6 +104,10 @@ By default, `MakeTimestamp` is evaluated in the JVM using Spark's own code-gener
## NextDay
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Under ANSI mode, an invalid `dayOfWeek` surfaces as `CometNativeException` rather than Spark's `SparkIllegalArgumentException` with error class `ILLEGAL_DAY_OF_WEEK`. The throw/NULL decision is correct; only the exception class and error class differ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+
The following incompatibilities cause `NextDay` to fall back to Spark by default. Set `spark.comet.expression.NextDay.allowIncompatible=true` to enable Comet acceleration despite these differences.
- next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646)
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
index e5c0f79d10..85ce8af197 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/map.md.txt
@@ -34,12 +34,20 @@ IEEE total ordering for floating-point, which differs from Spark's `Double.compa
## MapFromArrays
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
The following incompatibilities cause `MapFromArrays` to fall back to Spark by default. Set `spark.comet.expression.MapFromArrays.allowIncompatible=true` to enable Comet acceleration despite these differences.
- `spark.sql.mapKeyDedupPolicy` is set to `LAST_WIN`; Comet's native map construction does not implement LAST_WIN dedup semantics.
## MapFromEntries
+The following differences from Spark are always present and do not require any additional configuration:
+
+- Spark rejects a `NULL` element inside the keys array with a `RuntimeException` (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries` does not detect a per-element `NULL` key and produces a map with a `NULL` key instead ([#4680](https://github.com/apache/datafusion-comet/issues/4680)).
+
By default, `MapFromEntries` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.MapFromEntries.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- `BinaryType` is not supported as a map key in `map_from_entries`
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/math.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/math.md.txt
index 14487e18b8..7783506a36 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/math.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/math.md.txt
@@ -27,6 +27,12 @@ The following cases are not supported by Comet and always fall back to Spark, re
- Only integral, floating-point, and decimal types are supported
+## IntegralDivide
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input, `Long.MinValue div -1` does not overflow-check: Comet's decimal-backed path returns the wrapped value `Long.MinValue` under legacy mode and silently returns the same wrapped value under ANSI, where Spark raises `ARITHMETIC_OVERFLOW` ([#5065](https://github.com/apache/datafusion-comet/issues/5065)).
+
## Rand
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
@@ -38,4 +44,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- The `seed` argument must be a literal value
+
+## Round
+
+The following differences from Spark are always present and do not require any additional configuration:
+
+- On `LongType` input with a negative `scale` of `-19` or lower, `10^(-scale)` overflows the native integer type and Comet returns `0` instead of Spark's overflowed value. Under ANSI mode Spark raises `ARITHMETIC_OVERFLOW` at the same input; Comet still returns `0` ([#5070](https://github.com/apache/datafusion-comet/issues/5070)).
<!--END:EXPR_COMPAT-->
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
index cf20bb6607..e62b869189 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
@@ -35,6 +35,10 @@ The following cases are not supported by Comet and always fall back to Spark, re
## MakeDecimal
+The following differences from Spark are always present and do not require any additional configuration:
+
+- The native implementation silently returns `NULL` on overflow regardless of the `nullOnOverflow` flag, so under ANSI mode Comet returns `NULL` where Spark raises `NUMERIC_VALUE_OUT_OF_RANGE`. `MakeDecimal` is inserted by the `DecimalAggregates` optimizer rule, so this affects `sum`/`avg` over low-precision decimals when the unscaled long overflows the target precision ([#5066](https://github.com/apache/datafusion-comet/issues/5066)).
+
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
- Only `LongType` input is supported
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
index 21d1b7a90a..303c5e047d 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
@@ -54,6 +54,7 @@ The following cases have no native implementation and always run in the JVM usin
By default, `GetJsonObject` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.GetJsonObject.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
- Spark allows single-quoted JSON and unescaped control characters which Comet does not support
+- For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet's native implementation returns the last occurrence ([#4947](https://github.com/apache/datafusion-comet/issues/4947))
## InitCap
diff --git a/_sources/user-guide/latest/compatibility/index.md.txt b/_sources/user-guide/latest/compatibility/index.md.txt
index b08d9a3bdb..95f496b57f 100644
--- a/_sources/user-guide/latest/compatibility/index.md.txt
+++ b/_sources/user-guide/latest/compatibility/index.md.txt
@@ -102,3 +102,59 @@ string paths is tracked by
Separately, Comet's native Parquet scan currently rejects string columns whose stored bytes are not
valid UTF-8 rather than reading them like Spark
([#4121](https://github.com/apache/datafusion-comet/issues/4121)).
+
+## ANSI-mode error classes and messages
+
+Under `spark.sql.ansi.enabled=true`, several native error paths raise the error at the correct
+input but with a different exception type, error class, SQLSTATE, or message text than Spark.
+Code that catches `SparkException` and only asserts on message substrings is unaffected; code that
+inspects the exception class, `getCondition()`, or the parameterised error class will observe
+divergence:
+
+- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` where Spark
+ 4.1 raises `BINARY_ARITHMETIC_OVERFLOW`, and Long overflow surfaces as `"integer overflow"`
+ rather than `"long overflow"`. `Abs` uses Rust type names (`Int8`, `Int64`, ...) in the message
+ instead of Spark's SQL type names, and the scalar path of `UnaryMinus` on Byte / Short emits a
+ malformed message. The `try_` suggestion is omitted from all of these
+ ([#5071](https://github.com/apache/datafusion-comet/issues/5071)).
+- Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
+ raise raw Arrow errors that bypass `SparkErrorConverter` and surface as `CometNativeException`
+ rather than `SparkArithmeticException` with the proper error class and query context
+ ([#5072](https://github.com/apache/datafusion-comet/issues/5072)).
+- `next_day` and `make_date` throw at the correct inputs but surface as `CometNativeException`
+ instead of `SparkIllegalArgumentException [ILLEGAL_DAY_OF_WEEK]` /
+ `SparkDateTimeException [DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION]`
+ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+- Spark 4.2 introduced additional ANSI arithmetic overflow behavior differences that Comet does
+ not yet track ([#4967](https://github.com/apache/datafusion-comet/issues/4967)).
+
+## Known result-value divergences
+
+The following native paths silently return values that differ from Spark for edge-case inputs.
+Most also have entries in the per-category expression pages linked above; they are collected here
+so users hunting an unexpected value have a single place to check:
+
+- `CAST(boolean AS DECIMAL(p, s))` where `10^s` exceeds the target precision (e.g.
+ `DECIMAL(1, 1)`) throws `NUMERIC_VALUE_OUT_OF_RANGE` regardless of the eval mode. Spark returns
+ `NULL` under legacy and try mode, and only throws under ANSI
+ ([#5068](https://github.com/apache/datafusion-comet/issues/5068)).
+- `CAST(string AS boolean)` trims only Unicode whitespace. Spark also trims the ASCII control
+ bytes `0x00-0x08`, `0x0E-0x1F`, and `0x7F`, so a string like `"true"` casts to
+ `true` in Spark and to `NULL` in Comet (or throws under ANSI)
+ ([#4959](https://github.com/apache/datafusion-comet/issues/4959)).
+- Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from
+ Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns
+ ([#5022](https://github.com/apache/datafusion-comet/issues/5022)).
+- For an empty `IN` list, Comet always returns `false` for a `NULL` operand. Spark returns `NULL`
+ when `spark.sql.legacy.nullInEmptyListBehavior=true` (Spark 4.0+)
+ ([#4786](https://github.com/apache/datafusion-comet/issues/4786)).
+
+## Object store cache
+
+When Comet's native scan reads Parquet files, it caches one object store instance per
+`(scheme + host + port, hadoop-config-hash)` key. For `abfss://container@account.dfs.core.windows.net/...`
+URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
+account currently collide on the same cache entry. Within a single executor process, reading from
+a second container after a first can be served by the first container's store instance and return
+its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
+component. Tracked by [#4993](https://github.com/apache/datafusion-comet/issues/4993).
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt b/_sources/user-guide/latest/compatibility/scans.md.txt
index b7dca43ae8..0c07b5c237 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -51,10 +51,16 @@ The following features are not supported and cause Comet to fall back to Spark:
The following limitation may produce incorrect results without falling back to Spark:
-- No support for datetime rebasing. When reading Parquet files containing dates or timestamps written before
- Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps will be read as if they were
- written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before
- October 15, 1582.
+- No support for datetime rebasing. When reading Parquet files containing dates or timestamps
+ written with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY` (which is Spark's default for
+ data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as
+ if they were written using the Proleptic Gregorian calendar. This produces silently-wrong
+ values for dates before October 15, 1582 in both projections and predicates. Comet also
+ ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level
+ `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. The
+ `spark.comet.exceptionOnDatetimeRebase` config is currently dead code and does not raise on
+ legacy-calendar data. Tracked by
+ [#5010](https://github.com/apache/datafusion-comet/issues/5010).
The following limitations raise an error at scan time rather than falling back to Spark:
diff --git a/searchindex.js b/searchindex.js
index 0f761297c0..aa041b0396 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&": [[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/": [[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators (nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&": [[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/": [[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators (nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html b/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
index b150e5738f..5462e76787 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
@@ -822,6 +822,20 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li><p>Only <code class="docutils literal notranslate"><span class="pre">TimestampType</span></code> and <code class="docutils literal notranslate"><span class="pre">TimestampNTZType</span></code> inputs are supported.</p></li>
</ul>
</section>
+<section id="makedate">
+<h2>MakeDate<a class="headerlink" href="#makedate" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an out-of-range <code class="docutils literal notranslate"><span class="pre">(year,</span> <span class="pre">month,</span> <span class="pre">day)</span></code> triple surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkDateTimeException</span></code> with error class <code class="docutils literal notranslate"><span class="pr [...]
+</ul>
+</section>
+<section id="nextday">
+<h2>NextDay<a class="headerlink" href="#nextday" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an invalid <code class="docutils literal notranslate"><span class="pre">dayOfWeek</span></code> surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkIllegalArgumentException</span></code> with error class <code class="docutils literal notranslate"><span class="pre">ILLEGAL_DAY_OF_WEEK</span></code>. The throw/NULL decision is [...]
+</ul>
+</section>
<section id="precisetimestampconversion">
<h2>PreciseTimestampConversion<a class="headerlink" href="#precisetimestampconversion" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -930,6 +944,8 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromutctimestamp">FromUTCTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromunixtime">FromUnixTime</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hours">Hours</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedate">MakeDate</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#nextday">NextDay</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#precisetimestampconversion">PreciseTimestampConversion</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#secondstotimestamp">SecondsToTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toutctimestamp">ToUTCTimestamp</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/map.html b/user-guide/latest/compatibility/expressions/spark-3.4/map.html
index 551ba8f631..125ded5d75 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/map.html
@@ -761,6 +761,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromarrays">
<h2>MapFromArrays<a class="headerlink" href="#mapfromarrays" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
@@ -768,6 +772,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromentries">
<h2>MapFromEntries<a class="headerlink" href="#mapfromentries" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromEntries</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromEntries.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/math.html b/user-guide/latest/compatibility/expressions/spark-3.4/math.html
index 5c38d45e8a..8f63a85844 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/math.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/math.html
@@ -756,6 +756,13 @@ under the License.
<li><p>Only integral, floating-point, and decimal types are supported</p></li>
</ul>
</section>
+<section id="integraldivide">
+<h2>IntegralDivide<a class="headerlink" href="#integraldivide" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input, <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span> <span class="pre">div</span> <span class="pre">-1</span></code> does not overflow-check: Comet’s decimal-backed path returns the wrapped value <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span></code> under legacy mode and silently returns the same wrapped value under ANSI, whe [...]
+</ul>
+</section>
<section id="rand">
<h2>Rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -769,6 +776,13 @@ under the License.
<ul class="simple">
<li><p>The <code class="docutils literal notranslate"><span class="pre">seed</span></code> argument must be a literal value</p></li>
</ul>
+</section>
+<section id="round">
+<h2>Round<a class="headerlink" href="#round" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input with a negative <code class="docutils literal notranslate"><span class="pre">scale</span></code> of <code class="docutils literal notranslate"><span class="pre">-19</span></code> or lower, <code class="docutils literal notranslate"><span class="pre">10^(-scale)</span></code> overflows the native integer type and Comet returns <code class="docutils literal notranslate"><span class="pre">0</ [...]
+</ul>
<!--END:EXPR_COMPAT-->
</section>
</section>
@@ -821,8 +835,10 @@ under the License.
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#abs">Abs</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#integraldivide">IntegralDivide</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rand">Rand</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#randn">Randn</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#round">Round</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
index bff7b91854..0bf11317f5 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
@@ -765,6 +765,10 @@ under the License.
</section>
<section id="makedecimal">
<h2>MakeDecimal<a class="headerlink" href="#makedecimal" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>The native implementation silently returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> on overflow regardless of the <code class="docutils literal notranslate"><span class="pre">nullOnOverflow</span></code> flag, so under ANSI mode Comet returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> where Spark raises <code class="docutils literal notranslate"><span class="pre">NUMERIC_VALUE_OUT_OF_RANGE</span></code>. < [...]
+</ul>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
<ul class="simple">
<li><p>Only <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input is supported</p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/string.html b/user-guide/latest/compatibility/expressions/spark-3.4/string.html
index 127d0a53c0..b4f52ed6ea 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/string.html
@@ -786,6 +786,7 @@ under the License.
<p>By default, <code class="docutils literal notranslate"><span class="pre">GetJsonObject</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p>Spark allows single-quoted JSON and unescaped control characters which Comet does not support</p></li>
+<li><p>For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet’s native implementation returns the last occurrence (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4947">#4947</a>)</p></li>
</ul>
</section>
<section id="initcap">
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html b/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
index 65834520b1..36447ac08c 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
@@ -822,6 +822,20 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li><p>Only <code class="docutils literal notranslate"><span class="pre">TimestampType</span></code> and <code class="docutils literal notranslate"><span class="pre">TimestampNTZType</span></code> inputs are supported.</p></li>
</ul>
</section>
+<section id="makedate">
+<h2>MakeDate<a class="headerlink" href="#makedate" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an out-of-range <code class="docutils literal notranslate"><span class="pre">(year,</span> <span class="pre">month,</span> <span class="pre">day)</span></code> triple surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkDateTimeException</span></code> with error class <code class="docutils literal notranslate"><span class="pr [...]
+</ul>
+</section>
+<section id="nextday">
+<h2>NextDay<a class="headerlink" href="#nextday" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an invalid <code class="docutils literal notranslate"><span class="pre">dayOfWeek</span></code> surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkIllegalArgumentException</span></code> with error class <code class="docutils literal notranslate"><span class="pre">ILLEGAL_DAY_OF_WEEK</span></code>. The throw/NULL decision is [...]
+</ul>
+</section>
<section id="precisetimestampconversion">
<h2>PreciseTimestampConversion<a class="headerlink" href="#precisetimestampconversion" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -930,6 +944,8 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromutctimestamp">FromUTCTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromunixtime">FromUnixTime</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hours">Hours</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedate">MakeDate</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#nextday">NextDay</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#precisetimestampconversion">PreciseTimestampConversion</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#secondstotimestamp">SecondsToTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toutctimestamp">ToUTCTimestamp</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/map.html b/user-guide/latest/compatibility/expressions/spark-3.5/map.html
index 6aebe970af..7f33c1776b 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/map.html
@@ -761,6 +761,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromarrays">
<h2>MapFromArrays<a class="headerlink" href="#mapfromarrays" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
@@ -768,6 +772,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromentries">
<h2>MapFromEntries<a class="headerlink" href="#mapfromentries" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromEntries</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromEntries.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/math.html b/user-guide/latest/compatibility/expressions/spark-3.5/math.html
index 61b55a7815..d897f54a6a 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/math.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/math.html
@@ -756,6 +756,13 @@ under the License.
<li><p>Only integral, floating-point, and decimal types are supported</p></li>
</ul>
</section>
+<section id="integraldivide">
+<h2>IntegralDivide<a class="headerlink" href="#integraldivide" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input, <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span> <span class="pre">div</span> <span class="pre">-1</span></code> does not overflow-check: Comet’s decimal-backed path returns the wrapped value <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span></code> under legacy mode and silently returns the same wrapped value under ANSI, whe [...]
+</ul>
+</section>
<section id="rand">
<h2>Rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -769,6 +776,13 @@ under the License.
<ul class="simple">
<li><p>The <code class="docutils literal notranslate"><span class="pre">seed</span></code> argument must be a literal value</p></li>
</ul>
+</section>
+<section id="round">
+<h2>Round<a class="headerlink" href="#round" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input with a negative <code class="docutils literal notranslate"><span class="pre">scale</span></code> of <code class="docutils literal notranslate"><span class="pre">-19</span></code> or lower, <code class="docutils literal notranslate"><span class="pre">10^(-scale)</span></code> overflows the native integer type and Comet returns <code class="docutils literal notranslate"><span class="pre">0</ [...]
+</ul>
<!--END:EXPR_COMPAT-->
</section>
</section>
@@ -821,8 +835,10 @@ under the License.
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#abs">Abs</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#integraldivide">IntegralDivide</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rand">Rand</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#randn">Randn</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#round">Round</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
index 39a3df08b4..fb3e33a692 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
@@ -765,6 +765,10 @@ under the License.
</section>
<section id="makedecimal">
<h2>MakeDecimal<a class="headerlink" href="#makedecimal" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>The native implementation silently returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> on overflow regardless of the <code class="docutils literal notranslate"><span class="pre">nullOnOverflow</span></code> flag, so under ANSI mode Comet returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> where Spark raises <code class="docutils literal notranslate"><span class="pre">NUMERIC_VALUE_OUT_OF_RANGE</span></code>. < [...]
+</ul>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
<ul class="simple">
<li><p>Only <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input is supported</p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/string.html b/user-guide/latest/compatibility/expressions/spark-3.5/string.html
index 4ac73531bb..e118b1f307 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/string.html
@@ -786,6 +786,7 @@ under the License.
<p>By default, <code class="docutils literal notranslate"><span class="pre">GetJsonObject</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p>Spark allows single-quoted JSON and unescaped control characters which Comet does not support</p></li>
+<li><p>For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet’s native implementation returns the last occurrence (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4947">#4947</a>)</p></li>
</ul>
</section>
<section id="initcap">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html b/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
index 1928209afc..d3c562eda0 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
@@ -824,6 +824,13 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li><p>Only <code class="docutils literal notranslate"><span class="pre">TimestampType</span></code> and <code class="docutils literal notranslate"><span class="pre">TimestampNTZType</span></code> inputs are supported.</p></li>
</ul>
</section>
+<section id="makedate">
+<h2>MakeDate<a class="headerlink" href="#makedate" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an out-of-range <code class="docutils literal notranslate"><span class="pre">(year,</span> <span class="pre">month,</span> <span class="pre">day)</span></code> triple surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkDateTimeException</span></code> with error class <code class="docutils literal notranslate"><span class="pr [...]
+</ul>
+</section>
<section id="maketimestamp">
<h2>MakeTimestamp<a class="headerlink" href="#maketimestamp" title="Link to this heading">#</a></h2>
<p>By default, <code class="docutils literal notranslate"><span class="pre">MakeTimestamp</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MakeTimestamp.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
@@ -833,6 +840,10 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
</section>
<section id="nextday">
<h2>NextDay<a class="headerlink" href="#nextday" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an invalid <code class="docutils literal notranslate"><span class="pre">dayOfWeek</span></code> surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkIllegalArgumentException</span></code> with error class <code class="docutils literal notranslate"><span class="pre">ILLEGAL_DAY_OF_WEEK</span></code>. The throw/NULL decision is [...]
+</ul>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">NextDay</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.NextDay.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
<li><p>next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646)</p></li>
@@ -959,6 +970,7 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromutctimestamp">FromUTCTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromunixtime">FromUnixTime</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hours">Hours</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedate">MakeDate</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maketimestamp">MakeTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#nextday">NextDay</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#precisetimestampconversion">PreciseTimestampConversion</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/map.html b/user-guide/latest/compatibility/expressions/spark-4.0/map.html
index 9b40109eca..be62c13405 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/map.html
@@ -761,6 +761,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromarrays">
<h2>MapFromArrays<a class="headerlink" href="#mapfromarrays" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
@@ -768,6 +772,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromentries">
<h2>MapFromEntries<a class="headerlink" href="#mapfromentries" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromEntries</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromEntries.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/math.html b/user-guide/latest/compatibility/expressions/spark-4.0/math.html
index 168c24cad8..357b36ce58 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/math.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/math.html
@@ -756,6 +756,13 @@ under the License.
<li><p>Only integral, floating-point, and decimal types are supported</p></li>
</ul>
</section>
+<section id="integraldivide">
+<h2>IntegralDivide<a class="headerlink" href="#integraldivide" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input, <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span> <span class="pre">div</span> <span class="pre">-1</span></code> does not overflow-check: Comet’s decimal-backed path returns the wrapped value <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span></code> under legacy mode and silently returns the same wrapped value under ANSI, whe [...]
+</ul>
+</section>
<section id="rand">
<h2>Rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -769,6 +776,13 @@ under the License.
<ul class="simple">
<li><p>The <code class="docutils literal notranslate"><span class="pre">seed</span></code> argument must be a literal value</p></li>
</ul>
+</section>
+<section id="round">
+<h2>Round<a class="headerlink" href="#round" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input with a negative <code class="docutils literal notranslate"><span class="pre">scale</span></code> of <code class="docutils literal notranslate"><span class="pre">-19</span></code> or lower, <code class="docutils literal notranslate"><span class="pre">10^(-scale)</span></code> overflows the native integer type and Comet returns <code class="docutils literal notranslate"><span class="pre">0</ [...]
+</ul>
<!--END:EXPR_COMPAT-->
</section>
</section>
@@ -821,8 +835,10 @@ under the License.
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#abs">Abs</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#integraldivide">IntegralDivide</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rand">Rand</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#randn">Randn</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#round">Round</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
index c34bb51168..270b48e3e2 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
@@ -765,6 +765,10 @@ under the License.
</section>
<section id="makedecimal">
<h2>MakeDecimal<a class="headerlink" href="#makedecimal" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>The native implementation silently returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> on overflow regardless of the <code class="docutils literal notranslate"><span class="pre">nullOnOverflow</span></code> flag, so under ANSI mode Comet returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> where Spark raises <code class="docutils literal notranslate"><span class="pre">NUMERIC_VALUE_OUT_OF_RANGE</span></code>. < [...]
+</ul>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
<ul class="simple">
<li><p>Only <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input is supported</p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/string.html b/user-guide/latest/compatibility/expressions/spark-4.0/string.html
index 7f34059229..d65656b3e3 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/string.html
@@ -786,6 +786,7 @@ under the License.
<p>By default, <code class="docutils literal notranslate"><span class="pre">GetJsonObject</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p>Spark allows single-quoted JSON and unescaped control characters which Comet does not support</p></li>
+<li><p>For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet’s native implementation returns the last occurrence (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4947">#4947</a>)</p></li>
</ul>
</section>
<section id="initcap">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html b/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
index f85d00c88e..c5c6e95235 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
@@ -824,6 +824,13 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li><p>Only <code class="docutils literal notranslate"><span class="pre">TimestampType</span></code> and <code class="docutils literal notranslate"><span class="pre">TimestampNTZType</span></code> inputs are supported.</p></li>
</ul>
</section>
+<section id="makedate">
+<h2>MakeDate<a class="headerlink" href="#makedate" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an out-of-range <code class="docutils literal notranslate"><span class="pre">(year,</span> <span class="pre">month,</span> <span class="pre">day)</span></code> triple surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkDateTimeException</span></code> with error class <code class="docutils literal notranslate"><span class="pr [...]
+</ul>
+</section>
<section id="maketimestamp">
<h2>MakeTimestamp<a class="headerlink" href="#maketimestamp" title="Link to this heading">#</a></h2>
<p>By default, <code class="docutils literal notranslate"><span class="pre">MakeTimestamp</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MakeTimestamp.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
@@ -833,6 +840,10 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
</section>
<section id="nextday">
<h2>NextDay<a class="headerlink" href="#nextday" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Under ANSI mode, an invalid <code class="docutils literal notranslate"><span class="pre">dayOfWeek</span></code> surfaces as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code> rather than Spark’s <code class="docutils literal notranslate"><span class="pre">SparkIllegalArgumentException</span></code> with error class <code class="docutils literal notranslate"><span class="pre">ILLEGAL_DAY_OF_WEEK</span></code>. The throw/NULL decision is [...]
+</ul>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">NextDay</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.NextDay.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
<li><p>next_day does not support non-UTF8_BINARY collations (https://github.com/apache/datafusion-comet/issues/4646)</p></li>
@@ -959,6 +970,7 @@ with Spark. For dates beyond this range, functions that involve timezone-aware c
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromutctimestamp">FromUTCTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fromunixtime">FromUnixTime</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#hours">Hours</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedate">MakeDate</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maketimestamp">MakeTimestamp</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#nextday">NextDay</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#precisetimestampconversion">PreciseTimestampConversion</a></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/map.html b/user-guide/latest/compatibility/expressions/spark-4.1/map.html
index 8be9ee89fa..562eafd5b0 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/map.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/map.html
@@ -761,6 +761,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromarrays">
<h2>MapFromArrays<a class="headerlink" href="#mapfromarrays" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>The following incompatibilities cause <code class="docutils literal notranslate"><span class="pre">MapFromArrays</span></code> to fall back to Spark by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromArrays.allowIncompatible=true</span></code> to enable Comet acceleration despite these differences.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.mapKeyDedupPolicy</span></code> is set to <code class="docutils literal notranslate"><span class="pre">LAST_WIN</span></code>; Comet’s native map construction does not implement LAST_WIN dedup semantics.</p></li>
@@ -768,6 +772,10 @@ IEEE total ordering for floating-point, which differs from Spark’s <code class
</section>
<section id="mapfromentries">
<h2>MapFromEntries<a class="headerlink" href="#mapfromentries" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>Spark rejects a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> element inside the keys array with a <code class="docutils literal notranslate"><span class="pre">RuntimeException</span></code> (<code class="docutils literal notranslate"><span class="pre">Cannot</span> <span class="pre">use</span> <span class="pre">null</span> <span class="pre">as</span> <span class="pre">map</span> <span class="pre">key</span></code>); Comet’s native <code class="do [...]
+</ul>
<p>By default, <code class="docutils literal notranslate"><span class="pre">MapFromEntries</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.MapFromEntries.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> is not supported as a map key in <code class="docutils literal notranslate"><span class="pre">map_from_entries</span></code></p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/math.html b/user-guide/latest/compatibility/expressions/spark-4.1/math.html
index eec2a4b25b..d87388a1d2 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/math.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/math.html
@@ -756,6 +756,13 @@ under the License.
<li><p>Only integral, floating-point, and decimal types are supported</p></li>
</ul>
</section>
+<section id="integraldivide">
+<h2>IntegralDivide<a class="headerlink" href="#integraldivide" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input, <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span> <span class="pre">div</span> <span class="pre">-1</span></code> does not overflow-check: Comet’s decimal-backed path returns the wrapped value <code class="docutils literal notranslate"><span class="pre">Long.MinValue</span></code> under legacy mode and silently returns the same wrapped value under ANSI, whe [...]
+</ul>
+</section>
<section id="rand">
<h2>Rand<a class="headerlink" href="#rand" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -769,6 +776,13 @@ under the License.
<ul class="simple">
<li><p>The <code class="docutils literal notranslate"><span class="pre">seed</span></code> argument must be a literal value</p></li>
</ul>
+</section>
+<section id="round">
+<h2>Round<a class="headerlink" href="#round" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>On <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input with a negative <code class="docutils literal notranslate"><span class="pre">scale</span></code> of <code class="docutils literal notranslate"><span class="pre">-19</span></code> or lower, <code class="docutils literal notranslate"><span class="pre">10^(-scale)</span></code> overflows the native integer type and Comet returns <code class="docutils literal notranslate"><span class="pre">0</ [...]
+</ul>
<!--END:EXPR_COMPAT-->
</section>
</section>
@@ -821,8 +835,10 @@ under the License.
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#abs">Abs</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#integraldivide">IntegralDivide</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rand">Rand</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#randn">Randn</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#round">Round</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
index 4c3c4fd470..46a67065a3 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
@@ -765,6 +765,10 @@ under the License.
</section>
<section id="makedecimal">
<h2>MakeDecimal<a class="headerlink" href="#makedecimal" title="Link to this heading">#</a></h2>
+<p>The following differences from Spark are always present and do not require any additional configuration:</p>
+<ul class="simple">
+<li><p>The native implementation silently returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> on overflow regardless of the <code class="docutils literal notranslate"><span class="pre">nullOnOverflow</span></code> flag, so under ANSI mode Comet returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code> where Spark raises <code class="docutils literal notranslate"><span class="pre">NUMERIC_VALUE_OUT_OF_RANGE</span></code>. < [...]
+</ul>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
<ul class="simple">
<li><p>Only <code class="docutils literal notranslate"><span class="pre">LongType</span></code> input is supported</p></li>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/string.html b/user-guide/latest/compatibility/expressions/spark-4.1/string.html
index c7f05ea193..9922fa7780 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/string.html
@@ -786,6 +786,7 @@ under the License.
<p>By default, <code class="docutils literal notranslate"><span class="pre">GetJsonObject</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.GetJsonObject.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
<li><p>Spark allows single-quoted JSON and unescaped control characters which Comet does not support</p></li>
+<li><p>For JSON objects containing duplicate keys, Spark returns the value of the first occurrence while Comet’s native implementation returns the last occurrence (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4947">#4947</a>)</p></li>
</ul>
</section>
<section id="initcap">
diff --git a/user-guide/latest/compatibility/index.html b/user-guide/latest/compatibility/index.html
index 6623c0173b..49a78515e1 100644
--- a/user-guide/latest/compatibility/index.html
+++ b/user-guide/latest/compatibility/index.html
@@ -820,6 +820,64 @@ string paths is tracked by
valid UTF-8 rather than reading them like Spark
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4121">#4121</a>).</p>
</section>
+<section id="ansi-mode-error-classes-and-messages">
+<h2>ANSI-mode error classes and messages<a class="headerlink" href="#ansi-mode-error-classes-and-messages" title="Link to this heading">#</a></h2>
+<p>Under <code class="docutils literal notranslate"><span class="pre">spark.sql.ansi.enabled=true</span></code>, several native error paths raise the error at the correct
+input but with a different exception type, error class, SQLSTATE, or message text than Spark.
+Code that catches <code class="docutils literal notranslate"><span class="pre">SparkException</span></code> and only asserts on message substrings is unaffected; code that
+inspects the exception class, <code class="docutils literal notranslate"><span class="pre">getCondition()</span></code>, or the parameterised error class will observe
+divergence:</p>
+<ul class="simple">
+<li><p>Byte / Short <code class="docutils literal notranslate"><span class="pre">Add</span></code>, <code class="docutils literal notranslate"><span class="pre">Subtract</span></code>, and <code class="docutils literal notranslate"><span class="pre">Multiply</span></code> overflow raises <code class="docutils literal notranslate"><span class="pre">ARITHMETIC_OVERFLOW</span></code> where Spark
+4.1 raises <code class="docutils literal notranslate"><span class="pre">BINARY_ARITHMETIC_OVERFLOW</span></code>, and Long overflow surfaces as <code class="docutils literal notranslate"><span class="pre">"integer</span> <span class="pre">overflow"</span></code>
+rather than <code class="docutils literal notranslate"><span class="pre">"long</span> <span class="pre">overflow"</span></code>. <code class="docutils literal notranslate"><span class="pre">Abs</span></code> uses Rust type names (<code class="docutils literal notranslate"><span class="pre">Int8</span></code>, <code class="docutils literal notranslate"><span class="pre">Int64</span></code>, …) in the message
+instead of Spark’s SQL type names, and the scalar path of <code class="docutils literal notranslate"><span class="pre">UnaryMinus</span></code> on Byte / Short emits a
+malformed message. The <code class="docutils literal notranslate"><span class="pre">try_</span></code> suggestion is omitted from all of these
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5071">#5071</a>).</p></li>
+<li><p>Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
+raise raw Arrow errors that bypass <code class="docutils literal notranslate"><span class="pre">SparkErrorConverter</span></code> and surface as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code>
+rather than <code class="docutils literal notranslate"><span class="pre">SparkArithmeticException</span></code> with the proper error class and query context
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5072">#5072</a>).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">next_day</span></code> and <code class="docutils literal notranslate"><span class="pre">make_date</span></code> throw at the correct inputs but surface as <code class="docutils literal notranslate"><span class="pre">CometNativeException</span></code>
+instead of <code class="docutils literal notranslate"><span class="pre">SparkIllegalArgumentException</span> <span class="pre">[ILLEGAL_DAY_OF_WEEK]</span></code> /
+<code class="docutils literal notranslate"><span class="pre">SparkDateTimeException</span> <span class="pre">[DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION]</span></code>
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5073">#5073</a>).</p></li>
+<li><p>Spark 4.2 introduced additional ANSI arithmetic overflow behavior differences that Comet does
+not yet track (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4967">#4967</a>).</p></li>
+</ul>
+</section>
+<section id="known-result-value-divergences">
+<h2>Known result-value divergences<a class="headerlink" href="#known-result-value-divergences" title="Link to this heading">#</a></h2>
+<p>The following native paths silently return values that differ from Spark for edge-case inputs.
+Most also have entries in the per-category expression pages linked above; they are collected here
+so users hunting an unexpected value have a single place to check:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(boolean</span> <span class="pre">AS</span> <span class="pre">DECIMAL(p,</span> <span class="pre">s))</span></code> where <code class="docutils literal notranslate"><span class="pre">10^s</span></code> exceeds the target precision (e.g.
+<code class="docutils literal notranslate"><span class="pre">DECIMAL(1,</span> <span class="pre">1)</span></code>) throws <code class="docutils literal notranslate"><span class="pre">NUMERIC_VALUE_OUT_OF_RANGE</span></code> regardless of the eval mode. Spark returns
+<code class="docutils literal notranslate"><span class="pre">NULL</span></code> under legacy and try mode, and only throws under ANSI
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5068">#5068</a>).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(string</span> <span class="pre">AS</span> <span class="pre">boolean)</span></code> trims only Unicode whitespace. Spark also trims the ASCII control
+bytes <code class="docutils literal notranslate"><span class="pre">0x00-0x08</span></code>, <code class="docutils literal notranslate"><span class="pre">0x0E-0x1F</span></code>, and <code class="docutils literal notranslate"><span class="pre">0x7F</span></code>, so a string like <code class="docutils literal notranslate"><span class="pre">"true"</span></code> casts to
+<code class="docutils literal notranslate"><span class="pre">true</span></code> in Spark and to <code class="docutils literal notranslate"><span class="pre">NULL</span></code> in Comet (or throws under ANSI)
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4959">#4959</a>).</p></li>
+<li><p>Native <code class="docutils literal notranslate"><span class="pre">RANGE</span></code> window frames with an explicit <code class="docutils literal notranslate"><span class="pre">PRECEDING</span></code> / <code class="docutils literal notranslate"><span class="pre">FOLLOWING</span></code> offset diverge from
+Spark when the boundary arithmetic overflows for <code class="docutils literal notranslate"><span class="pre">DATE</span></code> or <code class="docutils literal notranslate"><span class="pre">DECIMAL</span></code> <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> columns
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5022">#5022</a>).</p></li>
+<li><p>For an empty <code class="docutils literal notranslate"><span class="pre">IN</span></code> list, Comet always returns <code class="docutils literal notranslate"><span class="pre">false</span></code> for a <code class="docutils literal notranslate"><span class="pre">NULL</span></code> operand. Spark returns <code class="docutils literal notranslate"><span class="pre">NULL</span></code>
+when <code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.nullInEmptyListBehavior=true</span></code> (Spark 4.0+)
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4786">#4786</a>).</p></li>
+</ul>
+</section>
+<section id="object-store-cache">
+<h2>Object store cache<a class="headerlink" href="#object-store-cache" title="Link to this heading">#</a></h2>
+<p>When Comet’s native scan reads Parquet files, it caches one object store instance per
+<code class="docutils literal notranslate"><span class="pre">(scheme</span> <span class="pre">+</span> <span class="pre">host</span> <span class="pre">+</span> <span class="pre">port,</span> <span class="pre">hadoop-config-hash)</span></code> key. For <code class="docutils literal notranslate"><span class="pre">abfss://container@account.dfs.core.windows.net/...</span></code>
+URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
+account currently collide on the same cache entry. Within a single executor process, reading from
+a second container after a first can be served by the first container’s store instance and return
+its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
+component. Tracked by <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4993">#4993</a>.</p>
+</section>
</section>
@@ -872,6 +930,9 @@ valid UTF-8 rather than reading them like Spark
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#compatible-by-default-opt-in-to-native">Compatible by default, opt in to native</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#native-and-codegen-dispatch-implementations">Native and codegen-dispatch implementations</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#strings-with-non-utf-8-bytes">Strings with non-UTF-8 bytes</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ansi-mode-error-classes-and-messages">ANSI-mode error classes and messages</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#known-result-value-divergences">Known result-value divergences</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#object-store-cache">Object store cache</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/scans.html b/user-guide/latest/compatibility/scans.html
index f6bea5ada5..cf1f1ed0eb 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -773,10 +773,16 @@ Comet Parquet scan regardless.</p></li>
</ul>
<p>The following limitation may produce incorrect results without falling back to Spark:</p>
<ul class="simple">
-<li><p>No support for datetime rebasing. When reading Parquet files containing dates or timestamps written before
-Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps will be read as if they were
-written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before
-October 15, 1582.</p></li>
+<li><p>No support for datetime rebasing. When reading Parquet files containing dates or timestamps
+written with <code class="docutils literal notranslate"><span class="pre">spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY</span></code> (which is Spark’s default for
+data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as
+if they were written using the Proleptic Gregorian calendar. This produces silently-wrong
+values for dates before October 15, 1582 in both projections and predicates. Comet also
+ignores <code class="docutils literal notranslate"><span class="pre">spark.sql.parquet.datetimeRebaseModeInRead</span></code> and the file-level
+<code class="docutils literal notranslate"><span class="pre">org.apache.spark.legacyDateTime</span></code> metadata that would tell it to rebase. The
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exceptionOnDatetimeRebase</span></code> config is currently dead code and does not raise on
+legacy-calendar data. Tracked by
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5010">#5010</a>.</p></li>
</ul>
<p>The following limitations raise an error at scan time rather than falling back to Spark:</p>
<ul class="simple">
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org