You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/07/29 11:26:29 UTC
(datafusion-comet) branch branch-1.0 updated: docs: document known correctness issues in the compatibility guide (#5085) (#5117)
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch branch-1.0
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/branch-1.0 by this push:
new 00f3396acd docs: document known correctness issues in the compatibility guide (#5085) (#5117)
00f3396acd is described below
commit 00f3396acd779a53f1b3f7be7c239b23ede80262
Author: Andy Grove <ag...@apache.org>
AuthorDate: Wed Jul 29 05:26:23 2026 -0600
docs: document known correctness issues in the compatibility guide (#5085) (#5117)
---
.../user-guide/latest/compatibility/index.md | 56 ++++++++++++++++++++++
.../user-guide/latest/compatibility/scans.md | 14 ++++--
.../scala/org/apache/comet/serde/arithmetic.scala | 12 +++++
.../scala/org/apache/comet/serde/datetime.scala | 14 ++++++
.../apache/comet/serde/decimalExpressions.scala | 8 ++++
.../main/scala/org/apache/comet/serde/maps.scala | 12 +++++
.../scala/org/apache/comet/serde/predicates.scala | 5 ++
.../scala/org/apache/comet/serde/strings.scala | 5 +-
8 files changed, 121 insertions(+), 5 deletions(-)
diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md
index b08d9a3bdb..95f496b57f 100644
--- a/docs/source/user-guide/latest/compatibility/index.md
+++ b/docs/source/user-guide/latest/compatibility/index.md
@@ -102,3 +102,59 @@ string paths is tracked by
Separately, Comet's native Parquet scan currently rejects string columns whose stored bytes are not
valid UTF-8 rather than reading them like Spark
([#4121](https://github.com/apache/datafusion-comet/issues/4121)).
+
+## ANSI-mode error classes and messages
+
+Under `spark.sql.ansi.enabled=true`, several native error paths raise the error at the correct
+input but with a different exception type, error class, SQLSTATE, or message text than Spark.
+Code that catches `SparkException` and only asserts on message substrings is unaffected; code that
+inspects the exception class, `getCondition()`, or the parameterised error class will observe
+divergence:
+
+- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` where Spark
+ 4.1 raises `BINARY_ARITHMETIC_OVERFLOW`, and Long overflow surfaces as `"integer overflow"`
+ rather than `"long overflow"`. `Abs` uses Rust type names (`Int8`, `Int64`, ...) in the message
+ instead of Spark's SQL type names, and the scalar path of `UnaryMinus` on Byte / Short emits a
+ malformed message. The `try_` suggestion is omitted from all of these
+ ([#5071](https://github.com/apache/datafusion-comet/issues/5071)).
+- Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
+ raise raw Arrow errors that bypass `SparkErrorConverter` and surface as `CometNativeException`
+ rather than `SparkArithmeticException` with the proper error class and query context
+ ([#5072](https://github.com/apache/datafusion-comet/issues/5072)).
+- `next_day` and `make_date` throw at the correct inputs but surface as `CometNativeException`
+ instead of `SparkIllegalArgumentException [ILLEGAL_DAY_OF_WEEK]` /
+ `SparkDateTimeException [DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION]`
+ ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).
+- Spark 4.2 introduced additional ANSI arithmetic overflow behavior differences that Comet does
+ not yet track ([#4967](https://github.com/apache/datafusion-comet/issues/4967)).
+
+## Known result-value divergences
+
+The following native paths silently return values that differ from Spark for edge-case inputs.
+Most also have entries in the per-category expression pages linked above; they are collected here
+so users hunting an unexpected value have a single place to check:
+
+- `CAST(boolean AS DECIMAL(p, s))` where `10^s` exceeds the target precision (e.g.
+ `DECIMAL(1, 1)`) throws `NUMERIC_VALUE_OUT_OF_RANGE` regardless of the eval mode. Spark returns
+ `NULL` under legacy and try mode, and only throws under ANSI
+ ([#5068](https://github.com/apache/datafusion-comet/issues/5068)).
+- `CAST(string AS boolean)` trims only Unicode whitespace. Spark also trims the ASCII control
+ bytes `0x00-0x08`, `0x0E-0x1F`, and `0x7F`, so a string like `"true"` casts to
+ `true` in Spark and to `NULL` in Comet (or throws under ANSI)
+ ([#4959](https://github.com/apache/datafusion-comet/issues/4959)).
+- Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from
+ Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns
+ ([#5022](https://github.com/apache/datafusion-comet/issues/5022)).
+- For an empty `IN` list, Comet always returns `false` for a `NULL` operand. Spark returns `NULL`
+ when `spark.sql.legacy.nullInEmptyListBehavior=true` (Spark 4.0+)
+ ([#4786](https://github.com/apache/datafusion-comet/issues/4786)).
+
+## Object store cache
+
+When Comet's native scan reads Parquet files, it caches one object store instance per
+`(scheme + host + port, hadoop-config-hash)` key. For `abfss://container@account.dfs.core.windows.net/...`
+URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
+account currently collide on the same cache entry. Within a single executor process, reading from
+a second container after a first can be served by the first container's store instance and return
+its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
+component. Tracked by [#4993](https://github.com/apache/datafusion-comet/issues/4993).
diff --git a/docs/source/user-guide/latest/compatibility/scans.md b/docs/source/user-guide/latest/compatibility/scans.md
index b7dca43ae8..0c07b5c237 100644
--- a/docs/source/user-guide/latest/compatibility/scans.md
+++ b/docs/source/user-guide/latest/compatibility/scans.md
@@ -51,10 +51,16 @@ The following features are not supported and cause Comet to fall back to Spark:
The following limitation may produce incorrect results without falling back to Spark:
-- No support for datetime rebasing. When reading Parquet files containing dates or timestamps written before
- Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps will be read as if they were
- written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before
- October 15, 1582.
+- No support for datetime rebasing. When reading Parquet files containing dates or timestamps
+ written with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY` (which is Spark's default for
+ data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as
+ if they were written using the Proleptic Gregorian calendar. This produces silently-wrong
+ values for dates before October 15, 1582 in both projections and predicates. Comet also
+ ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level
+ `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. The
+ `spark.comet.exceptionOnDatetimeRebase` config is currently dead code and does not raise on
+ legacy-calendar data. Tracked by
+ [#5010](https://github.com/apache/datafusion-comet/issues/5010).
The following limitations raise an error at scan time rather than falling back to Spark:
diff --git a/spark/src/main/scala/org/apache/comet/serde/arithmetic.scala b/spark/src/main/scala/org/apache/comet/serde/arithmetic.scala
index 6d1e2304fc..e67d4f6720 100644
--- a/spark/src/main/scala/org/apache/comet/serde/arithmetic.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/arithmetic.scala
@@ -291,6 +291,12 @@ object CometDivide extends CometExpressionSerde[Divide] with MathBase {
object CometIntegralDivide extends CometExpressionSerde[IntegralDivide] with MathBase {
+ override def getCompatibleNotes(): Seq[String] = Seq(
+ "On `LongType` input, `Long.MinValue div -1` does not overflow-check: Comet's decimal-backed" +
+ " path returns the wrapped value `Long.MinValue` under legacy mode and silently returns" +
+ " the same wrapped value under ANSI, where Spark raises `ARITHMETIC_OVERFLOW`" +
+ " ([#5065](https://github.com/apache/datafusion-comet/issues/5065)).")
+
override def getSupportLevel(expr: IntegralDivide): SupportLevel =
mathDataTypeSupportLevel(expr.left.dataType)
@@ -375,6 +381,12 @@ object CometRemainder extends CometExpressionSerde[Remainder] with MathBase {
object CometRound extends CometExpressionSerde[Round] {
+ override def getCompatibleNotes(): Seq[String] = Seq(
+ "On `LongType` input with a negative `scale` of `-19` or lower, `10^(-scale)` overflows the" +
+ " native integer type and Comet returns `0` instead of Spark's overflowed value. Under" +
+ " ANSI mode Spark raises `ARITHMETIC_OVERFLOW` at the same input; Comet still returns `0`" +
+ " ([#5070](https://github.com/apache/datafusion-comet/issues/5070)).")
+
override def getSupportLevel(expr: Round): SupportLevel = expr.child.dataType match {
case t: DecimalType if t.scale < 0 => // Spark disallows negative scale SPARK-30252
Unsupported(Some("Decimal type has negative scale"))
diff --git a/spark/src/main/scala/org/apache/comet/serde/datetime.scala b/spark/src/main/scala/org/apache/comet/serde/datetime.scala
index d968c00cd1..a2600bf688 100644
--- a/spark/src/main/scala/org/apache/comet/serde/datetime.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/datetime.scala
@@ -450,6 +450,12 @@ object CometNextDay extends CometExpressionSerde[NextDay] {
override def getIncompatibleReasons(): Seq[String] =
DatetimeCollation.incompatibleReasons("next_day")
+ override def getCompatibleNotes(): Seq[String] = Seq(
+ "Under ANSI mode, an invalid `dayOfWeek` surfaces as `CometNativeException` rather than" +
+ " Spark's `SparkIllegalArgumentException` with error class `ILLEGAL_DAY_OF_WEEK`. The" +
+ " throw/NULL decision is correct; only the exception class and error class differ" +
+ " ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).")
+
override def getSupportLevel(expr: NextDay): SupportLevel = {
if (DatetimeCollation.hasNonDefaultCollation(expr)) {
Incompatible(Some(collationReason))
@@ -475,6 +481,14 @@ object CometMakeDate extends CometExpressionSerde[MakeDate] {
* `(year, month, day)` triple rather than returning NULL. The resolved flag is passed to native
* via the `ScalarFunc.fail_on_error` field.
*/
+
+ override def getCompatibleNotes(): Seq[String] = Seq(
+ "Under ANSI mode, an out-of-range `(year, month, day)` triple surfaces as" +
+ " `CometNativeException` rather than Spark's `SparkDateTimeException` with error class" +
+ " `DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION`. The throw/NULL decision is correct;" +
+ " only the exception class and error class differ" +
+ " ([#5073](https://github.com/apache/datafusion-comet/issues/5073)).")
+
override def convert(expr: MakeDate, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = {
val childExpr = expr.children.map(exprToProtoInternal(_, inputs, binding))
val optExpr = scalarFunctionExprToProtoWithReturnType(
diff --git a/spark/src/main/scala/org/apache/comet/serde/decimalExpressions.scala b/spark/src/main/scala/org/apache/comet/serde/decimalExpressions.scala
index f41e0fafaa..3af32e1d11 100644
--- a/spark/src/main/scala/org/apache/comet/serde/decimalExpressions.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/decimalExpressions.scala
@@ -41,6 +41,14 @@ object CometMakeDecimal extends CometExpressionSerde[MakeDecimal] {
override def getUnsupportedReasons(): Seq[String] = Seq("Only `LongType` input is supported")
+ override def getCompatibleNotes(): Seq[String] = Seq(
+ "The native implementation silently returns `NULL` on overflow regardless of the" +
+ " `nullOnOverflow` flag, so under ANSI mode Comet returns `NULL` where Spark raises" +
+ " `NUMERIC_VALUE_OUT_OF_RANGE`. `MakeDecimal` is inserted by the `DecimalAggregates`" +
+ " optimizer rule, so this affects `sum`/`avg` over low-precision decimals when the" +
+ " unscaled long overflows the target precision" +
+ " ([#5066](https://github.com/apache/datafusion-comet/issues/5066)).")
+
override def getSupportLevel(expr: MakeDecimal): SupportLevel = {
expr.child.dataType match {
case LongType => Compatible()
diff --git a/spark/src/main/scala/org/apache/comet/serde/maps.scala b/spark/src/main/scala/org/apache/comet/serde/maps.scala
index c8304440c3..817792e1d2 100644
--- a/spark/src/main/scala/org/apache/comet/serde/maps.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/maps.scala
@@ -81,6 +81,12 @@ private object MapKeyDedupPolicySupport {
s"`${SQLConf.MapKeyDedupPolicy.LAST_WIN}`; Comet's native map construction " +
"does not implement LAST_WIN dedup semantics."
+ val nullKeyReason: String =
+ "Spark rejects a `NULL` element inside the keys array with a `RuntimeException`" +
+ " (`Cannot use null as map key`); Comet's native `map_from_arrays` / `map_from_entries`" +
+ " does not detect a per-element `NULL` key and produces a map with a `NULL` key instead" +
+ " ([#4680](https://github.com/apache/datafusion-comet/issues/4680))."
+
def isLastWin: Boolean =
SQLConf.get
.getConf(SQLConf.MAP_KEY_DEDUP_POLICY)
@@ -93,6 +99,9 @@ object CometMapFromArrays extends CometExpressionSerde[MapFromArrays] {
override def getIncompatibleReasons(): Seq[String] =
Seq(MapKeyDedupPolicySupport.incompatibleReason)
+ override def getCompatibleNotes(): Seq[String] =
+ Seq(MapKeyDedupPolicySupport.nullKeyReason)
+
override def getSupportLevel(expr: MapFromArrays): SupportLevel = {
if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
@@ -153,6 +162,9 @@ object CometMapFromEntries
override def getIncompatibleReasons(): Seq[String] =
Seq(keyUnsupportedReason, valueUnsupportedReason, MapKeyDedupPolicySupport.incompatibleReason)
+ override def getCompatibleNotes(): Seq[String] =
+ Seq(MapKeyDedupPolicySupport.nullKeyReason)
+
override def getSupportLevel(expr: MapFromEntries): SupportLevel = {
if (SupportLevel.containsType(expr.dataType.keyType, classOf[BinaryType])) {
Incompatible(Some(keyUnsupportedReason))
diff --git a/spark/src/main/scala/org/apache/comet/serde/predicates.scala b/spark/src/main/scala/org/apache/comet/serde/predicates.scala
index c9a70a830f..6a08f10605 100644
--- a/spark/src/main/scala/org/apache/comet/serde/predicates.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/predicates.scala
@@ -249,6 +249,11 @@ object CometIn extends CometExpressionSerde[In] with CodegenDispatchFallback {
override def getUnsupportedReasons(): Seq[String] =
Seq(ComparisonUtils.nonDefaultCollationDocReason)
+ override def getCompatibleNotes(): Seq[String] = Seq(
+ "For an empty `IN` list, Comet always returns `false` for a `NULL` operand. Spark returns" +
+ " `NULL` when `spark.sql.legacy.nullInEmptyListBehavior=true` (Spark 4.0+)" +
+ " ([#4786](https://github.com/apache/datafusion-comet/issues/4786)).")
+
override def convert(
expr: In,
inputs: Seq[Attribute],
diff --git a/spark/src/main/scala/org/apache/comet/serde/strings.scala b/spark/src/main/scala/org/apache/comet/serde/strings.scala
index 1482d3e79e..7422dadd97 100644
--- a/spark/src/main/scala/org/apache/comet/serde/strings.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/strings.scala
@@ -647,7 +647,10 @@ object CometGetJsonObject extends CometCodegenDispatch[GetJsonObject] with Nativ
override def getIncompatibleReasons(): Seq[String] =
Seq(
"Spark allows single-quoted JSON and unescaped control characters" +
- " which Comet does not support")
+ " which Comet does not support",
+ "For JSON objects containing duplicate keys, Spark returns the value of the first" +
+ " occurrence while Comet's native implementation returns the last occurrence" +
+ " ([#4947](https://github.com/apache/datafusion-comet/issues/4947))")
override def getSupportLevel(expr: GetJsonObject): SupportLevel =
if (!CometConf.isExprAllowIncompat(getExprConfigName(expr))) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org