You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/06/29 21:46:24 UTC
(datafusion-comet) branch main updated: docs: refresh expression audit notes for resolved correctness issues (#4762)
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new f40c33cf83 docs: refresh expression audit notes for resolved correctness issues (#4762)
f40c33cf83 is described below
commit f40c33cf835869b0453e418dd92569258c3df2de
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Jun 29 15:46:19 2026 -0600
docs: refresh expression audit notes for resolved correctness issues (#4762)
Several expression-audit pages still described correctness gaps that have
since been fixed in code. Update them to reflect current behavior:
- array_contains/distinct/except/max/min/union: float/double NaN and
signed-zero now match Spark (DataFusion canonicalizes).
- array_intersect: now reports Incompatible with a codegen-dispatch
fallback, so it is Spark-correct by default.
- bit_length/octet_length: BinaryType input now reported Unsupported and
falls back cleanly.
- translate: now reports Incompatible and falls back by default.
- decode: now routed through the codegen dispatcher on all versions,
honouring charset and the Spark 4.0 legacy flags.
- try_make_timestamp: now routed through the codegen dispatcher, returning
NULL for invalid inputs.
- from_utc_timestamp/to_utc_timestamp: now report Incompatible with a
codegen-dispatch fallback, so legacy zone forms are Spark-correct by
default.
---
.../contributor-guide/expression-audits/array_funcs.md | 13 +++++++------
.../contributor-guide/expression-audits/datetime_funcs.md | 6 +++---
.../contributor-guide/expression-audits/string_funcs.md | 9 +++++----
3 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/docs/source/contributor-guide/expression-audits/array_funcs.md b/docs/source/contributor-guide/expression-audits/array_funcs.md
index 8af1d06698..194ffc6c56 100644
--- a/docs/source/contributor-guide/expression-audits/array_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/array_funcs.md
@@ -48,7 +48,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayContains(left, right) extends BinaryExpression with NullIntolerant with Predicate`; `inputTypes` uses `findWiderTypeWithoutStringPromotionForTwo`. Wired as `CometScalarFunction("array_contains")`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`; `checkInputDataTypes` adopts `DataTypeUtils.sameType` (collation-aware in 4.x).
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known limitation: no NaN-canonicalization guard in `getSupportLevel`. For `Float`/`Double` arrays containing NaN, Spark's `SQLOrderingUtil` may produce different results than DataFusion's IEEE comparison (https://github.com/apache/datafusion-comet/issues/4481).
+- Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them the same way as Spark's `SQLOrderingUtil`.
## array_distinct
@@ -56,7 +56,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet` so NaN and `+0.0`/`-0.0` are canonicalized. Wired as `CometScalarFunction("array_distinct")`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: DataFusion `array_distinct` uses hash-based equality without NaN/signed-zero canonicalization, so float/double arrays may produce different results (https://github.com/apache/datafusion-comet/issues/4481).
+- Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them like Spark's `SQLOpenHashSet`.
## array_except
@@ -64,7 +64,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayExcept(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result preserves left-side first occurrences not present in right. Comet routes via `CometArrayExcept` and unconditionally flags `Incompatible` ("Null handling and ordering may differ from Spark"); also falls back for `BinaryType` / `StructType` element types.
- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; the overflow path uses `arrayFunctionWithElementsExceedLimitError`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` for float/double arrays (https://github.com/apache/datafusion-comet/issues/4481).
+- Float/double NaN and signed-zero canonicalization matches `array_distinct`. (`array_except` still falls back by default for the null-handling/ordering reasons noted above.)
## array_insert
@@ -79,6 +79,7 @@
- Spark 3.5.8 audited 2026-04-24 (same ordering incompatibility as 3.4.3)
- Spark 4.0.1 audited 2026-04-24 (ordering incompatibility as above; collated strings now fall back to Spark)
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Current status: `CometArrayIntersect` reports `Incompatible` because native element ordering can differ from Spark when the right array is longer than the left (DataFusion probes the longer side). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed. Non-default string collations are reported `Unsupported` and fall back to Spark.
## array_join
@@ -93,7 +94,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayMax(child) extends UnaryExpression with ImplicitCastInputTypes`; skips NULL elements; for float/double Spark's `SQLOrderingUtil` treats NaN as greater than any non-NaN. Wired as `CometScalarFunction("array_max")`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: DataFusion's `array_max` uses Arrow `partial_cmp`-based ordering, so float/double arrays containing NaN may produce different results (https://github.com/apache/datafusion-comet/issues/4482).
+- Float/double arrays containing NaN match Spark: NaN is treated as greater than any non-NaN value.
## array_min
@@ -101,7 +102,7 @@
- Spark 3.5.8 (audited 2026-05-27): mirror of `ArrayMax` with `evalInternal` returning the minimum. Same NULL-skip and NaN-ordering semantics. Wired as `CometScalarFunction("array_min")`.
- Spark 4.0.1 (audited 2026-05-27): same trait refactor as `array_max`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: same NaN-handling gap as `array_max` (https://github.com/apache/datafusion-comet/issues/4482).
+- Float/double arrays containing NaN match Spark, mirroring `array_max`.
## array_position
@@ -137,7 +138,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; overflow path uses `arrayFunctionWithElementsExceedLimitError`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` (https://github.com/apache/datafusion-comet/issues/4481). Result ordering versus DataFusion is also unverified; compare the `array_intersect` ordering caveat.
+- Float/double NaN and signed-zero canonicalization matches `array_distinct`. Result element ordering also matches Spark (left-side distinct elements followed by new right-side elements).
## arrays_overlap
diff --git a/docs/source/contributor-guide/expression-audits/datetime_funcs.md b/docs/source/contributor-guide/expression-audits/datetime_funcs.md
index e2958b2eea..b20fad5899 100644
--- a/docs/source/contributor-guide/expression-audits/datetime_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/datetime_funcs.md
@@ -42,7 +42,7 @@
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-12): baseline.
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
-- Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
+- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution (https://github.com/apache/datafusion-com [...]
## make_timestamp_ltz
@@ -81,10 +81,10 @@
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-12): baseline.
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
-- Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
+- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution (https://github.com/apache/datafusion-com [...]
## try_make_timestamp
-- Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).
+- Rewrites to `MakeTimestamp(failOnError = false)` and runs through the codegen dispatcher (`CometMakeTimestamp`), so invalid inputs return NULL to match Spark.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/docs/source/contributor-guide/expression-audits/string_funcs.md b/docs/source/contributor-guide/expression-audits/string_funcs.md
index cd58089c3a..b362958a23 100644
--- a/docs/source/contributor-guide/expression-audits/string_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/string_funcs.md
@@ -32,7 +32,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numBytes * 8` for strings and `.length * 8` for binary.
- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
-- Known limitation: wired as a raw `CometScalarFunction("bit_length")` with no `BinaryType` guard. DataFusion's `BitLengthFunc` signature only accepts string types, so `bit_length(<binary>)` execute-fails on the native side instead of falling back cleanly (https://github.com/apache/datafusion-comet/issues/4464).
+- `BinaryType` input is reported `Unsupported` in `getSupportLevel`, so `bit_length(<binary>)` falls back to Spark cleanly (DataFusion's `BitLengthFunc` signature accepts string types only).
## btrim
@@ -81,7 +81,8 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringDecode(bin, charset)` evaluated directly; invalid sequences silently substitute replacement characters via `new String(bytes, charset)`.
- Spark 4.0.1 (audited 2026-05-27): refactored to `RuntimeReplaceable` whose `replacement` is a `StaticInvoke(StringDecode.decode, bin, charset, legacyCharsets, legacyErrorAction)`; the 4-arg form raises on malformed input unless legacy flags are set.
-- Known limitations: Comet handles `decode` via `CommonStringExprs.stringDecode` from the version shims (no `CometExpressionSerde[StringDecode]` registration, so the function does not surface in the auto-generated compatibility docs: https://github.com/apache/datafusion-comet/issues/4466). Only literal `charset = 'utf-8'` (case-insensitive) is supported; everything else falls back. The Spark 4.0 `legacyCharsets` / `legacyErrorAction` flags are ignored: Comet always lowers to `Cast(bin, S [...]
+- `decode` runs through the codegen dispatcher on all versions (Spark 3.x via `CometStringDecode`, Spark 4.0 via the `StaticInvoke` replacement routed to `CometStaticInvokeCodegenDispatch`), so Spark's own evaluation runs inside the Comet pipeline. This honours the `charset` argument and the Spark 4.0 `legacyCharsets` / `legacyErrorAction` flags, and falls back to Spark when the dispatcher is disabled.
+- Known limitation: because there is no `CometExpressionSerde[StringDecode]` registration, `decode` does not surface in the auto-generated compatibility docs (https://github.com/apache/datafusion-comet/issues/4466).
## endswith
@@ -149,7 +150,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numBytes` for strings and `.length` for binary.
- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
-- Known limitation: wired as a raw `CometScalarFunction("octet_length")` with no `BinaryType` guard. DataFusion's `OctetLengthFunc` signature only accepts string types, so `octet_length(<binary>)` execute-fails on the native side instead of falling back cleanly (https://github.com/apache/datafusion-comet/issues/4464).
+- `BinaryType` input is reported `Unsupported` in `getSupportLevel`, so `octet_length(<binary>)` falls back to Spark cleanly (DataFusion's `OctetLengthFunc` signature accepts string types only).
## regexp_replace
@@ -230,7 +231,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTranslate(src, from, to)`; `UTF8String.translate(dict)` is code-point based, and any character mapped explicitly to U+0000 in `to` is also deleted.
- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTranslate.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
-- Known divergence: DataFusion's `translate` is grapheme-based (Spark uses code points), and does not delete characters mapped to U+0000 in `to`. Currently the support level is `Compatible` (https://github.com/apache/datafusion-comet/issues/4463).
+- Marked `Incompatible`: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.
## trim
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org