You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/29 21:55:58 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by f40c33cf835869b0453e418dd92569258c3df2de
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7e5694e5a5 Publish built docs triggered by f40c33cf835869b0453e418dd92569258c3df2de
7e5694e5a5 is described below
commit 7e5694e5a5e5cd1da37cf8027aa42708c45d71f5
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Jun 29 21:55:53 2026 +0000
Publish built docs triggered by f40c33cf835869b0453e418dd92569258c3df2de
---
.../contributor-guide/expression-audits/array_funcs.md.txt | 13 +++++++------
.../expression-audits/datetime_funcs.md.txt | 6 +++---
.../contributor-guide/expression-audits/string_funcs.md.txt | 9 +++++----
contributor-guide/expression-audits/array_funcs.html | 13 +++++++------
contributor-guide/expression-audits/datetime_funcs.html | 6 +++---
contributor-guide/expression-audits/string_funcs.html | 9 +++++----
searchindex.js | 2 +-
7 files changed, 31 insertions(+), 27 deletions(-)
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
index 8af1d06698..194ffc6c56 100644
--- a/_sources/contributor-guide/expression-audits/array_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -48,7 +48,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayContains(left, right) extends BinaryExpression with NullIntolerant with Predicate`; `inputTypes` uses `findWiderTypeWithoutStringPromotionForTwo`. Wired as `CometScalarFunction("array_contains")`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`; `checkInputDataTypes` adopts `DataTypeUtils.sameType` (collation-aware in 4.x).
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known limitation: no NaN-canonicalization guard in `getSupportLevel`. For `Float`/`Double` arrays containing NaN, Spark's `SQLOrderingUtil` may produce different results than DataFusion's IEEE comparison (https://github.com/apache/datafusion-comet/issues/4481).
+- Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them the same way as Spark's `SQLOrderingUtil`.
## array_distinct
@@ -56,7 +56,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet` so NaN and `+0.0`/`-0.0` are canonicalized. Wired as `CometScalarFunction("array_distinct")`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: DataFusion `array_distinct` uses hash-based equality without NaN/signed-zero canonicalization, so float/double arrays may produce different results (https://github.com/apache/datafusion-comet/issues/4481).
+- Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them like Spark's `SQLOpenHashSet`.
## array_except
@@ -64,7 +64,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayExcept(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result preserves left-side first occurrences not present in right. Comet routes via `CometArrayExcept` and unconditionally flags `Incompatible` ("Null handling and ordering may differ from Spark"); also falls back for `BinaryType` / `StructType` element types.
- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; the overflow path uses `arrayFunctionWithElementsExceedLimitError`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` for float/double arrays (https://github.com/apache/datafusion-comet/issues/4481).
+- Float/double NaN and signed-zero canonicalization matches `array_distinct`. (`array_except` still falls back by default for the null-handling/ordering reasons noted above.)
## array_insert
@@ -79,6 +79,7 @@
- Spark 3.5.8 audited 2026-04-24 (same ordering incompatibility as 3.4.3)
- Spark 4.0.1 audited 2026-04-24 (ordering incompatibility as above; collated strings now fall back to Spark)
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Current status: `CometArrayIntersect` reports `Incompatible` because native element ordering can differ from Spark when the right array is longer than the left (DataFusion probes the longer side). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed. Non-default string collations are reported `Unsupported` and fall back to Spark.
## array_join
@@ -93,7 +94,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayMax(child) extends UnaryExpression with ImplicitCastInputTypes`; skips NULL elements; for float/double Spark's `SQLOrderingUtil` treats NaN as greater than any non-NaN. Wired as `CometScalarFunction("array_max")`.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: DataFusion's `array_max` uses Arrow `partial_cmp`-based ordering, so float/double arrays containing NaN may produce different results (https://github.com/apache/datafusion-comet/issues/4482).
+- Float/double arrays containing NaN match Spark: NaN is treated as greater than any non-NaN value.
## array_min
@@ -101,7 +102,7 @@
- Spark 3.5.8 (audited 2026-05-27): mirror of `ArrayMax` with `evalInternal` returning the minimum. Same NULL-skip and NaN-ordering semantics. Wired as `CometScalarFunction("array_min")`.
- Spark 4.0.1 (audited 2026-05-27): same trait refactor as `array_max`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: same NaN-handling gap as `array_max` (https://github.com/apache/datafusion-comet/issues/4482).
+- Float/double arrays containing NaN match Spark, mirroring `array_max`.
## array_position
@@ -137,7 +138,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; overflow path uses `arrayFunctionWithElementsExceedLimitError`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known divergence: same NaN/signed-zero canonicalization gap as `array_distinct` (https://github.com/apache/datafusion-comet/issues/4481). Result ordering versus DataFusion is also unverified; compare the `array_intersect` ordering caveat.
+- Float/double NaN and signed-zero canonicalization matches `array_distinct`. Result element ordering also matches Spark (left-side distinct elements followed by new right-side elements).
## arrays_overlap
diff --git a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
index e2958b2eea..b20fad5899 100644
--- a/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/datetime_funcs.md.txt
@@ -42,7 +42,7 @@
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-12): baseline.
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
-- Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
+- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution (https://github.com/apache/datafusion-com [...]
## make_timestamp_ltz
@@ -81,10 +81,10 @@
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-12): baseline.
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
-- Known divergence: Comet's native timezone parser does not accept Spark's legacy zone forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). Such timezones throw a native parse error at execution.
+- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution (https://github.com/apache/datafusion-com [...]
## try_make_timestamp
-- Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).
+- Rewrites to `MakeTimestamp(failOnError = false)` and runs through the codegen dispatcher (`CometMakeTimestamp`), so invalid inputs return NULL to match Spark.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/expression-audits/string_funcs.md.txt b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
index cd58089c3a..b362958a23 100644
--- a/_sources/contributor-guide/expression-audits/string_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
@@ -32,7 +32,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numBytes * 8` for strings and `.length * 8` for binary.
- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation(supportsTrimCollation = true)`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
-- Known limitation: wired as a raw `CometScalarFunction("bit_length")` with no `BinaryType` guard. DataFusion's `BitLengthFunc` signature only accepts string types, so `bit_length(<binary>)` execute-fails on the native side instead of falling back cleanly (https://github.com/apache/datafusion-comet/issues/4464).
+- `BinaryType` input is reported `Unsupported` in `getSupportLevel`, so `bit_length(<binary>)` falls back to Spark cleanly (DataFusion's `BitLengthFunc` signature accepts string types only).
## btrim
@@ -81,7 +81,8 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringDecode(bin, charset)` evaluated directly; invalid sequences silently substitute replacement characters via `new String(bytes, charset)`.
- Spark 4.0.1 (audited 2026-05-27): refactored to `RuntimeReplaceable` whose `replacement` is a `StaticInvoke(StringDecode.decode, bin, charset, legacyCharsets, legacyErrorAction)`; the 4-arg form raises on malformed input unless legacy flags are set.
-- Known limitations: Comet handles `decode` via `CommonStringExprs.stringDecode` from the version shims (no `CometExpressionSerde[StringDecode]` registration, so the function does not surface in the auto-generated compatibility docs: https://github.com/apache/datafusion-comet/issues/4466). Only literal `charset = 'utf-8'` (case-insensitive) is supported; everything else falls back. The Spark 4.0 `legacyCharsets` / `legacyErrorAction` flags are ignored: Comet always lowers to `Cast(bin, S [...]
+- `decode` runs through the codegen dispatcher on all versions (Spark 3.x via `CometStringDecode`, Spark 4.0 via the `StaticInvoke` replacement routed to `CometStaticInvokeCodegenDispatch`), so Spark's own evaluation runs inside the Comet pipeline. This honours the `charset` argument and the Spark 4.0 `legacyCharsets` / `legacyErrorAction` flags, and falls back to Spark when the dispatcher is disabled.
+- Known limitation: because there is no `CometExpressionSerde[StringDecode]` registration, `decode` does not surface in the auto-generated compatibility docs (https://github.com/apache/datafusion-comet/issues/4466).
## endswith
@@ -149,7 +150,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `(StringType|BinaryType) -> IntegerType`; eval returns `numBytes` for strings and `.length` for binary.
- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
-- Known limitation: wired as a raw `CometScalarFunction("octet_length")` with no `BinaryType` guard. DataFusion's `OctetLengthFunc` signature only accepts string types, so `octet_length(<binary>)` execute-fails on the native side instead of falling back cleanly (https://github.com/apache/datafusion-comet/issues/4464).
+- `BinaryType` input is reported `Unsupported` in `getSupportLevel`, so `octet_length(<binary>)` falls back to Spark cleanly (DataFusion's `OctetLengthFunc` signature accepts string types only).
## regexp_replace
@@ -230,7 +231,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `StringTranslate(src, from, to)`; `UTF8String.translate(dict)` is code-point based, and any character mapped explicitly to U+0000 in `to` is also deleted.
- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringTranslate.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).
-- Known divergence: DataFusion's `translate` is grapheme-based (Spark uses code points), and does not delete characters mapped to U+0000 in `to`. Currently the support level is `Compatible` (https://github.com/apache/datafusion-comet/issues/4463).
+- Marked `Incompatible`: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.
## trim
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
index a4a787195a..30354cede4 100644
--- a/contributor-guide/expression-audits/array_funcs.html
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -584,7 +584,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayContains(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">BinaryExpression</span> <span class="pre">with</span> <span class="pre">NullIntolerant</span> <span class="pre">with</span> <span class="pre">Predicate</span></code>; <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> uses <code class= [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> trait replaced by <code class="docutils literal notranslate"><span class="pre">nullIntolerant:</span> <span class="pre">Boolean</span></code>; <code class="docutils literal notranslate"><span class="pre">checkInputDataTypes</span></code> adopts <code class="docutils literal notranslate"><span class="pre">DataTypeUtils.sameType</span></code> (collation-aware i [...]
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known limitation: no NaN-canonicalization guard in <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>. For <code class="docutils literal notranslate"><span class="pre">Float</span></code>/<code class="docutils literal notranslate"><span class="pre">Double</span></code> arrays containing NaN, Spark’s <code class="docutils literal notranslate"><span class="pre">SQLOrderingUtil</span></code> may produce different results than DataFusion’s IEEE [...]
+<li><p>Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them the same way as Spark’s <code class="docutils literal notranslate"><span class="pre">SQLOrderingUtil</span></code>.</p></li>
</ul>
</section>
<section id="array-distinct">
@@ -594,7 +594,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayDistinct(child)</span></code> over <code class="docutils literal notranslate"><span class="pre">ArraySetLike</span></code>; uses <code class="docutils literal notranslate"><span class="pre">SQLOpenHashSet</span></code> so NaN and <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>/<code class="docutils literal notranslate"><span class="pre">-0.0 [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known divergence: DataFusion <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code> uses hash-based equality without NaN/signed-zero canonicalization, so float/double arrays may produce different results (https://github.com/apache/datafusion-comet/issues/4481).</p></li>
+<li><p>Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them like Spark’s <code class="docutils literal notranslate"><span class="pre">SQLOpenHashSet</span></code>.</p></li>
</ul>
</section>
<section id="array-except">
@@ -604,7 +604,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayExcept(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">ArrayBinaryLike</span> <span class="pre">with</span> <span class="pre">ComplexTypeMergingExpression</span></code>; result preserves left-side first occurrences not present in right. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayEx [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span> <span class="pre">=</span> <span class="pre">true</span></code> moves into <code class="docutils literal notranslate"><span class="pre">ArrayBinaryLike</span></code>; the overflow path uses <code class="docutils literal notranslate"><span class="pre">arrayFunctionWithElementsExceedLimitError</span></code>.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known divergence: same NaN/signed-zero canonicalization gap as <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code> for float/double arrays (https://github.com/apache/datafusion-comet/issues/4481).</p></li>
+<li><p>Float/double NaN and signed-zero canonicalization matches <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code>. (<code class="docutils literal notranslate"><span class="pre">array_except</span></code> still falls back by default for the null-handling/ordering reasons noted above.)</p></li>
</ul>
</section>
<section id="array-insert">
@@ -623,6 +623,7 @@ under the License.
<li><p>Spark 3.5.8 audited 2026-04-24 (same ordering incompatibility as 3.4.3)</p></li>
<li><p>Spark 4.0.1 audited 2026-04-24 (ordering incompatibility as above; collated strings now fall back to Spark)</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Current status: <code class="docutils literal notranslate"><span class="pre">CometArrayIntersect</span></code> reports <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code> because native element ordering can differ from Spark when the right array is longer than the left (DataFusion probes the longer side). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly a [...]
</ul>
</section>
<section id="array-join">
@@ -641,7 +642,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayMax(child)</span> <span class="pre">extends</span> <span class="pre">UnaryExpression</span> <span class="pre">with</span> <span class="pre">ImplicitCastInputTypes</span></code>; skips NULL elements; for float/double Spark’s <code class="docutils literal notranslate"><span class="pre">SQLOrderingUtil</span></code> treats NaN as greater than any non-NaN. Wired as <code class [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known divergence: DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_max</span></code> uses Arrow <code class="docutils literal notranslate"><span class="pre">partial_cmp</span></code>-based ordering, so float/double arrays containing NaN may produce different results (https://github.com/apache/datafusion-comet/issues/4482).</p></li>
+<li><p>Float/double arrays containing NaN match Spark: NaN is treated as greater than any non-NaN value.</p></li>
</ul>
</section>
<section id="array-min">
@@ -651,7 +652,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): mirror of <code class="docutils literal notranslate"><span class="pre">ArrayMax</span></code> with <code class="docutils literal notranslate"><span class="pre">evalInternal</span></code> returning the minimum. Same NULL-skip and NaN-ordering semantics. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("array_min")</span></code>.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-27): same trait refactor as <code class="docutils literal notranslate"><span class="pre">array_max</span></code>.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known divergence: same NaN-handling gap as <code class="docutils literal notranslate"><span class="pre">array_max</span></code> (https://github.com/apache/datafusion-comet/issues/4482).</p></li>
+<li><p>Float/double arrays containing NaN match Spark, mirroring <code class="docutils literal notranslate"><span class="pre">array_max</span></code>.</p></li>
</ul>
</section>
<section id="array-position">
@@ -697,7 +698,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayUnion(left,</span> <span class="pre">right)</span> <span class="pre">extends</span> <span class="pre">ArrayBinaryLike</span> <span class="pre">with</span> <span class="pre">ComplexTypeMergingExpression</span></code>; result is left-side distinct elements followed by new right-side elements. Wired as <code class="docutils literal notranslate"><span class="pre">CometScalarFu [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span> <span class="pre">=</span> <span class="pre">true</span></code> moves into <code class="docutils literal notranslate"><span class="pre">ArrayBinaryLike</span></code>; overflow path uses <code class="docutils literal notranslate"><span class="pre">arrayFunctionWithElementsExceedLimitError</span></code>.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known divergence: same NaN/signed-zero canonicalization gap as <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code> (https://github.com/apache/datafusion-comet/issues/4481). Result ordering versus DataFusion is also unverified; compare the <code class="docutils literal notranslate"><span class="pre">array_intersect</span></code> ordering caveat.</p></li>
+<li><p>Float/double NaN and signed-zero canonicalization matches <code class="docutils literal notranslate"><span class="pre">array_distinct</span></code>. Result element ordering also matches Spark (left-side distinct elements followed by new right-side elements).</p></li>
</ul>
</section>
<section id="arrays-overlap">
diff --git a/contributor-guide/expression-audits/datetime_funcs.html b/contributor-guide/expression-audits/datetime_funcs.html
index e0b3db8c99..e524cc2846 100644
--- a/contributor-guide/expression-audits/datetime_funcs.html
+++ b/contributor-guide/expression-audits/datetime_funcs.html
@@ -580,7 +580,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-12): baseline.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-12): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for ASCII timezone strings.</p></li>
-<li><p>Known divergence: Comet’s native timezone parser does not accept Spark’s legacy zone forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC+1</span></code>, three-letter abbreviations like <code class="docutils literal notranslate"><span class="pre">PST</span></code>). Such timezones throw a native parse error at execution.</p></li>
+<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: Comet’s native timezone parser only accepts IANA zone IDs (e.g. <code class="docutils literal notranslate"><span class="pre">America/Los_Angeles</span></code>) and fixed <code class="docutils literal notranslate"><span class="pre">+HH:MM</span></code> offsets, while Spark also accepts legacy forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code c [...]
</ul>
</section>
<section id="make-timestamp-ltz">
@@ -637,13 +637,13 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-12): baseline.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-12): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; behaviour unchanged for ASCII timezone strings.</p></li>
-<li><p>Known divergence: Comet’s native timezone parser does not accept Spark’s legacy zone forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code class="docutils literal notranslate"><span class="pre">UTC+1</span></code>, three-letter abbreviations like <code class="docutils literal notranslate"><span class="pre">PST</span></code>). Such timezones throw a native parse error at execution.</p></li>
+<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: Comet’s native timezone parser only accepts IANA zone IDs (e.g. <code class="docutils literal notranslate"><span class="pre">America/Los_Angeles</span></code>) and fixed <code class="docutils literal notranslate"><span class="pre">+HH:MM</span></code> offsets, while Spark also accepts legacy forms (<code class="docutils literal notranslate"><span class="pre">GMT+1</span></code>, <code c [...]
</ul>
</section>
<section id="try-make-timestamp">
<h2>try_make_timestamp<a class="headerlink" href="#try-make-timestamp" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Native for valid inputs; returns wrong values for invalid inputs instead of NULL (https://github.com/apache/datafusion-comet/issues/4554).</p></li>
+<li><p>Rewrites to <code class="docutils literal notranslate"><span class="pre">MakeTimestamp(failOnError</span> <span class="pre">=</span> <span class="pre">false)</span></code> and runs through the codegen dispatcher (<code class="docutils literal notranslate"><span class="pre">CometMakeTimestamp</span></code>), so invalid inputs return NULL to match Spark.</p></li>
</ul>
</section>
</section>
diff --git a/contributor-guide/expression-audits/string_funcs.html b/contributor-guide/expression-audits/string_funcs.html
index ec5ad4ed7e..b7fc495b35 100644
--- a/contributor-guide/expression-audits/string_funcs.html
+++ b/contributor-guide/expression-audits/string_funcs.html
@@ -564,7 +564,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">(StringType|BinaryType)</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; eval returns <code class="docutils literal notranslate"><span class="pre">numBytes</span> <span class="pre">*</span> <span class="pre">8</span></code> for strings and <code class="docutils literal notranslate"><span class="pre">.length</span> <span class="pre">*</span> <sp [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true)</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.c [...]
-<li><p>Known limitation: wired as a raw <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("bit_length")</span></code> with no <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> guard. DataFusion’s <code class="docutils literal notranslate"><span class="pre">BitLengthFunc</span></code> signature only accepts string types, so <code class="docutils literal notranslate"><span class="pre">bit_length(<binary>)</s [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> input is reported <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> in <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>, so <code class="docutils literal notranslate"><span class="pre">bit_length(<binary>)</span></code> falls back to Spark cleanly (DataFusion’s <code class="docutils literal notranslate"><span cla [...]
</ul>
</section>
<section id="btrim">
@@ -629,7 +629,8 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringDecode(bin,</span> <span class="pre">charset)</span></code> evaluated directly; invalid sequences silently substitute replacement characters via <code class="docutils literal notranslate"><span class="pre">new</span> <span class="pre">String(bytes,</span> <span class="pre">charset)</span></code>.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-27): refactored to <code class="docutils literal notranslate"><span class="pre">RuntimeReplaceable</span></code> whose <code class="docutils literal notranslate"><span class="pre">replacement</span></code> is a <code class="docutils literal notranslate"><span class="pre">StaticInvoke(StringDecode.decode,</span> <span class="pre">bin,</span> <span class="pre">charset,</span> <span class="pre">legacyCharsets,</span> <span class="pre">legacyErrorAction)</ [...]
-<li><p>Known limitations: Comet handles <code class="docutils literal notranslate"><span class="pre">decode</span></code> via <code class="docutils literal notranslate"><span class="pre">CommonStringExprs.stringDecode</span></code> from the version shims (no <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde[StringDecode]</span></code> registration, so the function does not surface in the auto-generated compatibility docs: https://github.com/apache/datafusi [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">decode</span></code> runs through the codegen dispatcher on all versions (Spark 3.x via <code class="docutils literal notranslate"><span class="pre">CometStringDecode</span></code>, Spark 4.0 via the <code class="docutils literal notranslate"><span class="pre">StaticInvoke</span></code> replacement routed to <code class="docutils literal notranslate"><span class="pre">CometStaticInvokeCodegenDispatch</span></code>), so S [...]
+<li><p>Known limitation: because there is no <code class="docutils literal notranslate"><span class="pre">CometExpressionSerde[StringDecode]</span></code> registration, <code class="docutils literal notranslate"><span class="pre">decode</span></code> does not surface in the auto-generated compatibility docs (https://github.com/apache/datafusion-comet/issues/4466).</p></li>
</ul>
</section>
<section id="endswith">
@@ -719,7 +720,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">(StringType|BinaryType)</span> <span class="pre">-></span> <span class="pre">IntegerType</span></code>; eval returns <code class="docutils literal notranslate"><span class="pre">numBytes</span></code> for strings and <code class="docutils literal notranslate"><span class="pre">.length</span></code> for binary.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
-<li><p>Known limitation: wired as a raw <code class="docutils literal notranslate"><span class="pre">CometScalarFunction("octet_length")</span></code> with no <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> guard. DataFusion’s <code class="docutils literal notranslate"><span class="pre">OctetLengthFunc</span></code> signature only accepts string types, so <code class="docutils literal notranslate"><span class="pre">octet_length(<binary&g [...]
+<li><p><code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> input is reported <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> in <code class="docutils literal notranslate"><span class="pre">getSupportLevel</span></code>, so <code class="docutils literal notranslate"><span class="pre">octet_length(<binary>)</span></code> falls back to Spark cleanly (DataFusion’s <code class="docutils literal notranslate"><span c [...]
</ul>
</section>
<section id="regexp-replace">
@@ -826,7 +827,7 @@ under the License.
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringTranslate(src,</span> <span class="pre">from,</span> <span class="pre">to)</span></code>; <code class="docutils literal notranslate"><span class="pre">UTF8String.translate(dict)</span></code> is code-point based, and any character mapped explicitly to U+0000 in <code class="docutils literal notranslate"><span class="pre">to</span></code> is also deleted.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringTranslate.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4496).</p></li>
-<li><p>Known divergence: DataFusion’s <code class="docutils literal notranslate"><span class="pre">translate</span></code> is grapheme-based (Spark uses code points), and does not delete characters mapped to U+0000 in <code class="docutils literal notranslate"><span class="pre">to</span></code>. Currently the support level is <code class="docutils literal notranslate"><span class="pre">Compatible</span></code> (https://github.com/apache/datafusion-comet/issues/4463).</p></li>
+<li><p>Marked <code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>: DataFusion’s <code class="docutils literal notranslate"><span class="pre">translate</span></code> iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.</p></li>
</ul>
</section>
<section id="trim">
diff --git a/searchindex.js b/searchindex.js
index ec651e6721..8a2a4dc286 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[50, "id1"]], "%": [[48, "id1"]], "&": [[38, "id1"]], "*": [[48, "id2"]], "+": [[48, "id3"]], "-": [[48, "id4"]], "/": [[48, "id5"]], "1. Format Your Code": [[35, "format-your-code"]], "1. Install Comet": [[56, "install-comet"], [65, "install-comet"]], "1. Native Operators (nativeExecs map)": [[24, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[35, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[56, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[50, "id1"]], "%": [[48, "id1"]], "&": [[38, "id1"]], "*": [[48, "id2"]], "+": [[48, "id3"]], "-": [[48, "id4"]], "/": [[48, "id5"]], "1. Format Your Code": [[35, "format-your-code"]], "1. Install Comet": [[56, "install-comet"], [65, "install-comet"]], "1. Native Operators (nativeExecs map)": [[24, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[35, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[56, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org