You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/04 01:24:17 UTC
(datafusion-comet) 01/01: Publish built docs triggered by 4219fc79354a66f7080640412ccde206cba09502
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit ecabd5f9d0d77c5e8259e480f4dbeef80d0c3d06
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 4 01:24:09 2026 +0000
Publish built docs triggered by 4219fc79354a66f7080640412ccde206cba09502
---
.../contributor-guide/expression-audits/array_funcs.md.txt | 3 ++-
.../latest/compatibility/expressions/spark-3.4/array.md.txt | 2 +-
.../latest/compatibility/expressions/spark-3.5/array.md.txt | 2 +-
.../latest/compatibility/expressions/spark-4.0/array.md.txt | 2 +-
.../latest/compatibility/expressions/spark-4.1/array.md.txt | 2 +-
_sources/user-guide/latest/expressions.md.txt | 2 +-
_sources/user-guide/latest/tuning.md.txt | 10 ++++++++--
contributor-guide/expression-audits/array_funcs.html | 3 ++-
searchindex.js | 2 +-
.../latest/compatibility/expressions/spark-3.4/array.html | 2 +-
.../latest/compatibility/expressions/spark-3.5/array.html | 2 +-
.../latest/compatibility/expressions/spark-4.0/array.html | 2 +-
.../latest/compatibility/expressions/spark-4.1/array.html | 2 +-
user-guide/latest/expressions.html | 2 +-
user-guide/latest/tuning.html | 9 +++++++--
15 files changed, 30 insertions(+), 17 deletions(-)
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
index 31261b2542..b4ae0b5b3a 100644
--- a/_sources/contributor-guide/expression-audits/array_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -84,9 +84,10 @@
## array_join
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayJoin(array, delimiter, nullReplacement)`. Comet routes via `CometArrayJoin` to DataFusion's `array_to_string` and is unconditionally flagged `Incompatible` ("Null handling may differ from Spark", [#3178](https://github.com/apache/datafusion-comet/issues/3178)).
+- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayJoin(array, delimiter, nullReplacement)`. Comet routes via `CometArrayJoin` to DataFusion's `array_to_string`.
- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`; non-binary collations not propagated ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
- Spark 4.1.1 (audited 2026-05-27): adds `contextIndependentFoldable` override; runtime unchanged.
+- Current status: `CometArrayJoin` reports `Compatible` when the delimiter and null replacement are literals or column reads; Spark short-circuits past those arguments and DataFusion does not, so anything else runs through the codegen dispatcher, as do non-default string collations ([#2190](https://github.com/apache/datafusion-comet/issues/2190)). A nullable replacement is wrapped in an `IsNull` guard, since `array_to_string` reads a null `null_string` as "omit nulls" ([#3178](https://gi [...]
## array_max
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/array.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/array.md.txt
index 4889051d5c..2babf1a06c 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/array.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/array.md.txt
@@ -44,8 +44,8 @@ By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-gene
By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
-- Null handling may differ from Spark
- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)
+- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)
## ArraysZip
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/array.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/array.md.txt
index 4889051d5c..2babf1a06c 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/array.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/array.md.txt
@@ -44,8 +44,8 @@ By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-gene
By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
-- Null handling may differ from Spark
- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)
+- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)
## ArraysZip
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/array.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/array.md.txt
index 4889051d5c..2babf1a06c 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/array.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/array.md.txt
@@ -44,8 +44,8 @@ By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-gene
By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
-- Null handling may differ from Spark
- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)
+- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)
## ArraysZip
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/array.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/array.md.txt
index 4889051d5c..2babf1a06c 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/array.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/array.md.txt
@@ -44,8 +44,8 @@ By default, `ArrayIntersect` is evaluated in the JVM using Spark's own code-gene
By default, `ArrayJoin` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.ArrayJoin.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
-- Null handling may differ from Spark
- array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)
+- array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)
## ArraysZip
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 7c1beed474..3fbcb12ded 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -155,7 +155,7 @@ The tables below list every Spark built-in expression with its current status.
| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
| `array_insert` | ✅ | Native | |
| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
-| `array_join` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
+| `array_join` | ✅ | Hybrid | Native for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) |
| `array_max` | ✅ | Native | NaN ordering may differ ([details](compatibility/floating-point.md)) |
| `array_min` | ✅ | Native | NaN ordering may differ ([details](compatibility/floating-point.md)) |
| `array_position` | ✅ | Native | Binary/struct/map/null elements fall back |
diff --git a/_sources/user-guide/latest/tuning.md.txt b/_sources/user-guide/latest/tuning.md.txt
index ceac6709a3..decb47a8c2 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -254,8 +254,14 @@ even when both its parent and child are non-Comet operators.
### Remote Shuffle with Celeborn
-Applications using Apache Celeborn can opt into Comet's native shuffle writer and reader with
-the composite shuffle manager:
+Applications using Apache Celeborn can use Comet's composite shuffle manager to retain ordinary
+Spark/Celeborn shuffle while accelerating other operators with Comet.
+
+Native shuffle requires reliable completion tracking for in-flight payloads. Released Celeborn
+0.6.0 and 0.7.0 clients do not provide the required guarantee, so these versions retain ordinary
+Spark/Celeborn shuffle even when `spark.comet.shuffle.mode=native`. Native shuffle support for
+these clients requires a safe Celeborn push-completion API. The following settings request
+native shuffle when the client passes Comet's compatibility checks:
```properties
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
index dcebec4706..665bc36d1e 100644
--- a/contributor-guide/expression-audits/array_funcs.html
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -632,9 +632,10 @@ under the License.
<h2>array_join<a class="headerlink" href="#array-join" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayJoin(array,</span> <span class="pre">delimiter,</span> <span class="pre">nullReplacement)</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayJoin</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_to_string</span></code> and is unconditionally flagged <code class="docutils lit [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArrayJoin(array,</span> <span class="pre">delimiter,</span> <span class="pre">nullReplacement)</span></code>. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArrayJoin</span></code> to DataFusion’s <code class="docutils literal notranslate"><span class="pre">array_to_string</span></code>.</p></li>
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">AbstractArrayType(StringTypeWithCollation(supportsTrimCollation</span> <span class="pre">=</span> <span class="pre">true))</span></code>; non-binary collations not propagated (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/2190">#2190</a>).</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): adds <code class="docutils literal notranslate"><span class="pre">contextIndependentFoldable</span></code> override; runtime unchanged.</p></li>
+<li><p>Current status: <code class="docutils literal notranslate"><span class="pre">CometArrayJoin</span></code> reports <code class="docutils literal notranslate"><span class="pre">Compatible</span></code> when the delimiter and null replacement are literals or column reads; Spark short-circuits past those arguments and DataFusion does not, so anything else runs through the codegen dispatcher, as do non-default string collations (<a class="reference external" href="https://github.com/ap [...]
</ul>
</section>
<section id="array-max">
diff --git a/searchindex.js b/searchindex.js
index e756b3c505..376e133ebd 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/array.html b/user-guide/latest/compatibility/expressions/spark-3.4/array.html
index 0bfe33108b..499cbb5f45 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/array.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/array.html
@@ -809,8 +809,8 @@ under the License.
<h2>ArrayJoin<a class="headerlink" href="#arrayjoin" title="Link to this heading">#</a></h2>
<p>By default, <code class="docutils literal notranslate"><span class="pre">ArrayJoin</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.ArrayJoin.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
-<li><p>Null handling may differ from Spark</p></li>
<li><p>array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)</p></li>
+<li><p>array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)</p></li>
</ul>
</section>
<section id="arrayszip">
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/array.html b/user-guide/latest/compatibility/expressions/spark-3.5/array.html
index 203ad781f8..c9e413e839 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/array.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/array.html
@@ -809,8 +809,8 @@ under the License.
<h2>ArrayJoin<a class="headerlink" href="#arrayjoin" title="Link to this heading">#</a></h2>
<p>By default, <code class="docutils literal notranslate"><span class="pre">ArrayJoin</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.ArrayJoin.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
-<li><p>Null handling may differ from Spark</p></li>
<li><p>array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)</p></li>
+<li><p>array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)</p></li>
</ul>
</section>
<section id="arrayszip">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/array.html b/user-guide/latest/compatibility/expressions/spark-4.0/array.html
index 7cd73668cc..d0ed42a3fd 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/array.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/array.html
@@ -809,8 +809,8 @@ under the License.
<h2>ArrayJoin<a class="headerlink" href="#arrayjoin" title="Link to this heading">#</a></h2>
<p>By default, <code class="docutils literal notranslate"><span class="pre">ArrayJoin</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.ArrayJoin.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
-<li><p>Null handling may differ from Spark</p></li>
<li><p>array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)</p></li>
+<li><p>array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)</p></li>
</ul>
</section>
<section id="arrayszip">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/array.html b/user-guide/latest/compatibility/expressions/spark-4.1/array.html
index 21750c054c..706c35ee59 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/array.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/array.html
@@ -809,8 +809,8 @@ under the License.
<h2>ArrayJoin<a class="headerlink" href="#arrayjoin" title="Link to this heading">#</a></h2>
<p>By default, <code class="docutils literal notranslate"><span class="pre">ArrayJoin</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.ArrayJoin.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
<ul class="simple">
-<li><p>Null handling may differ from Spark</p></li>
<li><p>array_join does not propagate non-UTF8_BINARY collations to the output string (https://github.com/apache/datafusion-comet/issues/2190)</p></li>
+<li><p>array_join evaluates its delimiter and null replacement eagerly, while Spark short-circuits past them (https://github.com/apache/datafusion-comet/issues/3178)</p></li>
</ul>
</section>
<section id="arrayszip">
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 327d3c941d..32f3e0f0b7 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -1219,7 +1219,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">array_join</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible (<a class="reference internal" href="#compatibility/expressions/array.md"><span class="xref myst">details</span></a>)</p></td>
+<td><p>Native for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher (<a class="reference internal" href="#compatibility/expressions/array.md"><span class="xref myst">details</span></a>)</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">array_max</span></code></p></td>
<td><p>✅</p></td>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index c85b9142a2..a53f040b86 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -980,8 +980,13 @@ even when both its parent and child are non-Comet operators.</p>
</section>
<section id="remote-shuffle-with-celeborn">
<h3>Remote Shuffle with Celeborn<a class="headerlink" href="#remote-shuffle-with-celeborn" title="Link to this heading">#</a></h3>
-<p>Applications using Apache Celeborn can opt into Comet’s native shuffle writer and reader with
-the composite shuffle manager:</p>
+<p>Applications using Apache Celeborn can use Comet’s composite shuffle manager to retain ordinary
+Spark/Celeborn shuffle while accelerating other operators with Comet.</p>
+<p>Native shuffle requires reliable completion tracking for in-flight payloads. Released Celeborn
+0.6.0 and 0.7.0 clients do not provide the required guarantee, so these versions retain ordinary
+Spark/Celeborn shuffle even when <code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.mode=native</span></code>. Native shuffle support for
+these clients requires a safe Celeborn push-completion API. The following settings request
+native shuffle when the client passes Comet’s compatibility checks:</p>
<div class="highlight-properties notranslate"><div class="highlight"><pre><span></span><span class="na">spark.shuffle.manager</span><span class="o">=</span><span class="s">org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager</span>
<span class="na">spark.comet.exec.enabled</span><span class="o">=</span><span class="s">true</span>
<span class="na">spark.comet.shuffle.enabled</span><span class="o">=</span><span class="s">true</span>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org