You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/08 18:32:55 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 8e6846850c525506dd2b9194f2014e8acd2ab60a
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 9d6dbd407c Publish built docs triggered by 8e6846850c525506dd2b9194f2014e8acd2ab60a
9d6dbd407c is described below
commit 9d6dbd407ca4f2786311670262ed457ee5be3392
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Sep 8 18:32:48 2026 +0000
Publish built docs triggered by 8e6846850c525506dd2b9194f2014e8acd2ab60a
---
_sources/contributor-guide/expression-audits/string_funcs.md.txt | 7 ++++---
.../latest/compatibility/expressions/spark-3.4/string.md.txt | 2 +-
.../latest/compatibility/expressions/spark-3.5/string.md.txt | 2 +-
.../latest/compatibility/expressions/spark-4.0/string.md.txt | 2 +-
.../latest/compatibility/expressions/spark-4.1/string.md.txt | 2 +-
_sources/user-guide/latest/expressions.md.txt | 2 +-
contributor-guide/expression-audits/string_funcs.html | 7 ++++---
searchindex.js | 2 +-
user-guide/latest/compatibility/expressions/spark-3.4/string.html | 2 +-
user-guide/latest/compatibility/expressions/spark-3.5/string.html | 2 +-
user-guide/latest/compatibility/expressions/spark-4.0/string.html | 2 +-
user-guide/latest/compatibility/expressions/spark-4.1/string.html | 2 +-
user-guide/latest/expressions.html | 2 +-
13 files changed, 19 insertions(+), 17 deletions(-)
diff --git a/_sources/contributor-guide/expression-audits/string_funcs.md.txt b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
index c1db191fdc..d9a4c92bd6 100644
--- a/_sources/contributor-guide/expression-audits/string_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
@@ -66,9 +66,10 @@
## concat_ws
-- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `Seq[Expression] -> StringType`; NULL separator yields NULL, NULL element values are skipped, children can be `StringType` or `ArrayType(StringType)`. Comet serde rewrites a NULL-literal separator to a NULL of the result type and bails out on all-foldable inputs so Spark's `ConstantFolding` handles them; otherwise delegates to DataFusion `concat_ws`.
-- Spark 4.0.1 (audited 2026-05-27): `inputTypes` widened to `StringTypeWithCollation` / `AbstractArrayType`; `dataType` becomes `children.head.dataType` (collation-derived). Semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
+- Spark 3.4.3 (audited 2026-09-06): identical to 3.5.8.
+- Spark 3.5.8 (audited 2026-09-06): accepts ordered mixtures of strings and string arrays, skips null arguments and array elements, and returns NULL for a NULL separator. Comet retains DataFusion's string kernel and uses DataFusion Spark's `SparkConcatWs` for arrays and separator-only inputs, with native column coverage for multiple arrays, empty arrays, nulls, empty strings, Unicode, and varying separators. The existing all-foldable codegen fallback remains. A native adapter evaluates r [...]
+- Spark 4.0.1 (audited 2026-09-06): collation-aware input and result types; concatenation still uses `UTF8String.concatWs` without collation-dependent comparisons.
+- Spark 4.1.1 (audited 2026-09-06): same concatenation and null semantics as 4.0.1.
## contains
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
index efb8de8506..12c1e2056c 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
@@ -41,7 +41,7 @@ The following cases have no native implementation and always run in the JVM usin
The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline):
-- `concat_ws` with `array<string>` arguments: Spark flattens the array elements into the strings to join, which DataFusion's `concat_ws` does not support (https://github.com/apache/datafusion-comet/issues/5675)
+- all arguments are foldable
## Contains
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
index efb8de8506..12c1e2056c 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
@@ -41,7 +41,7 @@ The following cases have no native implementation and always run in the JVM usin
The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline):
-- `concat_ws` with `array<string>` arguments: Spark flattens the array elements into the strings to join, which DataFusion's `concat_ws` does not support (https://github.com/apache/datafusion-comet/issues/5675)
+- all arguments are foldable
## Contains
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
index 126022b3af..1e7edc3aab 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
@@ -41,7 +41,7 @@ The following cases have no native implementation and always run in the JVM usin
The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline):
-- `concat_ws` with `array<string>` arguments: Spark flattens the array elements into the strings to join, which DataFusion's `concat_ws` does not support (https://github.com/apache/datafusion-comet/issues/5675)
+- all arguments are foldable
## Contains
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
index 126022b3af..1e7edc3aab 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
@@ -41,7 +41,7 @@ The following cases have no native implementation and always run in the JVM usin
The following cases have no native implementation and always run in the JVM using Spark's code-generated implementation (inside the Comet pipeline):
-- `concat_ws` with `array<string>` arguments: Spark flattens the array elements into the strings to join, which DataFusion's `concat_ws` does not support (https://github.com/apache/datafusion-comet/issues/5675)
+- all arguments are foldable
## Contains
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 5c36138431..35d9a6c38d 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -561,7 +561,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `chr` | ✅ | Native | |
| `collate` | 🔜 | — | Spark collation (umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)) |
| `collation` | ✅ | — | Constant-folded to a literal (Spark 4.0+) |
-| `concat_ws` | ✅ | Hybrid | Array arguments route through the JVM codegen dispatcher; string arguments run natively ([details](compatibility/expressions/string.md)) |
+| `concat_ws` | ✅ | Hybrid | Mixed string and array arguments run natively; all-foldable arguments use codegen dispatch |
| `contains` | ✅ | — | |
| `decode` | ✅ | — | |
| `elt` | ✅ | Codegen dispatch | |
diff --git a/contributor-guide/expression-audits/string_funcs.html b/contributor-guide/expression-audits/string_funcs.html
index c55e9ea1ca..cb11bc2899 100644
--- a/contributor-guide/expression-audits/string_funcs.html
+++ b/contributor-guide/expression-audits/string_funcs.html
@@ -612,9 +612,10 @@ under the License.
<section id="concat-ws">
<h2>concat_ws<a class="headerlink" href="#concat-ws" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Seq[Expression]</span> <span class="pre">-></span> <span class="pre">StringType</span></code>; NULL separator yields NULL, NULL element values are skipped, children can be <code class="docutils literal notranslate"><span class="pre">StringType</span></code> or <code class="docutils literal notranslate"><span class="pre">ArrayType(StringType)</span></code>. Comet serde rewrit [...]
-<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">inputTypes</span></code> widened to <code class="docutils literal notranslate"><span class="pre">StringTypeWithCollation</span></code> / <code class="docutils literal notranslate"><span class="pre">AbstractArrayType</span></code>; <code class="docutils literal notranslate"><span class="pre">dataType</span></code> becomes <code class="docutils literal notranslate"><span class="pre">childr [...]
+<li><p>Spark 3.4.3 (audited 2026-09-06): identical to 3.5.8.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-09-06): accepts ordered mixtures of strings and string arrays, skips null arguments and array elements, and returns NULL for a NULL separator. Comet retains DataFusion’s string kernel and uses DataFusion Spark’s <code class="docutils literal notranslate"><span class="pre">SparkConcatWs</span></code> for arrays and separator-only inputs, with native column coverage for multiple arrays, empty arrays, nulls, empty strings, Unicode, and varying separators. Th [...]
+<li><p>Spark 4.0.1 (audited 2026-09-06): collation-aware input and result types; concatenation still uses <code class="docutils literal notranslate"><span class="pre">UTF8String.concatWs</span></code> without collation-dependent comparisons.</p></li>
+<li><p>Spark 4.1.1 (audited 2026-09-06): same concatenation and null semantics as 4.0.1.</p></li>
</ul>
</section>
<section id="contains">
diff --git a/searchindex.js b/searchindex.js
index 8538b8ce38..a761c0b8a7 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/string.html b/user-guide/latest/compatibility/expressions/spark-3.4/string.html
index 4e54d03dbe..b8f01ac632 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/string.html
@@ -805,7 +805,7 @@ under the License.
<h2>ConcatWs<a class="headerlink" href="#concatws" title="Link to this heading">#</a></h2>
<p>The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):</p>
<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> with <code class="docutils literal notranslate"><span class="pre">array<string></span></code> arguments: Spark flattens the array elements into the strings to join, which DataFusion’s <code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> does not support (https://github.com/apache/datafusion-comet/issues/5675)</p></li>
+<li><p>all arguments are foldable</p></li>
</ul>
</section>
<section id="contains">
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/string.html b/user-guide/latest/compatibility/expressions/spark-3.5/string.html
index f66977e8df..c42e0e61ed 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/string.html
@@ -805,7 +805,7 @@ under the License.
<h2>ConcatWs<a class="headerlink" href="#concatws" title="Link to this heading">#</a></h2>
<p>The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):</p>
<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> with <code class="docutils literal notranslate"><span class="pre">array<string></span></code> arguments: Spark flattens the array elements into the strings to join, which DataFusion’s <code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> does not support (https://github.com/apache/datafusion-comet/issues/5675)</p></li>
+<li><p>all arguments are foldable</p></li>
</ul>
</section>
<section id="contains">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/string.html b/user-guide/latest/compatibility/expressions/spark-4.0/string.html
index 428d63d316..f142e54926 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/string.html
@@ -805,7 +805,7 @@ under the License.
<h2>ConcatWs<a class="headerlink" href="#concatws" title="Link to this heading">#</a></h2>
<p>The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):</p>
<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> with <code class="docutils literal notranslate"><span class="pre">array<string></span></code> arguments: Spark flattens the array elements into the strings to join, which DataFusion’s <code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> does not support (https://github.com/apache/datafusion-comet/issues/5675)</p></li>
+<li><p>all arguments are foldable</p></li>
</ul>
</section>
<section id="contains">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/string.html b/user-guide/latest/compatibility/expressions/spark-4.1/string.html
index 75a42aee5e..f9d63750e4 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/string.html
@@ -805,7 +805,7 @@ under the License.
<h2>ConcatWs<a class="headerlink" href="#concatws" title="Link to this heading">#</a></h2>
<p>The following cases have no native implementation and always run in the JVM using Spark’s code-generated implementation (inside the Comet pipeline):</p>
<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> with <code class="docutils literal notranslate"><span class="pre">array<string></span></code> arguments: Spark flattens the array elements into the strings to join, which DataFusion’s <code class="docutils literal notranslate"><span class="pre">concat_ws</span></code> does not support (https://github.com/apache/datafusion-comet/issues/5675)</p></li>
+<li><p>all arguments are foldable</p></li>
</ul>
</section>
<section id="contains">
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 144b3112bd..0848e22d86 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -2938,7 +2938,7 @@ expressions. The following function families are <strong>not currently planned</
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">concat_ws</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p>Array arguments route through the JVM codegen dispatcher; string arguments run natively (<a class="reference internal" href="#compatibility/expressions/string.md"><span class="xref myst">details</span></a>)</p></td>
+<td><p>Mixed string and array arguments run natively; all-foldable arguments use codegen dispatch</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">contains</span></code></p></td>
<td><p>✅</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org