You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/04 19:33:53 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 683f89dc993001f094aa2a6f4a2d00152a0698c8
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bca3cdae3b Publish built docs triggered by 683f89dc993001f094aa2a6f4a2d00152a0698c8
bca3cdae3b is described below
commit bca3cdae3b35614bbad74c2abeae24f0f2022862
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Oct 4 19:33:48 2026 +0000
Publish built docs triggered by 683f89dc993001f094aa2a6f4a2d00152a0698c8
---
_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/expression-audits/agg_funcs.md.txt | 2 +-
_sources/user-guide/latest/compatibility/index.md.txt | 2 --
contributor-guide/expression-audits/agg_funcs.html | 2 +-
searchindex.js | 2 +-
user-guide/latest/compatibility/index.html | 2 --
6 files changed, 4 insertions(+), 8 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index c0065912ea..0750fa2eb3 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/agg_funcs.md.txt b/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
index 75b7fbf225..10445f247a 100644
--- a/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/agg_funcs.md.txt
@@ -58,7 +58,7 @@
- Spark 4.0.1 (audited 2026-06-24): only structural change is adding `with UnaryLike[Expression]` to the case class (no behavior change).
- Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.
- Comet implementation: `CometCollectList` (`native/spark-expr/src/agg_funcs/collect.rs`) delegates the ungrouped path to `datafusion_spark::function::aggregate::collect::SparkCollectList`, which wraps `ArrayAggAccumulator` with `ignore_nulls = true` and converts a final NULL accumulator state to an empty array (matching Spark's `defaultResult`); grouped aggregation uses Comet's own `GroupsAccumulator`, which retains the input arrays and gathers them into group order on emit. The native [...]
-- Buffer shape (applies equally to `collect_set`): both are `TypedImperativeAggregate`s, so Spark's `aggBufferAttributes` declares the intermediate buffer as `BinaryType` (serialized state) while the native accumulator's `state_fields` is a `List`. `CometBaseAggregate.adjustOutputForNativeState` rewrites the Comet-side Partial output to the list shape. Neither collector can split a Partial and Final across Comet and Spark, and a multi-stage distinct rewrite (which inserts a `PartialMerge [...]
+- Buffer shape (applies equally to `collect_set`): both are `TypedImperativeAggregate`s, so Spark's `aggBufferAttributes` declares the intermediate buffer as `BinaryType` (serialized state) while the native accumulator's `state_fields` is a `List`. `CometBaseAggregate.adjustOutputForNativeState` rewrites Comet `Partial` and `PartialMerge` outputs to the list shape. When a Spark `Partial` feeds a Comet `PartialMerge`, Comet validates and decodes the single-field `UnsafeRow`/`UnsafeArrayDa [...]
- Performance (tuned 2026-09-09, PR [#5803](https://github.com/apache/datafusion-comet/pull/5803)): grouped `collect_list` used to run through DataFusion's `GroupsAccumulatorAdapter`, which keeps one boxed `Accumulator` per group and slices every batch into a per-group `update_batch` call. `CollectListGroupsAccumulator` instead records a `(group, row range)` contribution per batch and rearranges them with a counting sort on emit, gathering long runs with `concat` and scattered rows with [...]
- Spark 4.2 (preview): `CollectList` and `CollectSet` gain an `ignoreNulls` field (default `true`); `RESPECT NULLS` sets it to `false` and keeps null elements. The native path always drops nulls, so `CometCollectShim` reads the field per Spark version (always `true` on 3.4-4.1) and `CometCollectList` / `CometCollectSet` report `Unsupported` when it is `false`, falling back to Spark.
diff --git a/_sources/user-guide/latest/compatibility/index.md.txt b/_sources/user-guide/latest/compatibility/index.md.txt
index 9bae431207..017140b14d 100644
--- a/_sources/user-guide/latest/compatibility/index.md.txt
+++ b/_sources/user-guide/latest/compatibility/index.md.txt
@@ -116,8 +116,6 @@ divergence:
example `byte overflow`) where Spark raises `BINARY_ARITHMETIC_OVERFLOW`, and integral
`ARITHMETIC_OVERFLOW` messages omit Spark's `try_` suggestion
([#6217](https://github.com/apache/datafusion-comet/issues/6217)).
-- Wide-decimal arithmetic overflow reports a different value in the error message than Spark
- ([#5211](https://github.com/apache/datafusion-comet/issues/5211)).
- Spark 4.2 introduced additional ANSI arithmetic overflow behavior differences that Comet does
not yet track ([#4967](https://github.com/apache/datafusion-comet/issues/4967)).
diff --git a/contributor-guide/expression-audits/agg_funcs.html b/contributor-guide/expression-audits/agg_funcs.html
index 0773815745..9623fb5977 100644
--- a/contributor-guide/expression-audits/agg_funcs.html
+++ b/contributor-guide/expression-audits/agg_funcs.html
@@ -605,7 +605,7 @@ under the License.
<li><p>Spark 4.0.1 (audited 2026-06-24): only structural change is adding <code class="docutils literal notranslate"><span class="pre">with</span> <span class="pre">UnaryLike[Expression]</span></code> to the case class (no behavior change).</p></li>
<li><p>Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.</p></li>
<li><p>Comet implementation: <code class="docutils literal notranslate"><span class="pre">CometCollectList</span></code> (<code class="docutils literal notranslate"><span class="pre">native/spark-expr/src/agg_funcs/collect.rs</span></code>) delegates the ungrouped path to <code class="docutils literal notranslate"><span class="pre">datafusion_spark::function::aggregate::collect::SparkCollectList</span></code>, which wraps <code class="docutils literal notranslate"><span class="pre">Array [...]
-<li><p>Buffer shape (applies equally to <code class="docutils literal notranslate"><span class="pre">collect_set</span></code>): both are <code class="docutils literal notranslate"><span class="pre">TypedImperativeAggregate</span></code>s, so Spark’s <code class="docutils literal notranslate"><span class="pre">aggBufferAttributes</span></code> declares the intermediate buffer as <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> (serialized state) while [...]
+<li><p>Buffer shape (applies equally to <code class="docutils literal notranslate"><span class="pre">collect_set</span></code>): both are <code class="docutils literal notranslate"><span class="pre">TypedImperativeAggregate</span></code>s, so Spark’s <code class="docutils literal notranslate"><span class="pre">aggBufferAttributes</span></code> declares the intermediate buffer as <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> (serialized state) while [...]
<li><p>Performance (tuned 2026-09-09, PR <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5803">#5803</a>): grouped <code class="docutils literal notranslate"><span class="pre">collect_list</span></code> used to run through DataFusion’s <code class="docutils literal notranslate"><span class="pre">GroupsAccumulatorAdapter</span></code>, which keeps one boxed <code class="docutils literal notranslate"><span class="pre">Accumulator</span></code> per group [...]
<li><p>Spark 4.2 (preview): <code class="docutils literal notranslate"><span class="pre">CollectList</span></code> and <code class="docutils literal notranslate"><span class="pre">CollectSet</span></code> gain an <code class="docutils literal notranslate"><span class="pre">ignoreNulls</span></code> field (default <code class="docutils literal notranslate"><span class="pre">true</span></code>); <code class="docutils literal notranslate"><span class="pre">RESPECT</span> <span class="pre">N [...]
</ul>
diff --git a/searchindex.js b/searchindex.js
index 403bb6677e..d17c90e3cb 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/index.html b/user-guide/latest/compatibility/index.html
index 141f73b598..20f7591fee 100644
--- a/user-guide/latest/compatibility/index.html
+++ b/user-guide/latest/compatibility/index.html
@@ -891,8 +891,6 @@ divergence:</p>
example <code class="docutils literal notranslate"><span class="pre">byte</span> <span class="pre">overflow</span></code>) where Spark raises <code class="docutils literal notranslate"><span class="pre">BINARY_ARITHMETIC_OVERFLOW</span></code>, and integral
<code class="docutils literal notranslate"><span class="pre">ARITHMETIC_OVERFLOW</span></code> messages omit Spark’s <code class="docutils literal notranslate"><span class="pre">try_</span></code> suggestion
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6217">#6217</a>).</p></li>
-<li><p>Wide-decimal arithmetic overflow reports a different value in the error message than Spark
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5211">#5211</a>).</p></li>
<li><p>Spark 4.2 introduced additional ANSI arithmetic overflow behavior differences that Comet does
not yet track (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4967">#4967</a>).</p></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org