You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/20 05:10:21 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 30ae34c3a8 Publish built docs triggered by 3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83
30ae34c3a8 is described below
commit 30ae34c3a8c2a6fdbac0c64e3521a29f218ef781
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Oct 20 05:10:16 2026 +0000
Publish built docs triggered by 3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83
---
_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../contributor-guide/expression-audits/array_funcs.md.txt | 1 +
_sources/contributor-guide/floating_point.md.txt | 5 +++++
.../user-guide/latest/compatibility/floating-point.md.txt | 12 ++++++++++++
contributor-guide/expression-audits/array_funcs.html | 1 +
contributor-guide/floating_point.html | 5 +++++
searchindex.js | 2 +-
user-guide/latest/compatibility/floating-point.html | 12 ++++++++++++
8 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index 27633b65b0..f36f31d4ff 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
index af635270e7..87ff575307 100644
--- a/_sources/contributor-guide/expression-audits/array_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -149,6 +149,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysOverlap(left, right)`; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via `CometArraysOverlap` to the native `spark_arrays_overlap` UDF, which implements the same three-valued logic.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates, 2026-10-14): `fastEval` and its generated code normalize each flat `FLOAT` or `DOUBLE` element with `NormalizeFloatingNumbers.FLOAT_NORMALIZER` / `DOUBLE_NORMALIZER` before the `java.util.HashSet` lookup (SPARK-59602), so `-0.0` and `0.0` are one value. Spark 4.2.0 normalizes the arguments in the plan instead (SPARK-54918). Earlier releases keep the zeros apart, and every release treats all NaNs as one value. On the versions where `Arra [...]
## arrays_zip
diff --git a/_sources/contributor-guide/floating_point.md.txt b/_sources/contributor-guide/floating_point.md.txt
index 758350db8f..220a7e4ddb 100644
--- a/_sources/contributor-guide/floating_point.md.txt
+++ b/_sources/contributor-guide/floating_point.md.txt
@@ -62,6 +62,11 @@ it runs against:
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).
+- `arrays_overlap` looks up the elements of a flat array in a `java.util.HashSet` of boxed values,
+ so it follows `Double.equals` and keeps signed zeros apart up to Spark 4.0.4 and 4.1.3. The same
+ two changes make the zeros one value: Spark 4.2.0 normalizes its arguments in the plan
+ (SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize each flat `FLOAT` or `DOUBLE` element before
+ the lookup (SPARK-59602). Nested elements compare in the SQL ordering on every version.
- Map construction (`ArrayBasedMapBuilder`) finds a duplicate `FLOAT` or `DOUBLE` key in a
`HashMap` of boxed values, which follows `Double.equals`, and a duplicate struct or array key in a
`TreeMap` in the SQL ordering. From Spark 4.0 it normalizes a `FLOAT` or `DOUBLE` key before the
diff --git a/_sources/user-guide/latest/compatibility/floating-point.md.txt b/_sources/user-guide/latest/compatibility/floating-point.md.txt
index 77d76f53c6..c94fc7dc68 100644
--- a/_sources/user-guide/latest/compatibility/floating-point.md.txt
+++ b/_sources/user-guide/latest/compatibility/floating-point.md.txt
@@ -148,6 +148,18 @@ differences are acceptable for your data.
The check uses the Spark version number, not the changes a build contains. A vendor build that
reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.
+## `arrays_overlap`
+
+`arrays_overlap` runs natively on every Spark version and follows that version's rule for `FLOAT`
+and `DOUBLE` elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat `-0.0` and `0.0` as one
+value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
+one value on every version, and nested arrays and structs compare their elements in Spark's order,
+in which the two zeros are equal.
+
+Comet chooses the rule by the Spark version number. On a vendor build that reports an older
+version but includes SPARK-59602, Comet returns `false` for two flat arrays whose only common value
+is a zero of opposite sign, where Spark returns `true`.
+
## `array_contains`, `array_remove` and `sort_array`
`array_contains` compares `FLOAT` and `DOUBLE` elements as Spark does: `-0.0` equals `0.0`, and all
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
index 39c176f9d0..b5e6631685 100644
--- a/contributor-guide/expression-audits/array_funcs.html
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -724,6 +724,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArraysOverlap(left,</span> <span class="pre">right)</span></code>; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArraysOverlap</span></code> to the native <code class="docutils literal notranslate"><span [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates, 2026-10-14): <code class="docutils literal notranslate"><span class="pre">fastEval</span></code> and its generated code normalize each flat <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> element with <code class="docutils literal notranslate"><span class="pre">NormalizeFloatingNumbers.FLOAT_NORMALIZER</span></code> [...]
</ul>
</section>
<section id="arrays-zip">
diff --git a/contributor-guide/floating_point.html b/contributor-guide/floating_point.html
index f8d504df61..8e1b6d1ff0 100644
--- a/contributor-guide/floating_point.html
+++ b/contributor-guide/floating_point.html
@@ -631,6 +631,11 @@ into <code class="docutils literal notranslate"><span class="pre">0.0</span></co
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code> looks up the elements of a flat array in a <code class="docutils literal notranslate"><span class="pre">java.util.HashSet</span></code> of boxed values,
+so it follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code> and keeps signed zeros apart up to Spark 4.0.4 and 4.1.3. The same
+two changes make the zeros one value: Spark 4.2.0 normalizes its arguments in the plan
+(SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize each flat <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> element before
+the lookup (SPARK-59602). Nested elements compare in the SQL ordering on every version.</p></li>
<li><p>Map construction (<code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code>) finds a duplicate <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key in a
<code class="docutils literal notranslate"><span class="pre">HashMap</span></code> of boxed values, which follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code>, and a duplicate struct or array key in a
<code class="docutils literal notranslate"><span class="pre">TreeMap</span></code> in the SQL ordering. From Spark 4.0 it normalizes a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key before the
diff --git a/searchindex.js b/searchindex.js
index a85f2525bd..a49eddc3f1 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/floating-point.html b/user-guide/latest/compatibility/floating-point.html
index 3fd44710b6..54b6aa4856 100644
--- a/user-guide/latest/compatibility/floating-point.html
+++ b/user-guide/latest/compatibility/floating-point.html
@@ -960,6 +960,17 @@ differences are acceptable for your data.</p>
<p>The check uses the Spark version number, not the changes a build contains. A vendor build that
reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.</p>
</section>
+<section id="arrays-overlap">
+<h2><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code><a class="headerlink" href="#arrays-overlap" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code> runs natively on every Spark version and follows that version’s rule for <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code>
+and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> as one
+value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
+one value on every version, and nested arrays and structs compare their elements in Spark’s order,
+in which the two zeros are equal.</p>
+<p>Comet chooses the rule by the Spark version number. On a vendor build that reports an older
+version but includes SPARK-59602, Comet returns <code class="docutils literal notranslate"><span class="pre">false</span></code> for two flat arrays whose only common value
+is a zero of opposite sign, where Spark returns <code class="docutils literal notranslate"><span class="pre">true</span></code>.</p>
+</section>
<section id="array-contains-array-remove-and-sort-array">
<h2><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code><a class="headerlink" href="#array-contains-array-remove-and-sort-array" title="Link to this heading">#</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">array_contains</span></code> compares <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> elements as Spark does: <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> equals <code class="docutils literal notranslate"><span class="pre">0.0</span></code>, and all
@@ -1036,6 +1047,7 @@ values don’t affect either function. The expressions’ entries in the
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ordering-nan-and-signed-zero-0-0-vs-0-0">Ordering: NaN and signed zero (<code class="docutils literal notranslate"><span class="pre">-0.0</span></code> vs <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#min-max-greatest-and-least"><code class="docutils literal notranslate"><span class="pre">min</span></code>, <code class="docutils literal notranslate"><span class="pre">max</span></code>, <code class="docutils literal notranslate"><span class="pre">greatest</span></code>, and <code class="docutils literal notranslate"><span class="pre">least</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-distinct-and-union">Array distinct and union</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#arrays-overlap"><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-contains-array-remove-and-sort-array"><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#map-construction">Map construction</a></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org