You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/20 05:33:24 UTC
(datafusion-comet) branch asf-site updated (9728d052ab -> 0813634dd3)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard 9728d052ab Publish built docs triggered by cae599ace3bbc76fa2e29133e3a9504ff7b15d61
new 30ae34c3a8 Publish built docs triggered by 3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83
new 0813634dd3 Publish built docs triggered by 137c8945398d77523871876c6f939f038136d847
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9728d052ab)
\
N -- N -- N refs/heads/asf-site (0813634dd3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../expression-audits/hash_funcs.md.txt | 7 +++++--
_sources/contributor-guide/jvm_shuffle.md.txt | 4 +---
_sources/contributor-guide/native_shuffle.md.txt | 9 +++------
_sources/user-guide/latest/configs.md.txt | 2 +-
_sources/user-guide/latest/expressions.md.txt | 4 ++--
_sources/user-guide/latest/tuning/shuffle.md.txt | 17 +++--------------
contributor-guide/expression-audits/hash_funcs.html | 6 ++++--
contributor-guide/jvm_shuffle.html | 4 +---
contributor-guide/native_shuffle.html | 11 ++++-------
searchindex.js | 2 +-
user-guide/latest/configs.html | 2 +-
user-guide/latest/expressions.html | 4 ++--
user-guide/latest/tuning/shuffle.html | 15 +++------------
14 files changed, 32 insertions(+), 57 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/02: Publish built docs triggered by 3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 30ae34c3a8c2a6fdbac0c64e3521a29f218ef781
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Oct 20 05:10:16 2026 +0000
Publish built docs triggered by 3565f2f9b1351ddcd63a76dff1532ddcdcbd7d83
---
_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../contributor-guide/expression-audits/array_funcs.md.txt | 1 +
_sources/contributor-guide/floating_point.md.txt | 5 +++++
.../user-guide/latest/compatibility/floating-point.md.txt | 12 ++++++++++++
contributor-guide/expression-audits/array_funcs.html | 1 +
contributor-guide/floating_point.html | 5 +++++
searchindex.js | 2 +-
user-guide/latest/compatibility/floating-point.html | 12 ++++++++++++
8 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index 27633b65b0..f36f31d4ff 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/array_funcs.md.txt b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
index af635270e7..87ff575307 100644
--- a/_sources/contributor-guide/expression-audits/array_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/array_funcs.md.txt
@@ -149,6 +149,7 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArraysOverlap(left, right)`; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via `CometArraysOverlap` to the native `spark_arrays_overlap` UDF, which implements the same three-valued logic.
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
+- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates, 2026-10-14): `fastEval` and its generated code normalize each flat `FLOAT` or `DOUBLE` element with `NormalizeFloatingNumbers.FLOAT_NORMALIZER` / `DOUBLE_NORMALIZER` before the `java.util.HashSet` lookup (SPARK-59602), so `-0.0` and `0.0` are one value. Spark 4.2.0 normalizes the arguments in the plan instead (SPARK-54918). Earlier releases keep the zeros apart, and every release treats all NaNs as one value. On the versions where `Arra [...]
## arrays_zip
diff --git a/_sources/contributor-guide/floating_point.md.txt b/_sources/contributor-guide/floating_point.md.txt
index 758350db8f..220a7e4ddb 100644
--- a/_sources/contributor-guide/floating_point.md.txt
+++ b/_sources/contributor-guide/floating_point.md.txt
@@ -62,6 +62,11 @@ it runs against:
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).
+- `arrays_overlap` looks up the elements of a flat array in a `java.util.HashSet` of boxed values,
+ so it follows `Double.equals` and keeps signed zeros apart up to Spark 4.0.4 and 4.1.3. The same
+ two changes make the zeros one value: Spark 4.2.0 normalizes its arguments in the plan
+ (SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize each flat `FLOAT` or `DOUBLE` element before
+ the lookup (SPARK-59602). Nested elements compare in the SQL ordering on every version.
- Map construction (`ArrayBasedMapBuilder`) finds a duplicate `FLOAT` or `DOUBLE` key in a
`HashMap` of boxed values, which follows `Double.equals`, and a duplicate struct or array key in a
`TreeMap` in the SQL ordering. From Spark 4.0 it normalizes a `FLOAT` or `DOUBLE` key before the
diff --git a/_sources/user-guide/latest/compatibility/floating-point.md.txt b/_sources/user-guide/latest/compatibility/floating-point.md.txt
index 77d76f53c6..c94fc7dc68 100644
--- a/_sources/user-guide/latest/compatibility/floating-point.md.txt
+++ b/_sources/user-guide/latest/compatibility/floating-point.md.txt
@@ -148,6 +148,18 @@ differences are acceptable for your data.
The check uses the Spark version number, not the changes a build contains. A vendor build that
reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.
+## `arrays_overlap`
+
+`arrays_overlap` runs natively on every Spark version and follows that version's rule for `FLOAT`
+and `DOUBLE` elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat `-0.0` and `0.0` as one
+value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
+one value on every version, and nested arrays and structs compare their elements in Spark's order,
+in which the two zeros are equal.
+
+Comet chooses the rule by the Spark version number. On a vendor build that reports an older
+version but includes SPARK-59602, Comet returns `false` for two flat arrays whose only common value
+is a zero of opposite sign, where Spark returns `true`.
+
## `array_contains`, `array_remove` and `sort_array`
`array_contains` compares `FLOAT` and `DOUBLE` elements as Spark does: `-0.0` equals `0.0`, and all
diff --git a/contributor-guide/expression-audits/array_funcs.html b/contributor-guide/expression-audits/array_funcs.html
index 39c176f9d0..b5e6631685 100644
--- a/contributor-guide/expression-audits/array_funcs.html
+++ b/contributor-guide/expression-audits/array_funcs.html
@@ -724,6 +724,7 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">ArraysOverlap(left,</span> <span class="pre">right)</span></code>; three-valued logic (TRUE if any common non-null element, NULL if a null is present and no overlap is found in non-nulls, FALSE otherwise). Comet routes via <code class="docutils literal notranslate"><span class="pre">CometArraysOverlap</span></code> to the native <code class="docutils literal notranslate"><span [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">NullIntolerant</span></code> -> <code class="docutils literal notranslate"><span class="pre">nullIntolerant</span></code> field refactor.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
+<li><p>Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates, 2026-10-14): <code class="docutils literal notranslate"><span class="pre">fastEval</span></code> and its generated code normalize each flat <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> element with <code class="docutils literal notranslate"><span class="pre">NormalizeFloatingNumbers.FLOAT_NORMALIZER</span></code> [...]
</ul>
</section>
<section id="arrays-zip">
diff --git a/contributor-guide/floating_point.html b/contributor-guide/floating_point.html
index f8d504df61..8e1b6d1ff0 100644
--- a/contributor-guide/floating_point.html
+++ b/contributor-guide/floating_point.html
@@ -631,6 +631,11 @@ into <code class="docutils literal notranslate"><span class="pre">0.0</span></co
(before 3.5.2, unless probing merges them). Spark 4.2.0 normalizes their arguments in the plan
(SPARK-54918). From 4.0.5, 4.1.4 and 4.2.1 they normalize while they evaluate instead
(SPARK-59602).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code> looks up the elements of a flat array in a <code class="docutils literal notranslate"><span class="pre">java.util.HashSet</span></code> of boxed values,
+so it follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code> and keeps signed zeros apart up to Spark 4.0.4 and 4.1.3. The same
+two changes make the zeros one value: Spark 4.2.0 normalizes its arguments in the plan
+(SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize each flat <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> element before
+the lookup (SPARK-59602). Nested elements compare in the SQL ordering on every version.</p></li>
<li><p>Map construction (<code class="docutils literal notranslate"><span class="pre">ArrayBasedMapBuilder</span></code>) finds a duplicate <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key in a
<code class="docutils literal notranslate"><span class="pre">HashMap</span></code> of boxed values, which follows <code class="docutils literal notranslate"><span class="pre">Double.equals</span></code>, and a duplicate struct or array key in a
<code class="docutils literal notranslate"><span class="pre">TreeMap</span></code> in the SQL ordering. From Spark 4.0 it normalizes a <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> or <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> key before the
diff --git a/searchindex.js b/searchindex.js
index a85f2525bd..a49eddc3f1 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/floating-point.html b/user-guide/latest/compatibility/floating-point.html
index 3fd44710b6..54b6aa4856 100644
--- a/user-guide/latest/compatibility/floating-point.html
+++ b/user-guide/latest/compatibility/floating-point.html
@@ -960,6 +960,17 @@ differences are acceptable for your data.</p>
<p>The check uses the Spark version number, not the changes a build contains. A vendor build that
reports an older version but includes SPARK-54918 or SPARK-59602 still falls back.</p>
</section>
+<section id="arrays-overlap">
+<h2><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code><a class="headerlink" href="#arrays-overlap" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code> runs natively on every Spark version and follows that version’s rule for <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code>
+and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> and <code class="docutils literal notranslate"><span class="pre">0.0</span></code> as one
+value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is
+one value on every version, and nested arrays and structs compare their elements in Spark’s order,
+in which the two zeros are equal.</p>
+<p>Comet chooses the rule by the Spark version number. On a vendor build that reports an older
+version but includes SPARK-59602, Comet returns <code class="docutils literal notranslate"><span class="pre">false</span></code> for two flat arrays whose only common value
+is a zero of opposite sign, where Spark returns <code class="docutils literal notranslate"><span class="pre">true</span></code>.</p>
+</section>
<section id="array-contains-array-remove-and-sort-array">
<h2><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code><a class="headerlink" href="#array-contains-array-remove-and-sort-array" title="Link to this heading">#</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">array_contains</span></code> compares <code class="docutils literal notranslate"><span class="pre">FLOAT</span></code> and <code class="docutils literal notranslate"><span class="pre">DOUBLE</span></code> elements as Spark does: <code class="docutils literal notranslate"><span class="pre">-0.0</span></code> equals <code class="docutils literal notranslate"><span class="pre">0.0</span></code>, and all
@@ -1036,6 +1047,7 @@ values don’t affect either function. The expressions’ entries in the
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#ordering-nan-and-signed-zero-0-0-vs-0-0">Ordering: NaN and signed zero (<code class="docutils literal notranslate"><span class="pre">-0.0</span></code> vs <code class="docutils literal notranslate"><span class="pre">+0.0</span></code>)</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#min-max-greatest-and-least"><code class="docutils literal notranslate"><span class="pre">min</span></code>, <code class="docutils literal notranslate"><span class="pre">max</span></code>, <code class="docutils literal notranslate"><span class="pre">greatest</span></code>, and <code class="docutils literal notranslate"><span class="pre">least</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-distinct-and-union">Array distinct and union</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#arrays-overlap"><code class="docutils literal notranslate"><span class="pre">arrays_overlap</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#array-contains-array-remove-and-sort-array"><code class="docutils literal notranslate"><span class="pre">array_contains</span></code>, <code class="docutils literal notranslate"><span class="pre">array_remove</span></code> and <code class="docutils literal notranslate"><span class="pre">sort_array</span></code></a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#map-construction">Map construction</a></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 02/02: Publish built docs triggered by 137c8945398d77523871876c6f939f038136d847
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 0813634dd39592aeeec1ea306add1bd289cbd0f3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Oct 20 05:33:18 2026 +0000
Publish built docs triggered by 137c8945398d77523871876c6f939f038136d847
---
...id-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../expression-audits/hash_funcs.md.txt | 7 +++-
_sources/contributor-guide/jvm_shuffle.md.txt | 4 +-
_sources/contributor-guide/native_shuffle.md.txt | 9 ++---
_sources/user-guide/latest/configs.md.txt | 4 +-
_sources/user-guide/latest/expressions.md.txt | 4 +-
_sources/user-guide/latest/rust_udfs.md.txt | 31 +++++++++++++--
_sources/user-guide/latest/tuning/shuffle.md.txt | 17 ++------
.../expression-audits/hash_funcs.html | 6 ++-
contributor-guide/jvm_shuffle.html | 4 +-
contributor-guide/native_shuffle.html | 11 ++----
searchindex.js | 2 +-
user-guide/latest/configs.html | 10 ++++-
user-guide/latest/expressions.html | 4 +-
user-guide/latest/rust_udfs.html | 46 +++++++++++++++++++---
user-guide/latest/tuning/shuffle.html | 15 ++-----
16 files changed, 110 insertions(+), 66 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index f36f31d4ff..6e6f75bc9d 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/hash_funcs.md.txt b/_sources/contributor-guide/expression-audits/hash_funcs.md.txt
index c3ad4efe7b..a294069df1 100644
--- a/_sources/contributor-guide/expression-audits/hash_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/hash_funcs.md.txt
@@ -34,7 +34,8 @@
- Spark 3.5.8 (audited 2026-05-27): baseline. `Murmur3Hash(children, seed) extends HashExpression[Int]`; produces a Murmur3 hash with a configurable Int seed and `IntegerType` result. Comet routes via `CometMurmur3Hash` to the native `murmur3_hash` UDF.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; some inner helper refactors only.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Known limitation: `DecimalType` children with precision > 18 fall back because Spark hashes them through Java `BigDecimal`; `TimeType` (Spark 4.0+) is also unsupported. The same limitations apply to `xxhash64`, `sha1`, `sha2` through the shared `HashUtils`.
+- Decimals with precision 19–38, including nested decimals, hash natively using the minimal signed big-endian bytes of the unscaled value, matching Spark's `BigInteger.toByteArray()`. Precision ≤18 retains unscaled-long hashing. `TimeType` remains unsupported through the shared `HashUtils`.
+- Performance (tuned 2026-09-28, issue #5994): typed wide-decimal list scans avoid per-element Arrow slices and recursive dispatch. Local Criterion measurements were 5.0–14.2x faster across 8,192-row `decimal(38,0)` List, LargeList and FixedSizeList inputs with 2/32 elements and no/sparse/dense nulls; paired scalar remeasurement found no significant slowdown. Benchmark: `benches/decimal_hash.rs`.
## md5
@@ -66,10 +67,12 @@
## xxhash64
+- Performance (tuned 2026-09-28, issue #5994): the shared typed wide-decimal list path was 2.1–15.8x faster in local Criterion measurements across 8,192-row `decimal(38,0)` List, LargeList and FixedSizeList inputs with 2/32 elements and no/sparse/dense nulls; paired scalar remeasurement found no significant slowdown. Benchmark: `benches/decimal_hash.rs`.
+
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `XxHash64(children, seed) extends HashExpression[Long]`; produces an xxHash64 hash with a configurable Long seed and `LongType` result. Comet routes via `CometXxHash64` to the native `xxhash64` UDF.
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
-- Upstream (2026-09-15): Comet's native `xxhash64` UDF delegates compatible arguments at Spark's default seed (`42`) to `datafusion-spark::SparkXxhash64`. Differential tests in `native/spark-expr/src/hash_funcs/xxhash64_diff.rs` compare Comet's kernel against `SparkXxhash64` for primitives, both Decimal128 widths, dictionaries, lists, maps, and nested combinations, and record where the two differ on a NaN whose bits are not canonical. The Comet kernel is retained for a non-default seed ( [...]
+- Native routing: Comet's native `xxhash64` UDF delegates compatible arguments at Spark's default seed (`42`) to `datafusion-spark::SparkXxhash64`. Differential tests in `native/spark-expr/src/hash_funcs/xxhash64_diff.rs` compare Comet's kernel against `SparkXxhash64` for primitives, narrow decimals, dictionaries, lists, maps, and nested combinations, and record where the two differ on a NaN whose bits are not canonical. Wide-decimal tests verify that the expression stays on Comet's kern [...]
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/contributor-guide/jvm_shuffle.md.txt b/_sources/contributor-guide/jvm_shuffle.md.txt
index 711e57ea9d..43acc7a655 100644
--- a/_sources/contributor-guide/jvm_shuffle.md.txt
+++ b/_sources/contributor-guide/jvm_shuffle.md.txt
@@ -52,9 +52,7 @@ JVM shuffle (`CometColumnarExchange`) is used instead of native shuffle (`CometE
range key always falls back here. `HashPartitioning` keys must be primitive only by default:
setting `spark.comet.shuffle.native.partitioning.hash.nested.enabled` to `true` keeps struct and
array keys, and map keys on Spark 4.0 and later, on the native path. The config defaults to
- `false`, so a complex hash key falls back to JVM columnar shuffle unless it is enabled.
- Decimal hash keys with precision greater than 18, including nested decimal leaves, also use
- JVM shuffle when there is more than one partition and the shuffle mode allows it. See
+ `false`, so a complex hash key falls back to JVM columnar shuffle unless it is enabled. See
[Supported partition key types](native_shuffle.md#when-native-shuffle-is-used) for the exact
rules. Complex types are fully supported as data columns in both implementations.
diff --git a/_sources/contributor-guide/native_shuffle.md.txt b/_sources/contributor-guide/native_shuffle.md.txt
index 25055afc85..a9d47f0f57 100644
--- a/_sources/contributor-guide/native_shuffle.md.txt
+++ b/_sources/contributor-guide/native_shuffle.md.txt
@@ -56,9 +56,9 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
`spark.comet.convert.shuffleInput.enabled` is set. Then a shuffle that JVM shuffle would take
uses native shuffle instead, provided native shuffle supports the partitioning and the
columns, and a `CometSparkToColumnarExec` converts the child's rows to Arrow for it. A shuffle
- that hashes a string, a value computed from a string, or a decimal wider than 18 digits stays
- on JVM shuffle, because native shuffle would not put every row in the partition Spark's
- partitioner does (`CometShuffleExchangeExec.convertsInputForNativeShuffle`).
+ that hashes a string or a value computed from a string stays on JVM shuffle, because native
+ shuffle would not put every row in the partition Spark's partitioner does
+ (`CometShuffleExchangeExec.convertsInputForNativeShuffle`).
3. **Supported partitioning type**: Native shuffle supports:
- `HashPartitioning`
@@ -83,9 +83,6 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
Spark's `mapsort` normalization makes physical entry order irrelevant. A collated string at any
depth still disqualifies the key. The config defaults to `false` pending measurement of the
nested hashing paths, so by default a complex hash key falls back to JVM shuffle.
- Decimal leaves with precision greater than 18 also disqualify a hash key when there is more
- than one output partition: native decimal hashing differs from Spark's. A one-partition hash
- exchange stays native because the writer serializes it as `SinglePartition` without hashing.
## Architecture
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index cc891408bb..a201c61986 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -64,7 +64,7 @@ Comet provides the following configuration settings.
| `spark.comet.convert.range.enabled` | When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format. | false |
| `spark.comet.convert.rdd.enabled` | When enabled, data from a DataFrame created from an RDD of rows, for example with `spark.createDataFrame(rdd, schema)`, will be converted to Arrow format. | false |
| `spark.comet.convert.rowDataSource.enabled` | When enabled, data from Data Source V1 relations that are not file-based, such as JDBC tables, will be converted to Arrow format. Spark scans these relations with `RowDataSourceScanExec`. | false |
-| `spark.comet.convert.shuffleInput.enabled` | When enabled, a shuffle that would use Comet's JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input's columns, which rules out calendar intervals. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a string, stay [...]
+| `spark.comet.convert.shuffleInput.enabled` | When enabled, a shuffle that would use Comet's JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input's columns, which rules out calendar intervals. A shuffle that hashes a string or a value computed from a string stays on the JVM columnar shuffle, so th [...]
| `spark.comet.convert.typedDataset.enabled` | When enabled, the output of typed Dataset operations, such as `map`, `flatMap`, `mapPartitions` and `groupByKey(...).mapGroups`, will be converted to Arrow format so that the operators above them can run natively. The user function still runs in Spark. This pays off when the operators above do enough work, such as an aggregation over many groups, and can be slower when they are cheap, such as an aggregation over a few groups after a selectiv [...]
| `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose. | false |
| `spark.comet.enabled` | Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and `spark.comet.exec.enabled` need to be enabled. It can be overridden by the environment variable `ENABLE_COMET`. | true |
@@ -89,6 +89,8 @@ Comet provides the following configuration settings.
| `spark.comet.metrics.enabled` | Whether to enable Comet metrics reporting through Spark's external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be se [...]
| `spark.comet.metrics.updateInterval` | The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion. | 3000 |
| `spark.comet.nativeLoadRequired` | Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted. | false |
+| `spark.comet.nativeUdf.allowedPaths` | A comma-separated list of directories that native UDF libraries may be loaded from. When set, a library is loaded only if its path is absolute and, after symlinks and `..` are resolved, lies under one of these directories, which are themselves resolved the same way. A path outside them fails on the driver at registration and again on the executor. When empty, any path is allowed. This limits which files Comet will load, not what the code in them d [...]
+| `spark.comet.nativeUdf.enabled` | Whether native UDF libraries may be loaded. A native UDF is a shared library that runs with the full privileges of the JVM process, so an operator who does not want any can turn the feature off. When false, `CometNativeUDF.register` fails on the driver, and a plan that names a native UDF library fails on the executor rather than loading it. | true |
| `spark.comet.operator.DataWritingCommandExec.allowIncompatible` | Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE`. | false |
| `spark.comet.operator.WriteFilesExec.allowIncompatible` | Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE`. | false |
| `spark.comet.sparkToColumnar.enabled` | Whether to convert the output of the leaf operators named in `spark.comet.sparkToColumnar.supportedOperatorList` to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have `spark.comet.convert` configs of their own. When the list is not set, this converts those four instead, which is deprecated and will be removed in a future major release. | false |
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 7a54c665e5..b981d3c0d1 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -358,12 +358,12 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
| Function | Status | Implementation | Notes |
| --- | --- | --- | --- |
| `crc32` | ✅ | Native | |
-| `hash` | ✅ | Native | |
+| `hash` | ✅ | Native | Includes decimals with precision 19–38 and nested decimals |
| `md5` | ✅ | Native | |
| `sha` | ✅ | Native | |
| `sha1` | ✅ | Native | |
| `sha2` | ✅ | Native | |
-| `xxhash64` | ✅ | Native | |
+| `xxhash64` | ✅ | Native | Includes decimals with precision 19–38 and nested decimals |
---
diff --git a/_sources/user-guide/latest/rust_udfs.md.txt b/_sources/user-guide/latest/rust_udfs.md.txt
index 8620a7676d..ae1a0957f1 100644
--- a/_sources/user-guide/latest/rust_udfs.md.txt
+++ b/_sources/user-guide/latest/rust_udfs.md.txt
@@ -176,9 +176,30 @@ rather than accept a function whose volatility it would go on to ignore.
`libraryPath` is passed to the platform's dynamic loader. An absolute path is what you want in
practice, and it is what the rest of this page assumes, but a bare library name resolves the same
way it would for any other shared object, through `LD_LIBRARY_PATH` on Linux and
-`DYLD_LIBRARY_PATH` on macOS. That is a convenience, not a sandbox: Comet does not restrict which
-paths may be loaded, so it makes no difference to the trust decision described under
-[Limitations](#limitations).
+`DYLD_LIBRARY_PATH` on macOS. That is a convenience, not a sandbox: unless you set the configs
+below, Comet does not restrict which paths may be loaded, so it makes no difference to the trust
+decision described under [Limitations](#limitations).
+
+## Restricting which libraries can be loaded
+
+Loading a library runs its code with the privileges of the JVM, so an operator can limit it:
+
+| Config | Default | Effect |
+| ------------------------------------ | ------- | ------------------------------------------------------------ |
+| `spark.comet.nativeUdf.enabled` | `true` | When `false`, no native UDF library is loaded. |
+| `spark.comet.nativeUdf.allowedPaths` | empty | Comma-separated directories a library must be under, if set. |
+
+With `allowedPaths` set, a library is loaded only if its path is absolute and, once symlinks and
+`..` are resolved, lies under one of the listed directories (which are resolved the same way). A bare
+library name is refused, because the dynamic loader's search path is not something Comet can check.
+The check applies on the driver, where `register` fails with `CometNativeUdfNotAllowedException`,
+and again on each executor before it loads a library named in a plan.
+
+These are ordinary Spark configs, so they are set in the cluster's defaults and are not a defense
+against a user who can change session configuration: such a user can set them back. The allow-list
+limits which files Comet will load, not what the code in them does, and a file that is replaced
+between the check and the load is not detected. List only directories that are writable by users
+you trust.
### When Spark evaluates the call
@@ -308,7 +329,9 @@ This feature is at an early stage. The current limitations are:
layout.
- **Loading a library is loading native code.** It runs with the full privileges of the executor
process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
- Only register libraries you trust and control.
+ Only register libraries you trust and control. See
+ [Restricting which libraries can be loaded](#restricting-which-libraries-can-be-loaded) for the
+ configs that turn the feature off or limit the directories libraries may come from.
- Once loaded, a library stays loaded for the life of the process, and `register` loads it on the
driver as well as on the executors. To deploy a new build, write it to a new path and register
that, or restart the executors and the driver. Never copy a new build over a library that is
diff --git a/_sources/user-guide/latest/tuning/shuffle.md.txt b/_sources/user-guide/latest/tuning/shuffle.md.txt
index b01a82ccff..edcf3b4970 100644
--- a/_sources/user-guide/latest/tuning/shuffle.md.txt
+++ b/_sources/user-guide/latest/tuning/shuffle.md.txt
@@ -53,17 +53,6 @@ scalar types. Hash partitioning keys must be scalar types unless
and later) map keys. That setting is disabled by default until the performance of the nested hashing paths has been
measured. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.
-Hash partitioning into multiple partitions on decimal keys with precision greater than 18 falls back because native
-hashing does not match Spark's partition assignments. Mixing the two partitioners can silently lose join rows.
-The restriction also applies to decimal leaves in nested hash keys. Wider decimals remain supported as payload
-columns, range partitioning keys, and in single-partition shuffles, including one-partition hash exchanges.
-
-With `spark.comet.shuffle.mode=auto`, Comet uses Columnar Shuffle when eligible, adding a columnar-to-row-to-columnar
-conversion. With `native`, it uses Spark shuffle, which can also restore aggregates such as `collect_list` and
-`collect_set` to Spark to preserve buffer compatibility. Celeborn uses Spark/Celeborn shuffle for unsupported native
-keys because it has no Comet columnar fallback. Spark-compatible native decimal hashing is tracked in
-[#5994](https://github.com/apache/datafusion-comet/issues/5994).
-
### Columnar (JVM) Shuffle
Comet Columnar shuffle is JVM-based and supports `HashPartitioning`, `RoundRobinPartitioning`, `RangePartitioning`, and
@@ -82,9 +71,9 @@ converts the child's rows to Arrow, as the `spark.comet.convert.*` settings do f
batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning
the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert
all of the child's columns, which rules out calendar intervals, and arrays and maps other than `array<string>` and
-`map<string,string>`. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a
-string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on
-the JVM outside Comet's memory pools. The setting is disabled by default.
+`map<string,string>`. A shuffle that hashes a string or a value computed from a string stays on columnar shuffle, so
+that it partitions rows as Spark does. The converted batches are allocated on the JVM outside Comet's memory pools.
+The setting is disabled by default.
### Automatic Revert to Spark Shuffle
diff --git a/contributor-guide/expression-audits/hash_funcs.html b/contributor-guide/expression-audits/hash_funcs.html
index 6b47335966..15c59dbd9d 100644
--- a/contributor-guide/expression-audits/hash_funcs.html
+++ b/contributor-guide/expression-audits/hash_funcs.html
@@ -579,7 +579,8 @@ under the License.
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Murmur3Hash(children,</span> <span class="pre">seed)</span> <span class="pre">extends</span> <span class="pre">HashExpression[Int]</span></code>; produces a Murmur3 hash with a configurable Int seed and <code class="docutils literal notranslate"><span class="pre">IntegerType</span></code> result. Comet routes via <code class="docutils literal notranslate"><span class="pre">Come [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged; some inner helper refactors only.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Known limitation: <code class="docutils literal notranslate"><span class="pre">DecimalType</span></code> children with precision > 18 fall back because Spark hashes them through Java <code class="docutils literal notranslate"><span class="pre">BigDecimal</span></code>; <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> (Spark 4.0+) is also unsupported. The same limitations apply to <code class="docutils literal notranslate"><span class="pre">xxh [...]
+<li><p>Decimals with precision 19–38, including nested decimals, hash natively using the minimal signed big-endian bytes of the unscaled value, matching Spark’s <code class="docutils literal notranslate"><span class="pre">BigInteger.toByteArray()</span></code>. Precision ≤18 retains unscaled-long hashing. <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> remains unsupported through the shared <code class="docutils literal notranslate"><span class="pre">H [...]
+<li><p>Performance (tuned 2026-09-28, issue #5994): typed wide-decimal list scans avoid per-element Arrow slices and recursive dispatch. Local Criterion measurements were 5.0–14.2x faster across 8,192-row <code class="docutils literal notranslate"><span class="pre">decimal(38,0)</span></code> List, LargeList and FixedSizeList inputs with 2/32 elements and no/sparse/dense nulls; paired scalar remeasurement found no significant slowdown. Benchmark: <code class="docutils literal notranslate [...]
</ul>
</section>
<section id="md5">
@@ -621,11 +622,12 @@ under the License.
<section id="xxhash64">
<h2>xxhash64<a class="headerlink" href="#xxhash64" title="Link to this heading">#</a></h2>
<ul class="simple">
+<li><p>Performance (tuned 2026-09-28, issue #5994): the shared typed wide-decimal list path was 2.1–15.8x faster in local Criterion measurements across 8,192-row <code class="docutils literal notranslate"><span class="pre">decimal(38,0)</span></code> List, LargeList and FixedSizeList inputs with 2/32 elements and no/sparse/dense nulls; paired scalar remeasurement found no significant slowdown. Benchmark: <code class="docutils literal notranslate"><span class="pre">benches/decimal_hash.rs [...]
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">XxHash64(children,</span> <span class="pre">seed)</span> <span class="pre">extends</span> <span class="pre">HashExpression[Long]</span></code>; produces an xxHash64 hash with a configurable Long seed and <code class="docutils literal notranslate"><span class="pre">LongType</span></code> result. Comet routes via <code class="docutils literal notranslate"><span class="pre">CometX [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): semantics unchanged.</p></li>
<li><p>Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.</p></li>
-<li><p>Upstream (2026-09-15): Comet’s native <code class="docutils literal notranslate"><span class="pre">xxhash64</span></code> UDF delegates compatible arguments at Spark’s default seed (<code class="docutils literal notranslate"><span class="pre">42</span></code>) to <code class="docutils literal notranslate"><span class="pre">datafusion-spark::SparkXxhash64</span></code>. Differential tests in <code class="docutils literal notranslate"><span class="pre">native/spark-expr/src/hash_fun [...]
+<li><p>Native routing: Comet’s native <code class="docutils literal notranslate"><span class="pre">xxhash64</span></code> UDF delegates compatible arguments at Spark’s default seed (<code class="docutils literal notranslate"><span class="pre">42</span></code>) to <code class="docutils literal notranslate"><span class="pre">datafusion-spark::SparkXxhash64</span></code>. Differential tests in <code class="docutils literal notranslate"><span class="pre">native/spark-expr/src/hash_funcs/xxha [...]
</ul>
</section>
</section>
diff --git a/contributor-guide/jvm_shuffle.html b/contributor-guide/jvm_shuffle.html
index a4c841ea46..3fcd3de81b 100644
--- a/contributor-guide/jvm_shuffle.html
+++ b/contributor-guide/jvm_shuffle.html
@@ -587,9 +587,7 @@ shuffle instead. See <a class="reference internal" href="native_shuffle.html#whe
range key always falls back here. <code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code> keys must be primitive only by default:
setting <code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.native.partitioning.hash.nested.enabled</span></code> to <code class="docutils literal notranslate"><span class="pre">true</span></code> keeps struct and
array keys, and map keys on Spark 4.0 and later, on the native path. The config defaults to
-<code class="docutils literal notranslate"><span class="pre">false</span></code>, so a complex hash key falls back to JVM columnar shuffle unless it is enabled.
-Decimal hash keys with precision greater than 18, including nested decimal leaves, also use
-JVM shuffle when there is more than one partition and the shuffle mode allows it. See
+<code class="docutils literal notranslate"><span class="pre">false</span></code>, so a complex hash key falls back to JVM columnar shuffle unless it is enabled. See
<a class="reference internal" href="native_shuffle.html#when-native-shuffle-is-used"><span class="std std-ref">Supported partition key types</span></a> for the exact
rules. Complex types are fully supported as data columns in both implementations.</p></li>
</ol>
diff --git a/contributor-guide/native_shuffle.html b/contributor-guide/native_shuffle.html
index 0fd53a5d3d..fb09184791 100644
--- a/contributor-guide/native_shuffle.html
+++ b/contributor-guide/native_shuffle.html
@@ -588,9 +588,9 @@ columnar output. Row-based Spark operators require JVM shuffle, unless
<code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled</span></code> is set. Then a shuffle that JVM shuffle would take
uses native shuffle instead, provided native shuffle supports the partitioning and the
columns, and a <code class="docutils literal notranslate"><span class="pre">CometSparkToColumnarExec</span></code> converts the child’s rows to Arrow for it. A shuffle
-that hashes a string, a value computed from a string, or a decimal wider than 18 digits stays
-on JVM shuffle, because native shuffle would not put every row in the partition Spark’s
-partitioner does (<code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec.convertsInputForNativeShuffle</span></code>).</p></li>
+that hashes a string or a value computed from a string stays on JVM shuffle, because native
+shuffle would not put every row in the partition Spark’s partitioner does
+(<code class="docutils literal notranslate"><span class="pre">CometShuffleExchangeExec.convertsInputForNativeShuffle</span></code>).</p></li>
<li><p><strong>Supported partitioning type</strong>: Native shuffle supports:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">HashPartitioning</span></code></p></li>
@@ -616,10 +616,7 @@ but those keys are rejected as range keys for being nested.</p></li>
arrays as keys, checked recursively to their leaves, and maps on Spark 4.0 and later, where
Spark’s <code class="docutils literal notranslate"><span class="pre">mapsort</span></code> normalization makes physical entry order irrelevant. A collated string at any
depth still disqualifies the key. The config defaults to <code class="docutils literal notranslate"><span class="pre">false</span></code> pending measurement of the
-nested hashing paths, so by default a complex hash key falls back to JVM shuffle.
-Decimal leaves with precision greater than 18 also disqualify a hash key when there is more
-than one output partition: native decimal hashing differs from Spark’s. A one-partition hash
-exchange stays native because the writer serializes it as <code class="docutils literal notranslate"><span class="pre">SinglePartition</span></code> without hashing.</p></li>
+nested hashing paths, so by default a complex hash key falls back to JVM shuffle.</p></li>
</ul>
</li>
</ol>
diff --git a/searchindex.js b/searchindex.js
index a49eddc3f1..a2247532aa 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index c401d58121..ae217133ea 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -973,7 +973,7 @@ under the License.
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.shuffleInput.enabled</span></code></p></td>
-<td><p>When enabled, a shuffle that would use Comet’s JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input’s columns, which rules out calendar intervals. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a string, stays on the JVM columnar shuffle, so that i [...]
+<td><p>When enabled, a shuffle that would use Comet’s JVM columnar shuffle because its input comes from a Spark operator converts that input to Arrow format and uses Comet native shuffle instead. This applies only where native shuffle supports the partitioning and Comet can convert all of the input’s columns, which rules out calendar intervals. A shuffle that hashes a string or a value computed from a string stays on the JVM columnar shuffle, so that it partitions rows as Spark does.</p></td>
<td><p>false</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.typedDataset.enabled</span></code></p></td>
@@ -1072,6 +1072,14 @@ under the License.
<td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
<td><p>false</p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.allowedPaths</span></code></p></td>
+<td><p>A comma-separated list of directories that native UDF libraries may be loaded from. When set, a library is loaded only if its path is absolute and, after symlinks and <code class="docutils literal notranslate"><span class="pre">..</span></code> are resolved, lies under one of these directories, which are themselves resolved the same way. A path outside them fails on the driver at registration and again on the executor. When empty, any path is allowed. This limits which files Comet [...]
+<td><p></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.enabled</span></code></p></td>
+<td><p>Whether native UDF libraries may be loaded. A native UDF is a shared library that runs with the full privileges of the JVM process, so an operator who does not want any can turn the feature off. When false, <code class="docutils literal notranslate"><span class="pre">CometNativeUDF.register</span></code> fails on the driver, and a plan that names a native UDF library fails on the executor rather than loading it.</p></td>
+<td><p>true</p></td>
+</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index fc9ce72990..8c74aa1f3c 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -2114,7 +2114,7 @@ to Spark (<a class="reference external" href="https://github.com/apache/datafusi
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">hash</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
-<td><p></p></td>
+<td><p>Includes decimals with precision 19–38 and nested decimals</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">md5</span></code></p></td>
<td><p>✅</p></td>
@@ -2139,7 +2139,7 @@ to Spark (<a class="reference external" href="https://github.com/apache/datafusi
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">xxhash64</span></code></p></td>
<td><p>✅</p></td>
<td><p>Native</p></td>
-<td><p></p></td>
+<td><p>Includes decimals with precision 19–38 and nested decimals</p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/latest/rust_udfs.html b/user-guide/latest/rust_udfs.html
index 7ac399243f..3fbd4b94a5 100644
--- a/user-guide/latest/rust_udfs.html
+++ b/user-guide/latest/rust_udfs.html
@@ -983,9 +983,42 @@ rather than accept a function whose volatility it would go on to ignore.</p>
<p><code class="docutils literal notranslate"><span class="pre">libraryPath</span></code> is passed to the platform’s dynamic loader. An absolute path is what you want in
practice, and it is what the rest of this page assumes, but a bare library name resolves the same
way it would for any other shared object, through <code class="docutils literal notranslate"><span class="pre">LD_LIBRARY_PATH</span></code> on Linux and
-<code class="docutils literal notranslate"><span class="pre">DYLD_LIBRARY_PATH</span></code> on macOS. That is a convenience, not a sandbox: Comet does not restrict which
-paths may be loaded, so it makes no difference to the trust decision described under
-<a class="reference internal" href="#limitations">Limitations</a>.</p>
+<code class="docutils literal notranslate"><span class="pre">DYLD_LIBRARY_PATH</span></code> on macOS. That is a convenience, not a sandbox: unless you set the configs
+below, Comet does not restrict which paths may be loaded, so it makes no difference to the trust
+decision described under <a class="reference internal" href="#limitations">Limitations</a>.</p>
+</section>
+<section id="restricting-which-libraries-can-be-loaded">
+<h2>Restricting which libraries can be loaded<a class="headerlink" href="#restricting-which-libraries-can-be-loaded" title="Link to this heading">#</a></h2>
+<p>Loading a library runs its code with the privileges of the JVM, so an operator can limit it:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Config</p></th>
+<th class="head"><p>Default</p></th>
+<th class="head"><p>Effect</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.enabled</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">true</span></code></p></td>
+<td><p>When <code class="docutils literal notranslate"><span class="pre">false</span></code>, no native UDF library is loaded.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeUdf.allowedPaths</span></code></p></td>
+<td><p>empty</p></td>
+<td><p>Comma-separated directories a library must be under, if set.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>With <code class="docutils literal notranslate"><span class="pre">allowedPaths</span></code> set, a library is loaded only if its path is absolute and, once symlinks and
+<code class="docutils literal notranslate"><span class="pre">..</span></code> are resolved, lies under one of the listed directories (which are resolved the same way). A bare
+library name is refused, because the dynamic loader’s search path is not something Comet can check.
+The check applies on the driver, where <code class="docutils literal notranslate"><span class="pre">register</span></code> fails with <code class="docutils literal notranslate"><span class="pre">CometNativeUdfNotAllowedException</span></code>,
+and again on each executor before it loads a library named in a plan.</p>
+<p>These are ordinary Spark configs, so they are set in the cluster’s defaults and are not a defense
+against a user who can change session configuration: such a user can set them back. The allow-list
+limits which files Comet will load, not what the code in them does, and a file that is replaced
+between the check and the load is not detected. List only directories that are writable by users
+you trust.</p>
<section id="when-spark-evaluates-the-call">
<h3>When Spark evaluates the call<a class="headerlink" href="#when-spark-evaluates-the-call" title="Link to this heading">#</a></h3>
<p>A call runs only inside Comet’s native execution. Spark cannot run the function, so wherever Spark
@@ -1150,7 +1183,9 @@ Downcast defensively in <code class="docutils literal notranslate"><span class="
layout.</p></li>
<li><p><strong>Loading a library is loading native code.</strong> It runs with the full privileges of the executor
process and Comet cannot sandbox it: a bug in a UDF can corrupt memory or crash the executor.
-Only register libraries you trust and control.</p></li>
+Only register libraries you trust and control. See
+<a class="reference internal" href="#restricting-which-libraries-can-be-loaded">Restricting which libraries can be loaded</a> for the
+configs that turn the feature off or limit the directories libraries may come from.</p></li>
<li><p>Once loaded, a library stays loaded for the life of the process, and <code class="docutils literal notranslate"><span class="pre">register</span></code> loads it on the
driver as well as on the executors. To deploy a new build, write it to a new path and register
that, or restart the executors and the driver. Never copy a new build over a library that is
@@ -1212,7 +1247,8 @@ Rebuild your UDF library when upgrading Comet.</p></li>
<nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#writing-a-udf">Writing a UDF</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#registering-and-calling-a-udf">Registering and calling a UDF</a><ul class="nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#registering-and-calling-a-udf">Registering and calling a UDF</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#restricting-which-libraries-can-be-loaded">Restricting which libraries can be loaded</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#when-spark-evaluates-the-call">When Spark evaluates the call</a></li>
</ul>
</li>
diff --git a/user-guide/latest/tuning/shuffle.html b/user-guide/latest/tuning/shuffle.html
index 8bbc0e764c..34aadd4d89 100644
--- a/user-guide/latest/tuning/shuffle.html
+++ b/user-guide/latest/tuning/shuffle.html
@@ -874,15 +874,6 @@ scalar types. Hash partitioning keys must be scalar types unless
<code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.native.partitioning.hash.nested.enabled=true</span></code>, which also admits struct, array, and (Spark 4.0
and later) map keys. That setting is disabled by default until the performance of the nested hashing paths has been
measured. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.</p>
-<p>Hash partitioning into multiple partitions on decimal keys with precision greater than 18 falls back because native
-hashing does not match Spark’s partition assignments. Mixing the two partitioners can silently lose join rows.
-The restriction also applies to decimal leaves in nested hash keys. Wider decimals remain supported as payload
-columns, range partitioning keys, and in single-partition shuffles, including one-partition hash exchanges.</p>
-<p>With <code class="docutils literal notranslate"><span class="pre">spark.comet.shuffle.mode=auto</span></code>, Comet uses Columnar Shuffle when eligible, adding a columnar-to-row-to-columnar
-conversion. With <code class="docutils literal notranslate"><span class="pre">native</span></code>, it uses Spark shuffle, which can also restore aggregates such as <code class="docutils literal notranslate"><span class="pre">collect_list</span></code> and
-<code class="docutils literal notranslate"><span class="pre">collect_set</span></code> to Spark to preserve buffer compatibility. Celeborn uses Spark/Celeborn shuffle for unsupported native
-keys because it has no Comet columnar fallback. Spark-compatible native decimal hashing is tracked in
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5994">#5994</a>.</p>
</section>
<section id="columnar-jvm-shuffle">
<h3>Columnar (JVM) Shuffle<a class="headerlink" href="#columnar-jvm-shuffle" title="Link to this heading">#</a></h3>
@@ -900,9 +891,9 @@ converts the child’s rows to Arrow, as the <code class="docutils literal notra
batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning
the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert
all of the child’s columns, which rules out calendar intervals, and arrays and maps other than <code class="docutils literal notranslate"><span class="pre">array<string></span></code> and
-<code class="docutils literal notranslate"><span class="pre">map<string,string></span></code>. A shuffle that hashes a decimal wider than 18 digits, or a string or a value computed from a
-string, stays on columnar shuffle, so that it partitions rows as Spark does. The converted batches are allocated on
-the JVM outside Comet’s memory pools. The setting is disabled by default.</p>
+<code class="docutils literal notranslate"><span class="pre">map<string,string></span></code>. A shuffle that hashes a string or a value computed from a string stays on columnar shuffle, so
+that it partitions rows as Spark does. The converted batches are allocated on the JVM outside Comet’s memory pools.
+The setting is disabled by default.</p>
</section>
<section id="automatic-revert-to-spark-shuffle">
<h3>Automatic Revert to Spark Shuffle<a class="headerlink" href="#automatic-revert-to-spark-shuffle" title="Link to this heading">#</a></h3>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org