You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/04 11:10:02 UTC
(datafusion-comet) branch asf-site updated (f3d7e86c02 -> 70582778de)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard f3d7e86c02 Publish built docs triggered by eaf6426a97d71a02af7b0751d5c16029106f123e
new 70582778de Publish built docs triggered by 9cac7164a430e7d7addfb62c99b2c30f983b5217
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f3d7e86c02)
\
N -- N -- N refs/heads/asf-site (70582778de)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_sources/contributor-guide/iceberg-spark-tests.md.txt | 5 +++++
contributor-guide/iceberg-spark-tests.html | 4 ++++
searchindex.js | 2 +-
3 files changed, 10 insertions(+), 1 deletion(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: Publish built docs triggered by 9cac7164a430e7d7addfb62c99b2c30f983b5217
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 70582778de3fd8b6fddd91b55547a378e9a3fdc3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 4 11:09:56 2026 +0000
Publish built docs triggered by 9cac7164a430e7d7addfb62c99b2c30f983b5217
---
.../contributor-guide/iceberg-spark-tests.md.txt | 5 ++++
.../expressions/spark-3.4/misc.md.txt | 6 ++++
.../expressions/spark-3.5/misc.md.txt | 6 ++++
.../expressions/spark-4.0/misc.md.txt | 6 ++++
.../expressions/spark-4.1/misc.md.txt | 6 ++++
_sources/user-guide/latest/iceberg-writes.md.txt | 5 +++-
_sources/user-guide/latest/iceberg.md.txt | 35 ++++++++++++++++++++++
contributor-guide/iceberg-spark-tests.html | 4 +++
searchindex.js | 2 +-
.../compatibility/expressions/spark-3.4/misc.html | 8 +++++
.../compatibility/expressions/spark-3.5/misc.html | 8 +++++
.../compatibility/expressions/spark-4.0/misc.html | 8 +++++
.../compatibility/expressions/spark-4.1/misc.html | 8 +++++
user-guide/latest/iceberg-writes.html | 5 +++-
user-guide/latest/iceberg.html | 34 +++++++++++++++++++++
15 files changed, 143 insertions(+), 3 deletions(-)
diff --git a/_sources/contributor-guide/iceberg-spark-tests.md.txt b/_sources/contributor-guide/iceberg-spark-tests.md.txt
index 6bfe57a4aa..df6c168af6 100644
--- a/_sources/contributor-guide/iceberg-spark-tests.md.txt
+++ b/_sources/contributor-guide/iceberg-spark-tests.md.txt
@@ -31,8 +31,13 @@ Here is an overview of the changes that the diffs make to Iceberg:
uses a native Iceberg scan, these classes fail to compile and must be removed.
- Configure test base classes (`TestBase`, `ExtensionsTestBase`, `ScanTestBase`, etc.) to load the Comet Spark
plugin and shuffle manager
+- Enable the Iceberg write split-operator plan (`spark.comet.write.iceberg.splitOperator.enabled`) alongside the
+ native scan in every Comet-configured session. The flag is off by default for users, so Iceberg's own suites
+ are the only place the split plan (`IcebergCommit -> IcebergWrite`) is exercised against Iceberg's write,
+ commit, and row-level-operation tests. See [#5259]
[#3739]: https://github.com/apache/datafusion-comet/pull/3739
+[#5259]: https://github.com/apache/datafusion-comet/issues/5259
[apache/iceberg#15674]: https://github.com/apache/iceberg/pull/15674
## 1. Install Comet
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
index ec617cec61..b9a6d05e52 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## Uuid
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
index dd0a84b74e..7c643ab030 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## ToPrettyString
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
index dd0a84b74e..7c643ab030 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## ToPrettyString
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
index dd0a84b74e..7c643ab030 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## ToPrettyString
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/iceberg-writes.md.txt b/_sources/user-guide/latest/iceberg-writes.md.txt
index e9227dea8b..49f3500920 100644
--- a/_sources/user-guide/latest/iceberg-writes.md.txt
+++ b/_sources/user-guide/latest/iceberg-writes.md.txt
@@ -128,7 +128,10 @@ per-task Parquet write is delegated to [iceberg-rust](https://github.com/apache/
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
statistics, and manifest metadata — so a write is only eligible when every table property it
depends on is one the native path reproduces exactly, and additionally only when the plan
-feeding the write is fully Comet-native. Ineligible writes run through iceberg-java unchanged,
+feeding the write is fully Comet-native. For a partitioned table that plan includes the hash
+distribution and local sort Iceberg requests on its partition transforms; those stay native
+because the transforms themselves have native implementations (see
+[Iceberg system functions](iceberg.md)). Ineligible writes run through iceberg-java unchanged,
with the reason reported as a fall-back reason in Comet's extended EXPLAIN output.
**Most Iceberg write settings are not supported.** Detection is an allowlist: a write is
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 156fe41808..300670a45d 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -196,6 +196,41 @@ the project, exchange, and sort operators around them stay on the Comet path end
Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuffle from
`CometExchange` to `CometColumnarExchange`.
+### Iceberg system functions
+
+Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (the SQL
+form of its partition transforms, for example `SELECT system.bucket(16, id) FROM t`) run natively.
+Spark binds them as static invocations of Iceberg's per-type implementations under
+`org.apache.iceberg.spark.functions`, and Comet recognizes those classes wherever the expression
+appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle.
+
+The native kernels reproduce Iceberg's Java semantics exactly rather than approximately:
+
+- `bucket` hashes the spec's byte encoding of each value (8-byte little-endian for integers, dates,
+ and timestamps; UTF-8 for strings; raw bytes for binary; the minimal big-endian two's complement
+ of the unscaled value for decimals) with 32-bit Murmur3 and masks the sign bit before taking the
+ modulus.
+- `truncate` uses Java's wrapping integer arithmetic and counts code points (not bytes) for
+ strings. Decimal inputs are the one case that stays with Spark, see below.
+- `years`, `months`, `days`, and `hours` are evaluated in UTC regardless of the session timezone
+ and go negative before the epoch; `days` returns a date, the other three an int. They cover the
+ whole `DATE` and `TIMESTAMP` domain, as Iceberg's `DateTimeUtil` does.
+
+`truncate` on a `decimal` column falls back to Spark. Truncating a negative decimal grows its
+magnitude, so the result can need one more digit than the column's precision allows:
+`truncate(10, v)` on a `decimal(18,4)` value of `-99999999999999.9999` is
+`-100000000000000.0000`, which has 19 digits. Iceberg's `TruncateDecimal` hands that oversized
+value back unchanged and Spark turns it into null only when the row is materialized. An Arrow
+`Decimal128(precision, scale)` array has no encoding for that intermediate, so a native kernel
+would have to null it during evaluation, which changes what an enclosing predicate or hash sees.
+Every other `truncate` input type, and `bucket` on decimals, runs natively.
+
+This matters most for writes. A partitioned table with the default `write.distribution-mode`
+(`hash`) is planned with a shuffle and a local sort keyed on the partition transforms, and with
+these functions native the whole sub-plan feeding the [native Iceberg writer](iceberg-writes.md)
+stays in Comet. A `numBuckets` or `width` argument that is not a positive integer literal makes
+the expression fall back to Spark.
+
### Task input metrics
The native Iceberg reader populates Spark's task-level `inputMetrics.bytesRead` (visible in the Spark UI Stages tab) using the `bytes_read` counter from iceberg-rust's `ScanMetrics`. This counter includes bytes read from both data files and delete files.
diff --git a/contributor-guide/iceberg-spark-tests.html b/contributor-guide/iceberg-spark-tests.html
index 95d6c60f3b..caa16d9242 100644
--- a/contributor-guide/iceberg-spark-tests.html
+++ b/contributor-guide/iceberg-spark-tests.html
@@ -557,6 +557,10 @@ added upstream in <a class="reference external" href="https://github.com/apache/
uses a native Iceberg scan, these classes fail to compile and must be removed.</p></li>
<li><p>Configure test base classes (<code class="docutils literal notranslate"><span class="pre">TestBase</span></code>, <code class="docutils literal notranslate"><span class="pre">ExtensionsTestBase</span></code>, <code class="docutils literal notranslate"><span class="pre">ScanTestBase</span></code>, etc.) to load the Comet Spark
plugin and shuffle manager</p></li>
+<li><p>Enable the Iceberg write split-operator plan (<code class="docutils literal notranslate"><span class="pre">spark.comet.write.iceberg.splitOperator.enabled</span></code>) alongside the
+native scan in every Comet-configured session. The flag is off by default for users, so Iceberg’s own suites
+are the only place the split plan (<code class="docutils literal notranslate"><span class="pre">IcebergCommit</span> <span class="pre">-></span> <span class="pre">IcebergWrite</span></code>) is exercised against Iceberg’s write,
+commit, and row-level-operation tests. See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5259">#5259</a></p></li>
</ul>
<section id="install-comet">
<h2>1. Install Comet<a class="headerlink" href="#install-comet" title="Link to this heading">#</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 3cefeb955a..34b8035ac0 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
index 32510beb64..92e1852607 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="uuid">
<h2>Uuid<a class="headerlink" href="#uuid" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -880,6 +887,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
index 22f1ad66e2..899401979a 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="toprettystring">
<h2>ToPrettyString<a class="headerlink" href="#toprettystring" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -887,6 +894,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toprettystring">ToPrettyString</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
index 667a647c0c..516e5b0cfa 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="toprettystring">
<h2>ToPrettyString<a class="headerlink" href="#toprettystring" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -887,6 +894,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toprettystring">ToPrettyString</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
index e2831900e5..31106fe258 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="toprettystring">
<h2>ToPrettyString<a class="headerlink" href="#toprettystring" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -887,6 +894,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toprettystring">ToPrettyString</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
diff --git a/user-guide/latest/iceberg-writes.html b/user-guide/latest/iceberg-writes.html
index 5a52e4154a..60a94a7861 100644
--- a/user-guide/latest/iceberg-writes.html
+++ b/user-guide/latest/iceberg-writes.html
@@ -881,7 +881,10 @@ per-task Parquet write is delegated to <a class="reference external" href="https
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
statistics, and manifest metadata — so a write is only eligible when every table property it
depends on is one the native path reproduces exactly, and additionally only when the plan
-feeding the write is fully Comet-native. Ineligible writes run through iceberg-java unchanged,
+feeding the write is fully Comet-native. For a partitioned table that plan includes the hash
+distribution and local sort Iceberg requests on its partition transforms; those stay native
+because the transforms themselves have native implementations (see
+<a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg system functions</span></a>). Ineligible writes run through iceberg-java unchanged,
with the reason reported as a fall-back reason in Comet’s extended EXPLAIN output.</p>
<p><strong>Most Iceberg write settings are not supported.</strong> Detection is an allowlist: a write is
eligible only when its entire effective configuration matches the table below, and anything
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 222454f39d..f47f08727a 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -943,6 +943,39 @@ the project, exchange, and sort operators around them stay on the Comet path end
Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuffle from
<code class="docutils literal notranslate"><span class="pre">CometExchange</span></code> to <code class="docutils literal notranslate"><span class="pre">CometColumnarExchange</span></code>.</p>
</section>
+<section id="iceberg-system-functions">
+<h3>Iceberg system functions<a class="headerlink" href="#iceberg-system-functions" title="Link to this heading">#</a></h3>
+<p>Iceberg’s system functions <code class="docutils literal notranslate"><span class="pre">bucket</span></code>, <code class="docutils literal notranslate"><span class="pre">truncate</span></code>, <code class="docutils literal notranslate"><span class="pre">years</span></code>, <code class="docutils literal notranslate"><span class="pre">months</span></code>, <code class="docutils literal notranslate"><span class="pre">days</span></code>, and <code class="docutils literal notranslate">< [...]
+form of its partition transforms, for example <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">system.bucket(16,</span> <span class="pre">id)</span> <span class="pre">FROM</span> <span class="pre">t</span></code>) run natively.
+Spark binds them as static invocations of Iceberg’s per-type implementations under
+<code class="docutils literal notranslate"><span class="pre">org.apache.iceberg.spark.functions</span></code>, and Comet recognizes those classes wherever the expression
+appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle.</p>
+<p>The native kernels reproduce Iceberg’s Java semantics exactly rather than approximately:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">bucket</span></code> hashes the spec’s byte encoding of each value (8-byte little-endian for integers, dates,
+and timestamps; UTF-8 for strings; raw bytes for binary; the minimal big-endian two’s complement
+of the unscaled value for decimals) with 32-bit Murmur3 and masks the sign bit before taking the
+modulus.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">truncate</span></code> uses Java’s wrapping integer arithmetic and counts code points (not bytes) for
+strings. Decimal inputs are the one case that stays with Spark, see below.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">years</span></code>, <code class="docutils literal notranslate"><span class="pre">months</span></code>, <code class="docutils literal notranslate"><span class="pre">days</span></code>, and <code class="docutils literal notranslate"><span class="pre">hours</span></code> are evaluated in UTC regardless of the session timezone
+and go negative before the epoch; <code class="docutils literal notranslate"><span class="pre">days</span></code> returns a date, the other three an int. They cover the
+whole <code class="docutils literal notranslate"><span class="pre">DATE</span></code> and <code class="docutils literal notranslate"><span class="pre">TIMESTAMP</span></code> domain, as Iceberg’s <code class="docutils literal notranslate"><span class="pre">DateTimeUtil</span></code> does.</p></li>
+</ul>
+<p><code class="docutils literal notranslate"><span class="pre">truncate</span></code> on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column falls back to Spark. Truncating a negative decimal grows its
+magnitude, so the result can need one more digit than the column’s precision allows:
+<code class="docutils literal notranslate"><span class="pre">truncate(10,</span> <span class="pre">v)</span></code> on a <code class="docutils literal notranslate"><span class="pre">decimal(18,4)</span></code> value of <code class="docutils literal notranslate"><span class="pre">-99999999999999.9999</span></code> is
+<code class="docutils literal notranslate"><span class="pre">-100000000000000.0000</span></code>, which has 19 digits. Iceberg’s <code class="docutils literal notranslate"><span class="pre">TruncateDecimal</span></code> hands that oversized
+value back unchanged and Spark turns it into null only when the row is materialized. An Arrow
+<code class="docutils literal notranslate"><span class="pre">Decimal128(precision,</span> <span class="pre">scale)</span></code> array has no encoding for that intermediate, so a native kernel
+would have to null it during evaluation, which changes what an enclosing predicate or hash sees.
+Every other <code class="docutils literal notranslate"><span class="pre">truncate</span></code> input type, and <code class="docutils literal notranslate"><span class="pre">bucket</span></code> on decimals, runs natively.</p>
+<p>This matters most for writes. A partitioned table with the default <code class="docutils literal notranslate"><span class="pre">write.distribution-mode</span></code>
+(<code class="docutils literal notranslate"><span class="pre">hash</span></code>) is planned with a shuffle and a local sort keyed on the partition transforms, and with
+these functions native the whole sub-plan feeding the <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">native Iceberg writer</span></a>
+stays in Comet. A <code class="docutils literal notranslate"><span class="pre">numBuckets</span></code> or <code class="docutils literal notranslate"><span class="pre">width</span></code> argument that is not a positive integer literal makes
+the expression fall back to Spark.</p>
+</section>
<section id="task-input-metrics">
<h3>Task input metrics<a class="headerlink" href="#task-input-metrics" title="Link to this heading">#</a></h3>
<p>The native Iceberg reader populates Spark’s task-level <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> (visible in the Spark UI Stages tab) using the <code class="docutils literal notranslate"><span class="pre">bytes_read</span></code> counter from iceberg-rust’s <code class="docutils literal notranslate"><span class="pre">ScanMetrics</span></code>. This counter includes bytes read from both data files and delete files.</p>
@@ -1005,6 +1038,7 @@ Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuf
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#object-store-configuration-s3">Object store configuration (S3)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#current-limitations">Current limitations</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#iceberg-udfs">Iceberg UDFs</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#iceberg-system-functions">Iceberg system functions</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#task-input-metrics">Task input metrics</a></li>
</ul>
</li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org