You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/04 11:11:11 UTC
(datafusion-comet) 01/01: Publish built docs triggered by 81d637b9bf40a5be6f4f0c65ad6f497b34746e69
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 7807d0f20fba7e377146b075e96b0b44d4472e2f
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 4 11:11:04 2026 +0000
Publish built docs triggered by 81d637b9bf40a5be6f4f0c65ad6f497b34746e69
---
.../contributor-guide/iceberg-spark-tests.md.txt | 5 ++++
.../expressions/spark-3.4/misc.md.txt | 6 ++++
.../expressions/spark-3.5/misc.md.txt | 6 ++++
.../expressions/spark-4.0/misc.md.txt | 6 ++++
.../expressions/spark-4.1/misc.md.txt | 6 ++++
_sources/user-guide/latest/configs.md.txt | 5 ++++
_sources/user-guide/latest/expressions.md.txt | 12 ++++----
_sources/user-guide/latest/iceberg-writes.md.txt | 5 +++-
_sources/user-guide/latest/iceberg.md.txt | 35 ++++++++++++++++++++++
contributor-guide/iceberg-spark-tests.html | 4 +++
searchindex.js | 2 +-
.../compatibility/expressions/spark-3.4/misc.html | 8 +++++
.../compatibility/expressions/spark-3.5/misc.html | 8 +++++
.../compatibility/expressions/spark-4.0/misc.html | 8 +++++
.../compatibility/expressions/spark-4.1/misc.html | 8 +++++
user-guide/latest/configs.html | 30 +++++++++++++++----
user-guide/latest/expressions.html | 32 ++++++++++----------
user-guide/latest/iceberg-writes.html | 5 +++-
user-guide/latest/iceberg.html | 34 +++++++++++++++++++++
19 files changed, 195 insertions(+), 30 deletions(-)
diff --git a/_sources/contributor-guide/iceberg-spark-tests.md.txt b/_sources/contributor-guide/iceberg-spark-tests.md.txt
index 6bfe57a4aa..df6c168af6 100644
--- a/_sources/contributor-guide/iceberg-spark-tests.md.txt
+++ b/_sources/contributor-guide/iceberg-spark-tests.md.txt
@@ -31,8 +31,13 @@ Here is an overview of the changes that the diffs make to Iceberg:
uses a native Iceberg scan, these classes fail to compile and must be removed.
- Configure test base classes (`TestBase`, `ExtensionsTestBase`, `ScanTestBase`, etc.) to load the Comet Spark
plugin and shuffle manager
+- Enable the Iceberg write split-operator plan (`spark.comet.write.iceberg.splitOperator.enabled`) alongside the
+ native scan in every Comet-configured session. The flag is off by default for users, so Iceberg's own suites
+ are the only place the split plan (`IcebergCommit -> IcebergWrite`) is exercised against Iceberg's write,
+ commit, and row-level-operation tests. See [#5259]
[#3739]: https://github.com/apache/datafusion-comet/pull/3739
+[#5259]: https://github.com/apache/datafusion-comet/issues/5259
[apache/iceberg#15674]: https://github.com/apache/iceberg/pull/15674
## 1. Install Comet
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
index ec617cec61..b9a6d05e52 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## Uuid
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
index dd0a84b74e..7c643ab030 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## ToPrettyString
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
index dd0a84b74e..7c643ab030 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## ToPrettyString
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
index dd0a84b74e..7c643ab030 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/misc.md.txt
@@ -51,6 +51,12 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+## StaticInvoke
+
+The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
+
+- Iceberg's `truncate(width, value)` system function on a `decimal` column. Iceberg's TruncateDecimal returns a Decimal that can exceed the column's declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a native kernel would null it during evaluation and change what an enclosing predicate or hash sees. Truncating `-99999999999999.9999` in a `decimal(18,4)` column by a width of 1 [...]
+
## ToPrettyString
The following cases are not supported by Comet and always fall back to Spark, regardless of any `allowIncompatible` setting:
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 294b62cfb3..494b0fc573 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -492,6 +492,11 @@ These settings can be used to determine which parts of the plan are accelerated
| `spark.comet.expression.Max.enabled` | Enable Comet acceleration for `Max` | true |
| `spark.comet.expression.Min.enabled` | Enable Comet acceleration for `Min` | true |
| `spark.comet.expression.Percentile.enabled` | Enable Comet acceleration for `Percentile` | true |
+| `spark.comet.expression.RegrIntercept.enabled` | Enable Comet acceleration for `RegrIntercept` | true |
+| `spark.comet.expression.RegrR2.enabled` | Enable Comet acceleration for `RegrR2` | true |
+| `spark.comet.expression.RegrReplacement.enabled` | Enable Comet acceleration for `RegrReplacement` | true |
+| `spark.comet.expression.RegrSXY.enabled` | Enable Comet acceleration for `RegrSXY` | true |
+| `spark.comet.expression.RegrSlope.enabled` | Enable Comet acceleration for `RegrSlope` | true |
| `spark.comet.expression.StddevPop.enabled` | Enable Comet acceleration for `StddevPop` | true |
| `spark.comet.expression.StddevSamp.enabled` | Enable Comet acceleration for `StddevSamp` | true |
| `spark.comet.expression.Sum.enabled` | Enable Comet acceleration for `Sum` | true |
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 3fbcb12ded..73926b8972 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -121,12 +121,12 @@ The tables below list every Spark built-in expression with its current status.
| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
-| `regr_intercept` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop` accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_r2` | 🔜 | — | Falls back; can reuse the `corr` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_slope` | 🔜 | — | Falls back; can reuse `covar_pop`/`var_pop` accumulators ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_sxx` | 🔜 | — | Falls back; can reuse `var_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_sxy` | 🔜 | — | Falls back; can reuse `covar_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
-| `regr_syy` | 🔜 | — | Falls back; can reuse `var_pop` accumulator ([#4552](https://github.com/apache/datafusion-comet/issues/4552)) |
+| `regr_intercept` | ✅ | Native | |
+| `regr_r2` | ✅ | Native | |
+| `regr_slope` | ✅ | Native | |
+| `regr_sxx` | ✅ | — | |
+| `regr_sxy` | ✅ | Native | |
+| `regr_syy` | ✅ | — | |
| `skewness` | 🔜 | — | Not yet implemented natively |
| `some` | ✅ | — | |
| `std` | ✅ | Native | |
diff --git a/_sources/user-guide/latest/iceberg-writes.md.txt b/_sources/user-guide/latest/iceberg-writes.md.txt
index e9227dea8b..49f3500920 100644
--- a/_sources/user-guide/latest/iceberg-writes.md.txt
+++ b/_sources/user-guide/latest/iceberg-writes.md.txt
@@ -128,7 +128,10 @@ per-task Parquet write is delegated to [iceberg-rust](https://github.com/apache/
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
statistics, and manifest metadata — so a write is only eligible when every table property it
depends on is one the native path reproduces exactly, and additionally only when the plan
-feeding the write is fully Comet-native. Ineligible writes run through iceberg-java unchanged,
+feeding the write is fully Comet-native. For a partitioned table that plan includes the hash
+distribution and local sort Iceberg requests on its partition transforms; those stay native
+because the transforms themselves have native implementations (see
+[Iceberg system functions](iceberg.md)). Ineligible writes run through iceberg-java unchanged,
with the reason reported as a fall-back reason in Comet's extended EXPLAIN output.
**Most Iceberg write settings are not supported.** Detection is an allowlist: a write is
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 156fe41808..300670a45d 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -196,6 +196,41 @@ the project, exchange, and sort operators around them stay on the Comet path end
Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuffle from
`CometExchange` to `CometColumnarExchange`.
+### Iceberg system functions
+
+Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (the SQL
+form of its partition transforms, for example `SELECT system.bucket(16, id) FROM t`) run natively.
+Spark binds them as static invocations of Iceberg's per-type implementations under
+`org.apache.iceberg.spark.functions`, and Comet recognizes those classes wherever the expression
+appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle.
+
+The native kernels reproduce Iceberg's Java semantics exactly rather than approximately:
+
+- `bucket` hashes the spec's byte encoding of each value (8-byte little-endian for integers, dates,
+ and timestamps; UTF-8 for strings; raw bytes for binary; the minimal big-endian two's complement
+ of the unscaled value for decimals) with 32-bit Murmur3 and masks the sign bit before taking the
+ modulus.
+- `truncate` uses Java's wrapping integer arithmetic and counts code points (not bytes) for
+ strings. Decimal inputs are the one case that stays with Spark, see below.
+- `years`, `months`, `days`, and `hours` are evaluated in UTC regardless of the session timezone
+ and go negative before the epoch; `days` returns a date, the other three an int. They cover the
+ whole `DATE` and `TIMESTAMP` domain, as Iceberg's `DateTimeUtil` does.
+
+`truncate` on a `decimal` column falls back to Spark. Truncating a negative decimal grows its
+magnitude, so the result can need one more digit than the column's precision allows:
+`truncate(10, v)` on a `decimal(18,4)` value of `-99999999999999.9999` is
+`-100000000000000.0000`, which has 19 digits. Iceberg's `TruncateDecimal` hands that oversized
+value back unchanged and Spark turns it into null only when the row is materialized. An Arrow
+`Decimal128(precision, scale)` array has no encoding for that intermediate, so a native kernel
+would have to null it during evaluation, which changes what an enclosing predicate or hash sees.
+Every other `truncate` input type, and `bucket` on decimals, runs natively.
+
+This matters most for writes. A partitioned table with the default `write.distribution-mode`
+(`hash`) is planned with a shuffle and a local sort keyed on the partition transforms, and with
+these functions native the whole sub-plan feeding the [native Iceberg writer](iceberg-writes.md)
+stays in Comet. A `numBuckets` or `width` argument that is not a positive integer literal makes
+the expression fall back to Spark.
+
### Task input metrics
The native Iceberg reader populates Spark's task-level `inputMetrics.bytesRead` (visible in the Spark UI Stages tab) using the `bytes_read` counter from iceberg-rust's `ScanMetrics`. This counter includes bytes read from both data files and delete files.
diff --git a/contributor-guide/iceberg-spark-tests.html b/contributor-guide/iceberg-spark-tests.html
index 95d6c60f3b..caa16d9242 100644
--- a/contributor-guide/iceberg-spark-tests.html
+++ b/contributor-guide/iceberg-spark-tests.html
@@ -557,6 +557,10 @@ added upstream in <a class="reference external" href="https://github.com/apache/
uses a native Iceberg scan, these classes fail to compile and must be removed.</p></li>
<li><p>Configure test base classes (<code class="docutils literal notranslate"><span class="pre">TestBase</span></code>, <code class="docutils literal notranslate"><span class="pre">ExtensionsTestBase</span></code>, <code class="docutils literal notranslate"><span class="pre">ScanTestBase</span></code>, etc.) to load the Comet Spark
plugin and shuffle manager</p></li>
+<li><p>Enable the Iceberg write split-operator plan (<code class="docutils literal notranslate"><span class="pre">spark.comet.write.iceberg.splitOperator.enabled</span></code>) alongside the
+native scan in every Comet-configured session. The flag is off by default for users, so Iceberg’s own suites
+are the only place the split plan (<code class="docutils literal notranslate"><span class="pre">IcebergCommit</span> <span class="pre">-></span> <span class="pre">IcebergWrite</span></code>) is exercised against Iceberg’s write,
+commit, and row-level-operation tests. See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5259">#5259</a></p></li>
</ul>
<section id="install-comet">
<h2>1. Install Comet<a class="headerlink" href="#install-comet" title="Link to this heading">#</a></h2>
diff --git a/searchindex.js b/searchindex.js
index 3cefeb955a..348c9abcfa 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
index 32510beb64..92e1852607 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="uuid">
<h2>Uuid<a class="headerlink" href="#uuid" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -880,6 +887,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
</nav></div>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
index 22f1ad66e2..899401979a 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="toprettystring">
<h2>ToPrettyString<a class="headerlink" href="#toprettystring" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -887,6 +894,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toprettystring">ToPrettyString</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
index 667a647c0c..516e5b0cfa 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="toprettystring">
<h2>ToPrettyString<a class="headerlink" href="#toprettystring" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -887,6 +894,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toprettystring">ToPrettyString</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
index e2831900e5..31106fe258 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/misc.html
@@ -818,6 +818,13 @@ under the License.
<li><p>When <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint=true</span></code>, sorting on floating-point types is not 100% compatible with Spark</p></li>
</ul>
</section>
+<section id="staticinvoke">
+<h2>StaticInvoke<a class="headerlink" href="#staticinvoke" title="Link to this heading">#</a></h2>
+<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
+<ul class="simple">
+<li><p>Iceberg’s <code class="docutils literal notranslate"><span class="pre">truncate(width,</span> <span class="pre">value)</span></code> system function on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column. Iceberg’s TruncateDecimal returns a Decimal that can exceed the column’s declared precision, and Spark only turns that into null when the row is materialized. An Arrow Decimal128(precision, scale) array cannot carry that intermediate, so a [...]
+</ul>
+</section>
<section id="toprettystring">
<h2>ToPrettyString<a class="headerlink" href="#toprettystring" title="Link to this heading">#</a></h2>
<p>The following cases are not supported by Comet and always fall back to Spark, regardless of any <code class="docutils literal notranslate"><span class="pre">allowIncompatible</span></code> setting:</p>
@@ -887,6 +894,7 @@ under the License.
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#makedecimal">MakeDecimal</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scalarsubquery">ScalarSubquery</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#sortorder">SortOrder</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#staticinvoke">StaticInvoke</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#toprettystring">ToPrettyString</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#uuid">Uuid</a></li>
</ul>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 018a2b52e5..ef04483c99 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -2480,23 +2480,43 @@ under the License.
<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">Percentile</span></code></p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StddevPop.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.RegrIntercept.enabled</span></code></p></td>
+<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">RegrIntercept</span></code></p></td>
+<td><p>true</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.RegrR2.enabled</span></code></p></td>
+<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">RegrR2</span></code></p></td>
+<td><p>true</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.RegrReplacement.enabled</span></code></p></td>
+<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">RegrReplacement</span></code></p></td>
+<td><p>true</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.RegrSXY.enabled</span></code></p></td>
+<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">RegrSXY</span></code></p></td>
+<td><p>true</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.RegrSlope.enabled</span></code></p></td>
+<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">RegrSlope</span></code></p></td>
+<td><p>true</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StddevPop.enabled</span></code></p></td>
<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">StddevPop</span></code></p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StddevSamp.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StddevSamp.enabled</span></code></p></td>
<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">StddevSamp</span></code></p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Sum.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Sum.enabled</span></code></p></td>
<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">Sum</span></code></p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.VariancePop.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.VariancePop.enabled</span></code></p></td>
<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">VariancePop</span></code></p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.VarianceSamp.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.expression.VarianceSamp.enabled</span></code></p></td>
<td><p>Enable Comet acceleration for <code class="docutils literal notranslate"><span class="pre">VarianceSamp</span></code></p></td>
<td><p>true</p></td>
</tr>
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 32f3e0f0b7..cb89ea400f 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -1066,34 +1066,34 @@ expressions. The following function families are <strong>not currently planned</
<td><p>Native: Spark rewrites to <code class="docutils literal notranslate"><span class="pre">Count</span></code> (tests in <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/4551">#4551</a>)</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_intercept</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>—</p></td>
-<td><p>Falls back; can reuse <code class="docutils literal notranslate"><span class="pre">covar_pop</span></code>/<code class="docutils literal notranslate"><span class="pre">var_pop</span></code> accumulators (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4552">#4552</a>)</p></td>
+<td><p>✅</p></td>
+<td><p>Native</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_r2</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>—</p></td>
-<td><p>Falls back; can reuse the <code class="docutils literal notranslate"><span class="pre">corr</span></code> accumulator (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4552">#4552</a>)</p></td>
+<td><p>✅</p></td>
+<td><p>Native</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_slope</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>—</p></td>
-<td><p>Falls back; can reuse <code class="docutils literal notranslate"><span class="pre">covar_pop</span></code>/<code class="docutils literal notranslate"><span class="pre">var_pop</span></code> accumulators (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4552">#4552</a>)</p></td>
+<td><p>✅</p></td>
+<td><p>Native</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_sxx</span></code></p></td>
-<td><p>🔜</p></td>
+<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Falls back; can reuse <code class="docutils literal notranslate"><span class="pre">var_pop</span></code> accumulator (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4552">#4552</a>)</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">regr_sxy</span></code></p></td>
-<td><p>🔜</p></td>
-<td><p>—</p></td>
-<td><p>Falls back; can reuse <code class="docutils literal notranslate"><span class="pre">covar_pop</span></code> accumulator (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4552">#4552</a>)</p></td>
+<td><p>✅</p></td>
+<td><p>Native</p></td>
+<td><p></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">regr_syy</span></code></p></td>
-<td><p>🔜</p></td>
+<td><p>✅</p></td>
<td><p>—</p></td>
-<td><p>Falls back; can reuse <code class="docutils literal notranslate"><span class="pre">var_pop</span></code> accumulator (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4552">#4552</a>)</p></td>
+<td><p></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">skewness</span></code></p></td>
<td><p>🔜</p></td>
diff --git a/user-guide/latest/iceberg-writes.html b/user-guide/latest/iceberg-writes.html
index 5a52e4154a..60a94a7861 100644
--- a/user-guide/latest/iceberg-writes.html
+++ b/user-guide/latest/iceberg-writes.html
@@ -881,7 +881,10 @@ per-task Parquet write is delegated to <a class="reference external" href="https
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
statistics, and manifest metadata — so a write is only eligible when every table property it
depends on is one the native path reproduces exactly, and additionally only when the plan
-feeding the write is fully Comet-native. Ineligible writes run through iceberg-java unchanged,
+feeding the write is fully Comet-native. For a partitioned table that plan includes the hash
+distribution and local sort Iceberg requests on its partition transforms; those stay native
+because the transforms themselves have native implementations (see
+<a class="reference internal" href="iceberg.html"><span class="std std-doc">Iceberg system functions</span></a>). Ineligible writes run through iceberg-java unchanged,
with the reason reported as a fall-back reason in Comet’s extended EXPLAIN output.</p>
<p><strong>Most Iceberg write settings are not supported.</strong> Detection is an allowlist: a write is
eligible only when its entire effective configuration matches the table below, and anything
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 222454f39d..f47f08727a 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -943,6 +943,39 @@ the project, exchange, and sort operators around them stay on the Comet path end
Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuffle from
<code class="docutils literal notranslate"><span class="pre">CometExchange</span></code> to <code class="docutils literal notranslate"><span class="pre">CometColumnarExchange</span></code>.</p>
</section>
+<section id="iceberg-system-functions">
+<h3>Iceberg system functions<a class="headerlink" href="#iceberg-system-functions" title="Link to this heading">#</a></h3>
+<p>Iceberg’s system functions <code class="docutils literal notranslate"><span class="pre">bucket</span></code>, <code class="docutils literal notranslate"><span class="pre">truncate</span></code>, <code class="docutils literal notranslate"><span class="pre">years</span></code>, <code class="docutils literal notranslate"><span class="pre">months</span></code>, <code class="docutils literal notranslate"><span class="pre">days</span></code>, and <code class="docutils literal notranslate">< [...]
+form of its partition transforms, for example <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">system.bucket(16,</span> <span class="pre">id)</span> <span class="pre">FROM</span> <span class="pre">t</span></code>) run natively.
+Spark binds them as static invocations of Iceberg’s per-type implementations under
+<code class="docutils literal notranslate"><span class="pre">org.apache.iceberg.spark.functions</span></code>, and Comet recognizes those classes wherever the expression
+appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle.</p>
+<p>The native kernels reproduce Iceberg’s Java semantics exactly rather than approximately:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">bucket</span></code> hashes the spec’s byte encoding of each value (8-byte little-endian for integers, dates,
+and timestamps; UTF-8 for strings; raw bytes for binary; the minimal big-endian two’s complement
+of the unscaled value for decimals) with 32-bit Murmur3 and masks the sign bit before taking the
+modulus.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">truncate</span></code> uses Java’s wrapping integer arithmetic and counts code points (not bytes) for
+strings. Decimal inputs are the one case that stays with Spark, see below.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">years</span></code>, <code class="docutils literal notranslate"><span class="pre">months</span></code>, <code class="docutils literal notranslate"><span class="pre">days</span></code>, and <code class="docutils literal notranslate"><span class="pre">hours</span></code> are evaluated in UTC regardless of the session timezone
+and go negative before the epoch; <code class="docutils literal notranslate"><span class="pre">days</span></code> returns a date, the other three an int. They cover the
+whole <code class="docutils literal notranslate"><span class="pre">DATE</span></code> and <code class="docutils literal notranslate"><span class="pre">TIMESTAMP</span></code> domain, as Iceberg’s <code class="docutils literal notranslate"><span class="pre">DateTimeUtil</span></code> does.</p></li>
+</ul>
+<p><code class="docutils literal notranslate"><span class="pre">truncate</span></code> on a <code class="docutils literal notranslate"><span class="pre">decimal</span></code> column falls back to Spark. Truncating a negative decimal grows its
+magnitude, so the result can need one more digit than the column’s precision allows:
+<code class="docutils literal notranslate"><span class="pre">truncate(10,</span> <span class="pre">v)</span></code> on a <code class="docutils literal notranslate"><span class="pre">decimal(18,4)</span></code> value of <code class="docutils literal notranslate"><span class="pre">-99999999999999.9999</span></code> is
+<code class="docutils literal notranslate"><span class="pre">-100000000000000.0000</span></code>, which has 19 digits. Iceberg’s <code class="docutils literal notranslate"><span class="pre">TruncateDecimal</span></code> hands that oversized
+value back unchanged and Spark turns it into null only when the row is materialized. An Arrow
+<code class="docutils literal notranslate"><span class="pre">Decimal128(precision,</span> <span class="pre">scale)</span></code> array has no encoding for that intermediate, so a native kernel
+would have to null it during evaluation, which changes what an enclosing predicate or hash sees.
+Every other <code class="docutils literal notranslate"><span class="pre">truncate</span></code> input type, and <code class="docutils literal notranslate"><span class="pre">bucket</span></code> on decimals, runs natively.</p>
+<p>This matters most for writes. A partitioned table with the default <code class="docutils literal notranslate"><span class="pre">write.distribution-mode</span></code>
+(<code class="docutils literal notranslate"><span class="pre">hash</span></code>) is planned with a shuffle and a local sort keyed on the partition transforms, and with
+these functions native the whole sub-plan feeding the <a class="reference internal" href="iceberg-writes.html"><span class="std std-doc">native Iceberg writer</span></a>
+stays in Comet. A <code class="docutils literal notranslate"><span class="pre">numBuckets</span></code> or <code class="docutils literal notranslate"><span class="pre">width</span></code> argument that is not a positive integer literal makes
+the expression fall back to Spark.</p>
+</section>
<section id="task-input-metrics">
<h3>Task input metrics<a class="headerlink" href="#task-input-metrics" title="Link to this heading">#</a></h3>
<p>The native Iceberg reader populates Spark’s task-level <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> (visible in the Spark UI Stages tab) using the <code class="docutils literal notranslate"><span class="pre">bytes_read</span></code> counter from iceberg-rust’s <code class="docutils literal notranslate"><span class="pre">ScanMetrics</span></code>. This counter includes bytes read from both data files and delete files.</p>
@@ -1005,6 +1038,7 @@ Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuf
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#object-store-configuration-s3">Object store configuration (S3)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#current-limitations">Current limitations</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#iceberg-udfs">Iceberg UDFs</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#iceberg-system-functions">Iceberg system functions</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#task-input-metrics">Task input metrics</a></li>
</ul>
</li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org